My self-hosted Talos Kubernetes cluster running on Tailscale, configured using terraform and flux
This holds everything for my bare metal Talos k8s cluster.
The cluster is initialized with Terraform, including flux, which then runs off of this repo.
This configuration depends on:
The cluster nodes have to be assigned IPs reachable from your machine.
Each image:
stable_secret
set as a kernel paramCreating the image factory config and fetching each image is handled in
Terraform and exposed as outputs. The justfile
makes it easy to grab the
update image and the ISOs.
Each node is added to the Terraform variable nodes
before it's added to either
control_plane_nodes
or worker_nodes
.
This Terraform uses Cloudflare to assign a public DNS name to load balance between the Tailscale IPs.
Another option would be to run a something like coredns-tailscale to handle this.
This has two problems:
This terraform can use IPv6 addresses, which Tailscale provisions dynamically.
Depending on which DNS you're using, note that there might be a large delay between when the nodes start querying the API endpoint /we try to contact the API endpoint and when the DNS records are propagated.
Somewhat out of scope but for ease of use I also assign IPv4 Tailscale addresses
statically.
Assuming cluster_name = "k8rn"
, this can be achieved via ACLs:
"nodeAttrs": [
{
"target": ["tag:k8rn-cp-0"],
"ipPool": ["100.64.1.10/32"],
},
{
"target": ["tag:k8rn-cp-1"],
"ipPool": ["100.64.1.11/32"],
},
{
"target": ["tag:k8rn-cp-2"],
"ipPool": ["100.64.1.12/32"],
},
],
Each node is assigned tag:<node hostname>
which Tailscale then uses to assign an IP pool,
in this case a pool of size one.
Cilium is set up with native routing. This is a bit tricky with Tailscale
because each node needs to advertise routes for its podCIDRs
but we can't
directly put this in the ExtensionServiceConfig
since it isn't known when
we initialize the extension.
Instead, two extra init containers run with the cilium agent.
The first retrieves the node's podCIDRs
and the second calls
tailscale set --advertise-routes
with those podCIDRs
.
The Tailscale extension is configured with --accept-routes
and --snat-subnet-routes=false
,
since we don't want or need to SNAT between Pods on different nodes.
Ideally this functionality would be upstreamed but for now, this just works.
This requires some ACLs set up in Tailscale:
{
"ipsets": {
"ipset:k8rn-pods": [
"add 10.244.0.0/16",
],
},
"autoApprovers": {
"routes": {
"10.244.0.0/16": [ // can't use ipset here
"tag:k8rn-node",
],
},
},
"acls": {
...
// k8rn: nodes and pods can reach each other
{
"action": "accept",
"src": ["tag:k8rn-node", "ipset:k8rn-pods"],
"dst": [
"tag:k8rn-node:*",
"ipset:k8rn-pods:*",
],
},
},
}
My nodes are Beelink EQ12s, their N100 CPU is a 12th generation Intel, so it has Intel Quick Sync Video support for hardware acceleration with Jellyfin.
This repo uses
Intel's GPU plugin
to make everything available to the Jellyfin Pods
.
k8rn is running Envoy gateway with a relatively complicated architecture.
Ideally we'd use the Tailscale operator to expose the gateway as a LoadBalancer
Service
and the Gateway
would get its own Tailscale machine.
However due to tailscale/tailscale#12393 this is currently infeasible.
Instead we rely on the nodes being Tailscale machines:
net.ipv4.ip_unprivileged_port_start=0
on the nodes so we can listen on 443
without rootPods
run with hostNetwork: true
useListenerPortAsContainerPort: true
so the container really listens on 443
EnvoyPatchPolicy
modifies the listener to bind only to tailscale0
with SO_BINDTODEVICE
NodePort
so that external-dns
creates A/AAAA records pointing to each of the node IPs.
DaemonSet
Some services are running on a different server in the tailnet and a Service
with a manually managed EndpointSlice
handles forwarding the traffic.