🔗 Use Cilium Cluster Mesh for multi-cluster networking.
Learn Cilium and Cilium Cluster Mesh by creating a Multi-cluster Kubernetes setup.
[!WARNING] For MacOS users it is recommended to use colima instead of Docker Desktop. See https://github.com/cilium/cilium/issues/30278
colima
(optional for MacOS users) - Installation instructions
docker
- Installation instructions
kind
- Installation instructions
kubectl
- Installation instructions
cilium-cli
- Installation instructions
colima
configurationEnsure inotify
setting are correct:
cp colima/override.yaml ~/.colima/_lima/_config
Start a virtual machine with 4 cpu and 8 Gb of RAM:
colima start --cpu 4 --memory 8
Prepare environment with just one command:
make all
This command can help you with:
kind
;cilium-cli
;[!TIP] Feel free to check out Makefile. It is self-explanatory
rebel-base
service from Cluster 1We can observe that all requests are load balanced between Cluster 1 and Cluster 2.
kubectl --context kind-cluster1 exec -ti deployments/x-wing -- /bin/sh -c 'for i in $(seq 1 10); do curl rebel-base; done'
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
rebel-base
service from Cluster 2Similarly to scenario above all requests are load balanced between Cluster 2 and Cluster 1.
kubectl --context kind-cluster2 exec -ti deployments/x-wing -- /bin/sh -c 'for i in $(seq 1 10); do curl rebel-base; done'
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
Set service.cilium.io/affinity
annotation for local
, the Global Service will load-balance across healthy local backends, and only use remote endpoints if all of local backends are not available or unhealthy.
kubectl --context kind-cluster1 annotate service rebel-base io.cilium/service-affinity=local --overwrite
Check the destination. As you can see the preferred endpoint destination is Cluster 1 (local
endpoints).
kubectl --context kind-cluster1 exec -ti deployments/x-wing -- /bin/sh -c 'for i in $(seq 1 10); do curl rebel-base; done'
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
Remove service.cilium.io/affinity
annotation.
kubectl --context kind-cluster1 annotate service rebel-base io.cilium/service-affinity-
You will see replies from pods in both clusters as usual.
kubectl --context kind-cluster1 exec -ti deployments/x-wing -- /bin/sh -c 'for i in $(seq 1 10); do curl rebel-base; done'
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
[!IMPORTANT] Try to use
service.cilium.io/affinity
withremote
configuration in Cluster 2 and observe the results.
[!TIP] Documentation for Global Service Affinity - https://docs.cilium.io/en/latest/network/clustermesh/affinity/
rebel-base
service from Cluster 3That's the fun part. In Cluster 3 we don't have any deployments for rebel-base
. But we can use global rebel-base
service to reach pods in Cluster 1 and Cluster 2.
Let's try it out. We got responses from Cluster 1 and Cluster 2.
kubectl --context kind-cluster3 exec -ti deployments/x-wing -- /bin/sh -c 'for i in $(seq 1 10); do curl rebel-base; done'
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
[!NOTE] Feature proposal for global service cluster affinity.
Let's try to make service temporarily unavailable in Cluster 1 and observe failover to Cluster 2.
kubectl --context kind-cluster1 scale deploy --replicas=0 rebel-base
Send some http requests to rebel-base
in Cluster 1, see failover in action.
kubectl --context kind-cluster1 exec -ti deployments/x-wing -- /bin/sh -c 'for i in $(seq 1 10); do curl rebel-base; done'
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
Check rebel-base
service from Cluster 2.
kubectl --context kind-cluster2 exec -ti deployments/x-wing -- /bin/sh -c 'for i in $(seq 1 10); do curl rebel-base; done'
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
{"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
Return rebel-base
deployment in Cluster 1 to normal operation.
kubectl --context kind-cluster1 scale deploy --replicas=2 rebel-base
[!IMPORTANT] Try scale up/down
rebel-base
deployment in Cluster 2 and observe the result.
make clean
It will delete all three clusters.