Running Knative on Raspberry Pi

Setup Knative on Raspberry Pi in 5 minutes

Hardware

• Get Raspberry 4, pick one that has 2GB or more of RAM. I would recommend getting the 8GB.
• Accesories:
  • Micro SD Card
  • Raspberry Power Supply (USB-C)
• Extras:
  • Fan, heat sinks, case

Setup Operating System (OS)

This instructions are based on Ubuntu Post

• You will need a 64 bit OS, you can use Ubuntu or Raspberry OS. I will be using Ubuntu 20.10 64bit
• Get an application to image the OS into the SD Card, you can use RaspBerry Imager or any other tool. Choose MacOS or Windows

1. Scroll down and Select Ubuntu Server 20.10 64bit (RPi 3/4/400) (Do not select Desktop)
   
2. Select SD Card
3. WRITE
4. WAIT for ~10 minutes

Configure some environment variables based on your operating system, you can use the template provided

cp template.env .env

Enter the values based on your Operating system in to the .env file

BOOT_DRIVE=/Volumes/system-boot/
WIFI_SSID=SSID
WIFI_PASSWORD=PASSWORD

Load the environment variables

source .env

Enable cgroups for the kernel

Enable cgroups for the kernel by appending to the cmdline.txt file

• Append to the one line, do not add a new line, a backup copy cmdline.txt.bak is made.

source .env
sed -i .bak 's/$/ cgroup_memory=1 cgroup_enable=memory/' ${BOOT_DRIVE}/cmdline.txt
cat ${BOOT_DRIVE}/cmdline.txt

Configure Network

After the SD Card is imaged, remove the card and re-insert. The boot filesystem should be visible from example on MacOS under /Volumes/system-boot/

Create a file network-config with the wifi info, replace the values for SSID and PASSWORD below

source .env
cp ${BOOT_DRIVE}/network-config ${BOOT_DRIVE}/network-config.bak
cat <<EOF >${BOOT_DRIVE}/network-config
version: 2
ethernets:
  eth0:
    match:
      driver: bcmgenet smsc95xx lan78xx
    set-name: eth0
    dhcp4: true
    optional: true
wifis:
  wlan0:
    dhcp4: true
    optional: true
    access-points:
      ${WIFI_SSID}:
        password: "${WIFI_PASSWORD}"
EOF

Boot up

Umount/Eject SD Card from your computer, insert into raspberry pi and power on the raspberry pi.

You can discover the IP via your router, or plug a monitor and keyboard and use the command ip a to get the ip address.

You can also use a network scan

nmap -sn 192.168.7.0/24

This would be something like 192.168.x.x in my case is 192.168.7.101

Update the file .env

IP=192.168.7.101
echo IP=$IP >>.env

Setup ssh

Login and change the default password ubuntu to something different

source .env
ssh ubuntu@$IP

Check if you have a local ssh, if not then create a new one with ssh-keygen

ls ~/.ssh/id_rsa*

If you have more than one ssh key you can use -i to be sure it picks the correct one.

source .env
ssh-copy-id -i $HOME/.ssh/id_rsa ubuntu@$IP

Now you can ssh into the pi without a password

source .env
ssh ubuntu@$IP uname -m

It should print aarch64 to indicate arm64

I recommend to change the hostname of the pi from the default ubuntu before you install kubernetes to something useful like pi4

source .env
NEW_HOSTNAME=pi4
ssh ubuntu@$IP "sudo hostnamectl set-hostname ${NEW_HOSTNAME}; sudo sed -i.bak s/localhost/${NEW_HOSTNAME}/g /etc/hosts"

Install Kubernetes

• Install the k3sup command line tool, check that you have latest version I'm using 0.9.7

  k3sup version
  

• Run the following command to install k3s on the pi as a master node.

  source .env
  k3sup install \
  --ip $IP \
  --user ubuntu \
  --merge \
  --local-path $HOME/.kube/config \
  --context knative-pi \
  --k3s-channel latest \
  --k3s-extra-args '--disable=traefik'
  

  • The --ip specifies the IP Address to ssh in to the pi
  • The --user specifies the user to ssh as
  • The --merge specified to merge the kubeconfig of the new cluster into an existing kubeconfig file
  • The --local-path specifies which file to write the kubeconfig file, in our case specify an existing one to merge in the new config
  • The --context specifies the context for the cluster when you use kubectl or kn
  • The --k3s-channel specifies which version of kubernetes to install
  • The --k3s-extra-args with --disable=traefik is to avoid the installation of traefik as we are going to use knative networking

• If you ever need to recover the kubeconfig from the cluster you can use the flag --skip-install1

  source .env
  k3sup install \
  --ip $IP \
  --user ubuntu \
  --merge \
  --local-path $HOME/.kube/config \
  --context knative-pi \
  --skip-install
  

Add worker nodes

• To add worker nodes use the join command, $AGENT_IP is the ip of the next raspberry ip, and $IP is the IP of the first pi acting as master node.

• Update .env file with AGENT_IP

  AGENT_IP=192.168.7.103
  echo AGENT_IP=$AGENT_IP >>.env
  

  source .env
  k3sup join \
  --ip $AGENT_IP \
  --server-ip $IP \
  --user ubuntu \
  --k3s-channel latest
  

• You con set a role for the worker node:

  NODE=pi-worker
  kubectl label node ${NODE} node-role.kubernetes.io/worker=true
  

Verify Kubernetes installed

1. switch your context using kubectx

   kubectx knative-pi
   

2. List the nodes

   kubectl get nodes -o wide
   

3. The output should look like this

   NAME     STATUS   ROLES    AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION     CONTAINER-RUNTIME
   ubuntu   Ready    master   54m   v1.19.3+k3s2   192.168.7.218   <none>        Ubuntu 20.10   5.8.0-1006-raspi   containerd://1.4.0-k3s1
   

Install Knative Serving

TLDR; ./01-knative-serving.sh

I will be using the pre-release build and install instructions

1. Install Knative Serving

   kubectl apply -f https://storage.googleapis.com/knative-nightly/serving/latest/serving-crds.yaml
   kubectl wait --for=condition=Established --all crd
   kubectl apply -f https://storage.googleapis.com/knative-nightly/serving/latest/serving-core.yaml
   kubectl wait pod --timeout=-1s --for=condition=Ready -n knative-serving -l '!job-name'
   

2. Install Contour 1.10 that uses arm enable envoy with 1.16

   kubectl apply -f https://storage.googleapis.com/knative-nightly/net-contour/latest/contour.yaml
   kubectl wait --for=condition=Established --all crd
   kubectl wait pod --timeout=-1s --for=condition=Ready -n contour-external -l '!job-name'
   kubectl wait pod --timeout=-1s --for=condition=Ready -n contour-internal -l '!job-name'
   

3. Install the Knative Network Controller:

   kubectl apply --filename https://storage.googleapis.com/knative-nightly/net-contour/latest/net-contour.yaml
   kubectl wait pod --timeout=-1s --for=condition=Ready -n knative-serving -l '!job-name'
   

4. To configure Knative Serving to use previous installed Network Controller by default:

   kubectl patch configmap/config-network \
   --namespace knative-serving \
   --type merge \
   --patch '{"data":{"ingress.class":"contour.ingress.networking.knative.dev"}}'
   

5. Set the environment variable EXTERNAL_IP to External IP Address of the Worker Node

   EXTERNAL_IP="$(kubectl get svc envoy -n contour-external  -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')"
   echo EXTERNAL_IP==$EXTERNAL_IP
   

6. Set the environment variable KNATIVE_DOMAIN as the DNS domain using nip.io

   KNATIVE_DOMAIN="$EXTERNAL_IP.nip.io"
   echo KNATIVE_DOMAIN=$KNATIVE_DOMAIN
   

   Double check DNS is resolving

   dig $KNATIVE_DOMAIN
   

7. Configure DNS for Knative Serving

   kubectl patch configmap -n knative-serving config-domain -p "{\"data\": {\"$KNATIVE_DOMAIN\": \"\"}}"
   

8. Verify that Knative is Installed properly all pods should be in Running state and contour-external service configured.

   kubectl get pods -n knative-serving
   kubectl get pods,svc -n contour-external
   

Deploy Knative Application

Deploy using Knative CLI kn:

kn service create hello --port 8080 --image csantanapr/helloworld-go:latest

Optional: Deploy a Knative Service using a yaml manifest:

cat <<EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hello
spec:
  template:
    spec:
      containers:
      - image: csantanapr/helloworld-go:latest
        ports:
        - containerPort: 8080
        env:
        - name: TARGET
          value: "Knative"
EOF

Wait for Knative Service to be Ready

kubectl wait ksvc hello --all --timeout=-1s --for=condition=Ready

Get the URL of the new Service

SERVICE_URL=$(kubectl get ksvc hello -o jsonpath='{.status.url}')
echo $SERVICE_URL

Test the App

curl $SERVICE_URL

Output should be:

Hello World from Golang!

Check the knative pods that scaled from zero

kubectl get pod -l serving.knative.dev/service=hello

Output should be:

NAME                                      READY   STATUS    RESTARTS   AGE
hello-00001-deployment-68bb96f946-wfzxc   2/2     Running   0          93s

Try the service url on your browser (command works on linux and macos)

open $SERVICE_URL

You can watch the pods and see how they scale down to zero after http traffic stops to the url

kubectl get pod -l serving.knative.dev/service=hello -w

Output should look like this:

NAME                                     READY   STATUS
hello-00001-deployment-68bb96f946-wfzxc   2/2     Running
hello-00001-deployment-68bb96f946-wfzxc   2/2     Terminating
hello-00001-deployment-68bb96f946-wfzxc   1/2     Terminating
hello-00001-deployment-68bb96f946-wfzxc   0/2     Terminating

Try to access the url again, and you will see a new pod running again.

NAME                                     READY   STATUS
hello-r4vz7-deployment-c5d4b88f7-rr8cd   0/2     Pending
hello-r4vz7-deployment-c5d4b88f7-rr8cd   0/2     ContainerCreating
hello-r4vz7-deployment-c5d4b88f7-rr8cd   1/2     Running
hello-r4vz7-deployment-c5d4b88f7-rr8cd   2/2     Running

Some people call this Serverless 🎉 🌮 🔥

If you want to see how much CPU and Memory all the pods are consuming use kubectl top

kubectl top pods -A --sort-by=cpu
NAMESPACE          NAME                                          CPU(cores)   MEMORY(bytes)
knative-serving    webhook-c76796d6f-4rh2n                       52m          13Mi
knative-serving    controller-7f6d88c75b-bmm5v                   29m          24Mi
knative-serving    autoscaler-b57c65d47-w8rs6                    21m          16Mi
contour-external   envoy-rqbd4                                   14m          25Mi
kube-system        coredns-66c464876b-nttx5                      12m          8Mi
contour-internal   envoy-kvk8d                                   12m          25Mi
knative-serving    contour-ingress-controller-5fdd4b6f94-hrqth   8m           14Mi
kube-system        metrics-server-7b4f8b595-6zbrf                6m           13Mi
contour-internal   contour-6b84898d48-rtw9j                      5m           19Mi
knative-serving    activator-5ff578d869-nxm4b                    5m           17Mi
contour-external   contour-5fd57c546b-2lgsl                      4m           19Mi
kube-system        local-path-provisioner-7ff9579c6-tph2s        4m           7Mi
contour-external   contour-5fd57c546b-qlxbm                      3m           29Mi
contour-internal   contour-6b84898d48-r77lj                      2m           16Mi
contour-external   svclb-envoy-wqwxm                             0m           3Mi
default            hello-00001-deployment-6d8674c767-2zx86       0m           2Mi

And this is what htop shows

Build your containers with multi architectures

I have two sample applications in apps/ one for go and one for nodejs

Build arm64 container image for go

1. Install ko version v0.6.0+ from https://github.com/google/ko
2. Setup your docker registry info, replace the value of DOCKER_HUB_USER

   export DOCKER_HUB_USER=csantanapr
   export KO_DOCKER_REPO="docker.io/${DOCKER_HUB_USER}"
   

3. change directory to the root of the go app

   cd apps/helloworld-go
   

4. Publish container image for all architecture types including arm64

   ko publish --platform=all -B .
   

5. Deploy as Knative service

   kn service create helloworld-go --image ${KO_DOCKER_REPO}/helloworld-go
   

6. Run the app

   curl $(kn service describe helloworld-go -o url)
   

Build arm64 container image for nodejs

1. Install Docker-Desktop with experimental enable follow instructions here https://www.docker.com/blog/multi-arch-images/
2. Create a new builder if you don't have one already

   docker buildx create --name mybuilder
   

3. Configure your docker registry

   export DOCKER_HUB_USER=csantanapr
   

4. change directory to the root of the go app

   cd apps/helloworld-nodejs
   

5. Build and Push the image

   docker buildx build \
    -t ${DOCKER_HUB_USER}/helloworld-nodejs:latest \
    --platform linux/amd64,linux/arm64 \
    --push .
   

6. Deploy as Knative service

   kn service create helloworld-nodejs --image docker.io/${DOCKER_HUB_USER}/helloworld-nodejs
   

7. Run the app

   curl $(kn service describe helloworld-nodejs -o url)
   

Delete Cluster

Uninstall kubernetes

ssh ubuntu@$IP sudo /usr/local/bin/k3s-uninstall.sh

If you have any issues with this instructions open an new issue please 🙏🏻

Related Posts

• https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/
• https://blog.alexellis.io/raspberry-pi-homelab-with-k3sup/
• https://itsmurugappan.medium.com/knative-on-raspberry-pi-1106984de5b8