https://www.cncf.io/certification/cka/
As of 05/2022
vim
- Text/Code editortmux
- Terminal multiplexorjq
- Working with JSON formatyq
- Working with YAML formatbase64
- Tool to convert to and from base 64grep
, wc
...The following are useful terminal shortcut aliases/shortcuts to use during the exam.
Add the following to the end of ~/.bashrc
file:
alias k='kubectl # <-- Most general and useful shortcut!
alias kd='kubectl delete --force --grace-period=0 # <-- Fast deletion of resources
alias kc="kubectl create" # <-- Create a resource
alias kc-dry='kubectl create --dry-run=client -o yaml # <-- Create a YAML template of resource
alias kr='kubectl run' # <-- Run/Create a resource (typically pod)
alias kr-dry='kubectl run --dry-run=client -o yaml # <-- Create a YAML template of resource
# If kc-dry and kr-dry do not autocomplete, add the following
export do="dry-run=client -o yaml" # <-- Create the YAML tamplate (usage: $do)
The following are some example usages:
k get nodes -o wide
kc deploymentmy my-dep --image=nginx --replicas=3
kr-dry my-pod --image=nginx --command sleep 36000
kr-dry --image=busybox -- "/bin/sh" "-c" "sleep 36000"
kr --image=busybox -- "/bin/sh" "-c" "sleep 36000" $do
The following is useful so that you can use the TAB key to auto-complete a command, allowing you to not always have to remember the exact keyword or spelling.
Type the following into the terminal:
kubectl completion bash >> ~/.bashrc
- kubectl
command completionkubeadm completion bash >> ~/.bashrc
- kubeadm
command completionexec $SHELL
- Reload shell to enable all added completionThe exam will have VIM or nano terminal text editor tools available. If you are using
VIM ensure that you create a ~/.vimrc
file and add the following:
set ts=2 " <-- tabstop - how many spaces is \t worth
set sw=2 " <-- shiftwidth - how many spaces is indentation
set et " <-- expandtab - Use spaces, never \t values
set mouse=a " <-- Enable mouse support
Or simply:
set ts=2 sw=2 et mouse=a
Also know VIM basics are as follows. Maybe a good idea to take a quick VIM course.
vim my-file.yaml
- If file exists, open it, else create it for editing:w
- Save:x
- Save and exit:q
- Exit:q!
- Exit without savingi
- Insert mode, regular text editor modev
- Visual mode for selectionESC
- Normal modeOften times you will want to paste text or code from the Kubernetes documentation into into a VIM terminal. If you simply do that, the tabs will do funky things.
Do the following inside VIM before pasting your copied text:
NORMAL
mode, type: :set paste
INSERT
mode-- INSERT (paste) --
at the bottom of the screenCTRL + SHIFT + v
tmux
tmux
will allow you to use multiple terminal windows in one (aka terminal multiplexing).
Make sure you know the basics for tmux
usage:
NOTE:
CTRL + b
is the prefix to anything intmux
tmux
- Turn and enter tmux
CTRL + b "
- Split the window vertically (line is horizontal)CTRL + b %
- Split the window horizontally (line is vertical)CTRL + b <ARROW KEY>
- Switch between window panesCTRL + b (hold) <ARROW KEY>
- Resize current window paneCTRL + b z
- Toggle full terminal/screen a pane (good for looking at a full document)CTRL + d
or exit
- Close a window paneIf you want to be able to click and select within tmux and tmux panes, you can also enable mouse support. This can be useful.
These steps must be done outside of tmux
Create a .tmux.conf
file and edit it
vim ~/.tmux.conf
Add the configuration, save, and exit file
set -g mouse on
Reload tmux configuration
tmux source .tmux.conf
kubeadm
kubectl
The following is a way of installing and setting up Kubernetes. However, you can see other methods here. The different method depend on how and where Kubernetes is set up. - https://itnext.io/kubernetes-installation-methods-the-complete-guide-1036c860a2b3
containerd
Installation (if needed)Docs: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd
containerd
is a container runtime (as is Docker), that is needed on each kubernetes node to deal with containers.
Perform these steps on both control and worker nodes. The following is on a Debian-based system.
Load needed kernel modules (ensures when system starts up, modules will be enables)
cat << EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br-netfilter
EOF
Enable the kernel modules right now
sudo modprobe overlay
sudo modprobe br-netfilter
System level networking settings
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
Command to make networking settings take effect immediately
sudo sysctl --system
Install containerd
sudo apt-get update
sudo apt-get install -y containerd
Create containerd configuration directory
sudo mkdir -p /etc/containerd
Set default configurations
sudo containerd config default | sudo tee /etc/containerd/config.toml
Restart containerd service
sudo systemctl restart containerd
kubelet
, kubeadm
, and kubectl
CLI toolsPerform these steps on both control and worker nodes.
Disable system swap memory to utilize all of the node's resources
sudo swapoff -a
Check system fstab
file for any entries that will turn swap back on
sudo cat /etc/fstab
Install dependencies
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
Add Google Cloud public signing key to apt
for the Kubernetes repository
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
Setup the Kubernetes repository entry in apt
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
Fetch newly added repository information
sudo apt-get update
Install Kubernetes tools (note the version)
sudo apt-get install -y kubelet=1.30.2-1.1 kubeadm=1.30.2-1.1 kubectl=1.30.2-1.1
kubelet
without the =1.23.0.00
)Prevent automatic updating of Kubernetes packages for more control
sudo apt-mark hold kubelet kubeadm kubectl
kubeadm
ClusterDocs: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
Setting up a cluster using kubeadm
Perform these steps on the control node.
Initialize the cluster
sudo kubeadm init
--pod-network-cidr 192.168.0.0/16
--kubernetes-version 1.23.0
From the previous command output, run the following. This commmand is to make kubectl
work
for non-root user.
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Verify clusters nodes are there
kubectl get nodes
A this point the cluster is running, however it is in a NotReady
status, because networking
has not been configured yet. For this we need a seperate networking plugin.
Setting up a network security plugin via a remote YAML file
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
Get a node join command to use on nodes to add to cluster
kubeadm token create --print-join-command
Run the following command on each Kubernetes node. Make sure to run as sudo
sudo kubeadm join <REST OF COMMAND>
minikube
ClusterDocs: https://minikube.sigs.k8s.io/docs/
minikube
is local Kubernetes, focusing on making it easy to learn and develop for Kubernetes.
minikube
allows you set up a local kubernetes cluster with one or many nodes. Each node is
a Docker container.
Installation
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
Useful minikube
commands
minikube start
minikube stop
minikube delete
minikube node add --control-plane
minikube node add --worker
minikube node delete
minikube dashboard
Docs: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
default
namespacekube-system
namespace is for system componentsListing existing namespaces
kubectl get namespaces
Can specify where the command can run with --namespace
kubectl get pods --namespace my-namespace
Can specify to look at all namespaces with --all-namespaces
(or -A
)
kubectl get pods --all-namespaces
Create a custom namespace
kubectl create namespace my-namespace
Set a default namespace for all subsequent commands
kubectl config set-context --current --namespace=<NAMESPACE-NAME>
Need multiple control plane nodes
kube-api-server
Design Patterns
Load Balancer
kubelet
Stacked etcd
etcd
runs on the same nodes as control plane componentsetcd
instancekubeadm
use this design patternExternal etcd
etcd
runs on complete separate nodes apart from the control planeetcd
nodesInterface and make it easier to setup or use Kubernetes.
kubectl
kubeadm
minikube
kind
helm
Kompose
Kustomize
Docs: https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/
When performing maintenance, you may sometimes need to remove a Kubernetes node from service/cluster. This allows all applications on the cluster to run without any interruptions. Containers will gracefully terminated.
These commands are run on the control plane node.
Draining a node
kubectl drain <NODE NAME>
--igrnoe-daemonsets
- Ignore DaemeonSets pods tied to node--force
- Ignore error messages such as DaemonSet-managed podsAfter maintenance is complete, allow pods to run on node again
kubectl uncordon <NODE NAME>
Docs: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
Drain node, putting it out of cluster service
kubectl drain <CONTROL PLANE NODE NAME> --ignore-daemonsets
Upgrade kubeadm
CLI tool
sudo apt-get update
sudo apt-get install -y --allow-change-held-packages kubeadm=1.22.2-00
Plan the cluster upgrade
sudo kubeadm upgrade plan v1.22.2
sudo kubeadm upgrade plan
Apply the cluster upgrade
sudo kubeadm upgrade apply -y v1.22.2
Upgrade kubelet
and kubectl
CLI tools
sudo apt-get install -y --allow-change-held-packages kubelet=1.22.2-00 kubectl=1.22.2-00
sudo systemctl daemon-reload
sudo systemctl restart kubelet
Uncordon the node, putting it back into cluster service
kubectl uncordon <CONTROL PLANE NODE NAME
Docs: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#upgrade-worker-nodes
Drain node, putting it out of cluster services
kubectl drain <WORKER NODE NAME> --ignore-daemonsets --force
--force
in case of stand-alone pods--force
Upgrade kubeadm
CLI tool
sudo apt-get update
sudo apt-get install -y --allow-change-held-packages kubeadm=1.22.2-00
Upgrade the kubelet
configuration
sudo kubeadm upgrade node
Upgrade kubelet
and kubectl
CLI tools
Uncordon the node, putting it back into cluster service
kubectl uncordon <WORKER PLANE NODE NAME
etcd
Cluster DataDocs: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/
etcd
is the backend key-value data storage solution for Kubernetes cluster.
All Kubernetes objects, applications, and configurations are stored in etcd.
etcd website: https://etcd.io/
Need to use etcdctl
CLI tool
ETCDCTL_API
environmental variableetcdctl get <KEY>
- Get a specific value for a keyetcdctl --endpoints=$ENDPOINTS endpoint health
- Check all etcd
endpoint healthetcd
typically runs on port 2379
etcd
could either be running as a system service or as a kubernetes pod
Important: Set environmental variable for etcd
version
export ETCDCTL_API=3
etcd
DatabaseFor HTTPS communication, must specify certificates
HTTPS Communication Example:
etcdctl snapshot save /home/me/etcd_backup.db \
--endpoints=https://etcd1:2379 \
--cert=/path/to/file/server.crt \
--key=/path/to/file/server.key \
--cacert=/path/to/file/ca.crt
For --endpoints
, can be IP, or DNS name
etcd
endpoint by looking at etcd
service or running podkubectl get pod -n kube-system etcd-pod-name -o yaml
--listen-clinet-urls
systemctl status etcd
For HTTP communication --cert
, --key
, and --cacert
are not needed
Encryption cert values can be found using in command section of kubectl -n kube-system describe pod etcd-controlplane
--cacert
: --trusted-ca-file
-> /path/to/file/<SOMETHING>.crt
--cert
: --cert-file
-> /path/to/file/<SOMETHING>.crt
--key
: --key-file
-> /path/to/file/<SOMETHING>.key
Verify database snapshot
etcdctl --write-out=table snapshot status /home/me/etcd_backup.db
etcd
DatabaseStop etcd
service
sudo systemctl stop etcd
ssh
into the cluster node/etc/kubernetes/manifests
mkdir -p ../backup
mv ./* ../backup
Remove existing etcd
database
sudo rm -rf /var/lib/etcd
Creates a new logical cluster
etcdctl snapshot restore <SNAPSHOT FILE NAME>
etcdctl --endpoints https://etcd1:2379 snapshot restore snapshotdb
Set ownership
sudo chown -R etcd:etcd /var/lib/etcd
Start etcd
sudo systemctl start etcd
mv /etc/kubernetes/backup/* /etc/kubernetes/manifests/
Object management is done with kubectl
CLI tool used to deploy applications, inspect
and manage cluster resources, and view logs.
Docs: https://kubernetes.io/docs/reference/kubectl/
Usage: kubectl [COMMAND] [OBJECT TYPE] [OBJECT NAME] [FLAGS]
kubectl api-resources
kubectl get ....
-o, --output
- Specify output format (i.e. json
, yaml
, wide
, etc)--sort-by
- Sort output using a JSONPath expression (i.e. {.metadata.name}
)
json
to see what the JSONPath is-l, --selector
- Filter results by label (i.e. key1=value1,key2=value2
)kubectl get pods
kubectl get pv -o wide
kubectl get pods my-pod -o yaml > my-pod.yml
kubectl describe ....
kubectl describe pods/nginx
kubectl describe -f pod.json
- Pod defined by pod.json
kubectl create ....
--dry-run=clinet
kubectl create -f pod.yaml
cat pod.json | kubectl create --filename -
kubectl create deployment my-dep --image alpine --dry-run=client -o yaml
kubectl apply ....
kubectl apply -f pod.yaml
kubectl apply -k my-dir/
- Will look into my-dir
directory for all resource configskubectl delete ....
pod myPod1 myPod2
)-l name=myLabel
)--all
- Delete all specified resources in current namespace--now
- Immediate shutdown, minimal delay--force
- Bypass graceful deletionkubectl delete -f ./pod.yaml
kubectl delete pod some-pod --now
kubectl exec ....
kubectl exec <POD NAME> -c <CONTAINER> -i -t -- bash
kubectl exec some-pod -- echo "yo"
- Using first container in podkubectl exec some-pod -c python-container -- printenv
kubectl exec deploy/myDeployment -- date
- Using first pod, first container, in deploymentkubectl exec svc/myService -- ls /dir
- Using first pod, first container, in serviceDeclarative
kubectl apply -f deployment.yml
Imperative
kubectl
commands and flags.kubectl create deployment myi-deploytment image=nginx
Role-based access control (RBAC) is a method of regulating access to resources based on the roles of individual users within the organization. Essentially, what users are allowed to do and access within the cluster.
Docs: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default # <-- Note namespace defined
name: pod-reader
rules:
- apiGroups: [""] # <-- "" indicates the core API group
resources: ["pods", "pods/logs"]
verbs: ["get", "watch", "list"]
namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: default # <-- Must be in the same namespace as Role
subjects: # <-- You can specify more than one "subject"
- kind: User
name: jane # <-- "name" is case sensitive
apiGroup: rbac.authorization.k8s.io
roleRef: # <-- "roleRef" specifies the binding to Role/ClusterRole
kind: Role # <-- This must be Role/ClusterRole
name: pod-reader # <-- This must match the name of Role/ClusterRole
apiGroup: rbac.authorization.k8s.io
ClusterRoleBinding
Get all roles defined in a specific namespace
kubectl get role --namespace <NAMESPACE>
Apply a role or role binding definition
kubectl apply -f <ROLE FILE>
Check if user has certain access
kubectl get pods --namespace <NAMESPACE> --kubeconfig <USER CONFIG>
Check permissions as current user
Example: Check to see if I can do everything in my current namespace ("*" means all)
kubectl auth can-i '*' '*'
Example: Check to see if I can create pods in any namespace
kubectl auth can-i create pods --all-namespaces
Example: Check to see if I can list deployments in my current namespace
kubectl auth can-i list deployments.extensions
Check permissions as someone else
john.blah
can create deployments
kubectl auth can-i create deployments --namespace default --as john.blah
Service account provides an identity for processes that run in a Pod. Users are authenticated to Kubernetes API with User Accounts, but processes in containers inside Pods are authenticated a Service Accounts.
Service accounts exist within namespaces
Can use RBAC objects to control service accounts
Can bind service accounts with ClusterRoles or ClusterRoleBindings to provide access
Get/list service accounts
kubectl get sa
Creating service accounts
kubectl create -f <SERVICE ACCOUNT CONFIG FILE>
kubectl create sa <NAME> -n <NAMESPACE>
Example: Service account definition
apiVersion: v1
kind: ServiceAccount
metadata:
name: build-robot
automountServiceAccountToken: false
Service account tokens and certificats are stored within a pod in this directory:
/var/run/kubernetes.io/serviceaccount
Kubernetes Metrics Server
Need add-on to collect and provide metrics data about resources, pods, and containers
Need Metrics API for this
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
Once a metrics add-on is installed, can use kubectl top
to view data about resource usage
in pods and nodes.
kubectl top pod --sort-by <JSONPATH> --selector <SELECTOR>
kubectl top pod --sort-by cpu
Can view resource usage by node
kubectl top node
Checking to see if metrics server is running and responsive
kubectl get --raw /apis/metrics.k8s.io/
This is needed if the cluster uses HTTPS communication. These steps must come before creating a role or rolebinding for the user.
Docs (General): https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/ Docs: https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#normal-user
openssl genrsa -out my-user.key 2048
openssl req -new -key my-user.key -out my-user.csr
.csr
filecat my-user.csr | base64 | tr -d "\n"
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: my-user # <-- User this aplies to
spec:
request: <BASE64 ENCRYPTED VALUE OF .csr FILE>
signerName: kubernetes.io/kube-apiserver-client
expirationSeconds: 86400 # <-- 24 hours in seconds
usages:
- client auth # < -- Must abe this value
kubectl create -f my-user-csr.yaml
kubectl get csr
--> Request should be pending for this userkubectl certificate approve my-user
kubectl get pod --as my-user
kubectl auth can-i get list --as my-user
Passing dynamic values to running applications/containers at runtime
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
namespace: default
data:
key1: value1 # <-- Each key can have value
key2: value2
key3: # <-- Can have tree structure
subkey:
more_keys: data
even_more: more data
key4: |
You can have
mulit-line
data.
key5.blah: | # <-- File-like keys
yo.cool=something
yo.moo=something
immutable: false
kind: Pod
...
spec:
containers:
- ...
env:
- name: MY_ENV_VAR # <-- Visible env. var in container
valueFrom:
configMapKeyRef:
name: app-config # <-- Name of ConfigMap object
key: key1 # <-- The value to read
...
kind: Pod
...
spec:
volumes:
- name: config-vol
configMap:
name: app-config
...
echo -n "some secret text" | base64
echo "<ENCRYPTED TEXT>" | base64 -d
stringData
instead of data
base64
encoding required apiVersion: v1
kind: Secret
metadata:
name: my-secret
type: Opaque # <-- Arbitary user-defined data
data:
username: 97sdf9== # <-- base64 encoded text
password: sd89sdfh/sd9f== # <-- base64 encoded text
immutable: false # <-- Default value is false
kubectl create secret generic my-secret --from-file <LOCAL FILENAME>
Resource Requests
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- name: busybox
image: busybox
resources: # <-- Definition
requests:
cpu: "250m" # <-- 0.25 CPU cores
memory: "128Mi" # <-- 128Mi memory
Resource Limits
...
resources:
limits:
cpu: "250m"
memory: "128Mi"
Kubernetes can automatically detect unhealthy containers by actively monitoring container health.
kubelet
uses different probes to gauge the health and readiness of the containers.
exec
- Run a command, if no error, successhttpGet
- If status code is over 200 and below 400, successtcpSocket
- TCP check if port is open, if it is, successmy-pod.yaml
apiVersion: v1
kind: Pod
...
spec:
containers:
- name: my-container
...
livenessProbe:
exec: # <-- exec type check
command: # <-- If command succeeds = healthy!
- cat # <-- Command
- /tmp/healthy # <-- Arguments to command
initialDelaySeconds: 5 # <-- Wait 5 seconds after container startup
periodSeconds: 5 # <-- Run every 5 seconds
apiVersion: v1
kind: Pod
...
startupProbe:
initialDelaySeconds: 1
periodSeconds: 2
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 30 # <-- Number of times allowed to fail
httpGet: # <-- Check HTTP endpoint
scheme: HTTP
path: /
httpHeaders:
- name: Host
value: myapplication1.com
port: 80
readinessProbe:
blockKubernetes allows you to customize when and how containers to be automatically restarted.
Three different values for restart policies: Always
, OnFailure
, and Never
.
Can view pod restart status with kubectl get pod <POD NAME>
Always
(default)
apiVersion: v1
kind: Pod
...
spec:
restartPolicy: Always # <-- Note
container:
...
OnFailure
Never
Error
More than one container running inside a single pod. Containers share pod resources such as network and storage. Containers can interact with each other.
sidecar
apiVersion: v1
kind: Pod
metadata:
name: sidecar-pod
spec:
containers:
- name: busybox1
image: busybox
command: ['sh', '-c', 'while true; do echo logs data > /output/output.log; sleep 5; done']
volumeMounts:
- name: sharedvol # <-- Same shared volume
mountPath: /output# # <-- Mounted here in container
- name: sidecar
image: busybox
command: ['sh', '-c', 'tail -f /input/output.log']
volumeMounts:
- name: sharedvol # <-- Same shared volume
mountPath: /input # <-- Mounted here in container
volumes:
- name: sharedvol # <-- The volume that is shared
emptyDir: {} # <-- Temp volume exsits only for life of Pod
Docs: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
Containers that run during startup process
Listed in InitContainer
spec section
Run before any other containers in containers
section
Must run to once and to completion
Run in the order they are listed
When starting a pod, status will read Init
if currently running init container
Why?
Possible use cases
Example: Init container with simple startup delay
apiVersion: v1
kind: Pod
metadata:
name: init-pod
spec:
containers:
- name: nginx
image: nginx:1.19.1
initContainers:
- name: delay
image: busybox
command: ['sleep', '30']
Example: Wait for a service
apiVersion: v1
kind: Pod
...
spec:
...
initContainers:
- name: my-init-container
image: busybox:1.28
command: ['sh', '-c', "until nslookup my-service; do echo waiting for my-service; sleep 2; done"]
Docs: https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/
In Kubernetes, scheduling refers to making sure that Pods are matched to Nodes so that Kubelet can run them.
nodeSelector
nodeSelector
kubectl label nodes <NODE NAME> <KEY>=<VALUE>
kubectl label nodes <worker1> mylabel=someValue
apiVersion: v1
kind: Pod
...
spec:
...
nodeSelector:
mylabel: someValue # <-- Only nodes with this label
nodeName
...
spec:
...
nodeName: worker0
kubectl get nodes
Taints applied to a node allow a node to repel a set of pods. Tollerations applied to pods allow pods to be scheduled onto nodes with matching taints. (Think pods tollerate a taint)
Docs: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
Usages
Taint on Nodes
NoSchedule
- Pod scheduling is not allowed on nodePreferNoSchedule
- Cluster will try to avoid placing a pod on this nodeNoExecute
- If pod is already running on node, keep it therekubectl train nodes my-node-1 my-key=my-value:NoSchedule
kubectl get nodes --output custom-columns=NODE_NAME:.metadata.name,TAINTS:.spec.taints
kubectl describe node my-node-1 | grep -i taint
Tolleration on Pods
apiVersion: v1
kind: Pod
metadata:
...
spec:
containers:
...
tolerations:
- key: "my-key" # <-- Matching taint key
operator: "Equal" # <-- Operator can be "Equal" or "Exists"
value: "my-value" # <-- If operator is "Equal", value is needed
effect: "NoSchedule" # <-- Node taint effect to tollerate
Docs: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
TODO
DaemonSets will automatically runs a copy of a Pod on each node in the cluster
Docs: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
apiVersion: apps/v1
template
apiVersion: apps/v1 # <-- NOTE
kind: DaemonSet
metadata:
name: my-daemonset
spec:
selector:
matchLabels:
app: my-daemonset # <-- Any Pods that have this label
template: # <-- The Pod template to create pods
metadata:
labels:
app: my-daemonset # <-- Typically matches the above selector
spec:
containers:
- name: nginx
image: nginx:1.19.1
kubectl get daemonset
Static Pods are managed directly by kubelet daemon on a specific node, without the API server observing them.
Docs: https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/
/etc/kubernetes/manifests/
kubelet
configuration:
ps -aux | grep kubelet
--config
value (ie. --config=/var/lib/kubelet/config.yaml
)cat <CONFIG FILE> | grep -i static
kubectl get pods
Defines a desired state (declaritive) for a ReplicaSet (set of replica Pods)
Docs: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
replicas
- Number of replica Pods that the Deployment will seek to maintinselector
- Label selector used to identify the replica Pods managed by the Deploymenttemplate
- Pod definition used to create all replica Pods for the Deployment apiVersion: apps/v1 # <-- Note
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3 # <-- 3 Pods in this deployment
selector:
matchLabels:
app: nginx # <-- Get any pods with this metadata label
template: # <-- Definition of the Pod for this deployment
metadata: # <-- Note no "name:". Deployment will give name.
labels:
app: nginx # <-- Match label in selector above
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
kubectl set image deployments/<DEPLOYMENT NAME> <IMAGE NAME AND VERSION>
kubectl set image deployments/my-deployment nginx:1.16.1
Docs: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#scaling-a-deployment
replicas
value in YAML deployment description then kubectl apply
kubectl scale
kubectl scale deployment/my-deployment --replicas=5
kubectl autoscale
to scale depending on other factors
kubectl autoscale deployment/my-deployment --min=10 --max=15 --cpu-percent=80
kubectl edit deployment <DEPLOYMENT NAME>
replicas
in the spec
section!Rolling Updates
Rolback
Deployment rollout management commands
kubectl rollout status deployment/<DEP. NAME>
- Checking deployment rolling update statuskubectl rollout history deployment/<DEP. NAME>
- Checking deployment rollout historykubectl rollout undo deployment/<DEP. NAME>
- Undo previous deployment rolloutkubectl rollout pause deployment/<DEP. NAME>
- Pause deployment rolloutkubectl rollout resume deployment/<DEP. NAME>
- Resume paused deployment rolloutDocs: https://kubernetes.io/docs/concepts/services-networking/
Docs: https://www.ibm.com/docs/en/cloud-private/3.1.2?topic=networking-kubernetes-network-model
Docs: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/
NotReady
state until a network plugin is installedkubectl apply -f <LOCAL FILE OR REMOTE URL YAML>
kubectl apply -f https://docs.projectcalico.org/manifests/calico-typha.yaml
By default kubelet looks into
/etc/cni/net.d
to discover the CNI plugin
Docs: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
some-name
instead of IP address (ie. 192.168.1.3
)kube-system
namespacekubectl get pods -n kube-system
kuabectl get service -n kube-system
<POD IP>.<NAMESPACE NAME>.pod.cluster.local
.
replaced by -
192-168-10-100.default.pod.cluster.local
Kubernetes object that allows you to control the flow of network communication to and from the Pods
Docs: https://kubernetes.io/docs/concepts/services-networking/network-policies/
from
sectionto
sectionfrom
and to
podSelector
- Traffic from and to specific podsnamespaceSelector
- Traffic from and to specific namespacesipBlock
- Traffic from a specific IP range using CIDR notation (ie. 10.0.1.0/16)port
- Specify one or more ports that will allow traffic, includes protocol (ie. TCP
)namespaceSelector.matchLabels
to apply the policy to specific namespace apiVersion: networking.k8s.io/v1 # <-- Note
kind: NetworkPolicy
metadata:
name: test-network-policy
namespace: default
spec:
podSelector: # <-- Which pod to apply this to, same namespace
matchLabels:
role: db
policyTypes: # <-- Should batch the sections below
- Ingress # <-- If "ingress" section omitted, no in-traffic
- Egress # <-- If "egress" section omitted, no out-traffic
ingress:
- from: # <-- "from" for ingress only
- ipBlock:
cidr: 172.17.0.0/16 # <-- Allow this IP range
except:
- 172.17.1.0/24
- namespaceSelector:
matchLabels:
project: myproject # <-- Allow from name space with this label
- podSelector:
matchLabels:
role: frontend # <-- Allow Pods with this label
ports: # <-- Rules applied only to these ports/protocol
- protocol: TCP
port: 6379
egress:
- to: # <-- "to" for egress only
- ipBlock:
cidr: 10.0.0.0/24
ports:
- protocol: TCP
port: 5978
Network is not set up
NotReady
network is not set upIP
is <none>
and/or STATUS
is Pending
, network is not set upkubectl describe node <NODE NAME>
shows event of Starting kube-proxy
kubectl get pods -n kube-system
shows no network plugin pod (ie. calico*
)kubectl apply -f <NETOWRK PLUGIN YAML>
Misconfigured NetworkPolicies
curl <IP ADDRESS>
from one pod cannot reach another podkubectl get networkpolicy
kubectl describe networkpolicy <NAME>
kubectl edit networkpolicy -n <NAMESPACE> <NAME>
kubectl apply -f <NETWORK POLOCY YAML>
All open traffic between Pods, namespaces, IP
Services expose applications running as a set of Pods.
Docs: https://kubernetes.io/docs/concepts/services-networking/service/
kubectl get endpoints <SERVICE NAME>
How and where the Service will expose the application.
kubectl create service clusterip svc-internal --tcp=80:80 --dry-run='client' -o yaml > svc-internal.yml
--help
--tcp=<INSIDE PORT>:<OUTSIDE TARGETPORT
apiVersion: v1 # <-- Note
kind: Service
metadata:
name: svc-clusterip
spec:
type: ClusterIP # <-- Can leave out, it is default anyways
selector:
app: svc-example # <-- Locate and attach to Pods with this label
ports: # <-- Can have many ports
- protocol: TCP
port: 80 # <-- Service listens on this port (outside pod)
targetPort: 80 # <-- Port on Pods attached to this service (inside pod)
<NODE IP>:<NodePort>
nodePort
keykubectl create service nodeport svc-external --tcp=80:80 --dry-run=client -o yaml > svc-external.yml
--help
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
type: NodePort # <-- NOTE
selector:
app: MyApp
ports:
# By default and for convenience, the `targetPort` is set to the same value as the `port` field.
- port: 80 # <-- Service listens on this port (outside pod)
targetPort: 80 # <-- Ports on Pods attached to this service (inside pod)
nodePort: 30007 # <-- Exposed port on nodes. Optional, by default chosen 30000-32767 (outside cluster)
externalName
field (ie. foo.bar.example.com)Docs: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#expose
kubectl expose pod my-pod --name my-service --type ClusterIP
kubectl expose deployment my-deploy --name my-service --type NodePort
kubectl expose pod my-pod --name my-service --type NodePort --dry-run=client --output yaml
Docs: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
<SERVICE NAME>.<NAMESPACE NAME>.svc.<CLUSTER DOMAIN>
<CLUSTER DOMAIN>
is cluster.local
my_service.default.svc.cluster.local
my_service.testing.svc.my_company.com
curl my-service
Docs: https://kubernetes.io/docs/concepts/services-networking/ingress/
NodePort
on each nodekubectl create ingress <NAME> --path:<SERVICE>:<PORT> [OPTIONS]
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
spec:
rules:
- http:
paths:
- path: /somepath # <-- When this path is requested
pathType: Prefix # <-- Type of URL matching
backend:
service:
name: my-service # <-- Route to this service, can be ClusterIP-based
port:
number: 80 # <-- To this service port
http://some-endpoint.com/somepath
my-service
at port 80
port
...
backend:
service:
name: my-service
port:
name: http-port # <-- In Pod: spec.ports.name
kubectl describe ingress <INGRESS NAME>
pathType
Exact
- Matches URL path exactly and is case sensitivePrefix
- Mataches based on URL path prefix split by /
. (ie. /yo
matches /yo/blah
)Docs: https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/
Allow storage of data outside of the container file system while allowing the container to access the data at runtime.
Docs: https://kubernetes.io/docs/concepts/storage/volumes/
apiVersion: v1
kind: Pod
metadata:
name: pod-with-volume
spec:
containers:
- name: busybox
image: busybox
volumeMounts: # <-- List of mounts within container
- name: my-volume # <-- Reference volume in volume section
mountPath: /output # <-- Directory within the container
- name: busybox2 # <-- Can have more than one container
image: busybox
volumeMounts:
- name: my-volume # <-- Same volume to share data between containers
mountPath: /input
volumes:
- name: my-volume
hostPath: # <-- Reference dir/file on host Node filesystem
path: /var/data # <-- Directory on Pod
- name: my-empty-dir
emptyDir: {} # <-- Temp. empty directory on Node only for life of Pod
Docs: https://kubernetes.io/docs/concepts/storage/volumes/#volume-types
Volumes and Persistent Volumes have a volume type, which determines how the storage is actually handled (mechanism for storage).
Various volumes types support storage methods such as:
hostPath
type
can be:
Directory
File
Socket
path
is the directory on the Node...
volumes:
- name: config-vol
hostPath:
path: /data # <-- Name where volume is mounted
type: Directory
emptyDir: {}
configMap
...
volumes:
- name: config-vol
configMap:
name: log-config # <-- Name of the ConfigMap
items:
- key: log_level
path: log_level
*
- Note, if StorageClass is defined, PersistentVolume can be dynamically allocated and doesDocs: https://kubernetes.io/docs/concepts/storage/storage-classes/
parameters
may be accepted depending on the provisioner
kubernetes.io/aws-ebs
we can have the following
provisioner: kubernetes.io/aws-ebs
parameters:
type: io1
iopsPerGB: "10"
fsType: ext4
allowVolumeExpansion
determines whether or not the StorageClass supports the ability to resizefalse
and try to resize, will get an errorfalse
apiVersion: storage.k8s.io/v1 # <-- Note
kind: StorageClass
metadata:
name: localdisk
provisioner: kubernetes.io/no-provisioner # <-- Type of service/platorm/provider
allowVolumeExpansion: true # <-- Volume can be resized after creation
Docs: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
storageClassName
references a StorageClass
objectpersistentVolumeReclaimPolicy
determines how the storage resources can be reused when theRetain
- Keep all data. Manual clean up and prepare for reuseRecycle
- Delete all data. Basic scrub (rm -rf /my-volume/*
)Delete
- Associated storage asset is deleted (only for cloud resources)kubectl get pv
Available
- Not bound to PersistentVolumeClaimBound
- Bound to a PersistentVolumeClaimReleased
- Claim has been deleted, resources not yet reclaimed by clusterFailed
- Voulme failed automatic reclamation apiVersion: v1
kind: PersistentVolume
metadata:
name: my-pv
spec:
storageClassName: localdisk # <-- References a StorageClass
persistentVolumeReclaimPolicy: Recycle # <-- Scrub and reuse
capacity:
storage: 1Gi # <-- Total available to be able to claim
accessModes:
- ReadWriteOnce # <-- Must match PersistentVolumeClaim!
hostPath: # <-- Type of storage
path: /var/output
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
storageClassName: localdisk # <-- Reference StorageClass
volumeMode: Filesystem # <-- Default value
accessModes:
- ReadWriteOnce # <-- Must match PersistentVolume!
resources:
requests:
storage: 100Mi # <-- Storage claimed from PersistentVolume
apiVersion: v1
kind: Pod
metadata:
name: pv-pod
spec:
containers:
- name: busybox
image: busyboxs
volumeMounts:
- name: pv-storage
mountPath: /output # <-- Volume mounted here
volumes:
- name: pv-storage
persistantVolumeClaim:
claimName: my-pvc # <-- Reference to pvc
kubectl edit pvc <PVC NAME>
kubectl apply -f <EDITED YAML FILE>
resources.requests.storage
kubectl
cannot interact with clusterThe connection to the server <ADDRESS>:<PORT> was refused = did you specify the right host or port?
docker --version
sudo systemctl status kubelet
Check Node status
kubectl get nodes
kubectl describe node <NODE NAME>
Check status and/or starting/enabling services
docker
and kubelet
sudo systemctl status kubelet
sudo systemctl start kubelet
sudo systemctl enable kubelet
kubeadm
cluster, several kubernetes components run as pods in kube-system
namespace
kube-system
component status
kubectl get pods -n kube-system
kubectl describe pod <POD NAME> -n kube-system
Can check logs for Kubernetes related services on each node using journalctl
sudo journalctl -uf kubelet
sudo journalctl -uf docker
Notes:
sudo
- Ensure to run command as root-u
- Show messages for specified systemd unit pattern-f
- Show latest logs and continously updateSHIFT-g
- Keyboard shortcut to jump to the end of logsKubernetes cluster components have log output redirected to /var/log
/var/log/kube-apiserver.log
/var/log/kube-scheduler.log
/var/log/kube-controller-manager.log
NOTE: kubeadm
clusters may not have these components because components run inside container.
In that case, access with the following:
- bash kubectl logs -n kube-system <SYSTEM POD NAME>
Getting status
kubectl get pod
kubectl describe pod <POD NAME>
Run command inside container in pod
kubectl exec <POD NAME> -c <CONTAINER NAME> -- <COMMAND>
Interactive shell inside the pod container
kubectl exec <POD NAME> -c <CONTAINER NAME> -it -- sh
Get container logs
kubectl logs <POD NAME> -c <CONTAINER NAME>
Check if kubernetes networking plugin is up and running
kubectl get pods --all-namespaces
Check kube-proxy
-kube-proxy
runs inside kube-system
namespace
kubectl logs
kubectl logs kube-porxy-XXXXX
Check kubernetes DNS
kube-system
namespacekubectl logs coredns-XXXXXX-XXXX
netshoot
Toolnicolaka/netshoot
Create the netshoot
container
netshoot
pod
apiVersion: v1
kind: Pod
metadata:
name: nginx-netshoot
spec:
containers:
- name: nginx
image: nicolaka/netshoot
command: ['sh', '-c', 'watch ls']
Create netshoot
container and use kubectl exec
into netshoot
container to explore network
kubectl exec -i -t netshoot -- sh
Explore network
curl <NETWORK RESOURCE>
- HTTP/HTTPS requestping <NETWORK RESOURECE>
- Check is something is up and reachablenslookup <NETWORK RESOURECE>
- Get DNS info on a IP or FAQN URLnetstat
python3
Create a sample YAML file of the resource to modify later using --dry-run -o yaml
kubectl create deployment my-deployment --image=nginx --dry-run=client -o yaml
Get the definition of a currently run pod
kubectl get pod <POD NAME> -o yaml > my-pod.yml
Record a command using --record
to add to the object's describe description.
describe
under Annotations:
kubectl scale development my-development replicas=5 --record
Use sample/template resource YAML configuration as a base, then add to it
Can edit an active resource object using kubectl edit <RESOURCE TYPE> <RESOURCE NAME>
kubectl edit deployment my-deployment
kubectl edit networkpolicy -n some-namespace my-networkpolicy
Adding CLI command completion to shell
kubeadm completion bash >> .bashrc
kubectl completion bash >> .bashrc
exec bash
) or open a new shellCheck reaching a pod/service from temporary pod
nikolaka/netshoot
image apiVersion: v1
kind: Pod
metadata:
name: netshoot
spec:
containers:
- name: netshoot
image: nicolaka/netshoot
command: ['watch', 'ls'] # <-- Runs "ls" every 2 seconds
curl
is not there, run wget
form this pod to other objects
kubectl exec curl-pod -- wget -O - <POD IP>:<PORT OF POD>
kubectl exec curl-pod -- wget -O - <SERVICE NAME>:<ClusterIP PORT>
Check DNS entries within a Pod
kubectl exec <POD NAME> -- nslookup <IP ADDRESS>
kubectl exec my-test-pod -- nslookup 10.104.162.248
Get specific field form YAML output using yq
jq
with -o json
or straigh build-in JSONPath with -o jsonpath
<COMMAND> -o yaml | yq r - <TOP LEVEL KEY>.<SUB KEY>.<SO ON>
kubectl get secret credentials -n demo -o yaml | yq r - data.password
| base64 -d
Count the number of lines of output with <COMMAND> | wc -l
kubectl get pod --namespace my-namespace | wc -l
To view information for a certificat
ps aux | grep kubelet
-> --cert-dir
-> pki
openssl req -noout -text -in ./server.csr
openssl x509 -noout -text -in ./server.crt
Reach the kubernetes API without kubectl
curl https://kubernetes.default
curl -k https://kubernetes.default
curl -k https://kubernetes.default/api/v1/secretes
Docs: https://kubernetes.io/docs/reference/kubectl/jsonpath/
Docs: https://dev.to/downey/how-to-make-kubectl-jsonpath-output-on-separate-lines-52bm
JSONPath is useful when trying to extract specific information from the information return by the
kubectl
command.
kubectl get nodes minikube-m02
will give something like:
NAME STATUS ROLES AGE VERSION
minikube-m02 Ready <none> 142m v1.23.3
Complete information about that node can shown with kubectl get nodes minikube-m02 -o json
{
"apiVersion": "v1",
"kind": "Node",
"metadata": {
"annotations": {
"kubeadm.alpha.kubernetes.io/cri-socket": "/var/run/dockershim.sock",
"node.alpha.kubernetes.io/ttl": "0",
"volumes.kubernetes.io/controller-managed-attach-detach": "true"
},
"creationTimestamp": "2022-04-28T13:34:34Z",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
<----- SNIP ----- >
"sizeBytes": 682696
}
],
"nodeInfo": {
"architecture": "amd64",
"bootID": "b44856c4-a1ef-4cf8-8319-9f95b2131580",
"containerRuntimeVersion": "docker://20.10.12",
"kernelVersion": "5.4.72-microsoft-standard-WSL2",
"kubeProxyVersion": "v1.23.3",
"kubeletVersion": "v1.23.3",
"machineID": "b6a262faae404a5db719705fd34b5c8b",
"operatingSystem": "linux",
"osImage": "Ubuntu 20.04.2 LTS",
"systemUUID": "b6a262faae404a5db719705fd34b5c8b"
}
}
}
To only get the the node apiVersion
you can apply JSONPATH to the output
kubectl get nodes minikube-m02 --output jsonpath="{.apiVersion}"
v1
To only get the value of status.nodeInfo.osImage
kubectl get nodes minikube-m02 --output jsonpath="{.apiVersion.nodeInfo.osImage}"
Ubuntu 20.04.2 LTS
TIP: To add a new line at the end of the output add {'\n'}
to the JSONpath
--output jsonpath="{.apiVersion.nodeInfo.osImage}{'\n'}"
kubectl get nodes
will give something like:
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 23d v1.23.3
minikube-m02 Ready <none> 6h32m v1.23.3
minikube-m03 Ready <none> 6h32m v1.23.3
Complete information about that node can shown with kubectl get nodes -o json
{
"apiVersion": "v1",
"items": [
{
"apiVersion": "v1",
"kind": "Node",
"metadata": {
"annotations": {
<----- SNIP ----- >
},
{
"apiVersion": "v1",
"kind": "Node",
"metadata": {
"annotations": {
<----- SNIP ----- >
},
{
"apiVersion": "v1",
"kind": "Node",
"metadata": {
"annotations": {
<----- SNIP ----- >
}
]
}
To only get the the node apiVersion
for all three nodes you can apply JSONPATH to the output
kubectl get nodes minikube-m02 --output jsonpath="{.items[*].metadata.creationTimestamp}"
2022-04-28T13:34:34Z 2022-05-21T14:26:26Z 2022-05-21T14:26:33Z
--output custom-columns
Docs: https://kubernetes.io/docs/reference/kubectl/#custom-columns
To be fancy you also use --output custom-columns
to nicely output information:
.items[*]
used in JSONPath is not needed, it will grab all items by default kubectl get nodes -o custom-columns="OS_IMAGE:{.status.nodeInfo.osImage},IP_ADDRESS:{.status.addresses[0].address}"
OS_IMAGE IP_ADDRESS
Ubuntu 20.04.2 LTS 192.168.49.2
Ubuntu 20.04.2 LTS 192.168.49.3
Ubuntu 20.04.2 LTS 192.168.49.4