Quickly provision jenkins blue ocean on kubernetes with persistent configuration
APACHE-2.0 License
Quickly provision Jenkins blue ocean on kubernetes v1.9 with RBAC and with persistent configuration.
Go from a simple 16.04 VM to a portable, scalable CI-CD pipeline on kubernetes with automated SSL provisioned and automatic building on push to github. The tutorial takes around 20 minutes.
The Jenkins instance can be deleted and moved between clouds while retaining the job configuration via the excellent Jenkins Helm chart https://github.com/kubernetes/charts/tree/master/stable/jenkins
.
The example shown will use a single Hetzner server ($0.04/hour), but this first step can skipped, and ssh access to an ubuntu 16.04 machine can be used instead.
The sample CI-CD pipeline and application uses Lachlan Evanson's excellent croc-hunter tutorial, which I would encourage you to go through after this. https://github.com/lachie83/croc-hunter
Inspiration was also taken from Nick Johnson's excellent upcoming book 'Continuous Deployment with Kubernetes' (https://withku.be/)
git clone https://github.com/EamonKeane/jenkins-blue.git
cd jenkins-blue
brew install hetznercloud/tap/hcloud
(https://github.com/hetznercloud/cli)hcloud ssh-key create --name $KEY_NAME --public-key-from-file ~/.ssh/id_rsa.pub
hcloud context create jenkins-blue-ocean
. Enter token when promptedhcloud ssh-key list
SERVER_NAME=jenkins-blue-ocean # replace this with your preferred name
SSH_KEY=7170 #replace with your ssh-key id here
SERVER_TYPE=cx41 # Machine with 16GB of ram, 4 vCPU, 160 GB disk (no commitment, $25 per month, 80% less than t2.xlarge on EC2)
To install a single node kubeadm on hetzner run (this will take around 4 minutes): https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
./kubernetes-hetzner.sh --SERVER_NAME=$SERVER_NAME --ssh-key=$SSH_KEY --SERVER_TYPE=$SERVER_TYPE
Set the jenkins ip variable:
JENKINS_IP=$(hcloud server list | grep -E $SERVER_NAME | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b");echo $JENKINS_IP
SSH_USER=root
JENKINS_IP=00.00.00.00 #Enter your machine IP here
To install a single node kubernetes kubeadm cluster run (this will take around 4 minutes): (kubernetes install commands from: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)
./kubernetes-ubuntu1604.sh --SSH_USER=$SSH_USER --JENKINS_IP=$JENKINS_IP
$JENKINS_IP
JENKINS_URL=jenkins.mysite.io # replace with your jenkins url
CROC_HUNTER_URL=croc-hunter.mysite.io # replace with your croc-hunter url
$JENKINS_IP
record exists at $JENKINS_URL
(this may take a minute or two depending on your DNS provider)watch -n 5 dig $JENKINS_URL
$CROC_HUNTER_URL
record exists at $JENKINS_URL
(this may take a minute or two depending on your DNS provider)watch -n 5 dig $CROC_HUNTER_URL
This contains a lot of best practice and contains a Jenkinsfile which is required to demonstrate Blue Ocean functionality. Alternatively specify your own project which has a Jenkinsfile.
https://github.com/lachie83/croc-hunter/
https://github.com/EamonKeane/croc-hunter
Returning to the shell, run the following commands:
ORGANISATION=EamonKeane # enter your organisation (github username)
cd ..
git clone https://github.com/$ORGANISATION/croc-hunter.git
cd croc-hunter
IMAGE_REPOSITORY_ORGANISATION=eamonkeane
IMAGE_REPOSITORY_URL=quay.io/eamonkeane/croc-hunter
Update the values for croc-hunter to match your values (brew install jq
)
jq ".app.hostname = \"$CROC_HUNTER_URL\"" Jenkinsfile.json > tmp.json && mv tmp.json Jenkinsfile.json
jq ".container_repo.master_acct = \"$IMAGE_REPOSITORY_ORGANISATION\"" Jenkinsfile.json > tmp.json && mv tmp.json Jenkinsfile.json
jq ".container_repo.image = \"$IMAGE_REPOSITORY_URL\"" Jenkinsfile.json > tmp.json && mv tmp.json Jenkinsfile.json
git add -A; git commit -m "changed croc hunter url and image repo"; git push origin master
Prerequisites:
brew install kubectl
(>= version 1.9.3)brew install kubernetes-helm
(>= version 2.8.1)cd ../jenkins-blue
export KUBECONFIG=$PWD/admin.conf
DOCKER_SERVER=quay.io
DOCKER_USERNAME=eamonkeane+crochunter
DOCKER_PASSWORD=
DOCKER_EMAIL=. # This is not important and can be left as a dot
kubectl create namespace croc-hunter
kubectl create secret docker-registry croc-hunter-secrets --namespace=croc-hunter --docker-server=$DOCKER_SERVER --docker-username=$DOCKER_USERNAME --docker-password=$DOCKER_PASSWORD --docker-email=$DOCKER_EMAIL
Replace your jenkins url in the hostname, TLS secret name, and TLS secret sections of jenkins-values-initial.yaml and jenkins-values.yaml
sed -i '' -e "s/jenkins\.mysite\.io/$JENKINS_URL/g" jenkins-values.yaml
sed -i '' -e "s/jenkins\.mysite\.io/$JENKINS_URL/g" jenkins-values-initial.yaml
Initial temporary installation of jenkins. This takes approx 4 minutes. This also installs nginx-ingress (configured for bare metal) and cert-manager (configured to auto-provision SSL certs) :
./jenkins-initial-install.sh
echo https://$JENKINS_URL
printf $(kubectl get secret --namespace jenkins jenkins-jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
Enter username admin
and password from terminal
Add docker credentials to jenkins:
$DOCKER_USERNAME
and $DOCKER_PASSWORD
as aboveGithub.com
, click on settings
, then developer settings
, then personal access tokens
, then generate new token
, tick read/write admin hooks, click generate token and copy to clipboardAUTH_TOKEN= #put your github API token here
ORGANISATION=EamonKeane #replace this with your github username or organisation
REPOSITORY=croc-hunter #replace this with your github repo if not using croc-hunter
github-webhook/create-github-webhook.sh --AUTH_TOKEN=$AUTH_TOKEN --SERVICE_URL=$JENKINS_URL --ORGANISATION=$ORGANISATION --REPOSITORY=$REPOSITORY
Click on the master branch on Jenkins blue ocean. After approximately 5 minutes, the application and test will have completed.
The croc-hunter application will be available on https://$CROC_HUNTER_URL
https://$CROC_HUNTER_URL
./copy-jenkins-config.sh
{{- $files := .Files }}
{{- range tuple "blue-ocean-config.xml" }}
{{ . }}: |-
{{ $files.Get . }}
{{- end }}
mkdir -p /var/jenkins_home/users/admin/;
cp -n /var/jenkins_config/blue_ocean_credentials.xml /var/jenkins_home/users/admin/config.xml;
echo " croc-hunter: |-" >> jenkins-jobs.yaml
cat jenkins-jobs/croc-hunter/config.xml | sed 's/^/ /' >> jenkins-jobs.yaml
The jenkins-jobs.yaml should look like the below
Master:
Jobs: |-
croc-hunter: |-
<?xml version='1.0' encoding='UTF-8'?>
helm del --purge jenkins
kubectl create -f kubernetes-yaml/jenkins-pv.yaml
kubectl create -f kubernetes-yaml/jenkins-pvc.yaml
kubectl get pvc -n jenkins
helm upgrade --install --namespace jenkins --wait --values jenkins-values.yaml --values jenkins-jobs.yaml jenkins jenkins/
cd ../croc-hunter
echo "change -- ignore" >> README.md
git add -A; git commit -m "made change to README.md"; git push origin master
printf $(kubectl get secret --namespace jenkins jenkins-jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
Go to Jenkins url at: https://$JENKINS_URL
Enter username admin
and password from clipboard
Check that the docker credentials quay_creds has been persisted in the credentials section.
Click on Jenkins blue ocean and you will now see master building and the croc-hunter app will update.
hcloud server delete $SERVER_NAME
Delete github API tokens and keys from bash history or from github.