A walkthrough of setting up a secure CI/CD pipeline for a containerized workloads using Google Cloud Build
APACHE-2.0 License
The purpose of this document is to provide a step by step guide and related artifacts to set up a secured CI/CD pipeline for a containerized workload. While the overall ecosystem security involves multiple layers, from securing underline physical infrastructure to actual code, this document focuses on the Application/Container security that can be automated in the CI/CD pipeline.
In this demo, we are using a simple java application (with Gradle build tool) that is containerized using Dockerfile and CI/CD pipeline is configured using cloudbuild.yaml. Other technologies/services used to implement this CI/CD pipelines are:
A standard CI/CD pipeline generally consists of 4-5 stages. For example, Build, Test, Static code Analysis and QA stage. These stages are more developer focused and It only makes sure code builds successfully, it follows general coding guidelines and it fulfills functional requirements.
Artifacts used in these stages are the following:
All of these artifacts are vulnerable to security lapses. Following is the list of vulnerabilities they can introduce:
Application Code
Scanning tool: SonarQube
Dockerfile
Scanning tool: Hadolint, Confest
Container Image
Scanning tool: Container Analysis
Kubermetes manifest file
Scanning tool: Kubesec, Confest
With a little more effort, we can handle mitigate the vulnerabilities discussed in the previous section.
CI/CD pipeline for this demo consists of 11 steps and out of them 7 steps are created to make this supply chain more secure.
Build steps can be broken down into categories like:
# Set environment variables on Mac/Linux
PROJECT\_ID=sec-soft-chain
REGION=australia-southeast1
ZONE=australia-southeast1-b
# Set your project as the current project
gcloud config set project $PROJECT_ID
# Or if you want a new project, Create a project and set as default
gcloud projects create $PROJECT\_ID –set-as-default
# Enable the Required API’s
gcloud services enable container.googleapis.com
gcloud services enable artifactregistry.googleapis.com
gcloud services enable ondemandscanning.googleapis.com
gcloud services enable cloudkms.googleapis.com
gcloud services enable binaryauthorization.googleapis.com
# Create a cluster for this demo, these commands do not provision a production ready cluster
gcloud beta container --project $PROJECT_ID clusters create "software-secure-supply" --no-enable-basic-auth --cluster-version "1.27.8-gke.1067004" --release-channel "regular" --machine-type "e2-medium" --image-type "COS_CONTAINERD" --disk-type "pd-balanced" --disk-size "100" --metadata disable-legacy-endpoints=true --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --num-nodes "3" --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM --enable-ip-alias --network "projects/$PROJECT_ID/global/networks/default" --subnetwork "projects/$PROJECT_ID/regions/$REGION/subnetworks/default" --no-enable-intra-node-visibility --default-max-pods-per-node "110" --security-posture=standard --workload-vulnerability-scanning=disabled --no-enable-master-authorized-networks --addons GcePersistentDiskCsiDriver --enable-autoupgrade --enable-autorepair --max-surge-upgrade 1 --max-unavailable-upgrade 1 --binauthz-evaluation-mode=PROJECT_SINGLETON_POLICY_ENFORCE --enable-shielded-nodes --node-locations $REGION
# Create an Artifact Repository
gcloud artifacts repositories create "$PROJECT_ID-repo" --location=$REGION --repository-format=docker
# Allow the Cloud Build Service Account to run scans
gcloud projects add-iam-policy-binding $PROJECT_ID --member=serviceAccount:$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")@cloudbuild.gserviceaccount.com --role=roles/ondemandscanning.admin
# Allow the Cloud Build Service Account to deploy to GKE
gcloud projects add-iam-policy-binding $PROJECT_ID --member=serviceAccount:$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")@cloudbuild.gserviceaccount.com --role=roles/container.developer
# Allow Cloud Build Service Account the permission to attest
gcloud projects add-iam-policy-binding $PROJECT_ID --member=serviceAccount:$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")@cloudbuild.gserviceaccount.com --role=roles/containeranalysis.notes.attacher
git clone <https://github.com/GoogleCloudPlatform/cloud-build-software-delivery>
# Create a repo called sec-soft-chain
gcloud source repos create sec-soft-chain
# Configure credentials
gcloud init && git config --global credential.https://source.developers.google.com.helper gcloud.sh
# Add your new repo as a remote repo called google
git remote add google https://source.developers.google.com/p/$PROJECT_ID/r/sec-soft-chain
# Push code to the new repo called google
git push --all google
Congratulations!!!
you have successfully configured most of the stages (other than a couple, highlighted in yellow). You can very easily replicate this in your actual project by just copying a couple of files (with a little modification).
Please note that you will have to comment these two stages if you want to test till this point.
The remaining two stages will require some more configurations outside the code, let us see how can we set them up one by one:
The Cloudbuild.yaml file in the repo needs a few changes these can be done manually by replacing each instance of the following variables:
Or the below scripts should set these as per the variables configured previously:giot
sed -i "s/<ZONE>/$ZONE/g" cloudbuild.yaml
sed -i "s/<REGION>/$REGION/g" cloudbuild.yaml
sed -i "s/<PROJECT\_ID>/$PROJECT\_ID/g" cloudbuild.yaml
sed -i "s/<REPO>/$PROJECT\_ID-repo/g" cloudbuild.yaml
sed -i "s/<IMAGE>/image/g" cloudbuild.yaml
https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/sonarqube
git clone https://github.com/GoogleCloudPlatform/cloud-builders-community
cd cloud-builders-community/sonarqube/
gcloud builds submit .
Login to https://sonarcloud.io with your github account
Note: Given below is the list of stripped down actions borrowed from following link:
https://cloud.google.com/architecture/binary-auth-with-cloud-build-and-gke
git clone <https://github.com/GoogleCloudPlatform/gke-binary-auth-tools> ~/binauthz-tools
gcloud builds submit --project $PROJECT\_ID --tag "gcr.io/$PROJECT\_ID/cloudbuild-attestor" ~/binauthz-tools
gcloud kms keyrings create "binauthz" \
--project "${PROJECT\_ID}" \
--location "${REGION}"
gcloud kms keys create "vulnz-signer" \
--project "${PROJECT_ID}" \
--location "${REGION}" \
--keyring "binauthz" \
--purpose "asymmetric-signing" \
--default-algorithm "rsa-sign-pkcs1-4096-sha512"
curl "https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/?noteId=vulnz-note" \
--request "POST" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $(gcloud auth print-access-token)" \
--header "X-Goog-User-Project: ${PROJECT_ID}" \
--data-binary @- <<EOF
{
"name": "projects/${PROJECT_ID}/notes/vulnz-note",
"attestation": {
"hint": {
"human_readable_name": "Vulnerability scan note"
}
}
}
EOF
export CLOUD_BUILD_SA_EMAIL=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")@cloudbuild.gserviceaccount.com
curl "https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/vulnz-note:setIamPolicy" \
--request POST \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $(gcloud auth print-access-token)" \
--header "X-Goog-User-Project: ${PROJECT_ID}" \
--data-binary @- <<EOF
{
"resource": "projects/${PROJECT_ID}/notes/vulnz-note",
"policy": {
"bindings": [
{
"role": "roles/containeranalysis.notes.occurrences.viewer",
"members": [
"serviceAccount:${CLOUD_BUILD_SA_EMAIL}"
]
},
{
"role": "roles/containeranalysis.notes.attacher",
"members": [
"serviceAccount:${CLOUD_BUILD_SA_EMAIL}"
]
}
]
}
}
EOF
gcloud container binauthz attestors create "vulnz-attestor" \_
--project "${PROJECT\_ID}" \
--attestation-authority-note-project "${PROJECT\_ID}" \
--attestation-authority-note "vulnz-note" \
--description "Vulnerability scan attestor"
gcloud beta container binauthz attestors public-keys add \
--project "${PROJECT\_ID}" \
--attestor "vulnz-attestor" \
--keyversion "1" \
--keyversion-key "vulnz-signer" \
--keyversion-keyring "binauthz" \
--keyversion-location "${REGION}" \
--keyversion-project "${PROJECT\_ID}"
gcloud container binauthz attestors add-iam-policy-binding "vulnz-attestor" \
--project "${PROJECT\_ID}" \
--member=serviceAccount:$(gcloud projects describe $PROJECT\_ID --format="value(projectNumber)")@cloudbuild.gserviceaccount.com \
--role "roles/binaryauthorization.attestorsViewer"
gcloud kms keys add-iam-policy-binding "vulnz-signer" \
--project "${PROJECT\_ID}" \
--location "${REGION}" \
--keyring "binauthz" \
--member serviceAccount:$(gcloud projects describe $PROJECT\_ID --format="value(projectNumber)")@cloudbuild.gserviceaccount.com \
--role 'roles/cloudkms.signerVerifier'
From Binary Authorization Policy page you can add the project and Attestor name as shown: