Comprehensive DevOps project tutorial covering Flask app development, Docker containerization, Jenkins CI/CD pipeline, Kubernetes orchestration, and Terraform infrastructure management. Designed for hands-on learning of essential DevOps tools and practices.
This project is a hands-on learning experience I undertook to understand how DevOps works and to get familiar with some of the essential tools used in DevOps workflows.
Here is the high-level process of this project:
Create a Flask Web Application:
Developed a small web application using Flask to understand the basics of web app development.
Run the Flask Application:
Ensured the Flask application was up and running smoothly on the local environment.
Docker Integration:
Created a DockerFile to containerize the Flask application, then built and ran the project using Docker for consistent and isolated environments.
CI/CD Pipeline with Jenkins:
Set up Jenkins to create a CI/CD pipeline, automating the build, test, and deployment processes for the Flask application.
Infrastructure Management with Terraform:
Utilized Terraform for efficient infrastructure provisioning, deployment, scaling, and monitoring. This allowed me to manage infrastructure as code.
Container Orchestration with Kubernetes:
Employed Kubernetes to automate the operational tasks of container management. This included deploying applications, rolling out updates, scaling as needed, and monitoring the application to maintain optimal performance.
Expose Services with Ngrok:
Used Ngrok to expose the local development environment to the internet securely, facilitating external access for testing and demonstrations.
Source Code Management with Git:
Managed the source code using Git, keeping track of changes and maintaining version control throughout the project lifecycle.
Repository Hosting with GitHub:
Stored, tracked, and version controlled the project on GitHub, utilizing its robust platform for collaborative development and project management.
Automation with GitHub Webhooks:
Implemented GitHub Webhooks to automatically trigger actions in response to specific events within the repository, streamlining the workflow and integrating various DevOps tools.
Hello CICD
.git // git folder
app // flask application
__pycache__
static // flask static files
css
style.css
img
ci-cd.png
templates // flask templates
index.html
__init__.py
views.py
env // environment variables
k8s // kubernetes files
deployment.yaml
ingress.yaml
service.yaml
terraform // terraform files
.terraform
.terraform.lock.hcl
terraform.tfstate.backup
main.tf
outputs.tf
variable.tf
terraform.tfstate
tests // python tests
__pycache__
__init__.py
test_views.py
.gitignore
Dockerfile // Docker file
Jenkinsfile // Jenkins file
README.md
run.py
requirements.txt
The first step for this project was to set up the main folder for this project. Let's name it Hello CICD
.
Inside the folder, we first initialize git to keep track of changes made within the directory.
Note: Go to GitHub and create a repository for this project.
# Initialize git in this folder.
git init
Note: The following commands will only work if there are files in the folder as Git tracks files, not folders. But since we mentioned the initialization, I'm mentioning the rest of the process below here.
For this purpose, create a .gitignore
file in the project folder.
# Add files to git
git add .
where . represents all files. if you want to push a single file, just mention the name of the file. For example:
# Add the .gitignore file to git
git add .gitignore
# Check the status of git.
git status
# Commit the changes made. Use -m to write a message. It helps.
git commit -m "Created .gitignore file"
# Create a connection to the repository.
git remote add origin <remote_repository_URL>
# Push the git commits to the main branch.
git push -u origin main
# Pull any changes to confirm that the changes were updated correctly.
git pull origin main
Inside the folder, we need to set up the virtual environment (env) for this project, so that any installs or updates done for the libraries used in this project won't impact the global python system.
# Creating a virtual environment.
python.exe -m venv env
# Activating the virtual environment.
./env/Scripts/activate
Note: Add the env/
folder to the .gitignore file. You really don't want to forget this step.
# If you accidentally forgot to add it to .gitignore and did git add. You can remove it from git
git rm -r --cached env
# Create a requirements file and do this command each time you download a library to stay up to date.
pip freeze > requirements.txt
For creating a flask application, we create an app
folder and inside we create folders for static
and templates
.
static
files will contain the img, css, (js...) and any assets for the project.templates
folder will contain static data as well as placeholder for dynamic data, usually HTML files that uses Jinja template.# Flask Application Folder Structure
Hello CICD
app
static
css
style.css
img
ci-cd.png
templates
index.html
__init__.py
views.py
run.py
We'll start with the ./app/templates/index.html
file. This is just going to be a small HTML document just for the purpose of running it.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Hello CICD</title>
<link
rel="stylesheet"
href="{{ url_for('static', filename='css/style.css') }}"
/>
</head>
<body>
<h1>Hello CI/CD!</h1>
<p>This is a test application being done to learn DevOps.</p>
<p>Author: Edwin Ronald Lambert</p>
<p>Last Updated: July 8, 2024</p>
<img
src="{{ url_for('static', filename='img/ci-cd.png') }}"
alt="Sample Image for CI/CS pipeline"
/>
</body>
</html>
Explanations:
<!DOCTYPE html>
declares the document type and the version of HTML used (HTML5).<html>
tags wraps the entire HTML document.<head>
section contains the meta-information of the document.
{{ url_for('static', filename='css/style.css') }}
is a Jinja2 template syntax used in Flask to dynamically generate the URL for the CSS file located in the static folder.<body>
section contains the content of the HTML document.
src="{{ url_for('static', filename='img/ci-cd.png') }}"
dynamically generates the URL for the image file located in the static folder using Jinja2 syntax.alt="Sample Image for CI/CD pipeline"
provides alternative text for the image, which is useful for accessibility and if the image fails to load.Note: Create the ./static/css/style.css
to add stylesheet and design to the HTML file.
Since we're using the Flask library. Let's install that into our virtual env.
# Install Flask
pip install Flask
We then create the views.py
file.
from flask import Blueprint, render_template
main = Blueprint("main", __name__)
@main.route("/")
def home():
return render_template("index.html")
Explanations:
Blueprint
helps organize the application into modules, making it easier to manage large applications.render_template
is used to render HTML templates.Blueprint()
method, we create a new Blueprint instance.
main
is the name of the blueprint.__name__
is the name of the current module.@main.route("/")
is a route decorator that is associated with the path '/' with the home function.home()
function handles request to the root URL and renders the index.html template.from flask import Flask
def create_app():
app = Flask(__name__)
from .views import main
app.register_blueprint(main)
return app
Explanations:
Flask
module is necessary to create flask applications.app = Flask(__name__)
creates an instance of the Flask application. __name__
is passed to Flask
constructor to determine the root path of the application, which it needs to location resources like static and template files.from .views import main
import the main
blueprint from views
module. The dot (.
) indicates that views
module is within the same package as the current module.app.register_blueprint(main)
registers the main
blueprint with the Flask application instance.create_app()
function returns the Flask application instance.We now create the run.py
file to run this on the terminal.
from app import create_app
app = create_app()
if __name__ == '__main__':
app.run(host="0.0.0.0", port=5000)
Explanations:
from app import create_app
imports the create_app
function from the app
module.app = create_app()
creates an instance of the Flask application and assigns it to app.app.run(host="0.0.0.0", port = 5000)
run the Flask application with the following settings:
host="0.0.0.0"
makes the server public available, listening on all available IP addresses. This is make sure that the application is accessible from other devices on the network.port=5000
is the default port for Flask where the app will listen for incoming connections.debug=True
was the previous command. Mentioned to make sure that the application enables debugging.We run the application first for testing using the command.
# Run the run.py file.
py run.py
This will run the Flask application on your web browser at localhost:5000
.
Ensure that your Flask application has a dedicated test directory. We'll use Pythons built-in unittest framework.
# Flask Test Folder Structure
Hello CICD
tests
__init__.py
test_views.py
import unittest
from flask import url_for
from app import create_app
class FlaskTestCase(unittest.TestCase):
def setUp(self):
# Set up test client before each test.
self.app = create_app()
self.app.testing = True
self.client = self.app.test_client()
def test_home_status_code(self):
# Test that the homepage is accessible.
response = self.client.get('/')
self.assertEqual(response.status_code, 200)
def test_home_data(self):
# Test the data returned by the home page.
response = self.client.get('/')
self.assertIn(b'Hello CI/CD!', response.data)
if __name__ =="__main__":
unittest.main()
Explanations:
unittest
is the Python's built-in module for creating and running tests.create_app
creates and configures an instance of the Flask application.unittest.Testcase
, providing a framework for writing tests.
setUp
is a special method that is run before each test method. It is used to set up the state that is shared among the test methods.
self.app = create_app()
: Creates an instance of the Flask application using the create_app function.self.app.testing = True
: Puts the Flask app in testing mode, which provides better error messages and ensures that exceptions propagate rather than being handled by the Flask error handlers.self.client = self.app.test_client()
: Creates a test client that can be used to simulate HTTP requests to the Flask application. This client is used to interact with the application during testing.test_home_status_code
is a test method to verify that the home page is accessible.
self.assertEqual(response.status_code, 200)
: Asserts that the HTTP status code of the response is 200, indicating success.test_home_data
is a test method to verify the content of the home page.
self.assertIn(b'Hello CI/CD!', response.data)
: Asserts that the response data contains the byte string b'Hello CI/CD!'. This ensures that the expected content is present on the home page.Now we will use Docker to build, test, and deploy applications quickly.
Note: I'm using WIndows (Yeah Yeah, I know. I hear you!)
# Verify Installation
docker --version
# Run a test container.
docker run hello-world
This command will download a test image and run it in a container. If successful, you'll see a "Hello from Docker!"
message.
A Dockerfile
is a text file that contains collections of instructions and commands that will be automatically executed in sequence in the docker environment for building a new docker image.
# Dockerfile folder structure
Hello CICD
Dockerfile
# Use an official Python runtime as a parent image
FROM python:3.9-slim
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
COPY . /app
# Intall any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
# Make port 5000 available to the world outside this container
EXPOSE 5000
# Defined environment variables
ENV NAME World
# RUN run.py when the container launches
CMD [ "python", "run.py" ]
Explanations:
FROM python:3.9-slim
specifies the base image for the Docker image. python:3.9-slim
is an official Docker Image for Python 3.9 with a minimal footprint.WORKDIR /app
sets the working directory inside the container to /app
. All subsequent instructions will be run in this directory.COPY . /app
copies all the contents of the current directory on the host machine into the /app
directory in the container.RUN pip install --no-cache-dir -r requirements.txt
installs all the Python packages specified in the requirements.txt
using pip
. The --no-cache-dir
option prevents pip
from caching the package files, which reduces the image size.ENV NAME World
sets an environment variable NAME
with the value World
inside the container.CMD [ "python", "run.py" ]
specifies the command to run when the container starts.Now, we build and run the docker container and verify that the applications runs inside the container by accessing it via localhost:5000
.
# Build the docker image.
docker build -t hello-cicd .
Explanations:
docker build
command is used to build a Docker image from a Dockerfile.-t hello-cicd
is used to tag the image with the name hello-cicd..
at th end of the command specifies the build context, which is the current directory. Docker will look for a Dockerfile
in this directory and use it to build the image.# Run the Docker Container
docker run -p 5000:5000 hello-cicd
Explanation:
docker run
is the command runs a Docker container from a specified Docker image.-p 5000:5000
is used to publish a container's port to the host. The format is host_port:container_post
.Verify that the application runs by opening the web browser and navigate to https://localhost:5000
.
C:\Windows\System32
or add the binary location to the environment variables PATH
variable or manually add it.terraform -v
to ensure it's correctly installed and variables are set.main.tf
: Main configuration file where you will define the provider and resources.variables.tf
: File to declare variables.outputs.tf
: File to declare outputs.# Terraform Folder Structure
Hello CICD
terraform
main.tf
outputs.tf
variable.tf
main.tf file
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
}
}
}
provider "kubernetes" {
config_path = "~/.kube/config"
}
resource "kubernetes_namespace" "hello-cicd" {
metadata {
name = "hello-cicd-namespace"
}
}
Explanations:
terraform
is the block is used to specify settings related to Terraform itself, including provider requirements.
required_providers
: This block specifies the providers that are required for this configuration.
kubernetes
is the name of the provider.
source = "hashicorp/kubernetes"
: This specifies the source of the provider, indicating that the Kubernetes provider plugin should be downloaded from the HashiCorp repository.provider "kubernetes"
block configures the Kubernetes provider.
config_path = "~/.kube/config"
: This specifies the path to the Kubernetes configuration file (kubeconfig).resource "kubernetes_namespace" "hello-cicd"
: This block defines a resource of type kubernetes_namespace with the identifier hello-cicd.
metadata
block specifies metadata for the Kubernetes namespace.
name = "hello-cicd-namespace"
: This sets the name of the namespace to hello-cicd-namespace.outputs.tf file
output "namespace" {
value = kubernetes_namespace.hello-cicd.metadata[0].name
}
Explanations:
output "namespace"
: This declares an output block named namespace. Outputs are used to make information about your infrastructure available after the configuration has been applied.
value
: This specifies the value that the output will contain. In this case, the value is being set to kubernetes_namespace.hello-cicd.metadata[0].name.Terraform
folder inside the main directory.terraform init
to initialize the directory with necessary Terraform configurations.Kubernetes can be used to automatically provisions, deploys, scales, and manages containerized applications without worrying about the underlying infrastructure.
# Terraform Folder Structure
Hello CICD
k8s
deployment.yaml
ingress.yaml
service.yaml
deployment.yaml file
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-cicd
spec:
replicas: 3
selector:
matchLabels:
app: hello-cicd
template:
metadata:
labels:
app: hello-cicd
spec:
containers:
- name: hello-cicd
image: {username}/hello-cicd:latest
ports:
- containerPort: 5000
Explanations:
apiVersion: apps/v1
specifies the API version for the Kubernetes Deployment object. apps/v1
is the stable API version for managing deployments.kind: Deployment
indicates that the resource being defined is a Deployment. A Deployment ensures that a specified number of pod replicas are running at any given time.metadata
provides metadata about the deployment.spec
describes the desired state of the Deployment.
replicas: 3
specifies that three replicas (pods) of the application should be running.selector
defines how to identify the pods managed by this Deployment.
matchLabels
specifies that pods with the label app: hello-cicd are selected.template
describes the pod that will be created by the Deployment.
containers
specifies the containers that will run in the pods.
name: hello-cicd
is the name of the container.image: {username}/hello-cicd:latest
is the Docker image to use for the container.ports
specifies the ports that the container will expose.
containerPort: 5000
mentions that the container will listen on port 5000.service.yaml file
apiVersion: v1
kind: Service
metadata:
name: hello-cicd-service
namespace: hello-cicd-namespace
spec:
type: LoadBalancer
ports:
- port: 5000
targetPort: 5000
protocol: TCP
selector:
app: hello-cicd
Explanations:
kind: Service
indicates that the resource being defined is a Service. A Service is an abstraction that defines a logical set of pods and a policy to access them.spec
describes the desired state of the Service.
type: LoadBalancer
specifies that the Service should be of type LoadBalancer, which exposes the Service externally using a cloud providers load balancer.ports
defines the ports that the Service will expose.
port: 5000
si the port on which the Service will be exposed.targetPort: 5000
is the port on the container that the traffic will be forwarded to.protocol: TCP
is the protocol used by the Service. In this case, its TCP.selector
defines how the Service will identify the pods it routes traffic to.kubectl apply -f k8s/
to apply your Kubernetes configurations.kubectl get services -n hello-cicd-namespace
to find the IP and port to access your Flask application.First, you need to have Jenkins installed on your server or local machine.
https://localhost:8080
.After installation, we need to set up the project in Jenkins.
hello-cicd
.Pipeline
and click OK.pipeline
" section.Pipeline script
" to write the Jenkinsfile script directly in the UI, or "Pipeline script from SCM
" to load it from your source control. Git
in my case.A Jenkinsfile
defines the steps that Jenkins should follow as part of the CI pipeline.
pipeline {
agent any
environment {
// Update this with your Docker Hub username and the repository name
DOCKER_IMAGE = '{username}/hello-cicd'
// Specify the path to your kubeconfig file correctly for a Windows machine.
KUBECONFIG = 'C:\\Users\\user\\.kube\\config'
}
stages {
stage('Preparation') {
steps {
script {
bat "python -m pip install -r requirements.txt"
}
}
}
stage('Test') {
steps {
script {
bat "python -m unittest discover -s tests"
}
}
}
stage('Build') {
steps {
script {
bat "docker build -t ${DOCKER_IMAGE}:${BUILD_NUMBER} ."
bat "docker push ${DOCKER_IMAGE}:${BUILD_NUMBER}"
}
}
}
stage('Deploy') {
steps {
script {
// Now using 'hello-cicd' as the container name as per your deployment details
bat "kubectl set image deployment/hello-cicd hello-cicd=${DOCKER_IMAGE}:${BUILD_NUMBER} -n hello-cicd-namespace"
}
}
}
}
}
Explanation:
pipeline
declares the beginning of the Jenkins pipeline.agent any
specifies that the pipeline can run on any available Jenkins agent.environment
defines the environment variables that are accessible throughout the pipeline.
DOCKER_IMAGE
is the docker image name.KUBECONFIG
is the path to Kubernetes configuration file (kubeconfig
)stages
consists of the following stages: Preparation
, Test
, Build
and Deploy
.
stage('Preparation')
declares a pipeline stage named "Preparation".
bat "python -m pip install -r requirements.txt"
is a batch command that runs updates pip (Pythons package installer) and then installs all the Python dependencies listed in the requirements.txt file.stage('Test')
declares a new stage named "Test".
bat "python -m unittest discover -s tests"
command runs the Python unittest module's discovery feature, which automatically identifies and runs tests. The -s tests option tells unittest to look for test files in the tests directory.stage("Build")
defines a stage named "Build".
bat "docker build -t ${DOCKER_IMAGE}:${BUILD_NUMBER} ."
uses the bat
command to execute a batch script on the Windows machine. It builds a Docker image with a tag that includes the Docker image name and the Jenkins build number.bat "docker push ${DOCKER_IMAGE}:${BUILD_NUMBER}"
pushes the docker image to the Docker Hub using the tag created in the previous step.stage("Deploy")
defines a stage named "Deploy".
bat "kubectl set image deployment/hello-cicd hello-cicd=${DOCKER_IMAGE}:${BUILD_NUMBER} -n hello-cicd-namespace"
uses the bat
command to execute a batch script to update the Kubernetes deployment named hello-cicd
in the hello-cicd-namespace
to use the new Docker image tagged with the current build number.Configure the Jenkins pipeline to pull the code from the GitHub repository, build the Docker image, run tests, and deploy the application.
Build Now
to start the pipeline.http://<jenkins-host-ip>:5000
in your web browser.If your Jenkins server is hosted locally and you want to expose it to the internet to handle incoming webhooks from services like GitHub, GitLab, or Bitbucket, you can use ngrok. Ngrok is a tool that creates a secure tunnel to your localhost, providing an externally accessible URL that forwards to your local development environment.
auth token
. This token is used to authenticate with ngrok's servers../ngrok authtoken <YOUR_AUTH_TOKEN>
You need to know on which port your Jenkins server is running. By default, Jenkins typically runs on port 8080.
In the terminal, change to where ngrock executable is kept and run the following command to start an ngrok tunnel to the Jenkins port:
./ngrok http 8080
This command tells ngrok to forward HTTP traffic from the ngrok URL to port 8080 on your localhost.
Once ngrok is running, it will display an URL in the terminal that looks something like http://<random-id>.ngrok-free.ap
. This URL is publicly accessible and routes to your local Jenkins server.
Note: Your ngrok tunnel must be kept running as long as you need external access to your local Jenkins. If ngrok stops, the URL will no longer function.
Configure webhooks to automate actions such as triggering Jenkins jobs or updating the deployment status.
Use the ngrok URL in your webhook configuration for GitHub, GitLab, or Bitbucket. Append any specific endpoint required by the plugin you are using (like /github-webhook/ for GitHub).
http://<random-id>.ngrok.io/github-webhook/
Settings
> Webhooks
> Add webhook
.
http://<random-id>.ngrok.io/github-webhook/
.application/json
.Just the push event.
"Active
box is checked to enable the webhook.Manage Jenkins
> Manage Plugins
> Available/Installed
.Manage Jenkins
> Configure System
.Configure
".Build Triggers
", check "GitHub hook trigger for GITScm polling
".Test the entire CI/CD pipeline from code commit to deployment.