A Spring Boot Camel boilerplate that aims to consume events from Apache Kafka, process it and send to a PostgreSQL database.
This is a Spring-Boot Camel Application model that you can use as a reference to study or even to use in your company. It contains a Sample Route that consumes events from a Kafka Topic called sample_topic, process the data and saves it into a PostgreSQL database.
In this project you will find references for the following study topics:
This project is configured to run using a docker-compose file or even at the command-line. The project is configured to use different configurations according to the environment which the project is being run.
Running the project with Docker uses different environment variables than running it from the command-line.
I highly recommend you to run this project using Docker instead of running locally otherwise you will need to guarantee that you have a Kafka and a PostgreSQL instance running or locally or remotely so the application can connect to it and do it's job.
Running the application without Kafka and PostgreSQL instances will result in errors, because Flyway will try to execute the migrations and will not be capable of.
This project contains a Dockerfile / Docker Compose that creates all the required infraestructure to develop and test the project:
Start-Up Containers
$ docker-compose up
Shutdown Containers
$ docker-compose down
Note in the application-docker.properties
that we have specific configurations to run the project using
Docker envinroment. More information in the Environment Variables topic.
When you run the project using Docker, it compiles the source code and creates a target folder containing the project's jar. After that you will not be able to compile the project using your IDE or your command line because the folder's owner now is Docker's user.
To solve this problem, just delete the target folder using sudo or administrative privileges when you want to run the application using your IDE or command line.
This project contains a maven plugin to execute the application. If you want to run it locally it is important that you guarantee that there's a Kafka and a PostgreSQL running in localhost.
You also should your Operational System's hosts file - that generally is located in /etc/hosts
- and
add the following line to it.
127.0.0.1 kafka
After this you just need to run the following maven command:
$ mvn clean install spring-boot:run -Dspring-boot.run.arguments=--database.password=postgres -Dspring-boot.run.profiles=local
The -Dspring-boot.run
command-line argument is defining a system variable called database.password
avoiding to expose
production credentials in the repository. The -Dspring-boot.run.profiles
tells Spring-Boot that it must read specific
local configurations from the application-local.properties
file.
If you have a Kafka and a PostgreSQL instance running somewhere else, you should change the
application-local.properties
to point to it's hosts. Don't forget to change thedatabase.password
value in the command-line.
To be able to send events to Kafka, you can connect into it's container and send events using the CLI. To do so, go into your terminal with the containers running and type the following commands to connect into the Kafka's container:
$ docker exec -it camel_boilerplate_kafka sh
$ cd /opt/kafka/bin
Inside Kafka's bin folder there's a sh script named kafka-console.producer.sh
.
This script allows you to create a producer and send events to the running instance on Kafka. Run the following command
to connect into the topic sample_topic.
$ kafka-console-producer.sh --topic sample_topic --bootstrap-server localhost:9092
You should see your cursor after a >
char if you've been connected successfully. Now everything that you type and
hit enter inside the console will produce a new event in the Kafka broker. You can use the following data as event
model to test the application:
Sample Event
{"name":"Product","quantity":12,"price":12.5}
All real applications contains different configurations for different environments. In other words you will always have different connection strings for the Production and Local environment, for example. At Spring it is called profiles.
In our case we need to set different hosts to run the project using the Docker environment and the local environment. This happens because each module of this application will run in 4 different containers, one for the application, one for the Zookeeper, one for the Kafka and another one for the PostgreSQL. Each of these containers "have it's own localhost definitions", meaning that localhost for the application if one host that is differente for Kafka.
In other words, when you are running the application using a Docker container and you configure it to connect into Kafka using a localhost definition it will not be able to connect to Kafka because there's not a Kafka running at the Application container because Kafka is in another container, another 'virtual machine'.
To solve this problem we need to tell the Application that Kafka is hosted at camel_boilerplate_kafka
- Kafka's
container name - and the PostgreSQL is hosted at camel_boilerplate_postgres
- PostgreSQL container name. These hosts
are defined at the docker-compose.yml
file.
So if we want to run the application in different environments we need to configure it to know how to manage this different environments. In the project we have 3 properties files:
The next step is to tell the application which property file it should consider other than the application.properties
file and in this case we do it using a special command-line argument -Dspring-boot.run.profiles={environment}
having the {environment}
part replaced by docker
or local
string. For example: -Dspring-boot.run.profiles=docker
or -Dspring-boot.run.profiles=local
.
A last consideration to have about Environment Variables is that I ommited the database password value from any
configuration file because you should not save credentials in your code repository for security reasons. That's why we
are passing the command-line argument -Dspring-boot.run.arguments=--database.password=postgres
when we execute the
application: we are telling to Spring-Boot that the value for database.password
property is postgres
While defining credentials will open a security issue for anyone who have access to your code repository, defining credentials in the command-line or event using Operational System variables will expose your credentials to anyone who has access to the server machine. There's better approaches to secure your application but it would make this Camel sample even more complex.
This project is configured with some endpoints in order to inform the state of the application.
Basically we have this endpoints:
A health application should return the following json when you make a GET request to /actuator/health
:
{
"status":"UP",
"components":{
"camelHealth":{
"status":"UP",
"details":{
"name":"camel-health-check",
"context":"UP",
"route:sample":"UP"
}
},
"db":{
"status":"UP",
"details":{
"database":"PostgreSQL",
"validationQuery":"isValid()"
}
},
"diskSpace":{
"status":"UP",
"details":{
"total":254356226048,
"free":170402906112,
"threshold":10485760,
"exists":true
}
},
"kafka":{
"status":"UP"
},
"livenessState":{
"status":"UP"
},
"ping":{
"status":"UP"
},
"readinessState":{
"status":"UP"
}
},
"groups":[
"liveness",
"readiness"
]
}
If Kafka or the database goes offline, the kafka property in the health json should looks like this:
"db": {
"status": "DOWN",
"details": {
"database": "PostgreSQL",
"validationQuery": "isValid()"
}
},
"kafka": {
"status": "DOWN",
"details": {
"Reason": "Kafka may be offline"
}
}
As soon as Kafka goes online again, the application will recover it's state.