Kafkamoon API

Welcome to Kafkamoon, a Kafka management application. This project demonstrates integration with Kafka APIs as part of a hiring test.

[!NOTE] This README documentation contains information on how to run the Kafkamoon application locally using Docker or Kubernetes. If you want to see all the information about the decisions made in this project, see here.

Table of Contents

• Kafkamoon API
  • Table of Contents
  • Getting Started
    • Prerequisites
    • Running the project locally with Docker Compose
    • Running the project locally on Kubernetes
    • Running the project on EKS

Getting Started

Prerequisites

Before you begin, make sure you have the following tools installed:

• Java: Version 17
• Maven: Version 3.9.6
• Docker
• Docker Compose
• Helm
• KinD
• eksctl

Running the project locally with Docker Compose

To run the project locally, follow these steps:

1. Clone the repository:

git clone [email protected]:mcruzdev/kafkamoon.git

2. Navigate to the project directory:

cd kafkamoon-api

3. Build the application:

make buildAll

4. Start the application using Docker Compose:

docker-compose --profile local up -d

Running with docker-compose you will have:

• Kafka cluster with 3 replicas
• Kafkamoon documentation
• Kafkamoon API

[!NOTE] If you want to run only the infrastructure (Kafka) and Documentation run the following command:

docker-compose --profile dev up -d

After the application is running, you can interact with the following resources:

• OpenAPI specification for exploring and interacting with the API.
• Kafkamoon Documentation for additional information and guidance.

Running the project locally on Kubernetes

To run the project locally on Kubernetes, follow these steps:

1. Create the Kubernetes cluster:

kind create cluster --name local --config=kind/kind-cluster.yaml

2. Configure the context:

kubectl cluster-info --context kind-local

3. Install Kafkamoon Helm Chart

make helm

This installation contains:

• Grafana
• Prometheus
• Kafka cluster (2 replicas)
• Kafkamoon API (2 replicas)
• Kafkamoon Documentation

[!IMPORTANT] Keycloak only lives on production, see here.

Accessing Grafana:

1. Get Grafana admin password to access Grafana Dashboards

kubectl get secret kafkamoon-grafana-operator-grafana-admin-credentials -o jsonpath="{.data['GF_SECURITY_ADMIN_PASSWORD']}" | base64 --decode

The output should like something like this:

abc123_d==%

[!IMPORTANT] Remove the last character (%) from the password.

2. Do a port-forward command:

kubectl port-forward svc/kafkamoon-grafana-operator-grafana-service 8888:3000

Access the Grafana through this url.

[!IMPORTANT] The username is admin.

Accessing the application:

1. Do a port-forward:

kubectl port-forward svc/kafkamoon-api 8080:80

Running the project on EKS

First of all, you need:

1. To have your AWS credentials (~/.aws/credentials) configured locally.

Run the following command: aws configure

2. To have a bucket created to store the terraform.tfstate file.

3. Configure the bucket created on main.tf file.

4. Go to terraform-gitops directory

cd terraform-gitops

5. Execute terraform init

terraform init

6. Execute terraform plan

terraform plan

Check the terraform plan output.

7. Apply the terraform manifests

terraform apply --auto-approve

8. Update current kubeconfig

aws eks update-kubeconfig --name <cluster_name> --region <region>

9. Configure Amazon EBS CSI for EKS

The Kafka application creates a PVC and the following configuration is necessary to give all necessary rights.

See here how to configure Amazon EBS CSI for EKS

10. Install helm charts

make helmUpdate && make helmPkg && make helmInstall