grafana-kafka-example
Example integration of a Kafka Producer, Kafka Broker and Promtail producing test data to Grafana Cloud Logs, see architecture
Requires Docker and Docker Compose
Configure the environment variables
Configure the environment variables below from your Grafana Cloud Account Logs Data Source settings:
- Log into your Grafana Cloud account to access the Cloud Portal
- Select the Loki Send Logs to set up and manage the Loki logging service from the Cloud Portal
- From the Grafana Data Source setting for Logs, use the hostname of the URL, the User and Password in the following environment variables:
Edit the file: envvars-grafana-cloud-stack1.sh
export GRAFANA_LOGS_HOST="logs-prod-us-central1.grafana.net"
export GRAFANA_LOGS_USERNAME="123456"
export GRAFANA_LOGS_API_KEY="eyJrIjoi...."
Configure Promtail
source envvars-grafana-cloud-stack1.sh
./ctl.sh configure
Check the file promtail-config.yml now has the correct LOGS credentials
Build the Docker container for the Kafka Producer
./ctl.sh build
Start the containers using Docker Compose
./ctl.sh up
This will start the containers: zookeeper, kafka, producer and promtail
Validate logs are being sent to Grafana Cloud Logs
Using the Logs Data Source and LogQL Check for log messages under the labels: {job="kafka"} and {job="varlogs"}
Import the example dashboard dashboard-example-1.json into Grafana and configure the Logs Data Source
Stop the containers
./ctl.sh down