This readme has the following sections:
The New Stack (TNS) is a simple three-tier demo application, fully instrumented with the 3 pillars of observability: metrics, logs, and traces. It offers an insight on what a modern observability stack looks like and experience what it's like to pivot among different types of observability data.
The TNS app is an example three-tier web app built by Weaveworks. It consists of a data layer, application logic layer, and load-balancing layer. To learn more about it, see How To Detect, Map and Monitor Docker Containers with Weave Scope from Weaveworks.
The instrumentation for the TNS app is as follows:
Metrics: Each tier of the TNS app exposes metrics on /metrics
endpoints, which are scraped by the Grafana Agent. Additionally, these metrics are tagged with exemplar information. The Grafana Agent then writes these metrics to Mimir for storage.
Logs: Each tier of the TNS app writes logs to standard output or standard error. It is captured by Kubernetes, which are then collected by the Grafana Agent. Finally, the Agent forwards them to Loki for storage.
Traces: Each tier of the TNS app sends traces in Jaeger format to the Grafana Agent, which then converts them to OTel format and forwards them to Tempo for storage.
Visualization: A Grafana instance configured to talk to the Mimir, Loki, and Tempo instances makes it possible to query and visualize the metrics, logs, and traces data.
If you are running the full metrics, logs and traces stack locally, install and configure all of the following software applications.
If you wish to only deploy the TNS app to an existing K8s cluster using the app-only
option, install and configure kubectl
, tanka
, and jsonnet-bundler
.
Since much of the program is written in Go, you should have have Go installed on your local machine. Click [here] (https://go.dev/doc/install) for installation instructions.
Make sure you have Docker installed and verify it is running with docker ps
. No errors means it is running. For Docker download and installation instructions, click here.
If using Linux and you see an error similar to permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
then make sure you are in the docker
group. This will ensure that you are able to run Docker without using the sudo
command.
$ sudo usermod -aG docker <username>
Logout and then login again for the changes to take effect.
To run the TNS demo, you need a Kubernetes cluster. The cluster creation script uses k3d
which runs as a single node cluster inside Docker. The specific version to use depends on your operating system:
brew install k3d
Note: Ensure that your Docker daemon has a minimum of 2.5 GB of total memory available for all pods in this deployment to be scheduled.
You can also run the TNS demo without Kubernetes. Click here for more information.
The TNS demo uses kubectl
to interact with the Kubernetes clusters. Click here for kubectl
installation instructions.
Tanka uses the Jsonnet language to interact with Kubernetes, via the kubectl
tool. Click here for installation instructions.
When you install the TNS demo application, it will create a tanka
directory in your TNS checkout. This directory contains all of the Jsonnet resources used to install this demo.
To find out more about Tanka, see https://tanka.dev.
The Jsonnet bundler downloads Jsonnet dependencies. Click here for installation instructions.
After the installation type jb
in the terminal to make sure it is added to the system path and working.
These instructions assume that you are using a local k3d
. If you plan to use a Kubernetes cluster other than a local k3d
one, you will need to modify these instructions for your setup.
Clone the TNS repository.
$ git clone https://github.com/grafana/tns
$ cd tns
Install K3D cluster.
$ ./create-k3d-cluster
$ export KUBECONFIG=$(k3d kubeconfig write tns)
Install TNS demo application:
$ ./install
Confirm yes
when prompted. You will be prompted five times during the installation.
Wait for the installation to finish. It can take over ten minutes for everything to download and then start up.
Verify the status of your cluster by running this command.
$ kubectl get pods -A
If all the pods are listed as either running
or completed
, your cluster is ready for use.
Access TNS using the URL http://localhost:8080/.
Note: If you need to re-do this process to get everything running, you can run k3d cluster delete tns
to delete the cluster, then run ./create-k3d-cluster
and re-start the process.
app-only
option)If you already have a K8s cluster and cloud metrics, logs, and traces services available to you, use the app-only
option to deploy only the instrumented TNS app to a Kubernetes cluster.
Get Kubernetes context
$ kubectl config get-contexts
Note down the context you'd like to use to deploy the app.
Deploy the app
$ ./install CONTEXT_YOU_NOTED app-only
Confirm yes
when prompted.
Verify the status of your cluster by running this command.
$ kubectl get pods -n tns-cloud
If all the pods are listed as either running
, your app is ready for use.
grafana-cloud
option)Note: this requires an existing K8S cluster.
Get Kubernetes context
$ kubectl config get-contexts
Note down the context you'd like to use to deploy the app.
Deploy the app
$ ./install CONTEXT_YOU_NOTED grafana-cloud
You will need to provide your organization name (slug), API Token, and stack.
Confirm yes
when prompted about applying changes to your cluster
Verify the status of your cluster by running this command.
$ kubectl get pods -n tns-cloud
If all the pods are listed as either running
, your app is ready for use.
After a few minutes, you will see metrics arriving in your Grafana instance. At this point, you can also enable the Kubernetes Integration - the agent is already configured for you!
The following instructions will help you go from metrics to logs to traces.
TraceID
field.TraceID
field to view the trace.The following instructions will help you go from metrics to traces to logs.
histogram_quantile(.99, sum(rate(tns_request_duration_seconds_bucket{}[1m])) by (le))
{job="tns/app"} | logfmt | level="info" | status>=500 and status <=599 and duration > 50ms
TraceID
field.TraceID
field to view the trace.To disable your cluster, run this command:
$ k3d cluster stop tns
To re-enable the cluster, run this command:
$ k3d cluster start tns
To remove your cluster, run this command:
$ k3d cluster delete tns
$ rm -rf tanka
make
to compile the code and tag new images in your local Docker image registry, after you have modified the source code of the TNS demo application,k3d
to pull the new images on a pod restart (and not use the image from its local cache): k3d image import -c tns grafana/tns-app && k3d image import -c tns grafana/tns-db && k3d image import -c tns grafana/tns-loadgen
.kubectl delete pod app-69db48747b-s6qq6 --namespace=tns
.tk apply --force environments/<ENV>/main.jsonnet
.tk apply --force environments/default/main.jsonnet
.The setup in this repo can help provide you with everything you need to use TNS' setup to work with local Grafana development. For more information, see the README in production/docker-compose.