terraform-grafana-prometheus-alerts

Terraform module to convert Prometheus Alerting rules to Grafana-managed alert rules

Motivation / Why using this module

There are plenty of apps (mostly out of CNCF's ecosystem) where the vendor or the community provides monitoring dashboards and alerts. Dashboards are normally provided as a JSON file which can be loaded into Grafana. Alerts are mostly provided as Prometheus Alerting rules.

There are users who already operate a Grafana instance or use a managed Grafana instance from a cloud provider (Grafana Cloud, Amazon Managed Grafana, Azure Managed Grafana, etc.). Why not using this Grafana instance for the alerting?

The problem is that Grafana's unified alerting uses another format for the alert definition but the concept with labels, annotations (provide description and runbook URLs) is almost identical. This module allows you to reuse the Prometheus Alerting rules and configure them inside Grafana.

Example usage

module "cert_manager_rules" {
  source = "github.com/mkilchhofer/terraform-grafana-prometheus-alerts"

  prometheus_alerts_file_path = file("/path/to/alerts/cert-manager.yaml")
  folder_uid                  = grafana_folder.test.uid
  datasource_uid              = grafana_data_source.prometheus.uid
}

Requirements

Grafana 8.0+ (Unified alerting)

Limitations

Defining multiple alerts with the same name is not supported in Grafana

Overriding definitions of Prometheus Alerting file

The alerts defined as Prometheus Alerting rules can be overridden without changing the input file. If your Prometheus Alerting rules YAML contains an alert alert: TooManyWriteErrors, you can override it like this:

module "cert_manager_rules" {
  source = "github.com/mkilchhofer/terraform-grafana-prometheus-alerts"

  prometheus_alerts_file_path = file("/path/to/alerts/cert-manager.yaml")
  folder_uid                  = grafana_folder.test.uid
  datasource_uid              = grafana_data_source.prometheus.uid

  # Overrides per alert
  overrides = {
    "TooManyWriteErrors" = {
      alert_threshold = 1
      is_paused       = true

      labels = {
        mycustomlabel = "foobar" # Add an extra label
        severity      = "notify" # Override severity
      }
    }
  }
}

Every alert supports the following overrides:

Override parameter	Type	Description
`alert_threshold`	`number`	Threshold of the Grafana alert. Defaults to `0`
`exec_err_state`	`string`	Describes what state to enter when the rule's query is invalid and the rule cannot be executed. Options are `OK`, `Error`, `KeepLast`, and `Alerting`. Defaults to `Error`.
`expr`	`string`	Use a custom PromQL expression for this Grafana alert.
`is_paused`	`bool`	Sets whether the alert should be paused or not. Defaults to `false`.
`no_data_state`	`string`	Describes what state to enter when the rule's query returns No Data. Options are `OK`, `NoData`, `KeepLast`, and `Alerting`. Defaults to `OK`.
`annotations`	`map(string)`	Extra annotations to add. It is also possible to override already defined annotations like `runbook_url`.
`labels`	`map(string)`	Extra labels to add. It is also possible to override already defined labels like `severity`.

TF module documentation

Providers

Name	Version
grafana	~> 3.2

Inputs

Name	Description	Type	Default	Required
datasource_uid	The UID of the Grafana datasource being queried with the expressions inside the Alerting rule file	`string`	n/a	yes
default_evaluation_interval_duration	How often is the rule evaluated by default. (When not defined inside your Alerting rules file)	`string`	`"5m"`	no
disable_provenance	Allow modifying the rule group from other sources than Terraform or the Grafana API.	`bool`	`false`	no
folder_uid	The UID of the Grafana folder that the alerts belongs to.	`string`	n/a	yes
org_id	The Organization ID of of the Grafana Alerting rule groups. (Only supported with basic auth, API keys are already org-scoped)	`string`	`null`	no
overrides	Overrides per Alert rule	map(object({ alert_threshold = optional(number) exec_err_state = optional(string) expr = optional(string) is_paused = optional(bool) no_data_state = optional(string) labels = optional(map(string)) annotations = optional(map(string)) }))	`{}`	no
prometheus_alerts_file_path	Path to the Prometheus Alerting rules file	`string`	n/a	yes

Outputs

No outputs.

Related Projects

terraform-aws-managed-service-prometheus

Terraform module to create AWS Managed Service for Prometheus (AMP) resources 🇺🇦

22 Oct 2021 27