Unbound Dashboard In Grafana With Prometheus & Loki
APACHE-2.0 License
🔸 Unbound dashboard in Grafana
🔸 Prometheus
time series database
🔸 Unbound metrics exporter in Go
🔸 Log aggregation with Loki
🔸 Unbound setup
is available at unbound-redis
🔸 Refer to info.md
for dashboard details and release.md
for changes
Grafana Prometheus Loki Go OS HW 11.1.0
2.42.0
3.1.0
1.21.5
raspios-bookworm-arm64-lite
Raspberry Pi 4 Model B
🔸 Grafana ➜ Prometheus ➜ Unbound Exporter ➜ Loki ➜ Import Dashboard
Download: There are 2 versions OSS and Enterprise. OSS version is more than enough. Enterprise version installs too many extra packages (like unattended-upgrades and more). Below cmd downloads Grafana OSS for arm64.
wget https://dl.grafana.com/oss/release/grafana_11.1.0_arm64.deb
Install:
sudo apt install musl
ℹ️
Note:musl
package is required since Grafana 10.2.3
sudo dpkg -i grafana_11.1.0_arm64.deb
ℹ️
Note: A tweakedgrafana.ini
is available in the release. It reduces memory footprint, removes usage collection, stops calls to grafana server/repo and has few more optimizations. You can use grafana.ini either from the release or the default one. Default config is located at/etc/grafana/grafana.ini
Start:
sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
UI:
http://<RP-IP>:3000/
Default user/pass ➟ admin/admin
Install:
sudo apt install prometheus
Config:
Enable unbound-exporter scraping in prometheus. A trimmed down prometheus config, prometheus.yml
is available in the release. Take a backup of existing prometheus.yml, if you are interested in the default config. Copy prometheus.yml
from the release to /etc/prometheus/
dir.
ℹ️
Note: Providedprometheus.yml
has only unbound-exporter metric collection enabled. Default metric collection for node and prometheus exporters are removed. Scraping interval is set to 5m.
Remove Node Exporter: Node exporter exports machine metrics. It is installed as part of prometheus pkg and runs as systemd service. It is not needed for unbound-dashboard. Unless you are already using it, remove node exporter. Below cmd will remove 8 node-exporter related pkgs.
Remove:
sudo apt --purge autoremove prometheus-node-exporter
Disable scrape config: Providedprometheus.yml
has node exporter scrapping config removed.
UI:
http://<RP-IP>:9090/
I wrote my own exporter in Go
. It is more efficient and tailored for this dashboard. A prebuilt binary (for arm64) is available in the release. Source code is available at unbound-exporter.
Config:
Modify Unbound config. Edit /etc/unbound/unbound.conf
Enable extended stats. Add option under server:
tag
extended-statistics: yes
Enable Unix domain socket for collecting stats. It is faster than default TCP. Add below options under remote-control:
tag
control-interface: "/var/run/unbound.sock"
control-use-cert: no
Install:
Copy unbound-exporter
binary from the release to /usr/local/bin/
dir. Make sure it is under ownership of root and executable.
Change ownership [If needed]:
sudo chown root:root /usr/local/bin/unbound-exporter
Make it executable [If needed]:sudo chmod +x /usr/local/bin/unbound-exporter
Service:
Create service to automatically run unbound-exporter at startup. Copy prometheus-unbound-exporter.service
from the release to /etc/systemd/system/
dir. Make sure it is under the ownership of root. Enable and start the service.
Change ownership [If needed]:
sudo chown root:root /etc/systemd/system/prometheus-unbound-exporter.service
Enable:sudo systemctl enable prometheus-unbound-exporter
Start:sudo systemctl start prometheus-unbound-exporter
Status:sudo systemctl status prometheus-unbound-exporter
ℹ️
Note: Providedprometheus-unbound-exporter.service
passes 2 paramaters. Blocklist file path and Unbound unix domain socket URI. Change them accordingly if you are using different path and name.
Usage:
unbound-exporter -h
Download:
Download Loki
and Promtail
curl -O -L "https://github.com/grafana/loki/releases/download/v3.1.0/loki_3.1.0_arm64.deb"
curl -O -L "https://github.com/grafana/loki/releases/download/v3.1.0/promtail_3.1.0_arm64.deb"
Install:
sudo dpkg -i loki_3.1.0_arm64.deb
sudo dpkg -i promtail_3.1.0_arm64.deb
Logging: Enable Unbound logging.
Edit /etc/unbound/unbound.conf
. Add/Modify below options under server:
tag
log-replies: yes
log-tag-queryreply: yes
log-local-actions: yes
logfile: /var/log/unbound/unbound.log
ℹ️
Note: Make sureverbosity:
is set to 0.
Create log dir
sudo mkdir /var/log/unbound
sudo chown unbound:unbound /var/log/unbound
Enable log rotation
Copy file
unbound
from the release under logrotate dir to/etc/logrotate.d/
dir. Make sure it is under the ownership of root.
Config:
Replace /etc/loki/config.yml
and /etc/promtail/config.yml
with the config.yml
files in the release under loki and promtail dirs respectively. Both should be under the ownership of root.
ℹ️
Note: Provided lokiconfig.yml
is optimized to process large data set and metrics calculation for Unbound logs. It is tweaked after some thorough testing in Unbound logs specific dashboard with 6 panels. Default loki config can hardly handle 2 panels with small data set, it throws errors and timeouts with large data set and multiple requests. Provided promtailconfig.yml
enables Unbound logs scrapping.
Restart:
sudo systemctl restart loki
sudo systemctl restart promtail
Open Grafana UI ➟ http://<RP-IP>:3000/
Click Data Sources
under Configuration
. Click Add data source
select Prometheus
. Add below options:
Name ➟
Prometheus
Default ➟On
Add URL ➟http://localhost:9090
Add Scrape interval ➟5m
Hit ➟Save & test
Click Data Sources
under Configuration
. Click Add data source
select Loki
. Add below options:
Name ➟
Loki
Add URL ➟http://localhost:3100
Add Maximum lines ➟100000
Hit ➟Save & test
Dashboard, unbound-dashboard.json
is available in the release. Click Import
under Dashboards
. Click Upload JSON file
. Select unbound-dashboard.json
. Add below options:
Folder ➟
Dashboards
Select Prometheus ➟Data Source
Select Loki ➟Data Source
Hit ➟Import
ℹ️
Tips & NotesGrafana:
How to ➟ Change grafana landing page
to unbound dashboard & Switch between Dark
(default) and Light
theme.
Open Grafana UI ➟
http://<RP-IP>:3000/
ClickProfile
under top right icon UnderPreferences
selectGeneral/Unbound
inHome Dashboard
drop down Change theme inInterface theme
drop down HitSave
There is an additional panel in the dashboard at the top right, not visible in the preview. It shows unbound-exporter status and may be beneficial. If you are not interested in that simply remove it. Screenshot below:
Prometheus:
How to ➟ Remove
time series (metrics) collected by prometheus instantly for fresh start & Reduce prometheus journal logging
.
Enable admin API:
sudo nano /etc/default/prometheus
Add at the top:ARGS="--web.enable-admin-api --log.level=warn"
Save & Exit
To delete all metrics of specific exporter add
job_name
as last argument in delete cmd: Delete:curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="node"}'
Purge:curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'
Restart:sudo systemctl restart prometheus