akvorado

• 🩹 inlet: fix versioning of metadata cache
• 🩹 orchestrator: fix panic in networks CSV refresher

Docker image: docker pull ghcr.io/akvorado/akvorado:1.10.1
Full changelog: https://github.com/akvorado/akvorado/compare/v1.10.0...v1.10.1

On this release, geo IP is now performed in ClickHouse instead of inlet. When
using the standard docker compose setup, the configuration should be
automatically migrated from the inlet component to the clickhouse component.
This also changes how geo IP is used for AS numbers: geo IP is used as last
resort when configured. It also increases memory usage (1.3GB for ClickHouse).

Another new feature is the ability to use a ClickHouse cluster deployment. This
is enabled when specifying a cluster name in clickhouse→cluster. There is no
automatic migration of an existing database. You should start from scratch and
copy data from the previous setup. Do not try to enable the cluster mode on
existing setup!

New installations should also get better compression and performance from the
main table, due to a change to the primary key used for this table. Check this
Altinity article if you want to apply the change on your installation.

Support for Docker Compose V1 (docker-compose command) has been removed in
favor of Docker Compose V2 (docker compose command). On Ubuntu/Debian systems,
this means you can no longer use the docker-compose package. On Ubuntu, you
can install the docker-compose-v2 package. For other options, check the
documentation for installing the Compose plugin.

• 💥 inlet: GeoIP data is moved from inlets to ClickHouse, add city and region
• 💥 console: persist console database on the default docker compose setup
• 💥 docker: remove support for docker-compose V1
• ✨ orchestrator: add support for ClickHouse clusters
• ✨ inlet: add gNMI metadata provider
• ✨ inlet: static metadata provider can provide exporter and interface metadata
• ✨ inlet: static metadata provider can fetch its configuration from an HTTP endpoint
• ✨ inlet: metadata can be fetched from multiple providers (eg, static, then SNMP)
• ✨ inlet: add support for several SNMPv2 communities
• ✨ inlet: timestamps for Netflow/IPFIX can now be retrieved from packet content, see inlet→flow→inputs→timestamp-source
• 🩹 cmd: fix parsing of inlet→metadata→provider→ports
• 🩹 console: fix use of InIfBoundary and OutIfBoundary as dimensions
• 🌱 orchestrator: add TLS support to connect to ClickHouse database
• 🌱 docker: update to Redis 7.2, Kafka 3.7, Kafka UI 0.7.1, and Zookeeper 3.8 (not mandatory)
• 🌱 orchestrator: improved ClickHouse schema to increase performance

Docker image: docker pull ghcr.io/akvorado/akvorado:1.10.0
Full changelog: https://github.com/akvorado/akvorado/compare/v1.9.3...v1.10.0

• 💥 inlet: many metrics renamed to match Prometheus best practices
• ✨ inlet: add the following collected data (disabled by default):
  MPLSLabels, MPLS1stLabel, MPLS2ndLabel, MPLS3rdLabel, and MPLS4thLabel
• 🩹 inlet: fix static metadata provider configuration validation
• 🩹 inlet: fix a performance regression when enriching flows
• 🩹 inlet: do not decode L4 header if IP packet is fragmented
• 🩹 inlet: handle exporters using several sampling rates
• 🌱 docker: update ClickHouse to 23.8 (this is not mandatory)
• 🌱 orchestrator: add orchestrator→clickhouse→prometheus-endpoint to configure an endpoint to expose metrics to Prometheus

Docker image: docker pull ghcr.io/akvorado/akvorado:1.9.3
Full changelog: https://github.com/akvorado/akvorado/compare/v1.9.2...v1.9.3

This is a bugfix release to workaround a change in ClickHouse Docker image where the init script is executed only once. It affects you if you modify the Akvorado's schema or ClickHouse settings.

• 🩹 docker: ensure ClickHouse init script is executed even when database already exists

Docker image: docker pull ghcr.io/akvorado/akvorado:1.9.2
Full changelog: https://github.com/akvorado/akvorado/compare/v1.9.1...v1.9.2

• 🌱 console: add filtering support for custom columns
• 🌱 inlet: update Expr, the language behind the
  classifiers: support for variables
• 🌱 inlet: support for RFC 7133 for IPFIX
• 🌱 orchestrator: improve performance when looking up for SrcNetPrefix and
  DstNetPrefix when these columns are materialized

Docker image: docker pull ghcr.io/akvorado/akvorado:1.9.1
Full changelog: https://github.com/akvorado/akvorado/compare/v1.9.0...v1.9.1

• 💥 cmd: use AKVORADO_CFG_ as a prefix for environment variables used to
  modify configuration (AKVORADO_CFG_ORCHESTRATOR_HTTP_LISTEN instead of
  AKVORADO_ORCHESTRATOR_HTTP_LISTEN)
• 💥 inlet: inlet→metadata→provider(snmp)→ports is now a map from
  exporter subnets to ports, instead of a map from agent subnets to ports. This
  is aligned with how communities and security-parameters options behave.
• ✨ inlet: support for IPinfo geo IP database and use
  it by default
• ✨ inlet: metadata retrieval is now pluggable. In addition to SNMP, it is
  now possible to set exporter names, interface names and descriptions directly
  in the configuration file. See inlet→metadata.
• ✨ inlet: routing information is now pluggable. See inlet→routing.
• ✨ inlet: BioRIS provider to retrieve routing information
• ✨ inlet: allow extraction of prefix length from routing information. See
  inlet→core→net-providers.
• ✨ inlet: add the following collected data (disabled by default):
  • IPTTL
  • IPTos
  • FragmentID and FragmentOffset
  • TCPFlags
  • ICMPv4Type, ICMPv4Code, ICMPv6Type, ICMPv6Code, ICMPv4, and ICMPv6
  • NextHop
• ✨ orchestrator: add custom dictionaries for additional flow hydration. See
  orchestrator→schema→custom-dictionaries. Currently, filtering on the
  generated data is not available.
• 🩹 inlet: fix Netflow processing when template is received with data
• 🩹 inlet: use sampling rate in Netflow data packet if available
• 🩹 console: fix display when using “%” units and interface speed is 0
• 🩹 orchestrator: create flows table with
  allow_suspicious_low_cardinality_types to ensure we can use
  LowCardinality(IPv6).
• 🌱 inlet: update Expr, the language behind the
  classifiers: new builtins are available
• 🌱 build: minimum supported Node version is now 16
• 🌱 docker: move Docker-related files to docker/
• 🌱 docker: update ClickHouse to 23.3 (not mandatory)
• 🌱 docker: update to Zookeeper 3.8 (not mandatory)
• 🌱 docker: update to Kafka 3.5 (not mandatory, but there is also a configuration change)
• 🌱 docker: add healthchecks for Redis and Zookeeper
• 🌱 console: emphasize trajectory on Sankey graphs

Docker image: docker pull ghcr.io/akvorado/akvorado:1.9.0
Full changelog: https://github.com/akvorado/akvorado/compare/v1.8.3...v1.9.0

This release only contains the change about the Docker image for Kafka and is
targeted for new installs. You don't need to upgrade. See
#832 for more details.

• 🩹 docker: pin Kafka Docker image

Docker image: docker pull ghcr.io/akvorado/akvorado:1.8.4
Full changelog: https://github.com/akvorado/akvorado/compare/v1.8.3...v1.8.4

• 🩹 docker: ensure Kafka is not using KRaft by default
• 🩹 console: fix SrcVlan and DstVlan as a dimension
• 🌱 orchestrator: add method and headers to specify HTTP method and
  additional headers to use when requesting a network source

Docker image: docker pull ghcr.io/akvorado/akvorado:1.8.3
Full changelog: https://github.com/akvorado/akvorado/compare/v1.8.2...v1.8.3

• ✨ orchestrator: add an option to materialize a column instead of using an alias
• 🩹 inlet: fix caching when setting interface name or description

Docker image: docker pull ghcr.io/akvorado/akvorado:1.8.2
Full changelog: https://github.com/akvorado/akvorado/compare/v1.8.1...v1.8.2

• 🩹 console: fix subnet aggregation when IPv4 or IPv6 is set to its default value
• 🩹 console: fix SrcNetPrefix, DstNetPrefix, PacketSize, and PacketSizeBucket dimensions

Docker image: docker pull ghcr.io/akvorado/akvorado:1.8.1
Full changelog: https://github.com/akvorado/akvorado/compare/v1.8.0...v1.8.1

• 💥 docker-compose: the configuration files are now shipped in a config/
  directory: you need to move your akvorado.yaml in config/ as well
• 💥 inlet: unknown interfaces are not skipped anymore
• ✨ console: add subnet aggregation for SrcAddr and DstAddr
• ✨ inlet: expose Interface.Index and Interface.VLAN to interface classification
• ✨ inlet: add Reject() to the set of classification functions to drop the current flow
• ✨ inlet: add SetName() and SetDescription() to modify interface name and description during classification
• ✨ inlet: add Format() to format a string during classification
• 🩹 inlet: fix parsing of sFlow containing IPv4/IPv6 headers
• 🌱 orchestrator: accept an !include tag to include other YAML files in akvorado.yaml

Docker image: docker pull ghcr.io/akvorado/akvorado:1.8.0
Full changelog: https://github.com/akvorado/akvorado/compare/v1.7.2...v1.8.0

When upgrading to this release, it takes some time to reduce the storage size
for a few columns.

• ✨ console: add “%” to available units
• 🩹 inlet: fix parsing of sFlow IPv4/IPv6 data
• 🩹 inlet: fix Bytes value for sFlow (this is the L3 length)
• 🩹 orchestrator: fix disabling of DstASPath
• 🩹 console: fix time range selection
• 🩹 console: fix calculation of the L2 overhead when selecting L2 bps
• 🩹 console: fix behavior of dimension limit field when empty
• 🌱 console: accept IN and NOTIN operators for ExporterAddr, SrcAddr, DstAddr, SrcAddrNAT, DstAddrNAT
• 🌱 inlet: optimize to reduce the number of queries to the system clock
• 🌱 orchestrator: reduce storage for InIfDescription, OutIfDescription, SrcAddr, DstAddr, Bytes, and Packets

Docker image: docker pull ghcr.io/akvorado/akvorado:1.7.2
Full changelog: https://github.com/akvorado/akvorado/compare/v1.7.1...v1.7.2

This is an important bugfix release. DstNet* values were classified using the
source address instead of the destination address.

• 🩹 orchestrator: fix DstNet* values
• 🌱 inlet: if available, use sFlow for DstASPath
• 🌱 docker: update Kafka UI image

Docker image: docker pull ghcr.io/akvorado/akvorado:1.7.1
Full changelog: https://github.com/akvorado/akvorado/compare/v1.7.0...v1.7.1

⚠️ This version has an important bug. Upgrade to 1.7.1 instead.

This version introduces the ability to customize the data schema used by
Akvorado. This change is quite invasive and you should be cautious when
deploying it. It requires a restart of ClickHouse after upgrading the
orchestrator. It also takes some time to reduce the storage size for SrcPort
and DstPort.

The orchestrator automatically defines the TTL for the system log tables (like
system.query_log). The default TTL is 30 days. You can disable that by setting
orchestrator.clickhouse.system-logs-ttl to 0.

• ✨ inlet: add schema.enabled, schema.disabled, schema.main-table-only,
  and schema.not-main-table-only to alter collected data
• ✨ inlet: add the following collected data (disabled by default):
  • SrcAddrNAT and DstAddrNAT
  • SrcPortNAT and DstPortNAT
  • SrcMAC and DstMAC
  • SrcVlan and DstVlan
• 🩹 inlet: handle correctly interfaces with high indexes for sFlow
• 🩹 docker: fix Kafka healthcheck
• 🌱 inlet: improve decoding/encoding performance (twice faster!)
• 🌱 orchestrator: set TTL for ClickHouse system log tables and exporters table
• 🌱 orchestrator: reduce storage size for SrcPort and DstPort
• 🌱 orchestrator: add clickhouse.kafka.engine-settings to configure additional settings for the Kafka engine
• 🌱 common: Go profiler endpoints are enabled by default

Docker image: docker pull ghcr.io/akvorado/akvorado:1.7.0
Full changelog: https://github.com/akvorado/akvorado/compare/v1.6.4...v1.7.0

There is a schema update in this version: you also have to restart ClickHouse
after upgrading for it to pick the new schema.

This version also introduces a cache for some HTTP requests, notably those to
plot the graphs in the “Visualize” tab. The default backend is in-memory,
however the shipped akvorado.yaml configuration file is using Redis instead.
The docker-compose setup has also been updated to start a Redis container for
this usage. Use of Redis is preferred but on upgrade, you need to enable it
explicitely by adding console.http.cache in your configuration.

• ✨ console: cache some costly requests to the backend
• ✨ console: add SrcNetPrefix and DstNetPrefix (as a dimension and a filter attribute)
• ✨ inlet: add inlet.flow.inputs.use-src-addr-for-exporter-addr to override exporter address
• 🌱 console: add limit and graph-type to console.default-visualize-options
• 🌱 docker: published docker-compose.yml file pins Akvorado image to the associated release
• 🌱 docker: update Zookeeper and Kafka images (this upgrade is optional)

Docker image: docker pull ghcr.io/akvorado/akvorado:1.6.4
Full changelog: https://github.com/akvorado/akvorado/compare/v1.6.3...v1.6.4

New Contributors

• @NoifP made their first contribution in https://github.com/akvorado/akvorado/pull/316
• @margau made their first contribution in https://github.com/akvorado/akvorado/pull/329

• ✨ console: add 100% stacked graph type
• 🩹 inlet: handle non-fatal BMP decoding errors more gracefully
• 🩹 inlet: fix a small memory leak in BMP collector
• 🩹 console: fix selection of the aggregate table to not get empty graphs
• 🩹 console: use configured dimensions limit for “Visualize” tab
• 🌱 inlet: optimize BMP CPU usage, memory usage, and lock times
• 🌱 inlet: replace LRU cache for classifiers by a time-based cache
• 🌱 inlet: add TLS support for Kafka transport
• 🌱 console: Ctrl-Enter or Cmd-Enter when editing a filter now applies the changes
• 🌱 console: switch to TypeScript for the frontend code

Docker image: docker pull ghcr.io/akvorado/akvorado:1.6.3
Full changelog: https://github.com/akvorado/akvorado/compare/v1.6.2...v1.6.3

• ✨ orchestrator: add orchestrator.network-sources to fetch network attributes with HTTP
• ✨ console: add console.database.saved-filters to populate filters from the configuration file
• 🩹 doc: durations must be written using a suffix (like 5s)
• 🌱 docker: provider a tarball with essential files to install or upgrade a docker-compose setup
• 🌱 inlet: skip unknown AFI/SAFI in BMP route monitoring messages

Docker image: docker pull ghcr.io/akvorado/akvorado:1.6.2
Full changelog: https://github.com/akvorado/akvorado/compare/v1.6.1...v1.6.2

• 🩹 inlet: fix SrcAS when receiving flows with sFlow
• 🩹 inlet: do not half-close BMP connection (a remote IOS XR closes its own end)
• 🌱 docker: split demo exporters out of docker-compose.yml
• 🌱 console: make the upper limit for dimensions configurable
  (console.dimensions-limit)

Docker image: docker pull ghcr.io/akvorado/akvorado:1.6.1
Full changelog: https://github.com/akvorado/akvorado/compare/v1.6.0...v1.6.1

This release features a BMP collector to grab BGP routes from one or
several routers. The routes can be used to determine source and
destination AS (instead of using GeoIP or information from the flows)
but also the AS paths and the communities. Check inlet.bmp and
inlet.core configuration settings for more information.

• ✨ inlet: BMP collector to get AS numbers, AS paths, and communities from BGP PR #155
• ✨ inlet: add inlet.snmp.agents to override exporter IP address for SNMP queries
• 🩹 inlet: handle sFlow specific interface number for locally
  originated/terminated traffic, discarded traffic and traffic sent to
  multiple interfaces.
• 🌱 build: Docker image is built using Nix instead of Alpine

Docker image: docker pull ghcr.io/akvorado/akvorado:1.6.0
Full changelog: https://github.com/akvorado/akvorado/compare/v1.5.8...v1.6.0

This release bumps the minimal required version for ClickHouse to
22.4. The docker-compose file has been updated to use ClickHouse
22.8 (which is a long term version). Moreover, Akvorado now has its
own organisation and the code is hosted at
akvorado/akvorado.

• 💥 console: make ClickHouse interpolate missing values (ClickHouse 22.4+ is required)
• 🩹 orchestrator: validate configuration of other services on start
• 🩹 inlet: correctly parse inlet.snmp.communities when it is just a string
• 🌱 cmd: print a shorter message when an internal error happens when parsing configuration
• 🌱 inlet: add inlet.snmp.ports to configure SNMP exporter ports

Docker image: docker pull ghcr.io/akvorado/akvorado:1.5.8
Full changelog: https://github.com/akvorado/akvorado/compare/v1.5.7...v1.5.8