carbon-relay-ng

Bump go mod version to 1.20 and update go images used to build carbon-relay-ng to 1.20.4 #518

• build using latest version of go, 1.17.3. #482
• tweaks to connection closing. #481
• experimental consistentHashing-v2 hashing scheme which aims to be compatible with more recent versions of carbon-relay. #477
• adopt the term "blocklist" for blocking metrics. The existing config options "blacklist" and commands "addBlack" are replaced
  with "blocklist" and "addBlock" respectively. The old option and command will keep working for the time being, but users are recommended
  to update their configuration because the old options will be removed at the next major release. #473
• grafanaNet route: make publishing storage-aggregation.conf optional. This is a breaking change for the addRoute grafanaNet ... command, as aggregationFile= now needs to be prepended to the aggregation file value. (Note that this command is only used by the experimental tcp admin interface, and the deprecated config init commands.) #485
• Update Jsonnet lib to work with latest Tanka version #487

docker image: https://hub.docker.com/layers/grafana/carbon-relay-ng/1.2/images/sha256-1e2311adc9ead7cee85c9b46716bf559f27c82f9956965110348b0846f6dd393?context=explore
packages and windows builds : https://app.circleci.com/pipelines/github/grafana/carbon-relay-ng/253/workflows/e501d36c-d4d1-4e0e-9d46-01e4399d6777/jobs/2509/artifacts

• improve error messages, especially for storage-aggregation.conf #470
• include example storage-schemas.conf and update k8s jsonnet mixin #472
• Support storage aggregation patterns special chars #471

build assets

• grafanaNet route: add errBackoffMin and errBackoffFactor controls to control backoff retry timing. #465
• grafanaNet route: automatically publish stripped (from comments) storage-aggregation.conf, similar to storage-schemas.conf
  This enables an upcoming new feature which allows customizing the configs on Grafana Cloud Graphite V5. #466
  IMPORTANT you must set the aggregationFile parameter for grafanaNet routes (pointing to your storage-aggregation.conf), similar to the existing schemasFile option (which points to your storage-schemas.conf)
• FreeBSD support. #468

build artifacts: https://app.circleci.com/pipelines/github/grafana/carbon-relay-ng/195/workflows/43170e91-7689-4480-889f-437f9f34e918/jobs/2209/artifacts

Important:
Docker images are no longer being pushed to raintank/carbon-relay-ng, only to grafana/carbon-relay-ng
See https://hub.docker.com/r/grafana/carbon-relay-ng for all Docker image updates.

• Deprecate (remove) pushing to "raintank" docker hub account. #449
• Kafkamdm: make sasl mechanism configurable. #447
• Fix panels on dashboard. #387
• Fix process reporter (/proc) to behave properly in windows, freeBSD. #450
• GrafanaNet route: periodically post the schemas parsed from storage-schemas.conf (minus file comments) #458
  (this enables an upcoming new feature which allows customizing the schemas on Grafana Cloud Graphite)
• Upgrade Sarama, our kafka library from v1.19.0 to v1.23.0 #445
• add bySeriesWithTagsFnv partitioner for kafka-mdm route #460

Packages and builds can be downloaded from CircleCI

• kafkaMdm route: allow partitionBy bySeriesWithTags. #381
• Aggregation point tracker (more detailed stats for aggregator). #382
• fix: Add error check for process reporter so we don't auto-crash for osx/windows. #390
• Add support for passing grafana.net credentials via environment variables. #393
• Add a flag to enable /debug/pprof/ on admin http port. #402
• Inverted filter criteria for matchers. #422
• Document k8s deployment. #427, #428
• substring filters have been renamed from "substr" to "sub" and "notSubstr" to "notSub" everywhere, for consistency.
  change is backwards compatible, the old "substr" parameters are still accepted in all places where they've been accepted before. #419, #423
• kafkaMdm route: TLS + SASL support. #426

• support pickle protocol versions 0, 1, 2, 3 & 4 + accept pickle arrays + send pickle tuples. #341
• fix different connections to same host:port using same spool file, by adding instance to destination key. #349
• fix aggregator flush timing + limit concurrent aggregator flushing #354
• support new tsdbgw response with detailed info about invalid metrics #338
• add 'count' aggregator. #355
• Some aggregator improvements and better stats. #361, specifically:
  • assure aggregation flushing is in timestamp asc order
  • new stat for number of metrics going into aggregator: service_is_carbon-relay-ng.instance_is_$instance.mtype_is_counter.unit_is_Metric.direction_is_in.aggregator_is_$aggregator
  • new stat for number of metrics going out of aggregator: service_is_carbon-relay-ng.instance_is_$instance.mtype_is_counter.unit_is_Metric.direction_is_out.aggregator_is_$aggregator
  • new stat for number of aggregators waiting to flush: service_is_carbon-relay-ng.instance_is_$instance.*.unit_is_aggregator.what_is_flush_waiting
  • new stats for CPU, memory and golang GC under `carbon-relay-ng.stats.$instance.*
  • give aggregators a "key" property for use in stats ($aggregator above), printed at startup and in web ui.
  • process pending aggregations more eagerly - instead of waiting - when relay is somewhat loaded.
• Fix bad regex cache clean up resulting in unbounded growth #365
• Fix deadlock in keepSafe #369
• Embed bootstrap, jquery, etc, instead of pulling in from the internet. #371

• BREAKING: switch to logrus for logging. #317, #326
  If you were previously using log level notice, you should now use info.
  See logging docs for more info on the log levels.
• IMPORTANT: refactor release process, docker tags and package repo. #330, #331
  see installation docs
• fix memory leak when connections should be closed. #259, #329
• Rewrite of the inputs. pickle, tcp, udp and data reading.
  They are now plugins, handle errors better, and support timeouts.
  #286, #287, #301, #303, #306, #313, #316, #318, #320
• simplify default configs, bump default stats interval, improve config docs. #276
• Make StorageResolution for CloudWatch routes configurable. #275
• re-organize docs into separate pages. #277
• track timestamps coming in to aggregators and being too old. #280
• include sysv init file in centos6/rhel6 rpm to accommodate aws images. #283
• rewriter: support "not" clause + DRY constructor. #284
• fix connection pooling for grafanaNet route. #289
• Fix sslVerify default=true for grafanaNet route using new syntax. #291
• add support for tag appendix validation. make validation inline with grafanaNet. #292
• add request logging for admin interface. #297
• make max_procs setting optional, honor GOMAXPROCS env var. #309
• switch to circleci 2.0. #294
• switch from govendor to dep. #310

• build windows binaries in CircleCI and save artifacts
• add support for dropping raw series consumed by an aggregator (#268, #273 )
• refactor concurrent sending for grafanaNet route. this should improve throughput and reduce buffering in some cases (#272)
• change max retry interval from 1 minute to 30 seconds, so service can restore quicker.

new routes:

• google pubsub #256
• amazon cloudwatch #261, #266

aggregators:

• percentile aggregators #265

other

• fix overlapping spool names / stats / log entries when you use same endpoint tcp address multiple times. #258
  names of spool files and stats will change to include route name
• packages for debian v9/v/10/stretch #246
• grafanaNet defaults update: timeout 5s->10s, flushMaxNum 10k->5k, concurrency 10->100 #254 , #263

various network setting fixes 351f807168c8185a8712f7ca5bd9ea6cff020d3e , #243
make windows builds work #245
Add tmpfiles.d config for centos/7 #235
Use useradd to support multiple distros #233
support variable substitution in instance, and default to $HOST #236
fix blacklist #237
add support for specifying explicit prefixFilter and/or substringFilter in aggregations, which can help perf a lot. #239
show target address for GrafanaNet routes #244

aggregators

• massive aggregation improvements. they now run faster (sometimes by 20x), use less memory and can handle much more load. see #227 , #230
• add derive (rate) aggregator #230
• aggregator regex cache, which lowers cpu usage and increases the max workload, in exchange for a bit more ram usage. #227 By default, the cache is enabled for aggregators set up via commands (init commands in the config) but disabled for aggregators configured via config sections (due to a limitation in our config library)

docs

• add dashboard explanation screenshot #208

packaging and versioning

• fix logging on cent6/amzn/el6 #224 43b265d267f27c7090f15c21ff356fee1da48734
• Fix the creation of /var/run/carbon-relay-ng directory #213
• version argument to get the version

other

• disable http2.0 support for grafanaNet route, since an incompatibility with nginx was resulting in bogus 400 Bad Request responses. note this fix does not properly work, use 0.9.3-1 instead.
• track allocated memory

• make blocking behavior configurable for kafkaMdm and grafanaNet routes. previously, grafanaNet was blocking, kafkaMdm was non-blocking. Now both default to non-blocking, but you can specify blocking=true.
  • nonblocking (default): when the route's buffer fills up, data will be discarded for that route, but everything else (e.g. other routes) will be unaffected. If you set your buffers large enough this won't be an issue.
    rule of thumb: rate in metrics/s times how many seconds you want to be able to buffer in case of downstream issues. memory used will be bufSize * 100B or use bufSize * 150B to be extra safe.
  • blocking: when the route's buffer fills up, ingestion into the route will slow down/block, providing backpressure to the clients, and also blocking other routes from making progress. use this only if you know what you're doing and have smart clients that can gracefully handle the backpressure
• monitor queue drops for non-blocking queues
• document route options better
• monitor queue size and ram used #218
• preliminary support for parsing out the new graphite tag format (kafkaMdm and grafanaNet route only)

the included, dashboard is updated accordingly. and also on https://grafana.com/dashboards/338

these new settings were previously hardcoded (to the values that are now the defaults):

connbuf=<int>                 connection buffer (how many metrics can be queued, not written into network conn). default 30k
iobuf=<int>                   buffered io connection buffer in bytes. default: 2M
spoolbuf=<int>                num of metrics to buffer across disk-write stalls. practically, tune this to number of metrics in a second. default: 10000
spoolmaxbytesperfile=<int>    max filesize for spool files. default: 200MiB (200 * 1024 * 1024)
spoolsyncevery=<int>          sync spool to disk every this many metrics. default: 10000
spoolsyncperiod=<int>         sync spool to disk every this many milliseconds. default 1000
spoolsleep=<int>              sleep this many microseconds(!) in between ingests from bulkdata/redo buffers into spool. default 500
unspoolsleep=<int>            sleep this many microseconds(!) in between reads from the spool, when replaying spooled data. default 10

• unrouteable messages should be debug not notice #198
• rewrite before aggregator; fix race condition, sometimes incorrect aggregations and occasional panics #199
• config parsing fix #175
• kafka-mdm: support multiple brokers. fix #195
• bugfix: make init section work again. fix #201
  attention init section must be changed from:

init = ...

to:

[init]
cmds = ...

note : this was previously released as 0.8.9 but the breaking config change warrants a major version bump.

inputs

better pickle input #174
better amqp input options #168, update amqp library 3e876644819001f0ad21ac0bc427a5eff9cb7332
kafka input logging fixes #188

config

more proper config format so you don't have to use init commands. #183

aggregations

add last, delta and stdev aggregator #191, #194

packaging

Add tmpfiles.d config for deb package #179
prevent erasing of configs #181
add root certs to docker container for better grafanaCloud experience #180

minor release

see #165

• fix metrics initialisation (#150)
• update to new metrics2.0 format (mtype instead of target_type)
• refactor docker build process (#158)
• amqp input (#160)
• allow grafanaNet route to make concurrent connections (#153) to Grafana hosted metrics
• add min/max aggregators (#161)
• add kafka-mdm route for metrictank (#161)