carbon-relay-ng

Bump go mod version to 1.20 and update go images used to build carbon-relay-ng to 1.20.4 #518

• retain Grafana Cloud/GEM fields when parsing storage-schemas.conf #502

• build using go 1.18 #497
• from now on embed binaries on github releases using goreleaser #498

• build using latest version of go, 1.17.3. #482
• tweaks to connection closing. #481
• experimental consistentHashing-v2 hashing scheme which aims to be compatible with more recent versions of carbon-relay. #477
• adopt the term "blocklist" for blocking metrics. The existing config options "blacklist" and commands "addBlack" are replaced
  with "blocklist" and "addBlock" respectively. The old option and command will keep working for the time being, but users are recommended
  to update their configuration because the old options will be removed at the next major release. #473
• grafanaNet route: make publishing storage-aggregation.conf optional. This is a breaking change for the addRoute grafanaNet ... command, as aggregationFile= now needs to be prepended to the aggregation file value. (Note that this command is only used by the experimental tcp admin interface, and the deprecated config init commands.) #485
• Update Jsonnet lib to work with latest Tanka version #487

docker image: https://hub.docker.com/layers/grafana/carbon-relay-ng/1.2/images/sha256-1e2311adc9ead7cee85c9b46716bf559f27c82f9956965110348b0846f6dd393?context=explore
packages and windows builds : https://app.circleci.com/pipelines/github/grafana/carbon-relay-ng/253/workflows/e501d36c-d4d1-4e0e-9d46-01e4399d6777/jobs/2509/artifacts

• improve error messages, especially for storage-aggregation.conf #470
• include example storage-schemas.conf and update k8s jsonnet mixin #472
• Support storage aggregation patterns special chars #471

build assets

• grafanaNet route: add errBackoffMin and errBackoffFactor controls to control backoff retry timing. #465
• grafanaNet route: automatically publish stripped (from comments) storage-aggregation.conf, similar to storage-schemas.conf
  This enables an upcoming new feature which allows customizing the configs on Grafana Cloud Graphite V5. #466
  IMPORTANT you must set the aggregationFile parameter for grafanaNet routes (pointing to your storage-aggregation.conf), similar to the existing schemasFile option (which points to your storage-schemas.conf)
• FreeBSD support. #468

build artifacts: https://app.circleci.com/pipelines/github/grafana/carbon-relay-ng/195/workflows/43170e91-7689-4480-889f-437f9f34e918/jobs/2209/artifacts

Important:
Docker images are no longer being pushed to raintank/carbon-relay-ng, only to grafana/carbon-relay-ng
See https://hub.docker.com/r/grafana/carbon-relay-ng for all Docker image updates.

• Deprecate (remove) pushing to "raintank" docker hub account. #449
• Kafkamdm: make sasl mechanism configurable. #447
• Fix panels on dashboard. #387
• Fix process reporter (/proc) to behave properly in windows, freeBSD. #450
• GrafanaNet route: periodically post the schemas parsed from storage-schemas.conf (minus file comments) #458
  (this enables an upcoming new feature which allows customizing the schemas on Grafana Cloud Graphite)
• Upgrade Sarama, our kafka library from v1.19.0 to v1.23.0 #445
• add bySeriesWithTagsFnv partitioner for kafka-mdm route #460

Packages and builds can be downloaded from CircleCI

• kafkaMdm route: allow partitionBy bySeriesWithTags. #381
• Aggregation point tracker (more detailed stats for aggregator). #382
• fix: Add error check for process reporter so we don't auto-crash for osx/windows. #390
• Add support for passing grafana.net credentials via environment variables. #393
• Add a flag to enable /debug/pprof/ on admin http port. #402
• Inverted filter criteria for matchers. #422
• Document k8s deployment. #427, #428
• substring filters have been renamed from "substr" to "sub" and "notSubstr" to "notSub" everywhere, for consistency.
  change is backwards compatible, the old "substr" parameters are still accepted in all places where they've been accepted before. #419, #423
• kafkaMdm route: TLS + SASL support. #426

• support pickle protocol versions 0, 1, 2, 3 & 4 + accept pickle arrays + send pickle tuples. #341
• fix different connections to same host:port using same spool file, by adding instance to destination key. #349
• fix aggregator flush timing + limit concurrent aggregator flushing #354
• support new tsdbgw response with detailed info about invalid metrics #338
• add 'count' aggregator. #355
• Some aggregator improvements and better stats. #361, specifically:
  • assure aggregation flushing is in timestamp asc order
  • new stat for number of metrics going into aggregator: service_is_carbon-relay-ng.instance_is_$instance.mtype_is_counter.unit_is_Metric.direction_is_in.aggregator_is_$aggregator
  • new stat for number of metrics going out of aggregator: service_is_carbon-relay-ng.instance_is_$instance.mtype_is_counter.unit_is_Metric.direction_is_out.aggregator_is_$aggregator
  • new stat for number of aggregators waiting to flush: service_is_carbon-relay-ng.instance_is_$instance.*.unit_is_aggregator.what_is_flush_waiting
  • new stats for CPU, memory and golang GC under `carbon-relay-ng.stats.$instance.*
  • give aggregators a "key" property for use in stats ($aggregator above), printed at startup and in web ui.
  • process pending aggregations more eagerly - instead of waiting - when relay is somewhat loaded.
• Fix bad regex cache clean up resulting in unbounded growth #365
• Fix deadlock in keepSafe #369
• Embed bootstrap, jquery, etc, instead of pulling in from the internet. #371

• BREAKING: switch to logrus for logging. #317, #326
  If you were previously using log level notice, you should now use info.
  See logging docs for more info on the log levels.
• IMPORTANT: refactor release process, docker tags and package repo. #330, #331
  see installation docs
• fix memory leak when connections should be closed. #259, #329
• Rewrite of the inputs. pickle, tcp, udp and data reading.
  They are now plugins, handle errors better, and support timeouts.
  #286, #287, #301, #303, #306, #313, #316, #318, #320
• simplify default configs, bump default stats interval, improve config docs. #276
• Make StorageResolution for CloudWatch routes configurable. #275
• re-organize docs into separate pages. #277
• track timestamps coming in to aggregators and being too old. #280
• include sysv init file in centos6/rhel6 rpm to accommodate aws images. #283
• rewriter: support "not" clause + DRY constructor. #284
• fix connection pooling for grafanaNet route. #289
• Fix sslVerify default=true for grafanaNet route using new syntax. #291
• add support for tag appendix validation. make validation inline with grafanaNet. #292
• add request logging for admin interface. #297
• make max_procs setting optional, honor GOMAXPROCS env var. #309
• switch to circleci 2.0. #294
• switch from govendor to dep. #310

• build windows binaries in CircleCI and save artifacts
• add support for dropping raw series consumed by an aggregator (#268, #273 )
• refactor concurrent sending for grafanaNet route. this should improve throughput and reduce buffering in some cases (#272)
• change max retry interval from 1 minute to 30 seconds, so service can restore quicker.

new routes:

• google pubsub #256
• amazon cloudwatch #261, #266

aggregators:

• percentile aggregators #265

other

• fix overlapping spool names / stats / log entries when you use same endpoint tcp address multiple times. #258
  names of spool files and stats will change to include route name
• packages for debian v9/v/10/stretch #246
• grafanaNet defaults update: timeout 5s->10s, flushMaxNum 10k->5k, concurrency 10->100 #254 , #263

various network setting fixes 351f807168c8185a8712f7ca5bd9ea6cff020d3e , #243
make windows builds work #245
Add tmpfiles.d config for centos/7 #235
Use useradd to support multiple distros #233
support variable substitution in instance, and default to $HOST #236
fix blacklist #237
add support for specifying explicit prefixFilter and/or substringFilter in aggregations, which can help perf a lot. #239
show target address for GrafanaNet routes #244

aggregators

• massive aggregation improvements. they now run faster (sometimes by 20x), use less memory and can handle much more load. see #227 , #230
• add derive (rate) aggregator #230
• aggregator regex cache, which lowers cpu usage and increases the max workload, in exchange for a bit more ram usage. #227 By default, the cache is enabled for aggregators set up via commands (init commands in the config) but disabled for aggregators configured via config sections (due to a limitation in our config library)

docs

• add dashboard explanation screenshot #208

packaging and versioning

• fix logging on cent6/amzn/el6 #224 43b265d267f27c7090f15c21ff356fee1da48734
• Fix the creation of /var/run/carbon-relay-ng directory #213
• version argument to get the version

other

• disable http2.0 support for grafanaNet route, since an incompatibility with nginx was resulting in bogus 400 Bad Request responses. note this fix does not properly work, use 0.9.3-1 instead.
• track allocated memory