Burrow

What's Changed

• Update dependencies by @bai in https://github.com/linkedin/Burrow/pull/812
• Update Go to 1.22 by @bai in https://github.com/linkedin/Burrow/pull/813
• Update Alpine to 3.19 by @bai in https://github.com/linkedin/Burrow/pull/814

Full Changelog: https://github.com/linkedin/Burrow/compare/v1.7.0...v1.8.0

What's Changed

• support deleting offset per group and topic by @nilsmagnus in https://github.com/linkedin/Burrow/pull/778
• Add DeleteConsumerMetrics for every delete consumer. by @ygsa in https://github.com/linkedin/Burrow/pull/776
• Update Golang to 1.20.4 by @bai in https://github.com/linkedin/Burrow/pull/780
• Add partition status metric by @javishere in https://github.com/linkedin/Burrow/pull/783
• chore(deps): bump sarama to v1.41.0 by @dnwe in https://github.com/linkedin/Burrow/pull/786
• Update CI badge by @toanju in https://github.com/linkedin/Burrow/pull/790
• Update Go to 1.21 by @bai in https://github.com/linkedin/Burrow/pull/792
• Bump golang.org/x/net from 0.15.0 to 0.17.0 by @dependabot in https://github.com/linkedin/Burrow/pull/793
• Build on loong64 by @212dandan in https://github.com/linkedin/Burrow/pull/794
• Update to golintci version that is compatible with go 1.21.x by @dpippenger in https://github.com/linkedin/Burrow/pull/807
• Update Go to 1.21.8 by @dpippenger in https://github.com/linkedin/Burrow/pull/805
• Bump google.golang.org/protobuf from 1.31.0 to 1.33.0 by @dependabot in https://github.com/linkedin/Burrow/pull/804
• Bump golang.org/x/crypto from 0.14.0 to 0.17.0 by @dependabot in https://github.com/linkedin/Burrow/pull/795
• Bump golang.org/x/net from 0.17.0 to 0.23.0 by @dependabot in https://github.com/linkedin/Burrow/pull/810

New Contributors

• @nilsmagnus made their first contribution in https://github.com/linkedin/Burrow/pull/778
• @ygsa made their first contribution in https://github.com/linkedin/Burrow/pull/776
• @javishere made their first contribution in https://github.com/linkedin/Burrow/pull/783
• @dnwe made their first contribution in https://github.com/linkedin/Burrow/pull/786
• @toanju made their first contribution in https://github.com/linkedin/Burrow/pull/790
• @dependabot made their first contribution in https://github.com/linkedin/Burrow/pull/793
• @212dandan made their first contribution in https://github.com/linkedin/Burrow/pull/794
• @dpippenger made their first contribution in https://github.com/linkedin/Burrow/pull/807

Full Changelog: https://github.com/linkedin/Burrow/compare/v1.6.0...v1.7.0

What's Changed

• #773 - @bai - Update Go to v1.20
• #767 - @curtiseppel - Fix metric leak
• #765 - @ronilichtman - A few typo fixes in storage.go comments
• #743 - @lavoiesl - Tweak build flags for internalDup2 on darwin arm64

What's Changed

• Update sarama to 1.31.1 by @bai in https://github.com/linkedin/Burrow/pull/739

Full Changelog: https://github.com/linkedin/Burrow/compare/v1.4.0...v1.5.0

What's Changed

• Handle consumer group tombstones by @ricky26 in https://github.com/linkedin/Burrow/pull/727
• Update Go dependencies + address recent Go CVEs (1.17.6) by @bai in https://github.com/linkedin/Burrow/pull/732
• Update Alpine image to 3.15 by @bai in https://github.com/linkedin/Burrow/pull/733
• Prefer http.NoBody to nil body by @bai in https://github.com/linkedin/Burrow/pull/734
• Update changelog for v1.4.0 by @bai in https://github.com/linkedin/Burrow/pull/735

New Contributors

• @ricky26 made their first contribution in https://github.com/linkedin/Burrow/pull/727

Full Changelog: https://github.com/linkedin/Burrow/compare/v1.3.8...v1.4.0

What's Changed

• Fix deprecated binaries goreleaser field by @bai in https://github.com/linkedin/Burrow/pull/724

Full Changelog: https://github.com/linkedin/Burrow/compare/v1.3.7...v1.3.8

What's Changed

• add a groups reaper to remove non existing groups by @d1egoaz in https://github.com/linkedin/Burrow/pull/657
• Add allowedLag setting to cache evaluator by @aboutNisblee in https://github.com/linkedin/Burrow/pull/591
• Update Golang to version 1.16 by @bai in https://github.com/linkedin/Burrow/pull/690
• fix metric reporting by @fr0stbyte in https://github.com/linkedin/Burrow/pull/638
• Fix to show metrics for incomplete partitions by @james-bjss in https://github.com/linkedin/Burrow/pull/700
• fix NPE for err & msg in kafka_client.go by @martianzhang in https://github.com/linkedin/Burrow/pull/695
• Expose network timeout consumer configs via client profile by @shrijeet-tesla in https://github.com/linkedin/Burrow/pull/712
• Update Sarama to version 1.30.0 by @bai in https://github.com/linkedin/Burrow/pull/715
• Allow rewind detection when previous zero lag by @jyates in https://github.com/linkedin/Burrow/pull/716
• Update go build version by @Thiago-Dantas in https://github.com/linkedin/Burrow/pull/718
• Use Go 1.17 with GitHub Actions by @bai in https://github.com/linkedin/Burrow/pull/719
• Continue to support losely specified older Kafka versions by @jyates in https://github.com/linkedin/Burrow/pull/720
• fix prometheus metrics leakage when nonexisting consumer removed from memory by @ArickNan in https://github.com/linkedin/Burrow/pull/708
• Revert "fix prometheus metrics leakage when nonexisting consumer removed from memory" by @bai in https://github.com/linkedin/Burrow/pull/722
• Update Go to 1.17.2 and bump golangci-lint by @bai in https://github.com/linkedin/Burrow/pull/723
• Add missing changelog entries by @bai in https://github.com/linkedin/Burrow/pull/721

New Contributors

• @d1egoaz made their first contribution in https://github.com/linkedin/Burrow/pull/657
• @aboutNisblee made their first contribution in https://github.com/linkedin/Burrow/pull/591
• @fr0stbyte made their first contribution in https://github.com/linkedin/Burrow/pull/638
• @james-bjss made their first contribution in https://github.com/linkedin/Burrow/pull/700
• @martianzhang made their first contribution in https://github.com/linkedin/Burrow/pull/695
• @shrijeet-tesla made their first contribution in https://github.com/linkedin/Burrow/pull/712
• @jyates made their first contribution in https://github.com/linkedin/Burrow/pull/716
• @Thiago-Dantas made their first contribution in https://github.com/linkedin/Burrow/pull/718
• @ArickNan made their first contribution in https://github.com/linkedin/Burrow/pull/708

Full Changelog: https://github.com/linkedin/Burrow/compare/v1.3.6...v1.3.7

• #687 - @bai - Fix goreleaser github action
• #686 - @bai - Update base alpine image from 3.12 to 3.13

• #684 - @bai - Add support for Kafka 2.7.0
• #677 - @CGA1123 - Fix reporting final offset in backfill
• #678 - @EladLeev - Adding Readiness Endpoint
• #679 - @bai - Fix broken CI on master
• #661 - @bai - Bump base alpine image to 3.12, go 1.15.2
• #652 - @bai - Add support for Kafka 2.6.0
• #647 - @toddpalino - Change configuration names to allowlist and denylist
• #576 - @dellalu - send notification only change of status #538

• #636 - @bai - Update sarama to the latest bugfix release, Go 1.14.3
• #632 - @bai - Update sarama with a fix for Shopify/sarama#1692
• #628 - @mwain - Add Prometheus Metrics Exporter
• #598 - @rjh-yext - Add SASL-SCRAM ability to Kafka connection
• #627 - @klDen - Fixes Travis CI build of PR 598
• #631 - @bai - Add support for Kafka 2.5.0 and Go 1.14

• #617 - @timbertson-zd - add and expose observedAt time for lag entries
• #616 - @timbertson - Fix self-reporting of burrow's own progress
• #615 - @bai - Run github actions when PR is created
• #545 - @Lukkie - Allow underscore in hostname

• #608 - @bai - Disable travis ci integration
• #607 - @bai - Build docker image with Go 1.13.7
• #605 - @alvarolmedo - Improve zk lock method to avoid trying lock errors
• #541 - @danudey - Use system SSL store for notifiers by default
• #606 - @bai - Update sarama with fixes for zstd and deadlock
• #548 - @hoesler - feature: configure sarama logger
• #601 - @bai - Add more linters and address select issues
• #597 - @bai - Use golangci-lint instead of assorted linters, build with github actions

This is a test release to verify new binary release process. Please ignore.

• #596 - @bai - Fix deprecated goreleaser config opts

• #595 - @bai - Update sarama to add support for kafka 2.4.0
• #592 - @jantebeest - Feature/zookeeper tls auth
• #585 - @chadjefferies - Make DisableKeepAlives configurable
• #590 - @thenom - Added 2.2.1 supporting version
• #574 - @mweigel - Set Order field on StorageRequests produced by KafkaZkClient
• #571 - @bai - Add support for Go 1.13
• #567 - @dellalu - Replace '-' with '_' for Env
• #563 - @zerowidth - Update the sarama TLS config to allow anonymous SSL connections
• #562 - @chrnola - Add support for group member metadata v3
• #557 - @khorshuheng - Remove reference to dep in Readme
• #556 - @khorshuheng - Migrate from dep to go module
• #399 - @MadDataScience - allow space in consumer name
• #555 - @khorshuheng - Upgrade sarama to support Kafka 2.3.0
• #488 - @timbertson - start-latest + backfill-earliest mode
• #390 - @vixns - parse http notifier open-url and close-url as templates.
• #528 - @timbertson - Report burrow's own progress as a fake consumer
• #524 - @bai - Run CI and build on Golang 1.12
• #523 - @bai - Update sarama to 1.22.1
• #521 - @asaf400 - Upgrade sarama to support Kafka 2.2.0
• #512 - @vvuibert - sarama 1.21.0
• #506 - @jbvmio - Groupmeta v2
• #477 - @mlongob - Only process metadata for groups with ProtocolType = consumer
• #415 - @lins05 - Fixed pid check in docker containers
• #443 - @Lavaerius - Enabling HTTPS endpoints by using ServeTLS
• #413 - @jorgelbg - Retrieve partition owners on the kafka zookeeper client
• #420 - @maxtuzz - Notifier TLS options
• #439 - @BewareMyPower - Replace topicMap with topicPartitions in cluster module
• #493 - @timbertson - travis: test output of gofmt as a single argument
• #502 - @him2994 - Fixed: *storage.brokerOffset has value nil when no leader election for partition

Release Highlights

• More fixes to binary release process.

Release Highlights

• Fix binary release process.
• Report ClientID for consumers.
• Fix division by zero error in evaluator.

Release Highlights

• Add support for Kafka up to version 2.1.0.
• Update sarama to version 1.20.1 with support for zstd compression.
• Support linux/arm64.
• Add blacklist for memory store.

Changes

• [d244fce922] - Bump sarama to 1.20.1 (Vlad Gorodetsky)
• [793430d249] - Golang 1.9.x is no longer supported (Vlad Gorodetsky)
• [735fcb7c82] - Replace deprecated megacheck with staticcheck (Vlad Gorodetsky)
• [3d49b2588b] - Link the README to the Compose file in the project (Jordan Moore)
• [3a59b36d94] - Tests fixed (Mikhail Chugunkov)
• [6684c5e4db] - Added unit test for v3 value decoding (Mikhail Chugunkov)
• [10d4dc39eb] - Added v3 messages protocol support (Mikhail Chugunkov)
• [d6b075b781] - Replace deprecated MAINTAINER directive with a label (Vlad Gorodetsky)
• [52606499a6] - Refactor parseKafkaVersion to reduce method complexity (gocyclo) (Vlad Gorodetsky)
• [b0440f9dea] - Add gcc to build zstd (Vlad Gorodetsky)
• [6898a8de26] - Add libc-dev to build zstd (Vlad Gorodetsky)
• [b81089aada] - Add support for Kafka 2.1.0 (Vlad Gorodetsky)
• [cb004f9405] - Build with Go 1.11 (Vlad Gorodetsky)
• [679a95fb38] - Fix golint import path (golint fixer)
• [f88bb7d3a8] - Update docker-compose Readme section with working url. (Daniel Wojda)
• [3f888cdb2d] - Upgrade sarama to support Kafka 2.0.0 (#440) (daniel)
• [1150f6fef9] - Support linux/arm64 using Dup3() instead of Dup2() (Mpampis Kostas)
• [1b65b4b2f2] - Add support for Kafka 1.1.0 (#403) (Vlad Gorodetsky)
• [74b309fc8d] - code coverage for newly added lines (Clemens Valiente)
• [279c75375c] - accidentally reverted this (Clemens Valiente)
• [192878c69c] - gofmt (Clemens Valiente)
• [33bc8defcd] - make first regex test case a proper match everything (Clemens Valiente)
• [279b256b27] - only set whitelist / blacklist if it's not empty string (Clemens Valiente)
• [b48d30d18c] - naming (Clemens Valiente)
• [7d6c6ccb03] - variable naming (Clemens Valiente)
• [4e051e973f] - add tests (Clemens Valiente)
• [545bec66d0] - add blacklist for memory store (Clemens Valiente)
• [07af26d2f1] - Updated burrow endpoint in README : #401 (Ratish Ravindran)
• [fecab1ea88] - pass custom headers to http notifications. (#357) (vixns)

Notes

This release contains some significant fixes for problems found in the initial 1.0.0 release, including:

• Updated Sarama version to support through Kafka 1.0 (#306)
• Fix Zookeeper watch handling (#328)

There are also some minor feature updates

• Store a ring of recent broker offsets to avoid false alerts for stopped partitions
• Add configurable notification intervals
• Add support for configuration via environment variables
• Support configurable queue depth in the storage module

Changelog

fecab1e pass custom headers to http notifications. (#357)
7c0b8b1 Add minimum-complete config for the evaluator (#388)
dc4cb84 Fix mail template (#369)
e2216d7 Fetch goreleaser via curl instead of 'go get' as compilation only works in 1.10 (#387)
f3659d1 Add a send-interval configuration parameter (#364)
3e488a2 Allow env vars to be used for configuration (#363)
b7428c9 Fix typo in slack close (#361)
5b546cc Create the broker offset rings earlier (#360)
61f097a Metadata refresh on detecting a deleted topic must not be for that topic (#359)
b890885 Make inmemory module request channel's size configurable (#352)
9911709 Update sarama to support 10.2.1 too. (#345)
a1bdcde Adjusting docker build to be self-contained (#344)
a91cf4d Fix an incorrect cast from #338 and add a test to cover it (#340)
389ef47 Store broker offset history (#338)
1a60efe Fix alert closing (#334)
b75a6f3 Fix typo in Cluster reference
cacf05e Reject offsets that are older than the group expiration time (#330)
b6184ff Fix typo in the config checked for TLS no-verify #316 (#329)
3b765ea Sync Gopkg.lock with Gopkg.toml (#312)
e47ec4c Fix ZK watch problem (#328)
846d785 Assume backward-compatible consumer protocol version (fix #313) (#327)
e3a1493 Update sarama to support Kafka 1.0.0 (#306)
946a425 Fixing requests for StorageFetchConsumersForTopic (#310)
52e3e5d Update burrow.toml (#300)
3a4372f Upgrade sarama dependency to support Kafka 0.11.0 (#297)
8993eb7 Fix goreleaser condition (#299)
d088c99 Add gitter webhook to travis config (#296)
08e9328 Merge branch 'gitter-badger-gitter-badge'
76db0a9 Fix positioning
dddd0ea Add Gitter badge

Docker images

• docker pull toddpalino/burrow:1.1.0
• docker pull toddpalino/burrow:latest

The new code is a significant improvement over the original version, and resolves a number of technical debt issues:

• All the bits are modular, allowing for new pieces (like notifiers, or consumer modules) to be added easily without a big impact on the rest of the code
• The internals have test coverage, which will make it a lot easier to accept PRs
• Configuration has been moved from gcfg to viper, which will make it more flexible
• Logging has been moved to uber/zap. This will look a lot different, as it's structured logging.
• The code is all documented for godoc now, and the wiki docs around config are being cleaned up.

There's also a lot of feature updates, and bug fixes:

• Biggest of all, topic deletion is now supported in Kafka clusters
• Evaluation logic has been fixed to have fewer false alerts on stopped partitions
• PID files are now more thoroughly checked to see if the process is actually running
• TLS and SASL support have been generalized, and are fully supported for Kafka connections
• Ownership info is tracked for new consumers
• Dependencies have all been updated, and dep has replaced gpm

We're also saying farewell to the Slack notifier. Slack messages can easily be sent with the HTTP notifier, and there are sample templates to do that. We'll be adding more docs later on setting that up. Most services can be handled with a generic HTTP notifier, so the direction will be to add samples and docs on how to do that, rather than creating custom notifiers for everything.

Docker images:

• toddpalino/burrow:1.0.0

Automated with GoReleaser
Built with go version go1.9.2 linux/amd64