kubernetes-kafka

The next release will change container arguments along with properties files' paths, and rename the broker service to kafka. Upgrades beyond this release might be messy.

In scale=1 scenarios (or whenever the number of zookeeper pods is equal to the number of kafka pods) zookeeper must allow additional clients, for example to run some admin commands. See 0515747.

This release bumps Kafka to the latest build, on the same Oracle JRE as before (see #263).

Also it includes a variant scale-3-5-nopzoo that has only one Zookeeper statefulset (#277).

Furthermore, anyone using Prometheus Operator might want to take a look at 50345f2.

Now the introduction of Kustomize support is backwards compatible, bar the change of default storage class name (#270).

Adds several examples for kubectl apply -k.

Note: the default storage class name has changed through #270. The old names are found in the gke-regional variant.

See #248, Cruise Control #218, Cruise Control UI https://github.com/solsson/dockerfiles/pull/22

See the upgrade suggestions in https://github.com/Yolean/kubernetes-kafka/pull/228#issuecomment-443536263 and https://github.com/Yolean/kubernetes-kafka/pull/228#issuecomment-443605983.

If anyone lost zookeeper state it should be possible to recover from snapshot files. We haven't tested that though.

See #231.

Things like #230 can be introduced now when we've destabilized.

Critical bugfix release for #89, see upgrade path in #228 and #227.

See the 5.0 milestone for details. There are quite some breaking changes, now that we'd destabilize anyway, such as a new zookeeper storageclass and auto.create.topics.enable changed to false.

Kafka 2.1.0 is the first Kafka release with Java 11 support. Previously it was Java 8, while support for Docker containers was greatly improved with Java 10. See https://medium.com/adorsys/jvm-memory-settings-in-a-container-environment-64b0840e1d9e and https://blog.docker.com/2018/04/improved-docker-container-integration-with-java-10/ for background.

We should take the leap sooner rather than later because memory limits with java containers is a huge improvement. Actually it's quite difficult to maintain a production cluster without resource limits on containers. Thus we encourage everyone to try to stabilize on Kafka 2.1.0 + Java 11 in test environments as soon as possible.

If you need a more well tested setup or rely on other tools in the Kafka ecosystem that are not yet available with 2.1+ support, see the v4.3.0 release of this repo.

This release makes shutdown graceful. See https://github.com/Yolean/kubernetes-kafka/pull/207 for details.

• Kafka upgraded to 1.0.2
• Added Mailgun Pixy
• Upgraded Burrow to 1.1.0
• Switched events-kube from curl+kafkacat to Eventrouter (#186)

See https://github.com/Yolean/kubernetes-kafka/tree/v4.0.0

• Config changes: #170 #148
• Init script updated for robustness, lower risk for partial success: #167
• Kafka 1.0.1, not #172

Apart from updating some yml to new apiVersions, this release also includes:

• Adds additional listener on port 9094 for clients outside the cluster (#78)
• Adds bootstrap.kafka service, for clients' --bootstrap.servers args to be independent of the current status of broker pods (#52)
• Dynamically provisioned volumes are now Retained by default (#85)
• Example kafka-manager manifests, to list topics w/ stats (#83).
• The readiness tests now stay online, to show the effect of broker/node disruptions (#79)
• The Kafka readiness probe has been simplified (#81)

Highlights:

• Removes the kafka.kafka service, to encourage use of bootstrap.servers and thus actual "listener" names (#21)
• Adds support for Balancing Replicas Across availability zones (#41)
• Uses dynamically provisioned volumes
• ZooKeeper too uses persistent volumes (#33)
• Support for RBAC (#59)
• Tested with Kubernetes 1.6 and 1.7

Additional features as "addons", notably:

• Support for pull-based monitoring such as Prometheus (#49, known issue #61)
• Schema Registry and REST Proxy (#45)
• Single node scale (#44)