strimzi-kafka-operator

Main changes since 0.20.0

This patch release contains several bug-fixes:

• Silent error from missing CRB RBAC in Kafka Connect when not needed (#4019)
• Fine-tune the Kafka Exporter health checks (#3885)
• Correct connect configuration comparison (#3987)
• Avoid changing custom resource status because of HashSet ordering (#4069)
• Fix the client rack-awareness in Kafka Connect (#3903)
• Connect default logging not expanded (#4057)
• Topic operator improvements (#3982)
• Do not use ownerReference in UO and TO bindings into a different namespace (#4080)
• Remove owner references from ClusterRoleBindings (#4077)
• Topic Operator metrics (#3883)

Main changes since 0.20.0

This patch release contains several bugfixes:

• Silent error from missing CRB RBAC in Kafka Connect when not needed (#4019)
• Fine-tune the Kafka Exporter health checks (#3885)
• Correct connect configuration comparison (#3987)
• Avoid changing custom resource status because of HashSet ordering (#4069)
• Fix the client rack-awareness in Kafka Connect (#3903)
• Connect default logging not expanded (#4057)
• Topic operator improvements (#3982)
• Do not use ownerReference in UO and TO bindings into a different namespace (#4080)
• Remove owner references from ClusterRoleBindings (#4077)
• Topic Operator metrics (#3883)

Main Changes since 0.19.0

Note: This is the last version of Strimzi that will support Kubernetes 1.11 and higher. Future versions will support only Kubernetes 1.16 and higher.

• Add support for Kafka 2.5.1 and 2.6.0. Remove support for 2.4.0 and 2.4.1
• Make listeners configurable as an array and add support for more different listeners in single cluster
• Remove TLS sidecars from Kafka pods => Kafka now uses native TLS to connect to ZooKeeper
• Updated to Cruise Control 2.5.11, which adds Kafka 2.6.0 support and fixes a previous issue with CPU utilization statistics for containers. As a result, the CPUCapacityGoal has now been enabled.
• Configure Cluster Operator logging using ConfigMap instead of environment variable and support dynamic changes  
• Switch to use the AclAuthorizer class for the simple Kafka authorization type. AclAuthorizer contains new features such as the ability to control the amount of authorization logs in the broker logs.
• Support dynamic logging configuration of Kafka, Kafka Connect, Kafka Connect S2I and Kafka MirrorMaker2
• Add support for client.rack property for Kafka Connect to use fetch from closest replica feature.
• Add support for configuring hostAliases in Pod templates
• Cruise Control metrics integration including a new Grafana dashboard
• Improved operators Grafana dashboard
• Add improvements for oauth authentication, and keycloak authorization:
  • Support for re-authentication was added, which also enforces access token lifespan on the Kafka client session
  • Permission changes through Keycloak Authorization Services are now detected by Kafka Brokers

All changes can be found under the 0.20.0 milestone.

Deprecations and removals

* Deprecation of the old listener configuration
* Removal of monitoring port on Kafka and ZooKeeper related services
* Deprecation of Kafka TLS sidecar configuration

Redesign of the .spec.kafka.listeners section

The .spec.kafka.listeners section of the Kafka CRD has been redesigned to allow configuring more different listeners.
The old listeners object which allowed only configuration of oneplain, one tls, and one external listener is now deprecated and will be removed in the future.
It is replaced with an array allowing configuration of multiple different listeners:

listeners:
  - name: local
    port: 9092
    type: internal
    tls: true
  - name: external1
    port: 9093
    type: loadbalancer
    tls: true
  - name: external2
    port: 9094
    type: nodeport
    tls: true

This change includes some other changes:

• The tls field is now required.
• The former overrides section is now merged with the configuration section.
• The dnsAnnotations field has been renamed to annotations since we found out it has wider use.
• Configuration of loadBalancerSourceRanges and externalTrafficPolicy has been moved into listener configuration. Its use in the template section is now deprecated.
• For type: internal listeners, you can now use the flag useServiceDnsDomain to define whether they should use the fully qualified DNS names including the cluster service suffix (usually .cluster.local). This option defaults to false.
• All listeners now support configuring the advertised hostname and port.
• preferredAddressType has been removed to preferredNodePortAddressType.

To convert the old format into the new format with backwards compatibility, you should use following names and types:

• For the old plain listener, use the name plain, port 9092 and type internal.
• For the old tls listener, use the name tls, port 9093 and type internal.
• For the old external listener, use the name external, port 9094.

For example the following old configuration:

listeners:
  plain:
    # ...
  tls: 
    # ...
  external:
    type: loadbalancer 
    # ...

Will look like this in the new format:

listeners:
  - name: plain
    port: 9092
    type: internal
    tls: false
  - name: tls
    port: 9093
    type: internal
    tls: true
  - name: external
    port: 9094
    type: loadbalancer
    tls: true

Removal of monitoring port on Kafka and ZooKeeper related services

The PodMonitor resource is now used instead of the ServiceMonitor for scraping metrics from Kafka, ZooKeeper, Kafka Connect and so on.
For this reason, we have removed the monitoring port tcp-prometheus (9404) on all the services where it is declared (Kafka bootstrap, ZooKeeper client and so on).
It was already deprecated in the previous 0.19.0 release.
Together with it we have also removed the Prometheus annotations from the services. If you want to add them, you can use the templates.
See here https://strimzi.io/docs/operators/master/using.html#assembly-customizing-kubernetes-resources-str for more details about templates usage.
Finally, the Kafka Exporter service was has been removed because it was used just for the monitoring port.

Deprecation of Kafka TLS sidecar configuration

Since the Kafka TLS sidecar has been removed, the related configuration options in the Kafka custom resource are now deprecated:

• .spec.kafka.tlsSidecar
• .spec.kafka.template.tlsSidecar

Upgrading from Strimzi 0.19.0

See the documentation for upgrade instructions.

Main Changes since 0.20.0-rc2

Release Candidate 3 contains several fixes to bugs found in Release Candidate 2:

• Fix rolling updates when all Kafka pods are Pending (#3848)
• Bump OPA Authorizer version to fix CVE in its dependencies (#3853)
• Fix rebalance examples (#3838)
• Fix cAdvisor metric labels (#3840)
• Several docs and system test fixes and improvements

Upgrading from Strimzi 0.19.0

See the documentation for upgrade instructions.

Main Changes since 0.20.0-rc1

Release Candidate 2 contains several fixes to bugs found in Release Candidate 1:

• Fix secrets with . in name (#3820)
• Fix leak of HTTPClients in Connector Operator (#3827)
• Fix handling of custom and OAuth certificates (#3825)
• Upgrade JMX Exporter to 0.14.0
• Several docs and system test fixes and improvements

Upgrading from Strimzi 0.19.0

See the documentation for upgrade instructions.

Main Changes since 0.19.0

Note: This is the last version of Strimzi that will support Kubernetes 1.11 and higher. Future versions will support only Kubernetes 1.16 and higher.

• Add support for Kafka 2.5.1 and 2.6.0. Remove support for 2.4.0 and 2.4.1
• Make listeners configurable as an array and add support for more different listeners in single cluster
• Remove TLS sidecars from Kafka pods => Kafka now uses native TLS to connect to ZooKeeper
• Updated to Cruise Control 2.5.11, which adds Kafka 2.6.0 support and fixes a previous issue with CPU utilization statistics for containers. As a result, the CPUCapacityGoal has now been enabled.
• Configure Cluster Operator logging using ConfigMap instead of environment variable and support dynamic changes  
• Switch to use the AclAuthorizer class for the simple Kafka authorization type. AclAuthorizer contains new features such as the ability to control the amount of authorization logs in the broker logs.
• Support dynamic logging configuration of Kafka, Kafka Connect, Kafka Connect S2I and Kafka MirrorMaker2
• Add support for client.rack property for Kafka Connect to use fetch from closest replica feature.
• Add support for configuring hostAliases in Pod templates
• Cruise Control metrics integration including a new Grafana dashboard
• Improved operators Grafana dashboard

Deprecations and removals

* Deprecation of the old listener configuration
* Removal of monitoring port on Kafka and ZooKeeper related services
* Deprecation of Kafka TLS sidecar configuration

Redesign of the .spec.kafka.listeners section

The .spec.kafka.listeners section of the Kafka CRD has been redesigned to allow configuring more different listeners.
The old listeners object which allowed only configuration of oneplain, one tls, and one external listener is now deprecated and will be removed in the future.
It is replaced with an array allowing configuration of multiple different listeners:

listeners:
  - name: local
    port: 9092
    type: internal
    tls: true
  - name: external1
    port: 9093
    type: loadbalancer
    tls: true
  - name: external2
    port: 9094
    type: nodeport
    tls: true

This change includes some other changes:

• The tls field is now required.
• The former overrides section is now merged with the configuration section.
• The dnsAnnotations field has been renamed to annotations since we found out it has wider use.
• Configuration of loadBalancerSourceRanges and externalTrafficPolicy has been moved into listener configuration. Its use in the template section is now deprecated.
• For type: internal listeners, you can now use the flag useServiceDnsDomain to define whether they should use the fully qualified DNS names including the cluster service suffix (usually .cluster.local). This option defaults to false.
• All listeners now support configuring the advertised hostname and port.
• preferredAddressType has been removed to preferredNodePortAddressType.

To convert the old format into the new format with backwards compatibility, you should use following names and types:

• For the old plain listener, use the name plain, port 9092 and type internal.
• For the old tls listener, use the name tls, port 9093 and type internal.
• For the old external listener, use the name external, port 9094.

For example the following old configuration:

listeners:
  plain:
    # ...
  tls: 
    # ...
  external:
    type: loadbalancer 
    # ...

Will look like this in the new format:

listeners:
  - name: plain
    port: 9092
    type: internal
    tls: false
  - name: tls
    port: 9093
    type: internal
    tls: true
  - name: external
    port: 9094
    type: loadbalancer
    tls: true

Removal of monitoring port on Kafka and ZooKeeper related services

The PodMonitor resource is now used instead of the ServiceMonitor for scraping metrics from Kafka, ZooKeeper, Kafka Connect and so on.
For this reason, we have removed the monitoring port tcp-prometheus (9404) on all the services where it is declared (Kafka bootstrap, ZooKeeper client and so on).
It was already deprecated in the previous 0.19.0 release.
Together with it we have also removed the Prometheus annotations from the services. If you want to add them, you can use the templates.
See here https://strimzi.io/docs/operators/master/using.html#assembly-customizing-kubernetes-resources-str for more details about templates usage.
Finally, the Kafka Exporter service was has been removed because it was used just for the monitoring port.

Deprecation of Kafka TLS sidecar configuration

Since the Kafka TLS sidecar has been removed, the related configuration options in the Kafka custom resource are now deprecated:

• .spec.kafka.tlsSidecar
• .spec.kafka.template.tlsSidecar

Upgrading from Strimzi 0.19.0

See the documentation for upgrade instructions.

Main Changes since 0.18.0

• Add support for authorization using Open Policy Agent
• Add support for scale subresource to make scaling of following resources easier:
  • KafkaConnect
  • KafkaConnectS2I
  • KafkaBridge
  • KafkaMirrorMaker
  • KafkaMirrorMaker2
  • KafkaConnector
• Remove deprecated Kafka.spec.topicOperator classes and deployment logic
• Use Java 11 as the Java runtime
• Removed the need to manually create Cruise Control metrics topics if topic auto creation is disabled.
• Migration to Helm 3
• Refactored the format of the KafkaRebalance resource's status. The state of the rebalance is now displayed in the associated Condition's type field rather than the status field. This was done so that the information would display correctly in various Kubernetes tools.
• Added performance tuning options to the KafkaRebalance CR and the ability to define a regular expression that will exclude matching topics from a rebalance optimization proposal.
• Use Strimzi Kafka Bridge 0.18.0
• Make it possible to configure labels and annotations for secrets created by the User Operator
• Strimzi Kafka Bridge metrics integration:
  • enable/disable metrics in the KafkaBridge custom resource
  • new Grafana dashboard for the bridge metrics
• Support dynamically changeable logging in the Entity Operator and Kafka Bridge

Full list of changes can be found under the 0.19.0 milestone.

Deprecations and removals

Deprecation of Helm v2 chart

The Helm v2 support will end soon.
Bug fixing should stop on August 13th 2020 and security fixes on November 13th.
See https://helm.sh/blog/covid-19-extending-helm-v2-bug-fixes/ for more details.

In sync with that, the Helm v2 chart of Strimzi Cluster Operator is now deprecated and will be removed in the future as Helm v2 support ends.
Since Strimzi 0.19.0, we have a new chart for Helm v3 which can be used instead.

Removal of v1alpha1 versions of several custom resources

In Strimzi 0.12.0, the v1alpha1 versions of the following resources have been deprecated and replaced by v1beta1:

• Kafka
• KafkaConnect
• KafkaConnectS2I
• KafkaMirrorMaker
• KafkaTopic
• KafkaUser

In the next release, the v1alpha1 versions of these resources will be removed.
Please follow the guide for upgrading the resources: https://strimzi.io/docs/operators/latest/deploying.html#assembly-upgrade-str.

Removal deprecated cadvisor metric labels

The pod_name and container_name labels provided on the cadvisor metrics are now just pod and container starting from Kubernetes 1.16.
We removed the old ones from the Prometheus scraping configuration/alerts and on the Kafka and ZooKeeper dashboard as well.
It means that the charts related to memory and CPU usage are not going to work on Kuvbernetes version previous 1.14.
For more information on what is changed: https://github.com/strimzi/strimzi-kafka-operator/pull/3312

Deprecation of monitoring port on Kafka and ZooKeeper related services

The PodMonitor resource is now used instead of the ServiceMonitor for scraping metrics from Kafka, ZooKeeper, Kafka Connect and so on.
For this reason, we are deprecating the monitoring port tcp-prometheus (9404) on all the services where it is declared (Kafka bootstrap, ZooKeeper client and so on).
This port will be removed in the next release.
Together with it we will also remove the Prometheus annotation from the service.

Removal warning of Cluster Operator log level

Because of the new Cluster Operator dynamic logging configuration via PR#3328 we are going to remove the STRIMZI_LOG_LEVEL environment variable from the Cluster Operator deployment YAML file in the 0.20.0 release.

Upgrading from Strimzi 0.18.0

See the documentation for upgrade instructions.

Main Changes since 0.18.0

• Add support for authorization using Open Policy Agent
• Add support for scale subresource to make scaling of following resources easier:
  • KafkaConnect
  • KafkaConnectS2I
  • KafkaBridge
  • KafkaMirrorMaker
  • KafkaMirrorMaker2
  • KafkaConnector
• Remove deprecated Kafka.spec.topicOperator classes and deployment logic
• Use Java 11 as the Java runtime
• Removed the need to manually create Cruise Control metrics topics if topic auto creation is disabled.
• Migration to Helm 3
• Refactored the format of the KafkaRebalance resource's status. The state of the rebalance is now displayed in the associated Condition's type field rather than the status field. This was done so that the information would display correctly in various Kubernetes tools.
• Added performance tuning options to the KafkaRebalance CR and the ability to define a regular expression that will exclude matching topics from a rebalance optimization proposal.
• Use Strimzi Kafka Bridge 0.18.0
• Make it possible to configure labels and annotations for secrets created by the User Operator
• Strimzi Kafka Bridge metrics integration:
  • enable/disable metrics in the KafkaBridge custom resource
  • new Grafana dashboard for the bridge metrics
• Support dynamically changeable logging in the Entity Operator and Kafka Bridge

Deprecations and removals

Deprecation of Helm v2 chart

The Helm v2 support will end soon.
Bug fixing should stop on August 13th 2020 and security fixes on November 13th.
See https://helm.sh/blog/covid-19-extending-helm-v2-bug-fixes/ for more details.

In sync with that, the Helm v2 chart of Strimzi Cluster Operator is now deprecated and will be removed in the future as Helm v2 support ends.
Since Strimzi 0.19.0, we have a new chart for Helm v3 which can be used instead.

Removal of v1alpha1 versions of several custom resources

In Strimzi 0.12.0, the v1alpha1 versions of the following resources have been deprecated and replaced by v1beta1:

• Kafka
• KafkaConnect
• KafkaConnectS2I
• KafkaMirrorMaker
• KafkaTopic
• KafkaUser

In the next release, the v1alpha1 versions of these resources will be removed.
Please follow the guide for upgrading the resources: https://strimzi.io/docs/operators/latest/deploying.html#assembly-upgrade-str.

Removal deprecated cadvisor metric labels

The pod_name and container_name labels provided on the cadvisor metrics are now just pod and container starting from Kubernetes 1.16.
We removed the old ones from the Prometheus scraping configuration/alerts and on the Kafka and ZooKeeper dashboard as well.
It means that the charts related to memory and CPU usage are not going to work on Kuvbernetes version previous 1.14.
For more information on what is changed: https://github.com/strimzi/strimzi-kafka-operator/pull/3312

Deprecation of monitoring port on Kafka and ZooKeeper related services

The PodMonitor resource is now used instead of the ServiceMonitor for scraping metrics from Kafka, ZooKeeper, Kafka Connect and so on.
For this reason, we are deprecating the monitoring port tcp-prometheus (9404) on all the services where it is declared (Kafka bootstrap, ZooKeeper client and so on).
This port will be removed in the next release.
Together with it we will also remove the Prometheus annotation from the service.

Full list of changes can be found under the 0.19.0 milestone.

Upgrading from Strimzi 0.18.0

See the documentation for upgrade instructions.

Main Changes since 0.17.0

• Add possibility to set Java System Properties via custom resources
• Make it possible to configure PodManagementPolicy for StatefulSets
• Update build system to use yq version 3 (https://github.com/mikefarah/yq)
• Add more metrics to Cluster, User and Topic Operators
• New Grafana dashboard for Operator monitoring
• Allow ssl.cipher.suites, ssl.protocol and ssl.enabled.protocols to be configurable for Kafka and the different components supported by Strimzi
• Add support for user configurable SecurityContext for each Strimzi container
• Add support for Kafka 2.4.1 and 2.5.0
• Remove TLS sidecars from ZooKeeper pods, using native ZooKeeper TLS support instead
• Use Strimzi Kafka Bridge 0.16.0 with support for CORS in the HTTP protocol
• Pass HTTP Proxy configuration from operator to operands
• Add Cruise Control support

Full list of changes can be found under the 0.18.0 milestone.

Upgrading from Strimzi 0.17.0

See the documentation for upgrade instructions.

Main Changes since 0.18.0-rc1

• Improve the KafkaRebalance resources
• Add the KafkaRebalance resource to the strimzi category
• Add KafkaRebalance resources to Operators Grafana dashboard
• Make sure TLS protocol can be configured in Kafka brokers
• Fix Prometheus installation guide

Full list of changes can be found under the 0.18.0 milestone.

Upgrading from Strimzi 0.17.0

See the documentation for upgrade instructions.

Main Changes since 0.17.0

• Add possibility to set Java System Properties via custom resources
• Make it possible to configure PodManagementPolicy for StatefulSets
• Update build system to use yq version 3 (https://github.com/mikefarah/yq)
• Add more metrics to Cluster, User and Topic Operators
• New Grafana dashboard for Operator monitoring
• Allow ssl.cipher.suites, ssl.protocol and ssl.enabled.protocols to be configurable for Kafka and the different components supported by Strimzi
• Add support for user configurable SecurityContext for each Strimzi container
• Add support for Kafka 2.4.1 and 2.5.0
• Remove TLS sidecars from ZooKeeper pods, using native ZooKeeper TLS support instead
• Use Strimzi Kafka Bridge 0.16.0 with support for CORS in the HTTP protocol
• Pass HTTP Proxy configuration from operator to operands
• Add Cruise Control support

Full list of changes can be found under the 0.18.0 milestone.

Upgrading from Strimzi 0.17.0

See the documentation for upgrade instructions.

Main Changes since 0.16.x

• Add possibility to set Java System Properties via CR yaml
• Add support for Mirror Maker 2.0
• Add Jmxtrans deployment
• Add public keys of TLS listeners to the status section of the Kafka CR
• Various bug-fixes
• Dependency upgrades to avoid CVEs
• Improved system tests and documentation

Note: The Connector operator is now using network policies to make sure it has access to Kafka Connect even when network policies deny all traffic. If you want to use the Connector operator while also using the REST API directly, you should create additional network policies to give access to the REST API to your applications as well.

Full list of changes can be found under the 0.17.0 milestone.

Upgrading from Strimzi 0.16.x

See the documentation for upgrade instructions.

Main Changes since 0.17.0-rc3

• Make sure the ZookeeperScaler does work will with custom CAs without the PKCS12 files

Full list of changes can be found under the 0.17.0 milestone.

Upgrading from Strimzi 0.16.x

See the documentation for upgrade instructions.

Main Changes since 0.17.0-rc2

• Fixed NPE which happen in KafkaRoller under rare circumstances
• Fix ordering of addresses in config map to avoid random rolling updates
• Fix scaling of Zookeeper 3.5
• Fix several issues with Connector operator and Mirror Maker 2
• Many other bugs, test and docs improvements

Note: The Connector operator is now using network policies to make sure it has access to Kafka Connect even when network policies deny all traffic. If you want to use the Connector operator while also using the REST API directly, you should create additional network policies to give access to the REST API to your applications as well.

Full list of changes can be found under the 0.17.0 milestone.

Upgrading from Strimzi 0.16.x

See the documentation for upgrade instructions.

Main Changes since 0.17.0-rc1

• Support for pausing / resuming MM2 connectors
• Fix bug in Kafka rack awareness configuration
• Add network policies for Kafka connect when using the connector operator
• Fix rolling update restart when configuration changes
• Fix ZooKeeper scale-up bug
• Validate the replication factor in relation to number of Kafka replicas
• Various bug-fixes
• Dependency upgrades to avoid CVEs
• Improved system tests and documentation

Note: The Connector operator is now using network policies to make sure it has access to Kafka Connect even when network policies deny all traffic. If you want to use the Connector operator while also using the REST API directly, you should create additional network policies to give access to the REST API to your applications as well.

Full list of changes can be found under the 0.17.0 milestone.

Upgrading from Strimzi 0.16.x

See the documentation for upgrade instructions.

Main Changes since 0.16.0

• Add possibility to set Java System Properties via CR yaml
• Add support for Mirror Maker 2.0
• Add Jmxtrans deployment
• Add public keys of TLS listeners to the status section of the Kafka CR

Full list of changes can be found under the 0.17.0 milestone.

Upgrading from Strimzi 0.16.x

See the documentation for upgrade instructions.

Main Changes since 0.16.1

• Fix bug in upgrade when the Kafka.spec.kafka.version field is not specified
• Fix network policies for port 9091 on older Kubernetes versions

Main Changes since 0.16.0

• Fix connector operator when Kafka Connect runs in a different namespace

Main Changes since 0.15.0

• Add support for Kafka 2.4.0 and upgrade from Zookeeper 3.4.x to 3.5.x
• Drop support for Kafka 2.2.1 and 2.3.0
• Add KafkaConnector resource and connector operator
• Let user choose which node address will be used as advertised host (ExternalDNS, ExternalIP, InternalDNS, InternalIP or Hostname)
• Add support for tini
• When not explicitly configured by the user in jvmOptions, -Xmx option is calculated from memory requests rather than from memory limits
• Expose JMX port on Kafka brokers via an internal service
• Add support for externalTrafficPolicy and loadBalancerSourceRanges properties on loadbalancer and nodeport type services
• Add support for user quotas
• Add service discovery labels and annotations
• Add possibility to specify custom server certificates to TLS based listeners
• Add support for Istio protocol selection in service port names
  • Note: Strimzi is essentially adding a tcp- prefix to the port names in Kafka services and headless services. (e.g clientstls -> tcp-clientstls)

Full list of changes can be found under the 0.16.0 milestone.

Upgrading from Strimzi 0.15.x

See the documentation for upgrade instructions.

Main changes since RC1

• Fix to user quotas deletion
• NPE in Kafka Roller
• Other minor bugfixes
• Improved documentation and system tests

Full list of changes can be found under the 0.16.0 milestone.

Upgrading from Strimzi 0.15.x

See the documentation for upgrade instructions.