Azure Event Hubs for Apache Kafka Ecosystems
OTHER License
An Azure Event Hubs Kafka endpoint enables users to connect to Azure Event Hubs using the Kafka protocol. By making minimal changes to a Kafka application, users will be able to connect to Azure Event Hubs and reap the benefits of the Azure ecosystem. Event Hubs for Kafka Ecosystems supports Apache Kafka version 1.0 and later.
When we built Kafka-enabled Event Hubs, we wanted to give Kafka users the stability, scalability, and support of Event Hubs without sacrificing their ability to connect to the network of Kafka supporting frameworks. With that in mind, we've started rolling out a set of tutorials to show how simple it is to connect Kafka-enabled Event Hubs with various platforms and frameworks. The tutorials in this directory all work right out of the box, but for those of you looking to connect with a framework we haven't covered, this guide will outline the generic steps needed to connect your preexisting Kafka application to an Event Hubs Kafka endpoint.
If you don't have an Azure subscription, create a free account before you begin.
An Event Hubs namespace is required to send or receive from any Event Hubs service. See Create Kafka-enabled Event Hubs for instructions on getting an Event Hubs Kafka endpoint. Make sure to copy the Event Hubs connection string for later use.
For these samples, you will need the connection string from the portal as well as the FQDN that points to your Event Hub namespace. The FQDN can be found within your connection string as follows:
Endpoint=sb://
mynamespace.servicebus.windows.net
/;SharedAccessKeyName=XXXXXX;SharedAccessKey=XXXXXX
If your Event Hubs namespace is deployed on a non-Public cloud, your domain name may differ (e.g. *.servicebus.chinacloudapi.cn, *.servicebus.usgovcloudapi.net, or *.servicebus.cloudapi.de).
To connect to a Kafka-enabled Event Hub, you'll need to update the Kafka client configs. If you're having trouble finding yours, try searching for where bootstrap.servers
is set in your application.
Insert the following configs wherever makes sense in your application. Make sure to update the bootstrap.servers
and sasl.jaas.config
values to direct the client to your Event Hubs Kafka endpoint with the correct authentication.
bootstrap.servers=mynamespace.servicebus.windows.net:9093
request.timeout.ms=60000
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
If sasl.jaas.config
is not a supported configuration in your framework, find the configurations that are used to set the SASL username and password and use those instead. Set the username to $ConnectionString
and the password to your Event Hubs connection string.
With Event Hubs AMQP clients, a ServerBusy exception is immediately returned upon service throttling, equivalent to a “try again later” message. In Kafka, messages are just delayed before being completed, and the delay length is returned in milliseconds as throttle_time_ms
in the produce/fetch response. In most cases, these delayed requests are not logged as ServerBusy exceptions on Event Hubs dashboards – instead, the response's throttle_time_ms
value should be used as an indicator that throughput has exceeded the provisioned quota.
If traffic is extremely excessive, the service has the following behavior:
Dedicated clusters do not have throttling mechanisms - you are free to consume all of your cluster resources. An overview on dedicated clusters can be found here.
There is no exception or error when this happens, but the Kafka logs will show that the consumers are stuck trying to re-join the group and assign partitions. There are a few possible causes:
request.timeout.ms
is at least the recommended value of 60000 and your session.timeout.ms
is at least the recommended value of 30000. Having these too low could cause consumer timeouts which then cause rebalances (which then cause more timeouts which then cause more rebalancing...)Kafka supports compression, and Event Hubs for Kafka currently does not. Errors that mention a message format version (e.g. The message format version on the broker does not support the request.
) are usually caused when a client tries to send compressed Kafka messages to our brokers.
If compressed data is necessary, compressing your data before sending it to the brokers and decompressing after receiving it is a valid workaround. The message body is just a byte array to the service, so client-side compression/decompression will not cause any issues.
Note: if you are using rdkafka
-based libraries and you are seeing issues where producers get 100% timeout failures, please upgrade your rdkafka
installation to the latest version (> v1.5.2).
The error will look something like this:
org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
Please open an issue. Debug-level logging and exception timestamps in UTC are extremely helpful.
Check the following items if experiencing issues when using Kafka on Event Hubs.
Firewall blocking traffic - Make sure that port 9093 isn't blocked by your firewall.
TopicAuthorizationException - The most common causes of this exception are:
Kafka version mismatch - Event Hubs for Kafka Ecosystems supports Kafka versions 1.0 and later. Some applications using Kafka version 0.10 and later could occasionally work because of the Kafka protocol's backwards compatability, but we heavily recommend against using old API versions. Kafka versions 0.9 and earlier do not support the required SASL protocols and will not be able to connect to Event Hubs.
Strange encodings on AMQP headers when consuming with Kafka - when sending to Event Hubs over AMQP, any AMQP payload headers are serialized in AMQP encoding. Kafka consumers will not deserialize the headers from AMQP - to read header values, you must manually decode the AMQP headers. (Alternatively, you can avoid using AMQP headers if you know that you will be consuming via Kafka protocol.) See here - https://github.com/Azure/azure-event-hubs-for-kafka/issues/56
SASL authentication - Getting your framework to cooperate with the SASL authentication protocol required by Event Hubs can be more difficult than meets the eye. See if you can troubleshoot the configuration using your framework's resources on SASL authentication. If you figure it out, let us know and we'll share it with other developers!
If you're still stuck (or if you know the secret to making it work with your framework), let us know by opening up a GitHub issue on this repo!
For the most part, the Event Hubs for Kafka Ecosystems has the same defaults, properties, error codes, and general behavior that Apache Kafka does. The instances where the two explicitly differ (or where Event Hubs imposes a limit that Kafka does not) are listed below:
group.id
property is 256 charactersoffset.metadata.max.bytes
is 1024 bytesAre you running Apache Kafka?
No. We execute Kafka API operations against Event Hubs infrastructure. Because there is a tight correlation between Apache Kafka and Event Hubs AMQP functionality (i.e. produce, receive, management, etc.), we are able to bring the known reliability of Event Hubs to the Kafka PaaS space.
What's the difference between an Event Hub consumer group and a Kafka consumer group on Event Hubs?
Kafka consumer groups on EH are fully distinct from standard Event Hubs consumer groups.
Event Hubs consumer groups are...
Kafka consumer groups are...
See the following articles in our Azure Event Hubs documentation: