Confluent's Kafka Python Client
OTHER License
Bot releases are visible (Hide)
describe_cluster()
and describe_topics()
. (@jainruchir, #1635)list_offsets
(#1576).Rack
to the Node
type, so AdminAPI calls can expose racks for brokersconfluent-kafka-python is based on librdkafka v2.3.0, see the
librdkafka release notes
for a complete list of changes, enhancements, fixes and upgrade considerations.
Published by pranavrth over 1 year ago
confluent-kafka-python is based on librdkafka v2.2.0, see the
librdkafka release notes
for a complete list of changes, enhancements, fixes and upgrade considerations.
Published by pranavrth over 1 year ago
confluent-kafka-python is based on librdkafka v2.1.1, see the
librdkafka release notes
for a complete list of changes, enhancements, fixes and upgrade considerations.
Published by pranavrth over 1 year ago
set_sasl_credentials
. This new method (on the Producer, Consumer, and AdminClient) allows modifying the stored SASL PLAIN/SCRAM credentials that will be used for subsequent (new) connections to a broker (#1511).confluent-kafka-python is based on librdkafka v2.1.0, see the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
Published by pranavrth over 1 year ago
list_consumer_groups
Admin operation. Supports listing by state.describe_consumer_groups
Admin operation. Supports multiple groups.delete_consumer_groups
Admin operation. Supports multiple groups.list_consumer_group_offsets
Admin operation. Currently, only supports 1 group with multiple partitions. Supports require_stable option.alter_consumer_group_offsets
Admin operation. Currently, only supports 1 group with multiple offsets.normalize.schemas
configuration property to Schema Registry client (@rayokota, #1406)TopicPartition
type and commit()
(#1410).consumer.memberid()
for getting member id assigned tonb_bool
method for the Producer, so that the default (which uses len)AvroProducer
and AvroConsumer
. Use AvroSerializer
and AvroDeserializer
instead.list_groups
. Use list_consumer_groups
and describe_consumer_groups
instead.OpenSSL 3.0.x upgrade in librdkafka requires a major version bump, as some
legacy ciphers need to be explicitly configured to continue working,
but it is highly recommended NOT to use them. The rest of the API remains
backward compatible.
confluent-kafka-python is based on librdkafka v2.0.2, see the
librdkafka v2.0.0 release notes
and later ones for a complete list of changes, enhancements, fixes and upgrade considerations.
Note: There were no v2.0.0 and v2.0.1 releases.
Published by emasab about 2 years ago
confluent-kafka-python is based on librdkafka v1.9.2, see the
librdkafka release notes
for a complete list of changes, enhancements, fixes and upgrade considerations.
Published by jliunyu over 2 years ago
confluent-kafka-python is based on librdkafka v1.9.0, see the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
Published by edenhill almost 3 years ago
v1.8.2 is a maintenance release with the following fixes and enhancements:
use.deprecated.format
to ProtobufSerializer
and ProtobufDeserializer
.use.latest.version
and skip.known.types
(Protobuf) to the Serializer classes. (Robert Yokota, #1133).list_topics()
and list_groups()
added to AdminClient.confluent-kafka-python is based on librdkafka v1.8.2, see the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
Note: There were no v1.8.0 and v1.8.1 releases.
Prior to this version the confluent-kafka-python client had a bug where nested protobuf schemas indexes were incorrectly serialized, causing incompatibility with other Schema-Registry protobuf consumers and producers.
This has now been fixed, but since the old defect serialization and the new correct serialization are mutually incompatible the user of confluent-kafka-python will need to make an explicit choice which serialization format to use during a transitory phase while old producers and consumers are upgraded.
The ProtobufSerializer
and ProtobufDeserializer
constructors now both take a (for the time being) configuration dictionary that requires
the use.deprecated.format
configuration property to be explicitly set.
Producers should be upgraded first and as long as there are old (<=v1.7.0) Python consumers reading from topics being produced to, the new (>=v1.8.2) Python producer must be configured with use.deprecated.format
set to True
.
When all existing messages in the topic have been consumed by older consumers the consumers should be upgraded and both new producers and the new consumers must set use.deprecated.format
to False
.
The requirement to explicitly set use.deprecated.format
will be removed in a future version and the setting will then default to False
(new format).
Published by jliunyu over 3 years ago
v1.7.0 is a maintenance release with the following fixes and enhancements:
confluent-kafka-python is based on librdkafka v1.7.0, see the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
Published by edenhill over 3 years ago
v1.6.1 is a feature release:
return_record_name=True
to AvroDeserializer (@slominskir, #1028)schema.Parse
call (@casperlehmann, #1006).**kwargs
to legacy AvroProducer and AvroConsumer constructors tologger
(@venthur, #699).producer.flush()
could return a non-zero value without hitting the specified timeout.confluent-kafka-python is based on librdkafka v1.6.1, see the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
Published by edenhill over 3 years ago
v1.6.0 is a feature release with the following features, fixes and enhancements:
Message.latency()
to retrieve the per-message produce latency.consumer.close()
must now be explicitly called if the applicationPY_SSIZE_T_CLEAN
warningproducer.purge()
to purge messages in-queue/flight (@peteryin21, #548)AdminClient.list_groups()
API (@messense, #948)confluent-kafka-python is based on librdkafka v1.6.0, see the librdkafka release notes
for a complete list of changes, enhancements, fixes and upgrade considerations.
Published by edenhill about 4 years ago
v1.5.0 is a maintenance release with the following fixes and enhancements:
confluent-kafka-python is based on librdkafka v1.5.0, see the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
Published by rnpridgeon over 4 years ago
v1.4.2 is a maintenance release with the following fixes and enhancements:
confluent-kafka-python is based on librdkafka v1.4.2, see the librdkafka v1.4.2 release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
Published by rnpridgeon over 4 years ago
v1.4.0 is a feature release:
confluent-kafka-python is based on librdkafka v1.4.0, see the librdkafka v1.4.0 release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
Release v1.4.0 for confluent-kafka-python adds complete Exactly-Once-Semantics (EOS) functionality, supporting the idempotent producer (since v1.0.0), a transaction-aware consumer (since v1.2.0) and full producer transaction support (v1.4.0).
This enables developers to create Exactly-Once applications with Apache Kafka.
See the Transactions in Apache Kafka page for an introduction and check the transactions example.
Release v1.4.0 introduces a new, experimental, API which adds serialization capabilities to Kafka Producer and Consumer. This feature provides the ability to configure Producer/Consumer key and value serializers/deserializers independently. Previously all serialization must be handled prior to calling Producer.produce
and after Consumer.poll
.
This release ships with 3 built-in, Java compatible, standard serializer and deserializer classes:
Name | Type | Format |
---|---|---|
Double | float | IEEE 764 binary64 |
Integer | int | int32 |
String | Unicode | bytes* |
* The StringSerializer codec is configurable and supports any one of Python's standard encodings. If left unspecified 'UTF-8' will be used.
Additional serialization implementations are possible through the extension of the Serializer and Deserializer base classes.
See avro_producer.py and avro_consumer.py for example usage.
Release v1.4.0 for confluent-kafka-python adds support for two new Schema Registry serialization formats with its Generic Serialization API; JSON and Protobuf. A new set of Avro Serialization classes have also been added to conform to the new API.
Format | Serializer Example | Deserializer Example |
---|---|---|
Avro | avro_producer.py | avro_consumer.py |
JSON | json_producer.py | json_consumer.py |
Protobuf | protobuf_producer.py | protobuf_consumer.py |
Two security issues have been identified in the SASL SCRAM protocol handler:
sasl.username
and sasl.password
contained characters that needed escaping, a buffer overflow and heap corruption would occur. This was protected, but too late, by an assertion.Both of these issues are fixed in this release.
General:
Schema Registry/Avro:
/
from Schema Registry base URL (@IvanProdaiko94 , #749)Also see the librdkafka v1.4.0 release notes for fixes to the underlying client implementation.
Published by rnpridgeon almost 5 years ago
confluent-kafka-python is based on librdkafka v1.3.0, see the librdkafka v1.3.0 release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
This is a feature release adding support for KIP-392 Fetch from follower, allowing a consumer to fetch messages from the closest replica to increase throughput and reduce cost.
Published by rnpridgeon about 5 years ago
confluent-kafka-python is based on librdkafka v1.2.0, see the librdkafka v1.2.0 release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
isolation.level=read_committed
) implemented by @mhowlett.linger.ms
) on the producer.This release adds consumer-side support for transactions.
In previous releases, the consumer always delivered all messages to the application, even those in aborted or not yet committed transactions. In this release, the consumer will by default skip messages in aborted transactions.
This is controlled through the new isolation.level
configuration property which
defaults to read_committed
(only read committed messages, filter out aborted and not-yet committed transactions), to consume all messages, including for aborted transactions, you may set this property to read_uncommitted
to get the behaviour of previous releases.
For consumers in read_committed
mode, the end of a partition is now defined to be the offset of the last message of a successfully committed transaction (referred to as the 'Last Stable Offset').
For non-transactional messages there is no change from previous releases, they will always be read, but a consumer will not advance into a not yet committed transaction on the partition.
linger.ms
default was changed from 0 to 0.5 ms to promote some level of batching even with default settings.isolation.level=read_committed
ensures the consumer will only read messages from successfully committed producer transactions. Default is read_committed
. To get the previous behaviour, set the property to read_uncommitted
, which will read all messages produced to a topic, regardless if the message was part of an aborted or not yet committed transaction.General:
linger.ms
, this reduces CPU load and lock contention for high throughput producer applications. (#2509)enable.ssl.certificate.verification=false
(@salisbury-espinosa)Consumer:
pause|resume()
synchronous, ensuring that a subsequent poll()
will not return messages for the paused partitions.Producer:
message.timeout.ms=0
is now accepted even if linger.ms
> 0 (by Jeff Snyder)Published by rnpridgeon over 5 years ago
confluent-kafka-python is based on librdkafka v1.1.0, see the librdkafka v1.1.0 release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
ssl.endpoint.identification.algorithm=https
(off by default) to validate the broker hostname matches the certificate. Requires OpenSSL >= 1.0.2(included with Wheel installations))ssl.ca.location
), librdkafka will load the CA certs by default from the Windows Root Certificate Store.enable.ssl.certificate.verification=false
)%{broker.name}
is no longer supported in sasl.kerberos.kinit.cmd
since kinit refresh is no longer executed per broker, but per client instance.New configuration properties:
ssl.key.pem
- client's private key as a string in PEM formatssl.certificate.pem
- client's public key as a string in PEM formatenable.ssl.certificate.verification
- enable(default)/disable OpenSSL's builtin broker certificate verification.enable.ssl.endpoint.identification.algorithm
- to verify the broker's hostname with its certificate (disabled by default).rd_kafka_conf_set_ssl_cert()
to pass PKCS#12, DER or PEM certs in (binary) memory form to the configuration object.message.timeout.ms
max value from 15 minutes to 24 days (@sarkanyi, workaround for #2015)sasl.kerberos.kinit.cmd
to first attempt ticket refresh, then acquire.max.poll.interval.ms
now correctly handles blocking poll calls, allowing a longer poll timeout than the max poll interval.Published by rnpridgeon over 5 years ago
confluent-kafka-python is based on librdkafka v1.0.1, see the librdkafka v1.0.1 release notes for a complete list of changes, enhancements, fixes and upgrade considerations.
v1.0.1 is a maintenance release with the following fixes:
Published by rnpridgeon over 5 years ago
confluent-kafka-python is based on librdkafka v1.0.0, see the librdkafka v1.0.0 release notes for a complete list of changes, enhancements and fixes and upgrade considerations.
v1.0.0 is a major feature release:
max.poll.interval.ms
support in the Consumer.This release also changes configuration defaults and deprecates a set
of configuration properties, make sure to read the Upgrade considerations
section below.
The following configuration properties have changed default values, which
may require application changes:
acks
(alias request.required.acks
) now defaults to all
; wait for all in-sync replica brokers to ack. The previous default, 1
, only waited for an ack from the partition leader. This change places a greater emphasis on durability at a slight cost to latency. It is not recommended that you lower this value unless latency takes a higher precedence than data durability in your application.
broker.version.fallback
now to defaults to 0.10
, previously 0.9
. broker.version.fallback.ms
now defaults to 0. Users on Apache Kafka <0.10 must set api.version.request=false
and broker.version.fallback=..
to their broker version. For users >=0.10 there is no longer any need to specify any of these properties.
enable.partition.eof
now defaults to false
. KafkaError._PARTITION_EOF
was previously emitted by default to signify the consumer has reached the end of a partition. Applications which rely on this behavior must now explicitly set enable.partition.eof=true
if this behavior is required. This change simplifies the more common case where consumer applications consume in an endless loop.
group.id
is now required for Python consumers.
The following configuration properties have been deprecated. Use of any deprecated configuration property will result in a warning when the client instance is created. The deprecated configuration properties will be removed in a future release.
offset.store.method=file
is deprecated.offset.store.path
is deprecated.offset.store.sync.interval.ms
is deprecated.produce.offset.report
is no longer used. Offsets are always reported.queuing.strategy
was an experimental property that is now deprecated.reconnect.backoff.jitter.ms
is no longer used, see reconnect.backoff.ms
and reconnect.backoff.max.ms
.socket.blocking.max.ms
is no longer used.topic.metadata.refresh.fast.cnt
is no longer used.default.topic.config
is deprecated.This release adds support for Idempotent Producer, providing exactly-once
producing and guaranteed ordering of messages.
Enabling idempotence is as simple as setting the enable.idempotence
configuration property to true
.
There are no required application changes, but it is recommended to add
support for the newly introduced fatal errors that will be triggered when the idempotent producer encounters an unrecoverable error that would break the ordering or duplication guarantees.
See Idempotent Producer in the manual and the Exactly once semantics blog post for more information.
In previous releases librdkafka would maintain open connections to all
brokers in the cluster and the bootstrap servers.
With this release librdkafka now connects to a single bootstrap server
to retrieve the full broker list, and then connects to the brokers
it needs to communicate with: partition leaders, group coordinators, etc.
For large scale deployments this greatly reduces the number of connections
between clients and brokers, and avoids the repeated idle connection closes
for unused connections.
Sparse connections is on by default (recommended setting), the old
behavior of connecting to all brokers in the cluster can be re-enabled
by setting enable.sparse.connections=false
.
See Sparse connections in the manual for more information.
Original issue librdkafka #825.
max.poll.interval.ms
is enforcedThis release adds support for max.poll.interval.ms
(KIP-62), which requires
the application to call consumer.poll()
at least every max.poll.interval.ms
.
Failure to do so will make the consumer automatically leave the group, causing a group rebalance,
and not rejoin the group until the application has called ..poll() again, triggering yet another group rebalance.
max.poll.interval.ms
is set to 5 minutes by default.
CachedSchemaRegistryClient
configuration with configuration dict for application configsDelete Schema
support to CachedSchemaRegistryClientConsumer.consume
without setting group.id
(now required)CachedSchemaRegistryClient
handles get_compatibility
properly./tests/run.sh
added to simplify unit and integration test executionPublished by rnpridgeon almost 6 years ago
See librdkafka v0.11.6 release notes for enhancements and fixes in librdkafka.
default.topic.configuration
has been deprecated and will be removed in 1.0, but still has precedence to topic configuration specified in the global configuration dictionary. (#446)debug
configuration property prior to plugin.library.paths
for enhanced debugging. (#464)