kafka-python | Python Ecosystem Directory

Bot releases are hidden (Show)

kafka-python - 1.2.2

Published by dpkp about 8 years ago

Bugfixes

Clarify timeout unit in KafkaProducer close and flush (ms7s PR 734)
Avoid busy poll during metadata refresh failure with retry_backoff_ms (dpkp PR 733)
Check_version should scan nodes until version found or timeout (dpkp PR 731)
Fix bug which could cause least_loaded_node to always return the same unavailable node (dpkp PR 730)
Fix producer garbage collection with weakref in atexit handler (dpkp PR 728)
Close client selector to fix fd leak (msmith PR 729)
Tweak spelling mistake in error const (steve8918 PR 719)
Rearrange connection tests to separate legacy KafkaConnection

kafka-python - 1.2.3

Published by dpkp about 8 years ago

Patch Improvements

Fix gc error log: avoid AttributeError in _unregister_cleanup (dpkp PR 747)
Wakeup socket optimizations (dpkp PR 740)
Assert will be disabled by "python -O" (tyronecai PR 736)
Randomize order of topics/partitions processed by fetcher to improve balance (dpkp PR 732)
Allow client.check_version timeout to be set in Producer and Consumer constructors (eastlondoner PR 647)

kafka-python - 1.2.4

Published by dpkp about 8 years ago

Bugfixes

Update consumer_timeout_ms docstring - KafkaConsumer raises StopIteration, no longer ConsumerTimeout
Use explicit subscription state flag to handle seek() during message iteration
Fix consumer iteration on compacted topics (dpkp PR 752)
Support ssl_password config when loading cert chains (amckemie PR 750)

kafka-python - 1.2.5

Published by dpkp about 8 years ago

Bugfixes

Fix bug causing KafkaProducer to double-compress message batches on retry
Check for double-compressed messages in KafkaConsumer, log warning and optionally skip
Drop recursion in _unpack_message_set; only decompress once

kafka-python - 1.3.0

Published by dpkp about 8 years ago

Incompatible Changes

Delete KafkaConnection class (dpkp 769)
Rename partition_assignment -> assignment in MemberMetadata for consistency
Move selectors34 and socketpair to kafka.vendor (dpkp 785)
Change api_version config to tuple; deprecate str with warning (dpkp 761)
Rename _DEFAULT_CONFIG -> DEFAULT_CONFIG in KafkaProducer (dpkp 788)

Improvements

Vendor six 1.10.0 to eliminate runtime dependency (dpkp 785)
Add KafkaProducer and KafkaConsumer.metrics() with instrumentation similar to java client (dpkp 754 / 772 / 794)
Support Sasl PLAIN authentication (larsjsol PR 779)
Add checksum and size to RecordMetadata and ConsumerRecord (KAFKA-3196 / 770 / 594)
Use MetadataRequest v1 for 0.10+ api_version (dpkp 762)
Fix KafkaConsumer autocommit for 0.8 brokers (dpkp 756 / 706)
Improve error logging (dpkp 760 / 759)
Adapt benchmark scripts from https://github.com/mrafayaleem/kafka-jython (dpkp 754)
Add api_version config to KafkaClient (dpkp 761)
New Metadata method with_partitions() (dpkp 787)
Use socket_options configuration to setsockopts(). Default TCP_NODELAY (dpkp 783)
Expose selector type as config option (dpkp 764)
Drain pending requests to the coordinator before initiating group rejoin (dpkp 798)
Send combined size and payload bytes to socket to avoid potentially split packets with TCP_NODELAY (dpkp 797)

Bugfixes

Ignore socket.error when checking for protocol out of sync prior to socket close (dpkp 792)
Fix offset fetch when partitions are manually assigned (KAFKA-3960 / 786)
Change pickle_method to use python3 special attributes (jpaulodit 777)
Fix ProduceResponse v2 throttle_time_ms
Always encode size with MessageSet (#771)
Avoid buffer overread when compressing messageset in KafkaProducer
Explicit format string argument indices for python 2.6 compatibility
Simplify RecordMetadata; short circuit callbacks (#768)
Fix autocommit when partitions assigned manually (KAFKA-3486 / #767 / #626)
Handle metadata updates during consumer rebalance (KAFKA-3117 / #766 / #701)
Add a consumer config option to exclude internal topics (KAFKA-2832 / #765)
Protect writes to wakeup socket with threading lock (#763 / #709)
Fetcher spending unnecessary time during metrics recording (KAFKA-3785)
Always use absolute_import (dpkp)

Test / Fixtures

Catch select errors while capturing test fixture logs
Fix consumer group test race condition (dpkp 795)
Retry fixture failures on a different port (dpkp 796)
Dump fixture logs on failure

Documentation

Fix misspelling of password (ssaamm 793)
Document the ssl_password config option (ssaamm 780)
Fix typo in KafkaConsumer documentation (ssaamm 775)
Expand consumer.fetcher inline comments
Update kafka configuration links -> 0.10.0.0 docs
Fixup metrics_sample_window_ms docstring in consumer

kafka-python - 1.0.2

Published by dpkp over 8 years ago

This release includes critical bugfixes -- upgrade strongly recommended

Consumers

Improve KafkaConsumer Heartbeat handling (dpkp PR 583)
Fix KafkaConsumer.position bug (stefanth PR 578)
Raise TypeError when partition is not a TopicPartition (dpkp PR 587)
KafkaConsumer.poll should sleep to prevent tight-loops (dpkp PR 597)

Producers

Fix producer threading bug that can crash sender (dpkp PR 590)
Fix bug in producer buffer pool reallocation (dpkp PR 585)
Remove spurious warnings when closing sync SimpleProducer (twm PR 567)
Fix FutureProduceResult.await() on python2.6 (dpkp)
Add optional timeout parameter to KafkaProducer.flush() (dpkp)
KafkaProducer Optimizations (zackdever PR 598)

Clients

Improve error handling in SimpleClient.load_metadata_for_topics (dpkp)
Improve handling of KafkaClient.least_loaded_node failure (dpkp PR 588)

Documentation

Fix KafkaError import error in docs (shichao-an PR 564)
Fix serializer / deserializer examples (scribu PR 573)

Internals

Update to Kafka 0.9.0.1 for integration testing
Fix ifr.future.failure in conn.py (mortenlj PR 566)
Improve Zookeeper / Kafka Fixture management (dpkp)

kafka-python - 1.0.1

Published by dpkp over 8 years ago

Consumers

Add RangePartitionAssignor (and use as default); add assignor tests (dpkp PR 550)
Make sure all consumers are in same generation before stopping group test
Verify node ready before sending offset fetch request from coordinator
Improve warning when offset fetch request returns unknown topic / partition

Producers

Warn if pending batches failed during flush
Fix concurrency bug in RecordAccumulator.ready()
Fix bug in SimpleBufferPool memory condition waiting / timeout
Support batch_size = 0 in producer buffers (dpkp PR 558)
Catch duplicate batch.done() calls [e.g., maybe_expire then a response errback]

Clients

Documentation

Improve kafka.cluster docstrings
Migrate load_example.py to KafkaProducer / KafkaConsumer

Internals

Dont override system rcvbuf or sndbuf unless configured explicitly (dpkp PR 557)
Some attributes may not exist in __del__ if we failed assertions
Break up some circular references and close client wake pipes on __del__ (aisch PR 554)

kafka-python - 0.9.5

Published by dpkp over 8 years ago

Consumers

Initial support for consumer coordinator: offsets only (toddpalino PR 420)
Allow blocking until some messages are received in SimpleConsumer (saaros PR 457)
Support subclass config changes in KafkaConsumer (zackdever PR 446)
Support retry semantics in MultiProcessConsumer (barricadeio PR 456)
Support partition_info in MultiProcessConsumer (scrapinghub PR 418)
Enable seek() to an absolute offset in SimpleConsumer (haosdent PR 412)
Add KafkaConsumer.close() (ucarion PR 426)

Producers

Catch client.reinit() exceptions in async producer (dpkp)
Producer.stop() now blocks until async thread completes (dpkp PR 485)
Catch errors during load_metadata_for_topics in async producer (bschopman PR 467)
Add compression-level support for codecs that support it (trbs PR 454)
Fix translation of Java murmur2 code, fix byte encoding for Python 3 (chrischamberlin PR 439)
Only call stop() on not-stopped producer objects (docker-hub PR 435)
Allow null payload for deletion feature (scrapinghub PR 409)

Clients

Use non-blocking io for broker aware requests (ecanzonieri PR 473)
Use debug logging level for metadata request (ecanzonieri PR 415)
Catch KafkaUnavailableError in _send_broker_aware_request (mutability PR 436)
Lower logging level on replica not available and commit (ecanzonieri PR 415)

Documentation

Update docs and links wrt maintainer change (mumrah -> dpkp)

Internals

Add py35 to tox testing
Update travis config to use container infrastructure
Add 0.8.2.2 and 0.9.0.0 resources for integration tests; update default official releases
new pylint disables for pylint 1.5.1 (zackdever PR 481)
Fix python3 / python2 comments re queue/Queue (dpkp)
Add Murmur2Partitioner to kafka all imports (dpkp Issue 471)
Include LICENSE in PyPI sdist (koobs PR 441)

kafka-python - 1.0.0

Published by dpkp over 8 years ago

This release includes significant code changes. Users of older kafka-python versions are encouraged to test upgrades before deploying to production as some interfaces and configuration options have changed.

Users of SimpleConsumer / SimpleProducer / SimpleClient (formerly KafkaClient) from prior releases should migrate to KafkaConsumer / KafkaProducer. Low-level APIs (Simple*) are no longer being actively maintained and will be removed in a future release.

For comprehensive API documentation, please see python help() / docstrings, kafka-python.readthedocs.org, or run tox -e docs from source to build documentation locally.

Consumers

KafkaConsumer re-written to emulate the new 0.9 kafka consumer (java client) and support coordinated consumer groups (feature requires >= 0.9.0.0 brokers)
- Methods no longer available:
  - configure [initialize a new consumer instead]
  - set_topic_partitions [use subscribe() or assign()]
  - fetch_messages [use poll() or iterator interface]
  - get_partition_offsets
  - offsets [use committed(partition)]
  - task_done [handled internally by auto-commit; or commit offsets manually]
- Configuration changes (consistent with updated java client):
  - lots of new configuration parameters -- see docs for details
  - auto_offset_reset: previously values were 'smallest' or 'largest', now values are 'earliest' or 'latest'
  - fetch_wait_max_ms is now fetch_max_wait_ms
  - max_partition_fetch_bytes is now max_partition_fetch_bytes
  - deserializer_class is now value_deserializer and key_deserializer
  - auto_commit_enable is now enable_auto_commit
  - auto_commit_interval_messages was removed
  - socket_timeout_ms was removed
  - refresh_leader_backoff_ms was removed
SimpleConsumer and MultiProcessConsumer are now deprecated and will be removed in a future release. Users are encouraged to migrate to KafkaConsumer.

Producers

new producer class: KafkaProducer. Exposes the same interface as official java client. Async by default; returned future.get() can be called for synchronous blocking
SimpleProducer is now deprecated and will be removed in a future release. Users are encouraged to migrate to KafkaProducer.

Clients

synchronous KafkaClient renamed to SimpleClient. For backwards compatibility, you will get a SimpleClient via from kafka import KafkaClient. This will change in a future release.
All client calls use non-blocking IO under the hood.
Add probe method check_version() to infer broker versions.

Documentation

Updated README and sphinx documentation to address new classes.
Docstring improvements to make python help() easier to use.

Internals

Old protocol stack is deprecated. It has been moved to kafka.protocol.legacy and may be removed in a future release.
Protocol layer re-written using Type classes, Schemas and Structs (modeled on the java client).
Add support for LZ4 compression (including broken framing header checksum).

kafka-python - 0.9.4

Published by dpkp over 9 years ago

Consumers

Refactor SimpleConsumer internal fetch handling (dpkp #399)
Handle exceptions in SimpleConsumer commit() and reset_partition_offset() (dpkp #404)
Improve FailedPayloadsError handling in KafkaConsumer (dpkp #398)
KafkaConsumer: avoid raising KeyError in task_done (dpkp #389)
MultiProcessConsumer -- support configured partitions list (dpkp #380)
Fix SimpleConsumer leadership change handling (dpkp #393)
Fix SimpleConsumer connection error handling (reAsOn2010 #392)
Improve Consumer handling of 'falsy' partition values (wting #342)
Fix _offsets call error in KafkaConsumer (hellais #376)
Fix str/bytes bug in KafkaConsumer (dpkp #365)
Register atexit handlers for consumer and producer thread/multiprocess cleanup (dpkp #360)
Always fetch commit offsets in base consumer unless group is None (dpkp #356)
Stop consumer threads on delete (dpkp #357)
Deprecate metadata_broker_list in favor of bootstrap_servers in KafkaConsumer (dpkp #340)
Support pass-through parameters in multiprocess consumer (scrapinghub #336)
Enable offset commit on SimpleConsumer.seek (ecanzonieri #350)
Improve multiprocess consumer partition distribution (scrapinghub #335)
Ignore messages with offset less than requested (wkiser #328)
Handle OffsetOutOfRange in SimpleConsumer (ecanzonieri #296)

Producers

Add Murmur2Partitioner (dpkp #378)
Log error types in SimpleProducer and SimpleConsumer (dpkp #405)
SimpleProducer support configuration of fail_on_error (dpkp #396)
Deprecate KeyedProducer.send() (dpkp #379)
Further improvements to async producer code (dpkp #388)
Add more configuration parameters for async producer (dpkp)
Deprecate SimpleProducer batch_send=True in favor of async (dpkp)
Improve async producer error handling and retry logic (vshlapakov #331)
Support message keys in async producer (vshlapakov #329)
Use threading instead of multiprocessing for Async Producer (vshlapakov #330)
Stop threads on __del__ (chmduquesne #324)
Fix leadership failover handling in KeyedProducer (dpkp #314)

KafkaClient

Add .topics property for list of known topics (dpkp)
Fix request / response order guarantee bug in KafkaClient (dpkp #403)
Improve KafkaClient handling of connection failures in _get_conn (dpkp)
Client clears local metadata cache before updating from server (dpkp #367)
KafkaClient should return a response or error for each request - enable better retry handling (dpkp #366)
Improve str/bytes conversion in KafkaClient and KafkaConsumer (dpkp #332)
Always return sorted partition ids in client.get_partition_ids_for_topic() (dpkp #315)

Documentation

Cleanup Usage Documentation
Improve KafkaConsumer documentation (dpkp #341)
Update consumer documentation (sontek #317)
Add doc configuration for tox (sontek #316)
Switch to .rst doc format (sontek #321)
Fixup google groups link in README (sontek #320)
Automate documentation at kafka-python.readthedocs.org

Internals

Switch integration testing from 0.8.2.0 to 0.8.2.1 (dpkp #402)
Fix most flaky tests, improve debug logging, improve fixture handling (dpkp)
General style cleanups (dpkp #394)
Raise error on duplicate topic-partition payloads in protocol grouping (dpkp)
Use module-level loggers instead of simply 'kafka' (dpkp)
Remove pkg_resources check for __version__ at runtime (dpkp #387)
Make external API consistently support python3 strings for topic (kecaps #361)
Fix correlation id overflow (dpkp #355)
Cleanup kafka/common structs (dpkp #338)
Use context managers in gzip_encode / gzip_decode (dpkp #337)
Save failed request as FailedPayloadsError attribute (jobevers #302)
Remove unused kafka.queue (mumrah)

kafka-python - 0.9.3

Published by dpkp over 9 years ago

Add coveralls.io support (sontek PR 307)
Fix python2.6 threading.Event bug in ReentrantTimer (dpkp PR 312)
Add kafka 0.8.2.0 to travis integration tests (dpkp PR 310)
Auto-convert topics to utf-8 bytes in Producer (sontek PR 306)
Fix reference cycle between SimpleConsumer and ReentrantTimer (zhaopengzp PR 309)
Add Sphinx API docs (wedaly PR 282)
Handle additional error cases exposed by 0.8.2.0 kafka server (dpkp PR 295)
Refactor error class management (alexcb PR 289)
Expose KafkaConsumer in all for easy imports (Dinoshauer PR 286)
SimpleProducer starts on random partition by default (alexcb PR 288)
Add keys to compressed messages (meandthewallaby PR 281)
Add new high-level KafkaConsumer class based on java client api (dpkp PR 234)
Add KeyedProducer.send_messages api (pubnub PR 277)
Fix consumer pending() method (jettify PR 276)
Update low-level demo in README (sunisdown PR 274)
Include key in KeyedProducer messages (se7entyse7en PR 268)
Fix SimpleConsumer timeout behavior in get_messages (dpkp PR 238)
Fix error in consumer.py test against max_buffer_size (rthille/wizzat PR 225/242)
Improve string concat performance on pypy / py3 (dpkp PR 233)
Reorg directory layout for consumer/producer/partitioners (dpkp/wizzat PR 232/243)
Add OffsetCommitContext (locationlabs PR 217)
Metadata Refactor (dpkp PR 223)
Add Python 3 support (brutasse/wizzat - PR 227)
Minor cleanups - imports / README / PyPI classifiers (dpkp - PR 221)
Fix socket test (dpkp - PR 222)
Fix exception catching bug in test_failover_integration (zever - PR 216)

kafka-python - 0.9.2

Published by dpkp about 10 years ago

Warn users that async producer does not reliably handle failures (dpkp - PR 213)
Fix spurious ConsumerFetchSizeTooSmall error in consumer (DataDog - PR 136)
Use PyLint for static error checking (dpkp - PR 208)
Strictly enforce str message type in producer.send_messages (dpkp - PR 211)
Add test timers via nose-timer plugin; list 10 slowest timings by default (dpkp)
Move fetching last known offset logic to a stand alone function (zever - PR 177)
Improve KafkaConnection and add more tests (dpkp - PR 196)
Raise TypeError if necessary when encoding strings (mdaniel - PR 204)
Use Travis-CI to publish tagged releases to pypi (tkuhlman / mumrah)
Use official binary tarballs for integration tests and parallelize travis tests (dpkp - PR 193)
Improve new-topic creation handling (wizzat - PR 174)