karafka

karafka - v2.2.7

Published by mensfeld about 1 year ago

[Feature] Introduce Inline Insights to both OSS and Pro. Inline Insights allow you to get the Kafka insights/metrics from the consumer instance and use them to alter the processing flow. In Pro, there's an extra filter flow allowing to ensure, that the insights exist during consumption.
[Enhancement] Make sure, that subscription groups ids are unique by including their consumer group id in them similar to how topics ids are handled (not a breaking change).
[Enhancement] Expose #attempt method on a consumer to directly indicate number of attempt of processing given data.
[Enhancement] Support Rails 7.1.

karafka - v2.2.6

Published by mensfeld about 1 year ago

[Enhancement] Retry Karafka::Admin#read_watermark_offsets fetching upon not_leader_for_partition that can occur mostly on newly created topics in KRaft and after crashes during leader selection.

karafka - v2.2.5

Published by mensfeld about 1 year ago

[Enhancement] Ensure, that when topic related operations end, the result is usable. There were few cases where admin operations on topics would finish successfully but internal Kafka caches would not report changes for a short period of time.
[Enhancement] Stabilize cooperative-sticky early shutdown procedure.
[Fix] use #nil? instead of #present? on DataDog::Tracing::SpanOperation (vitellochris)
[Maintenance] Align connection clearing API with Rails 7.1 deprecation warning.
[Maintenance] Make #subscription_group reference consistent in the Routing and Instrumentation.
[Maintenance] Align the consumer pause instrumentation with client pause instrumentation by adding subscription_group visibility to the consumer.

karafka - v2.2.4

Published by mensfeld about 1 year ago

[Enhancement] Compensate for potential Kafka cluster drifts vs consumer drift in batch metadata (#1611).

karafka - v2.2.3

Published by mensfeld about 1 year ago

[Fix] Karafka admin time based offset lookup can break for one non-default partition.

karafka - v2.2.2

Published by mensfeld about 1 year ago

[Feature] Provide ability to define routing defaults.
[Maintenance] Require karafka-core >= 2.2.2

karafka - v2.2.1

Published by mensfeld about 1 year ago

[Fix] Fix insufficient validation of named patterns
[Maintenance] Rely on 2.2.x karafka-core.

karafka - v2.2.0

Published by mensfeld about 1 year ago

[Feature] Introduce dynamic topic subscriptions based on patterns [Pro].
[Enhancement] Allow for Karafka::Admin setup reconfiguration via config.admin scope.
[Enhancement] Make sure that consumer group used by Karafka::Admin obeys the ConsumerMapper setup.
[Fix] Fix a case where subscription group would not accept a symbol name.

karafka - v2.1.13

Published by mensfeld about 1 year ago

[Feature] Introduce Cleaning API for much better memory management for iterative data processing [Pro].
[Enhancement] Automatically free message resources after processed for ActiveJob jobs [Pro]
[Enhancement] Free memory used by the raw payload as fast as possible after obtaining it from karafka-rdkafka.
[Enhancement] Support changing service_name in DataDog integration.

karafka - v2.1.12

Published by mensfeld about 1 year ago

[Fix] Fix a case where DLQ + VP without intermediate marking would mark earlier message then the last one.

karafka - v2.1.11

Published by mensfeld about 1 year ago

[Enhancement] Expand the error handling for offset related queries with timeout error retries.
[Enhancement] Allow for connection proxy timeouts configuration.

karafka - v2.1.10

Published by mensfeld about 1 year ago

[Enhancement] Introduce connection.client.rebalance_callback event for instrumentation of rebalances.
[Refactor] Introduce low level commands proxy to handle deviation in how we want to run certain commands and how rdkafka-ruby runs that by design.
[Fix] Do not report lags in the DD listener for cases where the assignment is not workable.
[Fix] Do not report negative lags in the DD listener.
[Fix] Extremely fast shutdown after boot in specs can cause process not to stop.
[Fix] Disable allow.auto.create.topics for admin by default to prevent accidental topics creation on topics metadata lookups.
[Fix] Improve the query_watermark_offsets operations by increasing too low timeout.
[Fix] Increase TplBuilder timeouts to compensate for remote clusters.
[Fix] Always try to unsubscribe short-lived consumers used throughout the system, especially in the admin APIs.
[Fix] Add missing connection.client.poll.error error type reference.

karafka - v2.1.9

Published by mensfeld about 1 year ago

[Feature] Introduce ability to customize pause strategy on a per topic basis (Pro).
[Improvement] Disable the extensive messages logging in the default karafka.rb template.
[Change] Require waterdrop >= 2.6.6 due to extra LoggerListener API.

karafka - v2.1.8

Published by mensfeld about 1 year ago

[Improvement] Introduce Karafka::BaseConsumer#used? method to indicate, that at least one invocation of #consume took or will take place. This can be used as a replacement to the non-direct messages.count check for shutdown and revocation to ensure, that the consumption took place or is taking place (in case of running LRJ).
[Improvement] Make messages#to_a return copy of the underlying array to prevent scenarios, where the mutation impacts offset management.
[Improvement] Mitigate a librdkafka cooperative-sticky rebalance crash issue.
[Improvement] Provide ability to overwrite consumer_persistence per subscribed topic. This is mostly useful for plugins and extensions developers.
[Fix] Fix a case where the performance tracker would crash in case of mutation of messages to an empty state.

karafka - v2.1.7

Published by mensfeld about 1 year ago

[Improvement] Always query for watermarks in the Iterator to improve the initial response time.
[Improvement] Add max_wait_time option to the Iterator.
[Fix] Fix a case where Admin#read_topic would wait for poll interval on non-existing messages instead of early exit.
[Fix] Fix a case where Iterator with per partition offsets with negative lookups would go below the number of available messages.
[Fix] Remove unused constant from Admin module.
[Fix] Add missing connection.client.rebalance_callback.error to the LoggerListener instrumentation hook.

karafka - v2.1.6

Published by mensfeld over 1 year ago

[Improvement] Provide time support for iterator
[Improvement] Provide time support for admin #read_topic
[Improvement] Provide time support for consumer #seek.
[Improvement] Remove no longer needed locks for client operations.
[Improvement] Raise Karafka::Errors::TopicNotFoundError when trying to iterate over non-existing topic.
[Improvement] Ensure that Kafka multi-command operations run under mutex together.
[Change] Require waterdrop >= 2.6.2
[Change] Require karafka-core >= 2.1.1
[Refactor] Clean-up iterator code.
[Fix] Improve performance in dev environment for a Rails app (juike)
[Fix] Rename InvalidRealOffsetUsage to InvalidRealOffsetUsageError to align with naming of other errors.
[Fix] Fix unstable spec.
[Fix] Fix a case where automatic #seek would overwrite manual seek of a user when running LRJ.
[Fix] Make sure, that user direct #seek and #pause operations take precedence over system actions.
[Fix] Make sure, that #pause and #resume with one underlying connection do not race-condition.

karafka - v2.1.5

Published by mensfeld over 1 year ago

[Improvement] Drastically improve #revoked? response quality by checking the real time assignment lost state on librdkafka.
[Improvement] Improve eviction of saturated jobs that would run on already revoked assignments.
[Improvement] Expose #commit_offsets and #commit_offsets! methods in the consumer to provide ability to commit offsets directly to Kafka without having to mark new messages as consumed.
[Improvement] No longer skip offset commit when no messages marked as consumed as librdkafka has fixed the crashes there.
[Improvement] Remove no longer needed patches.
[Improvement] Ensure, that the coordinator revocation status is switched upon revocation detection when using #revoked?
[Improvement] Add benchmarks for marking as consumed (sync and async).
[Change] Require karafka-core >= 2.1.0
[Change] Require waterdrop >= 2.6.1

karafka - v2.1.5.beta1

Published by mensfeld over 1 year ago

[Improvement] Drastically improve #revoked? response quality by checking the real time assignment lost state on librdkafka.
[Improvement] Improve eviction of saturated jobs that would run on already revoked assignments.
[Improvement] Expose #commit_offsets and #commit_offsets! methods in the consumer to provide ability to commit offsets directly to Kafka without having to mark new messages as consumed.
[Improvement] No longer skip offset commit when no messages marked as consumed as librdkafka has fixed the crashes there.
[Improvement] Remove no longer needed patches.
[Change] Require karafka-core >= 2.1.0
[Change] Require waterdrop >= 2.6.1

karafka - v2.1.4

Published by mensfeld over 1 year ago

[Fix] processing_lag and consumption_lag on empty batch fail on shutdown usage (#1475)

karafka - v2.1.3

Published by mensfeld over 1 year ago

[Maintenance] Add linter to ensure, that all integration specs end with _spec.rb.
[Fix] Fix #retrying? helper result value (Aerdayne).
[Fix] Fix mark_as_consumed! raising an error instead of false on unknown_member_id (#1461).
[Fix] Enable phantom tests.

Related Projects

karafka-testing

cqrs-ruby-example

ruby-kafka

karafka-web