karafka | Ruby Ecosystem Directory

Bot releases are hidden (Show)

karafka - v2.4.0 Latest Release

Published by mensfeld 6 months ago

This release contains BREAKING changes. Make sure to read and apply upgrade notes.

[Breaking] Drop Ruby 2.7 support.
[Breaking] Drop the concept of consumer group mapping.
[Breaking] karafka topics migrate will now perform declarative topics configuration alignment.
[Breaking] Replace deserializer config with #deserializers in routing to support key and lazy header deserializers.
[Breaking] Rename Karafka::Serializers::JSON::Deserializer to Karafka::Deserializers::Payload to reflect its role.
[Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
[Feature] Provide karafka topics alter for declarative topics alignment.
[Feature] Introduce ability to use direct assignments (Pro).
[Feature] Provide consumer piping API (Pro).
[Feature] Introduce karafka topics plan to describe changes that will be applied when migrating.
[Feature] Introduce ability to use custom message key deserializers.
[Feature] Introduce ability to use custom message headers deserializers.
[Feature] Provide Karafka::Admin::Configs API for cluster and topics configuration management.
[Enhancement] Protect critical rdkafka thread executable code sections.
[Enhancement] Assign names to internal threads for better debuggability when on TTIN.
[Enhancement] Provide log_polling setting to the Karafka::Instrumentation::LoggerListener to silence polling in any non-debug mode.
[Enhancement] Provide metadata#message to be able to retrieve message from metadata.
[Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
[Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
[Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
[Enhancement] Allow for usage of a custom Appsignal namespace when logging.
[Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
[Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
[Enhancement] Introduce ability to async lock other subscription groups polling.
[Enhancement] Improve shutdown when using long polling setup (high max_wait_time).
[Enhancement] Provide Karafka::Admin#read_lags_with_offsets for ability to query lags and offsets of a given CG.
[Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
[Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
[Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
[Enhancement] Support enable.partition.eof fast yielding.
[Enhancement] Provide #mark_as_consumed and #mark_as_consumed! to the iterator.
[Enhancement] Introduce graceful #stop to the iterator instead of recommending of usage of break.
[Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
[Enhancement] Publish listeners status lifecycle events.
[Enhancement] Use proxy wrapper for Admin metadata requests.
[Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
[Enhancement] No longer raise Karafka::UnsupportedCaseError for not recognized error types to support dynamic errors reporting.
[Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
[Change] Update karafka.erb template with a placeholder for waterdrop and karafka error instrumentation.
[Change] Replace statistics.emitted.error error type with callbacks.statistics.error to align naming conventions.
[Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
[Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
[Fix] Fix a case where connection conductor would not be released during manager state changes.
[Fix] Make sure, that all Admin operations go through stabilization proxy.
[Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
[Fix] Fix a case where critically crashed supervisor would raise incorrect error.
[Fix] Re-raise critical supervisor errors before shutdown.
[Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
[Fix] Make #enqueue_all output compatible with ActiveJob.perform_all_later (oozzal)
[Fix] Seek consumer group on a topic level is updating only recent partition.

Upgrade Notes

PLEASE MAKE SURE TO READ AND APPLY THEM!

Available here.

karafka - v2.4.0.rc1

Published by mensfeld 6 months ago

This release contains BREAKING changes. Make sure to read and apply upgrade notes.

[Breaking] Drop Ruby 2.7 support.
[Breaking] Drop the concept of consumer group mapping.
[Breaking] karafka topics migrate will now perform declarative topics configuration alignment.
[Breaking] Replace deserializer config with #deserializers in routing to support key and lazy header deserializers.
[Breaking] Rename Karafka::Serializers::JSON::Deserializer to Karafka::Deserializers::Payload to reflect its role.
[Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
[Feature] Provide karafka topics alter for declarative topics alignment.
[Feature] Introduce ability to use direct assignments (Pro).
[Feature] Provide consumer piping API (Pro).
[Feature] Introduce karafka topics plan to describe changes that will be applied when migrating.
[Feature] Introduce ability to use custom message key deserializers.
[Feature] Introduce ability to use custom message headers deserializers.
[Feature] Provide Karafka::Admin::Configs API for cluster and topics configuration management.
[Enhancement] Assign names to internal threads for better debuggability when on TTIN.
[Enhancement] Provide log_polling setting to the Karafka::Instrumentation::LoggerListener to silence polling in any non-debug mode.
[Enhancement] Provide metadata#message to be able to retrieve message from metadata.
[Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
[Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
[Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
[Enhancement] Allow for usage of a custom Appsignal namespace when logging.
[Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
[Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
[Enhancement] Introduce ability to async lock other subscription groups polling.
[Enhancement] Improve shutdown when using long polling setup (high max_wait_time).
[Enhancement] Provide Karafka::Admin#read_lags_with_offsets for ability to query lags and offsets of a given CG.
[Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
[Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
[Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
[Enhancement] Support enable.partition.eof fast yielding.
[Enhancement] Provide #mark_as_consumed and #mark_as_consumed! to the iterator.
[Enhancement] Introduce graceful #stop to the iterator instead of recommending of usage of break.
[Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
[Enhancement] Publish listeners status lifecycle events.
[Enhancement] Use proxy wrapper for Admin metadata requests.
[Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
[Enhancement] No longer raise Karafka::UnsupportedCaseError for not recognized error types to support dynamic errors reporting.
[Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
[Change] Update karafka.erb template with a placeholder for waterdrop and karafka error instrumentation.
[Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
[Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
[Fix] Fix a case where connection conductor would not be released during manager state changes.
[Fix] Make sure, that all Admin operations go through stabilization proxy.
[Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
[Fix] Fix a case where critically crashed supervisor would raise incorrect error.
[Fix] Re-raise critical supervisor errors before shutdown.
[Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
[Fix] Make #enqueue_all output compatible with ActiveJob.perform_all_later (oozzal)
[Fix] Seek consumer group on a topic level is updating only recent partition.

Upgrade Notes

PLEASE MAKE SURE TO READ AND APPLY THEM!

Available here.

karafka - v2.3.4

Published by mensfeld 6 months ago

[Fix] Seek consumer group on a topic level is updating only recent partition.

karafka - v2.4.0.beta2

Published by mensfeld 6 months ago

This release contains BREAKING changes. Make sure to read and apply upgrade notes.

[Breaking] Drop Ruby 2.7 support.
[Breaking] Drop the concept of consumer group mapping.
[Breaking] karafka topics migrate will now perform declarative topics configuration alignment.
[Breaking] Replace deserializer config with #deserializers in routing to support key and lazy header deserializers.
[Breaking] Rename Karafka::Serializers::JSON::Deserializer to Karafka::Deserializers::Payload to reflect its role.
[Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
[Feature] Provide karafka topics alter for declarative topics alignment.
[Feature] Introduce ability to use direct assignments (Pro).
[Feature] Provide consumer piping API (Pro).
[Feature] Introduce karafka topics plan to describe changes that will be applied when migrating.
[Feature] Introduce ability to use custom message key deserializers.
[Feature] Introduce ability to use custom message headers deserializers.
[Feature] Provide Karafka::Admin::Configs API for cluster and topics configuration management.
[Enhancement] Assign names to internal threads for better debuggability when on TTIN.
[Enhancement] Provide log_polling setting to the Karafka::Instrumentation::LoggerListener to silence polling in any non-debug mode.
[Enhancement] Provide metadata#message to be able to retrieve message from metadata.
[Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
[Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
[Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
[Enhancement] Allow for usage of a custom Appsignal namespace when logging.
[Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
[Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
[Enhancement] Introduce ability to async lock other subscription groups polling.
[Enhancement] Improve shutdown when using long polling setup (high max_wait_time).
[Enhancement] Provide Karafka::Admin#read_lags_with_offsets for ability to query lags and offsets of a given CG.
[Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
[Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
[Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
[Enhancement] Support enable.partition.eof fast yielding.
[Enhancement] Provide #mark_as_consumed and #mark_as_consumed! to the iterator.
[Enhancement] Introduce graceful #stop to the iterator instead of recommending of usage of break.
[Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
[Enhancement] Publish listeners status lifecycle events.
[Enhancement] Use proxy wrapper for Admin metadata requests.
[Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
[Enhancement] No longer raise Karafka::UnsupportedCaseError for not recognized error types to support dynamic errors reporting.
[Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
[Change] Update karafka.erb template with a placeholder for waterdrop and karafka error instrumentation.
[Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
[Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
[Fix] Fix a case where connection conductor would not be released during manager state changes.
[Fix] Make sure, that all Admin operations go through stabilization proxy.
[Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
[Fix] Fix a case where critically crashed supervisor would raise incorrect error.
[Fix] Re-raise critical supervisor errors before shutdown.
[Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
[Fix] Make #enqueue_all output compatible with ActiveJob.perform_all_later (oozzal)
[Fix] Seek consumer group on a topic level is updating only recent partition.

Upgrade Notes

PLEASE MAKE SURE TO READ AND APPLY THEM!

Available here.

karafka - v2.4.0.beta1

Published by mensfeld 6 months ago

This release contains BREAKING changes. Make sure to read and apply upgrade notes.

[Breaking] Drop Ruby 2.7 support.
[Breaking] Drop the concept of consumer group mapping.
[Breaking] karafka topics migrate will now perform declarative topics configuration alignment.
[Breaking] Replace deserializer config with #deserializers in routing to support key and lazy header deserializers.
[Breaking] Rename Karafka::Serializers::JSON::Deserializer to Karafka::Deserializers::Payload to reflect its role.
[Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
[Feature] Provide karafka topics alter for declarative topics alignment.
[Feature] Introduce ability to use direct assignments (Pro).
[Feature] Provide consumer piping API (Pro).
[Feature] Introduce karafka topics plan to describe changes that will be applied when migrating.
[Feature] Introduce ability to use custom message key deserializers.
[Feature] Introduce ability to use custom message headers deserializers.
[Enhancement] Assign names to internal threads for better debuggability when on TTIN.
[Enhancement] Provide log_polling setting to the Karafka::Instrumentation::LoggerListener to silence polling in any non-debug mode.
[Enhancement] Provide metadata#message to be able to retrieve message from metadata.
[Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
[Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
[Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
[Enhancement] Allow for usage of a custom Appsignal namespace when logging.
[Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
[Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
[Enhancement] Introduce ability to async lock other subscription groups polling.
[Enhancement] Improve shutdown when using long polling setup (high max_wait_time).
[Enhancement] Provide Karafka::Admin#read_lags_with_offsets for ability to query lags and offsets of a given CG.
[Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
[Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
[Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
[Enhancement] Support enable.partition.eof fast yielding.
[Enhancement] Provide #mark_as_consumed and #mark_as_consumed! to the iterator.
[Enhancement] Introduce graceful #stop to the iterator instead of recommending of usage of break.
[Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
[Enhancement] Publish listeners status lifecycle events.
[Enhancement] Use proxy wrapper for Admin metadata requests.
[Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
[Enhancement] No longer raise Karafka::UnsupportedCaseError for not recognized error types to support dynamic errors reporting.
[Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
[Change] Update karafka.erb template with a placeholder for waterdrop and karafka error instrumentation.
[Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
[Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
[Fix] Fix a case where connection conductor would not be released during manager state changes.
[Fix] Make sure, that all Admin operations go through stabilization proxy.
[Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
[Fix] Fix a case where critically crashed supervisor would raise incorrect error.
[Fix] Re-raise critical supervisor errors before shutdown.
[Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
[Fix] Make #enqueue_all output compatible with ActiveJob.perform_all_later (oozzal)

Upgrade Notes

PLEASE MAKE SURE TO READ AND APPLY THEM!

Available here.

karafka - v2.3.3

Published by mensfeld 8 months ago

[Enhancement] Routing based topics allocation for swarm (Pro)
[Enhancement] Publish the -1 shutdown reason status for a non-responding node in swarm.
[Enhancement] Allow for using the distribution mode for DataDog listener histogram reporting (Aerdayne).
[Change] Change internal.swarm.node_report_timeout to 60 seconds from 30 seconds to compensate for long pollings.
[Fix] Static membership routing evaluation happens too early in swarm.
[Fix] Close producer in supervisor prior to forking and warmup to prevent invalid memory states.

karafka - v2.3.2

Published by mensfeld 8 months ago

[Feature] Provide swarm capabilities to OSS and Pro.
[Feature] Provide ability to use complex strategies in DLQ (Pro).
[Enhancement] Support using :partition as the partition key for ActiveJob assignments.
[Enhancement] Expand Logger listener with swarm notifications.
[Enhancement] Introduce K8s swarm liveness listener.
[Enhancement] Use Process.warmup in Ruby 3.3+ prior to forks (in swarm) and prior to app start.
[Enhancement] Provide app.before_warmup event to allow hooking code loading tools prior to final warmup.
[Enhancement] Provide Consumer#errors_tracker to be able to get errors that occurred while doing complex recovery.
[Fix] Infinite consecutive error flow with VPs and without DLQ can cause endless offsets accumulation.
[Fix] Quieting mode causes too early unsubscribe.

karafka - v2.3.1

Published by mensfeld 8 months ago

[Refactor] Ensure that Karafka::Helpers::Async#async_call can run from multiple threads.

karafka - v2.3.0

Published by mensfeld 9 months ago

[Feature] Introduce Exactly-Once Semantics within consumers #transaction block (Pro)
[Feature] Provide ability to multiplex subscription groups (Pro)
[Feature] Provide Karafka::Admin::Acl for Kafka ACL management via the Admin APIs.
[Feature] Periodic Jobs (Pro)
[Feature] Offset Metadata storage (Pro)
[Feature] Provide low-level listeners management API for dynamic resources scaling (Pro)
[Enhancement] Improve shutdown process by allowing for parallel connections shutdown.
[Enhancement] Introduce non_blocking routing API that aliases LRJ to indicate a different use-case for LRJ flow approach.
[Enhancement] Allow to reset offset when seeking backwards by using the reset_offset keyword attribute set to true.
[Enhancement] Alias producer operations in consumer to skip #producer reference.
[Enhancement] Provide an :independent configuration to DLQ allowing to reset pause count track on each marking as consumed when retrying.
[Enhancement] Remove no longer needed shutdown patches for librdkafka improving multi-sg shutdown times for cooperative-sticky.
[Enhancement] Allow for parallel closing of connections from independent consumer groups.
[Enhancement] Provide recovery flow for cases where DLQ dispatch would fail.
[Change] Make Kubernetes::LivenessListener not start until Karafka app starts running.
[Change] Remove the legacy "inside of topics" way of defining subscription groups names
[Change] Update supported instrumentation to report on #tick.
[Refactor] Replace define_method with class_eval in some locations.
[Fix] Fix a case where internal Idle job scheduling would go via the consumption flow.
[Fix] Make the Iterator #stop_partition work with karafka-rdkafka 0.14.6.
[Fix] Ensure Pro components are not loaded during OSS specs execution (not affecting usage).
[Fix] Fix invalid action label for consumers in DataDog logger instrumentation.
[Fix] Fix a scenario where Karafka::Admin#seek_consumer_group would fail because reaching not the coordinator.
[Ignore] option --include-consumer-groups not working as intended after removal of "thor"

karafka - v2.3.0.rc1

Published by mensfeld 9 months ago

RC1 release of 2.3.0

karafka - v2.3.0.alpha2

Published by mensfeld 9 months ago

karafka - v2.3.0.alpha1

Published by mensfeld 9 months ago

karafka - v2.2.14

Published by mensfeld 11 months ago

[Feature] Provide Karafka::Admin#delete_consumer_group and Karafka::Admin#seek_consumer_group.
[Feature] Provide Karafka::App.assignments that will return real-time assignments tracking.
[Enhancement] Make sure that the Scheduling API is thread-safe by default and allow for lock-less schedulers when schedulers are stateless.
[Enhancement] "Blockless" topics with defaults
[Enhancement] Provide a finished? method to the jobs for advanced reference based job schedulers.
[Enhancement] Provide client.reset notification event.
[Enhancement] Remove all usage of concurrent-ruby from Karafka
[Change] Replace single #before_schedule with appropriate methods and events for scheduling various types of work. This is needed as we may run different framework logic on those and, second, for accurate job tracking with advanced schedulers.
[Change] Rename before_enqueue to before_schedule to reflect what it does and when (internal).
[Change] Remove not needed error catchers for strategies code. This code if errors, should be considered critical and should not be silenced.
[Change] Remove not used notifications events.

karafka - v2.2.13

Published by mensfeld 11 months ago

[Feature] Introduce low-level extended Scheduling API for granular control of schedulers and jobs execution [Pro].
[Improvement] Use separate lock for user-facing synchronization.
[Improvement] Instrument consumer.before_enqueue.
[Improvement] Limit usage of concurrent-ruby (plan to remove it as a dependency fully)
[Improvement] Provide #synchronize API same as in VPs for LRJs to allow for lifecycle events and consumption synchronization.

karafka - v2.2.12

Published by mensfeld 11 months ago

[Improvement] Rewrite the polling engine to update statistics and error callbacks despite longer non LRJ processing or long max_wait_time setups. This change provides stability to the statistics and background error emitting making them time-reliable.
[Improvement] Auto-update Inline Insights if new insights are present for all consumers and not only LRJ (OSS and Pro).
[Improvement] Alias #insights with #inline_insights and #insights? with #inline_insights?

karafka - v2.2.11