Ruby and Rails efficient Kafka processing framework
OTHER License
Bot releases are hidden (Show)
This release contains BREAKING changes. Make sure to read and apply upgrade notes.
2.7
support.karafka topics migrate
will now perform declarative topics configuration alignment.deserializer
config with #deserializers
in routing to support key and lazy header deserializers.Karafka::Serializers::JSON::Deserializer
to Karafka::Deserializers::Payload
to reflect its role.karafka topics alter
for declarative topics alignment.karafka topics plan
to describe changes that will be applied when migrating.Karafka::Admin::Configs
API for cluster and topics configuration management.rdkafka
thread executable code sections.TTIN
.log_polling
setting to the Karafka::Instrumentation::LoggerListener
to silence polling in any non-debug mode.metadata#message
to be able to retrieve message from metadata.max_wait_time
).Karafka::Admin#read_lags_with_offsets
for ability to query lags and offsets of a given CG.enable.partition.eof
fast yielding.#mark_as_consumed
and #mark_as_consumed!
to the iterator.#stop
to the iterator instead of recommending of usage of break
.Karafka::UnsupportedCaseError
for not recognized error types to support dynamic errors reporting.karafka.erb
template with a placeholder for waterdrop and karafka error instrumentation.statistics.emitted.error
error type with callbacks.statistics.error
to align naming conventions.Admin
operations go through stabilization proxy.#enqueue_all
output compatible with ActiveJob.perform_all_later
(oozzal)PLEASE MAKE SURE TO READ AND APPLY THEM!
Available here.
Published by mensfeld 6 months ago
This release contains BREAKING changes. Make sure to read and apply upgrade notes.
2.7
support.karafka topics migrate
will now perform declarative topics configuration alignment.deserializer
config with #deserializers
in routing to support key and lazy header deserializers.Karafka::Serializers::JSON::Deserializer
to Karafka::Deserializers::Payload
to reflect its role.karafka topics alter
for declarative topics alignment.karafka topics plan
to describe changes that will be applied when migrating.Karafka::Admin::Configs
API for cluster and topics configuration management.TTIN
.log_polling
setting to the Karafka::Instrumentation::LoggerListener
to silence polling in any non-debug mode.metadata#message
to be able to retrieve message from metadata.max_wait_time
).Karafka::Admin#read_lags_with_offsets
for ability to query lags and offsets of a given CG.enable.partition.eof
fast yielding.#mark_as_consumed
and #mark_as_consumed!
to the iterator.#stop
to the iterator instead of recommending of usage of break
.Karafka::UnsupportedCaseError
for not recognized error types to support dynamic errors reporting.karafka.erb
template with a placeholder for waterdrop and karafka error instrumentation.Admin
operations go through stabilization proxy.#enqueue_all
output compatible with ActiveJob.perform_all_later
(oozzal)PLEASE MAKE SURE TO READ AND APPLY THEM!
Available here.
Published by mensfeld 6 months ago
Published by mensfeld 6 months ago
This release contains BREAKING changes. Make sure to read and apply upgrade notes.
2.7
support.karafka topics migrate
will now perform declarative topics configuration alignment.deserializer
config with #deserializers
in routing to support key and lazy header deserializers.Karafka::Serializers::JSON::Deserializer
to Karafka::Deserializers::Payload
to reflect its role.karafka topics alter
for declarative topics alignment.karafka topics plan
to describe changes that will be applied when migrating.Karafka::Admin::Configs
API for cluster and topics configuration management.TTIN
.log_polling
setting to the Karafka::Instrumentation::LoggerListener
to silence polling in any non-debug mode.metadata#message
to be able to retrieve message from metadata.max_wait_time
).Karafka::Admin#read_lags_with_offsets
for ability to query lags and offsets of a given CG.enable.partition.eof
fast yielding.#mark_as_consumed
and #mark_as_consumed!
to the iterator.#stop
to the iterator instead of recommending of usage of break
.Karafka::UnsupportedCaseError
for not recognized error types to support dynamic errors reporting.karafka.erb
template with a placeholder for waterdrop and karafka error instrumentation.Admin
operations go through stabilization proxy.#enqueue_all
output compatible with ActiveJob.perform_all_later
(oozzal)PLEASE MAKE SURE TO READ AND APPLY THEM!
Available here.
Published by mensfeld 6 months ago
This release contains BREAKING changes. Make sure to read and apply upgrade notes.
2.7
support.karafka topics migrate
will now perform declarative topics configuration alignment.deserializer
config with #deserializers
in routing to support key and lazy header deserializers.Karafka::Serializers::JSON::Deserializer
to Karafka::Deserializers::Payload
to reflect its role.karafka topics alter
for declarative topics alignment.karafka topics plan
to describe changes that will be applied when migrating.TTIN
.log_polling
setting to the Karafka::Instrumentation::LoggerListener
to silence polling in any non-debug mode.metadata#message
to be able to retrieve message from metadata.max_wait_time
).Karafka::Admin#read_lags_with_offsets
for ability to query lags and offsets of a given CG.enable.partition.eof
fast yielding.#mark_as_consumed
and #mark_as_consumed!
to the iterator.#stop
to the iterator instead of recommending of usage of break
.Karafka::UnsupportedCaseError
for not recognized error types to support dynamic errors reporting.karafka.erb
template with a placeholder for waterdrop and karafka error instrumentation.Admin
operations go through stabilization proxy.#enqueue_all
output compatible with ActiveJob.perform_all_later
(oozzal)PLEASE MAKE SURE TO READ AND APPLY THEM!
Available here.
Published by mensfeld 8 months ago
-1
shutdown reason status for a non-responding node in swarm.distribution
mode for DataDog listener histogram reporting (Aerdayne).internal.swarm.node_report_timeout
to 60 seconds from 30 seconds to compensate for long pollings.Published by mensfeld 8 months ago
:partition
as the partition key for ActiveJob assignments.Process.warmup
in Ruby 3.3+ prior to forks (in swarm) and prior to app start.app.before_warmup
event to allow hooking code loading tools prior to final warmup.Consumer#errors_tracker
to be able to get errors that occurred while doing complex recovery.Published by mensfeld 8 months ago
Karafka::Helpers::Async#async_call
can run from multiple threads.Published by mensfeld 9 months ago
#transaction
block (Pro)Karafka::Admin::Acl
for Kafka ACL management via the Admin APIs.non_blocking
routing API that aliases LRJ to indicate a different use-case for LRJ flow approach.reset_offset
keyword attribute set to true
.#producer
reference.:independent
configuration to DLQ allowing to reset pause count track on each marking as consumed when retrying.librdkafka
improving multi-sg shutdown times for cooperative-sticky
.Kubernetes::LivenessListener
not start until Karafka app starts running.#tick
.define_method
with class_eval
in some locations.#stop_partition
work with karafka-rdkafka 0.14.6
.Karafka::Admin#seek_consumer_group
would fail because reaching not the coordinator.Published by mensfeld 9 months ago
RC1 release of 2.3.0
Published by mensfeld 9 months ago
Published by mensfeld 9 months ago
Published by mensfeld 11 months ago
Karafka::Admin#delete_consumer_group
and Karafka::Admin#seek_consumer_group
.Karafka::App.assignments
that will return real-time assignments tracking.finished?
method to the jobs for advanced reference based job schedulers.client.reset
notification event.before_enqueue
to before_schedule
to reflect what it does and when (internal).Published by mensfeld 11 months ago
consumer.before_enqueue
.concurrent-ruby
(plan to remove it as a dependency fully)#synchronize
API same as in VPs for LRJs to allow for lifecycle events and consumption synchronization.Published by mensfeld 11 months ago
max_wait_time
setups. This change provides stability to the statistics and background error emitting making them time-reliable.#insights
with #inline_insights
and #insights?
with #inline_insights?
Published by mensfeld 12 months ago
#synchronize
block.karafka server
, that is: karafka consumer
.Published by mensfeld 12 months ago
#pause
without specifying the offset (provide offset or :consecutive
). This allows for pausing on the consecutive message (last received + 1), so after resume we will get last message received + 1 effectively not using #seek
and not purging librdafka
buffer preserving on networking. Please be mindful that this uses notion of last message passed from librdkafka, and not the last one available in the consumer (messages.last
). While for regular cases they will be the same, when using things like DLQ, LRJs, VPs or Filtering API, those may not be the same.:consecutive
offset as default strategy for running LRJs without moving the offset in place and purging the data.In the latest Karafka release, there are no breaking changes. However, please note the updates to #pause and #seek. If you spot any issues, please report them immediately. Your feedback is crucial.
Published by mensfeld 12 months ago
Karafka::Admin#read_topic
.2.6.10
or higher is used with this release to support transactions fully and the Web-UI.Published by mensfeld 12 months ago
#synchronize
for VPs to allow for locks when cross-VP consumers work is needed.#collapse_until!
direct consumer API to allow for collapsed virtual partitions consumer operations together with the Filtering API for advanced use-cases.librdkafka
to Karafka. Replace connection.client.rebalance_callback
with rebalance.partitions_assigned
and rebalance.partitions_revoked
. Introduce two extra events: rebalance.partitions_assign
and rebalance.partitions_revoke
to handle pre-rebalance future work.thor
as a CLI layer and rely on Ruby OptParser
connection.client.rebalance_callback
which was considered private, nothing.thor
has been removed so please report if you find any bugs.Published by mensfeld 12 months ago