karafka

• [Feature] Provide out of the box encryption support for Pro.
• [Improvement] Add instrumentation upon #pause.
• [Improvement] Add instrumentation upon retries.
• [Improvement] Assign #id to consumers similar to other entities for ease of debugging.
• [Improvement] Add retries and pausing to the default LoggerListener.
• [Improvement] Introduce a new final terminated state that will kick in prior to exit but after all the instrumentation and other things are done.
• [Improvement] Ensure that state transitions are thread-safe and ensure state transitions can occur in one direction.
• [Improvement] Optimize status methods proxying to Karafka::App.
• [Improvement] Allow for easier state usage by introducing explicit #to_s for reporting.
• [Improvement] Change auto-generated id from SecureRandom#uuid to SecureRandom#hex(6)
• [Improvement] Emit statistic every 5 seconds by default.
• [Improvement] Introduce general messages parser that can be swapped when needed.
• [Fix] Do not trigger code reloading when consumer_persistence is enabled.
• [Fix] Shutdown producer after all the consumer components are down and the status is stopped. This will ensure, that any instrumentation related Kafka messaging can still operate.

Upgrade notes

If you want to disable librdkafka statistics because you do not use them at all, update the kafka statistics.interval.ms setting and set it to 0:

class KarafkaApp < Karafka::App
  setup do |config|
    # Other settings...
    config.kafka = {
      'statistics.interval.ms': 0
    }
  end
end

• [Maintenance] Align with waterdrop and karafka-core
• [Improvement] Provide Admin#read_topic API to get topic data without subscribing.
• [Improvement] Upon an end user #pause, do not commit the offset in automatic offset management mode. This will prevent from a scenario where pause is needed but during it a rebalance occurs and a different assigned process starts not from the pause location but from the automatic offset that may be different. This still allows for using the #mark_as_consumed.
• [Fix] Fix a scenario where manual #pause would be overwritten by a resume initiated by the strategy.
• [Fix] Fix a scenario where manual #pause in LRJ would cause infinite pause.

• [Improvement] Load Pro components upon Karafka require so they can be altered prior to setup.
• [Improvement] Do not run LRJ jobs that were added to the jobs queue but were revoked meanwhile.
• [Improvement] Allow running particular named subscription groups similar to consumer groups.
• [Improvement] Allow running particular topics similar to consumer groups.
• [Improvement] Raise configuration error when trying to run Karafka with options leading to no subscriptions.
• [Fix] Fix karafka info subscription groups count reporting as it was misleading.
• [Fix] Allow for defining subscription groups with symbols similar to consumer groups and topics to align the API.
• [Fix] Do not allow for an explicit nil as a subscription_group block argument.
• [Fix] Fix instability in subscription groups static members ids when using --consumer_groups CLI flag.
• [Fix] Fix a case in routing, where anonymous subscription group could not be used inside of a consumer group.
• [Fix] Fix a case where shutdown prior to listeners build would crash the server initialization.
• [Fix] Duplicated logs in development environment for Rails when logger set to $stdout.

• [Improvement] Make revocation jobs for LRJ topics non-blocking to prevent blocking polling when someone uses non-revocation-aware LRJ jobs and revocation happens.

• [Improvement] Support group.instance.id assignment (static group membership) for a case where a single consumer group has multiple subscription groups (#1173).

• [Feature] Provide the ability to skip failing messages without dispatching them to an alternative topic (DLQ).
• [Improvement] Improve the integration with Ruby on Rails by preventing double-requiring components.
• [Improvement] Improve stability of the shutdown process upon critical errors.
• [Improvement] Improve stability of the integrations spec suite.
• [Fix] Fix an issue where upon fast startup of multiple subscription groups from the same consumer group, a ghost queue would be created due to problems in Concurrent::Hash.

• [Feature] Support quiet mode via TSTP signal. When used, Karafka will finish processing current messages, run shutdown jobs, and switch to a quiet mode where no new work is being accepted. At the same time, it will keep the consumer group quiet, and thus no rebalance will be triggered. This can be particularly useful during deployments.
• [Improvement] Trigger #revoked for jobs in case revocation would happen during shutdown when jobs are still running. This should ensure we get a notion of revocation for Pro LRJ jobs even when revocation happens upon shutdown (#1150).
• [Improvement] Stabilize the shutdown procedure for consumer groups with many subscription groups that have non-aligned processing cost per batch.
• [Improvement] Remove double loading of Karafka via Rails railtie.
• [Fix] Fix invalid class references in YARD docs.
• [Fix] prevent parallel closing of many clients.
• [Fix] fix a case where information about revocation for a combination of LRJ + VP would not be dispatched until all VP work is done.

• [Fix] Few typos around DLQ and Pro DLQ Dispatch original metadata naming.
• [Fix] Narrow the components lookup to the appropriate scope (#1114)

Upgrade notes

1. Replace original-* references from DLQ dispatched metadata with original_*

# DLQ topic consumption
def consume
  messages.each do |broken_message|
    topic = broken_message.metadata['original_topic'] # was original-topic
    partition = broken_message.metadata['original_partition'] # was original-partition
    offset = broken_message.metadata['original_offset'] # was original-offset

    Rails.logger.error "This message is broken: #{topic}/#{partition}/#{offset}"
  end
end

• [Breaking] Disable the root manual_offset_management setting and require it to be configured per topic. This is part of "topic features" configuration extraction for better code organization.
• [Feature] Introduce Dead Letter Queue feature and Pro Enhanced Dead Letter Queue feature
• [Improvement] Align attributes available in the instrumentation bus for listener related events.
• [Improvement] Include consumer group id in consumption related events (#1093)
• [Improvement] Delegate pro components loading to Zeitwerk
• [Improvement] Include Datadog::LoggerListener for tracking logger data with DataDog (@bruno-b-martins)
• [Improvement] Include seek_offset in the consumer.consume.error event payload (#1113)
• [Refactor] Remove unused logger listener event handler.
• [Refactor] Internal refactoring of routing validations flow.
• [Refactor] Reorganize how routing related features are represented internally to simplify features management.
• [Refactor] Extract supported features combinations processing flow into separate strategies.
• [Refactor] Auto-create topics in the integration specs based on the defined routing
• [Refactor] Auto-inject Pro components via composition instead of requiring to use Karafka::Pro::BaseConsumer (#1116)
• [Fix] Fix a case where routing tags would not be injected when given routing definition would not be used with a block
• [Fix] Fix a case where using #active_job_topic without extra block options would cause manual_offset_management to stay false.
• [Fix] Fix a case when upon Pro ActiveJob usage with Virtual Partitions, correct offset would not be stored
• [Fix] Fix a case where upon Virtual Partitions usage, same underlying real partition would be resumed several times.
• [Fix] Fix LRJ enqueuing pause increases the coordinator counter (#115)
• [Fix] Release ActiveRecord connection to the pool after the work in non-dev envs (#1130)
• [Fix] Fix a case where post-initialization shutdown would not initiate shutdown procedures.
• [Fix] Prevent Karafka from committing offsets twice upon shutdown.
• [Fix] Fix for a case where fast consecutive stop signaling could hang the stopping listeners.
• [Specs] Split specs into regular and pro to simplify how resources are loaded
• [Specs] Add specs to ensure, that all the Pro components have a proper per-file license (#1099)

Upgrade notes

1. Remove the manual_offset_management setting from the main config if you use it:

class KarafkaApp < Karafka::App
  setup do |config|
    # ...

    # This line needs to be removed:
    config.manual_offset_management = true
  end
end

2. Set the manual_offset_management feature flag per each topic where you want to use it in the routing. Don't set it for topics where you want the default offset management strategy to be used.

class KarafkaApp < Karafka::App
  routes.draw do
    consumer_group :group_name do
      topic :example do
        consumer ExampleConsumer
        manual_offset_management true
      end

      topic :example2 do
        consumer ExampleConsumer2
        manual_offset_management true
      end
    end
  end
end

3. If you were using code to restart dead connections similar to this:

class ActiveRecordConnectionsCleaner
  def on_error_occurred(event)
    return unless event[:error].is_a?(ActiveRecord::StatementInvalid)

    ::ActiveRecord::Base.clear_active_connections!
  end
end

Karafka.monitor.subscribe(ActiveRecordConnectionsCleaner.new)

It should be removed. This code is no longer needed.

• Sanitize admin config before any admin action.
• Make messages partitioner outcomes for virtual partitions consistently distributed for concurrent processing.
• Improve DataDog/StatsD metrics reporting by reporting partition lags and trends per topic.
• Replace synchronous offset commit with async on resuming paused partition (#1087).

• Prevent consecutive stop signals from starting multiple supervision shutdowns.
• Provide Karafka::Embedded to simplify the start/stop process when running Karafka from within other process (Puma, Sidekiq, etc).
• Fix a race condition when un-pausing a long-running-job exactly upon listener resuming would crash the listener loop (#1072).

• Early exit upon attempts to commit current or earlier offset twice.
• Add more integration specs covering edge cases.
• Strip non-producer-related config when default producer is initialized (#776)

• Fix concurrent-ruby missing as a dependency (Azdaroth)
• Warn about upcoming end of 1.4 support.

• Commit stored offsets upon rebalancing revocation event to reduce the number of messages that are re-processed.
• Support cooperative-sticky rebalance strategy.
• Replace the offset commit after each batch with a per-rebalance commit.
• User instrumentation to publish internal rebalance errors.

This is an important release because:

• it allows you to use cooperative-sticky strategy for rebalancing (https://github.com/karafka/karafka/blob/66457b104a409734139116ec206ff5e1657b031d/spec/integrations/rebalancing/cooperative_sticky.rb) - this can drastically improve rebalancing friction and number of potential reprocessings
• upon rebalance the most recent offset is always flushed. Until now it would "probably" be flushed because of the async trigger request but there was no warranty. Now there is.
• Karafka will no longer trigger async flush after each batch due to latter. This should improve stability without compromising risk of reprocessing (again: as each rebalance will trigger a commit flush)

https://rubygems.org/gems/karafka/versions/2.0.12
https://my.diffend.io/gems/karafka/2.0.11/2.0.12

• Report early on errors related to network and on max poll interval being exceeded to indicate critical problems that will be retries but may mean some underlying problems in the system.
• Fix support of Ruby 2.7.0 to 2.7.2 (#1045)

• Improve error recovery by delegating the recovery to the existing librdkafka instance.

• Fix Singleton not visible when used in PORO (#1034)
• Divide pristine specs into pristine and poro. Pristine will still have helpers loaded, poro will have nothing.
• Fix a case where manual_offset_management offset upon error is not reverted to the first message in a case where there were no markings as consumed at all for multiple batches.
• Implement small reliability improvements around marking as consumed.
• Introduce a config sanity check to make sure Virtual Partitions are not used with manual offset management.
• Fix the possibility of using active_job_topic with Virtual Partitions and manual offset management (ActiveJob still can use due to the atomicity of jobs).
• Move seek offset ownership to the coordinator to allow Virtual Partitions further development.
• Improve client shutdown in specs.
• Do not reset client on network issue and rely on librdkafka to do so.
• Allow for nameless (anonymous) subscription groups (#1033)

• [Breaking change] Rename Virtual Partitions concurrency to max_partitions to avoid confusion (#1023).
• Allow for block-based subscription group management (#1030).

• [Breaking change] Redefine the Virtual Partitions routing DSL to accept concurrency
• Allow for concurrency setting in Virtual Partitions to extend or limit number of jobs per regular partition. This allows to make sure, we do not use all the threads on virtual partitions jobs
• Allow for creation of as many Virtual Partitions as needed, without taking global concurrency into consideration

• Improve client closing.
• Fix for: Multiple LRJ topics fetched concurrently block ability for LRJ to kick in (#1002)
• Introduce a pre-enqueue sync execution layer to prevent starvation cases for LRJ
• Close admin upon critical errors to prevent segmentation faults
• Add support for manual subscription group management (#852 / feature)

Subscription groups feature explanation can be found here: https://karafka.io/docs/Concurrency-and-multithreading/#consumer-groups-and-topics-structure