Features

• Adds configuration options to trigger cache trim before the cache reaches its maximum size. by @jcipar in [#19624](https://github.com/ redpanda-data/redpanda/pull/19624)
  • cloud_storage_cache_trim_threshold_size
  • cloud_storage_cache_trim_threshold_objects
    These mirror the options for controlling maximum size: cloud_storage_cache_size and cloud_storage_cache_max_objects
• The new default behavior, if these are not set, is to trigger a trim when the cache is 100% full. by @jcipar in [#19624](https://github.com/ redpanda-data/redpanda/pull/19624)
• #18739 Schema Registry: Support /mode endpoints for READONLY by @BenPope in #18742

Bug Fixes

• Fixes a bug where crashes within the redpanda http client could occur when encountering tls exceptions by @graphcareful in #18696
• #18633 rpk: fixes an error in rpk topic consume that prevented the usage of the --regex flag. by @r-vasquez in #18634
• #18734 Fixes incorrect ordering of arguments in the cloud cache trim admin endpoint. by @andrwng in #18764
• #18770 Fixes a bug that would allow requests to complete that created acls for topics with invalid kafka topic names by @graphcareful in #19791
• fixed overflow that may lead to unnecessary moves by @mmaslankaprv in #19805
• rpk cluster config get: does not round float numbers anymore. by @r-vasquez in #18849## Improvements
• Short description of how this PR improves existing behavior. by @jcipar in #19624
• #18645 rpk: topic describe supports --regex flag by @daisukebe in #18646
• made fast partition movements easier to debug. by @mmaslankaprv in #18689
• reduced the amount of data required to transfer over the network by @mmaslankaprv in [#19835](https://github.com/redpanda-data/redpanda/ pull/19835)
• PR #18741 [v23.3.x] cloud_storage_clients: check for BlobNotFound in abs_client:: do_delete_path() by @WillemKauf
• PR #18784 [v23.3.x] raft: fix node_id mismatch log message by @ztlpn
• PR #18855 [v23.3.x] Fixed possible log discrepancy when using forced reconfiguration by @mmaslankaprv
• PR #19838 [v23.3.x] s/disk_log_impl: don't prefix-truncate empty segments by @ztlpn
• PR #18573 rm_stm: couple of stability fixes noticed when down scaling max_concurrent_producer_ids by @bharathv

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.16...v23.3.17

Features

• Split cache into buckets using cloud_storage_cache_num_buckets configuration parameter. by @Lazin in #18780

Bug Fixes

• Fixes a bug that would allow requests to complete that created acls for topics with invalid kafka topic names by @graphcareful in #18769
• #18735 Fixes incorrect ordering of arguments in the cloud cache trim admin endpoint. by @andrwng in #18763

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v24.1.6...v24.1.7

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v24.1.5...v24.1.6

Bug fix

• Fix connect so it work's with rpk parent autocomplete

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v24.1.4...v24.1.5

Bug Fixes

• concurrent requests of set_log_level + expiration now work as expected by @andijcr in #18437

Improvements

• #18643 rpk: topic describe supports --regex flag by @daisukebe in #18644
• #18675 rpk now will exit (1) when running rpk with unknown commands by @r-vasquez in #18676
• made fast partition movements easier to debug. by @mmaslankaprv in #18690

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v24.1.3...v24.1.4

Features

• Schema Registry: Support /mode endpoints for READONLY by @BenPope in #18623
• Schema Registry: Support for deleted=true query parameter on POST /subjects/<subject>. by @BenPope in #18433
• #18458 rpk: ability to transfer partition leadership by @daisukebe in #18459

Bug Fixes

• Don't mark partition rebalance complete if some partitions are not moveable (e.g. due to partial recovery mode) by @ztlpn in #18518
• Enforce client quota throttling in a Kafka-compatible way, meaning we enforce the throttle delay on the next request if the client did not enforce it on its side. by @pgellert in #18568
• Fixes a bug in the http client where a crash may occur in the event certain tls verification errors are observed by @graphcareful in #18428
• #18439 Fixed an assertion triggering in a full-disk scenario by @andijcr in #18440
• #18565 Fix an edge case where a timequery returns no results if it races with tiered storage retention and garbage collection. This is important at least for consumers that fall behind retention. They interpret such response as the partition is empty and jump to the HWM instead of resuming consuming from the first available message. by @nvartolomei in #18597
• #18631 rpk: fixes an error in rpk topic consume that prevented the usage of the --regex flag. by @r-vasquez in #18632
• fixes possible stall in raft::state_machine_manger by @mmaslankaprv in #18638
• PR #18392 [v24.1.x] archival: clamp uploads to committed offset by @ nvartolomei

Improvements

• Made electing a leader faster by @mmaslankaprv in #18493
• PR #18448 [v24.1.x] cloud_storage: correct list_object() request headers and parameters (manual backport) by @WillemKauf
• PR #18476 [v24.1.x] rptest: be more permissive with errors in stress fibers test by @andrwng
• PR #18488 [v24.1.x] tests: wait for messages before adding a node to cluster by @mmaslankaprv
• PR #18503 [v24.1.x] storage: change map type for _db in kvstore by @ WillemKauf
• PR #18520 [v24.1.x] Made client id parsing vcluster aware by @ mmaslankaprv
• PR #18560 [v24.1.x] cst/ducktape: Accept errors due to gap in manifest by @abhijat
• PR #18588 [v24.1.x] archival: Disable housekeeping jobs on startup by @ Lazin
• PR #18620 [v24.1.x] tests: fix replaced segments accounting in TopicRecoveryTest by @ztlpn
• PR #18639 [v24.1.x] schema_registry: Make mode_mutability: true by default by @BenPope

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v24.1.2...v24.1.3

Features

• Schema Registry: Support for deleted=true query parameter on POST /subjects/<subject>. by @BenPope in #18432
• #18460 rpk: ability to transfer partition leadership by @daisukebe in #18461

Bug Fixes

• Fix initial_leader_epoch/KIP-320 handling in fetch requests. It was ignored until now which prevented consumers to correctly detect suffix truncation. For Redpanda (and Raft), this is a minor problem since suffix truncation is a very improbable event. by @nvartolomei in #17728
• #17957 Fix incorrect log truncations caused by delayed replication requests. by @ztlpn in #18523
• #18282 #18566 Fix a scenario where list_offset with a timestamp could return a lower offset than partition start after a trim-prefix command. This could lead to consumers being stuck with an out-of-range-offset exception if they began consuming from an offset below the one which was used in the trim-prefix command. by @nvartolomei in #18599
• #18282 #18566 Fix an edge case where a timequery returns no results if it races with tiered storage retention and garbage collection. This is important at least for consumers that fall behind retention. They interpret such response as the partition is empty and jump to the HWM instead of resuming consuming from the first available message. by @nvartolomei in #18599
• #18443 Fixed an assertion triggering in a full-disk scenario by @andijcr in #18444
• #18517 Don't mark partition rebalance complete if some partitions are not moveable (e.g. due to partial recovery mode) by @ztlpn in #18522
• #18569 Enforce client quota throttling in a Kafka-compatible way, meaning we enforce the throttle delay on the next request if the client did not enforce it on its side. by @pgellert in #18575
• concurrent requests of set_log_level + expiration now work as expected by @andijcr in #18438
• fixes possible stall in raft::state_machine_manger by @mmaslankaprv in #18637

Improvements

• Made electing a leader faster by @mmaslankaprv in #18625
• #17951 Schema Registry: Improve retry logic for delete_config and delete_subject_permanent by @BenPope in #18624
• #17951 Schema Registry: Improve tombstoning when deleting a subject by @BenPope in #18624

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.15...v23.3.16

Features

• Re-adds the fetch_read_strategy cluster config property to select between polling and non-polling fetch implementations. Uses the non-polling fetch implementation by default. by @StephanDollberg in #18176
• #18163 rpk container start: now starts a Redpanda Console container connected with the cluster. by @r-vasquez in #18164
• rpk container now has a set of flags to specify ports for node to start on. by @r-vasquez in #18148

Bug Fixes

• Fix a bug validating WebAssembly when global constants are specific values that have the encoded byte 0x0B. by @rockwotj in #18108
• Fix a bug where an invalid buffer passed into the WebAssembly host from the guest could cause Redpanda to abort. by @rockwotj in #18234
• Fix a scenario where list_offset with a timestamp could return a lower offset than partition start after a trim-prefix command. This could lead to consumers being stuck with an out-of-range-offset exception if they began consuming from an offset below the one which was used in the trim-prefix command. by @nvartolomei in #18281
• #18100 Better mapping of REST error codes by @mmaslankaprv in #18102
• #18158 Fix issuing timequeries to cloud storage if remote.read is not enabled. by @WillemKauf in #18159
• #18240 Fixes a crash caused by a race between a client disconnect and a segment reader in tiered storage. by @andrwng in #18241
• #18317 Fixes expiration for transactions that have begun and not produced any data batches. This prevents a stalling LSO. by @bharathv in #18324
• PR #18051 [v24.1.x] Address oversized allocs across kafka API and schema registry by @oleiman
• PR #18125 [v24.1.x] cluster_recovery_backend_test: fix unsafe iteration by @andrwng
• PR #18141 [v24.1.x] Fixes for wait_ms cpu profiler mode by @StephanDollberg
• PR #18216 [v24.1.x] controller_backend: prevent busy-looping when removing partitions by @ztlpn
• PR #18222 [v24.1.x] tx/tm_stm: fix unboundedness of _pid_tx_id by @bharathv
• PR #18328 [v24.1.x] Change information stored in _topic_node_index to avoid oversized alloc by @ballard26
• PR #18406 [v24.1.x] Fix some concurrent memory access problems in partition balancer by @ztlpn

Improvements

• Improve cloud storage cache to prevent readers from being blocked during cache eviction. by @Lazin in #18134
• #18150 rpk container start: You can now select the subnet and gateway to create your 'redpanda' network. by @r-vasquez in #18151
• allow interpreting 'retention_duration' = -1 in a topic_manifest.json file as infinite time retention by @andijcr in #18243
• rpk container now starts the seed broker using the default listener ports. by @r-vasquez in #18148
• PR #18117 [v24.1.x] wasm/parser: better global support by @rockwotj
• PR #18128 [v24.1.x] c/balancer_backend: first initialize planner and then call plan by @mmaslankaprv
• PR #18194 [v24.1.x] configuration to enable delete retention for consumer offsets by @bharathv
• PR #18228 [v24.1.x] CORE-1752: cst: Downgrade error logs to debug by @abhijat
• PR #18269 [v24.1.x] [CORE-2581] cst: move chunk downloads to remote segment bg loop by @abhijat
• PR #18321 [v24.1.x] rpk: stop using args[0] in cloud cluster select by @r-vasquez
• PR #18318 [v24.1.x] offline_log_viewer: fix get_control_record_type by @bharathv

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v24.1.1...v24.1.2

Bug Fixes

• Fix a bug where an invalid buffer passed into the WebAssembly host from the guest could cause Redpanda to abort. by @rockwotj in #18235
• Fixes expiration for transactions that have begun and not produced any data batches. This prevents a stalling LSO. by @bharathv in #18248
• #18237 Fixes a crash caused by a race between a client disconnect and a segment reader in tiered storage. by @andrwng in #18238
• PR #18223 [v23.3.x] tx/tm_stm: fix unboundedness of _pid_tx_id by @bharathv

Improvements

• allow interpreting 'retention_duration' = -1 in a topic_manifest.json file as infinite time retention by @andijcr in #18242

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.14...v23.3.15

Bug Fixes

• fixed a problem leading to UAF error while calculating cloud stage usage by @mmaslankaprv in #17981

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.2.28...v23.2.29

Features

• rpk: rpk cluster partitions list now supports filtering with broker IDs. by @daisukebe in [#18104](https://github.com/redpanda-data/ redpanda/pull/18104)

Bug Fixes

• Fix a bug validating WebAssembly when global constants are specific values that have the encoded byte 0x0B. by @rockwotj in #18109
• #18081 Fixes a crash that could happen when reading from local storage with a large number of segments that all do not contain user data. by @andrwng in #18088 * #18101 Better mapping of REST error codes by @mmaslankaprv in [#18103](https:// github.com/redpanda-data/redpanda/pull/18103)
• #18155 Fix issuing timequeries to cloud storage if remote.read is not enabled. by @WillemKauf in #18156
• PR #18123 [v23.3.x] cluster_recovery_backend_test: fix unsafe iteration by @andrwng

Improvements

• #18133 Improve cloud storage cache to prevent readers from being blocked during cache eviction. by @Lazin in #18138
• PR #18111 [v23.3.x] gh: fix lint-cpp for ubuntu noble by @dotnwat
• PR #18118 [v23.3.x] wasm/parser: better global support by @rockwotj
• PR #18154 [v23.3.x] c/topic_table: do not log duplicated lifecycle marker command by @mmaslankaprv
• PR #18166 [v23.3.x] rpk: bump docker version by @r-vasquez
• PR #18186 [v23.3.x] rpk: bump go deps by @r-vasquez
• PR #18191 [backport][23.3.x] configuration to enable delete retention for consumer offsets #18140 by @bharathv

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.13...v23.3.14

New Features

• Adds new cluster and topic level configurations for write caching feature. by @bharathv in #16924
• PR #17009 write caching - raft implementation by @bharathv
• Enables write caching by default in dev container mode. by @bharathv in #17677
• Add rpk security roles, a new command space to manage your Redpanda roles. by @r-vasquez in #17538
• Introduce --allow-role and --deny-role flags for rpk acl commands by @oleiman in #17416
• Introduces GET /v1/security/users/roles (Admin API) by @oleiman in #17155
• Introduces /v1/security/roles/{role}/members Admin API endpoint for reading and updating RBAC role members. by @oleiman in #17153
• #17679 rpk security acl list now supports --format=json by @rockwotj in #17684
• Data Transforms now support writing to multiple output topics. The REDPANDA_OUTPUT_TOPIC environment variable exposed in transforms is now removed for REDPANDA_OUTPUT_TOPIC_%d for each output topic specified. by @rockwotj in #16946
• rpk transform deploy now supports multiple output topics by @rockwotj in #16950
• The golang transform-sdk gains the ability to write to multiple output topics.
  This feature can only be used in Redpanda v24.1.x or newer. by @rockwotj in #16978
• The rust transform-sdk gains the ability to write to multiple output topics.
  This feature can only be used in Redpanda v24.1.x or newer. by @rockwotj in #17007
• Publish log (i.e. stderr/stdout) output from data transforms exclusively to an internally managed Redpanda topic (_redpanda.transform_logs). Data transform logs will no longer appear in broker logs. by @oleiman in #16485
• Introduce rpk transform logs NAME to view logs for a transform by @rockwotj in #16923
• #16075 Data Transform's Rust SDK now supports a Schema Registry Client. by @rockwotj in #16464
• Topic-aware partition balancing, which attempts to spread partition replicas topic-wise across a cluster. This behavior is controlled by the partition_autobalancing_topic_aware config property (enabled by default). by @ztlpn in #17263
• Tiered Storage now supports using Azure VM user-assigned managed identities for securely accessing
  Azure Blob Storage @andijcr in #17157
• Topic recovery and ‘whole-cluster restore’ from Tiered Storage now perform integrity checks on metadata to ensure that each partition can be recovered successfully by @andijcr in #16915
• You can now create namespaces in Redpanda Cloud using rpk cloud namespace. by @r-vasquez in #16685
• #13175 rpk debug bundle now includes a CPU profile of the requested nodes. by @r- vasquez in #16414
• #16107 You can print a schema now using rpk registry schema get --print-schema. by @r-vasquez in #16109
• #16623 rpk redpanda config bootstrap now supports bootstrapping your advertised addresses configuration. by @r-vasquez in #16652
• new metric vectorized_storage_log_compacted_away_bytes for compaction observability in local storage added by @andijcr in #17579
• new public metric redpanda_cluster_latest_cluster_metadata_manifest_age to track the age of the cluster_metadata_manifest in cloud storage added by @andijcr in #17404

Bug Fixes

• Aggregates partitions in some cloud storage metrics when the aggregate_metrics cluster config is set to true. by @ballard26 in #16336
• Fix a bug that could lead to raft log inconsistencies when 2 out of 3 nodes in a configuration are changed. by @ztlpn in #17675
• Fix a bug that resulted in Redpanda ignoring until the next restart config values that were reset to their defaults. by @ztlpn in #16504
• Fix a bug where logging in a transform could cause the transform to not make progress. by @rockwotj in #17186
• Fix a crash that happened when a cluster that was partially in recovery mode tried to upload consumer offsets to cloud storage. by @ztlpn in #17013
• Fix a memory leak when using transactions with many different producer IDs. by @rockwotj in #15797
• Fix a potential cloud storage cache access time tracker file corruption during shutdown. by @nvartolomei in #16648
• Fix a race condition between suffix truncation / delete records and adjacent segment compaction that can lead to crashes and data-loss. by @ nvartolomei in #17019
• Fix a rare bug where http client connections would vanish from the connection pool leading to various operations hanging while waiting for an http client. by @nvartolomei in #15681
• Fix an issue where rpk transform logs waits for records without the --follow flag specified. by @rockwotj in #17832
• Fix an issue with Cargo.toml when initializing a Rust Data Transform project via rpk transform init by @rockwotj in #15934
• Fix initial_leader_epoch/KIP-320 handling in fetch requests. It was ignored until now which prevented consumers to correctly detect suffix truncation. For Redpanda (and Raft), this is a minor problem since suffix truncation is a very improbable event. by @nvartolomei in #17674
• Fix internal RPC client connection stall after more than 2^32 requests are sent. by @ztlpn in #16156
• Fix large allocation in partition manifest. by @dotnwat in #16160
• Fix oversized allocation in storage. by @Lazin in #16642
• Fix the starter code for Rust projects in rpk transform init by @rockwotj in #16180
• Fix tiered-storage housekeeping problem that may cause replaced segments to pile up if the spillover is enabled. by @Lazin in #16163
• Fixed a few oversized allocations for some admin server endpoints. by @rockwotj in #16551
• Fixed the values for the rpc client in/out bytes metric by @ballard26 in #17933
• Fixes rpk transform init --install-deps so that an explicit true value is not needed. by @rockwotj in #17831
• Fixes a bug in windowed compaction that could cause Redpanda to crash when an error occurs while reading batches. by @andrwng in #16928
• Fixes a bug of config_frontend methods getting called on shards other than the controller shard. by @pgellert in #17088
• Fixes a bug that may prevent redpanda from shutting down cleanly when auditing is enabled by @graphcareful in #16315
• Fixes a concurrency issue in transform offset commits pertaining to taking/applying snapshots. by @bharathv in #17383
• Fixes a crash if a WebAssembly function is deployed that immediately crashes. by @rockwotj in #15939
• Fixes a crash that could happen when reading from local storage with a large number of segments that all do not contain user data. by @andrwng in #18075
• Fixes a plausible correctness issue with idempotent requests during replication failures. by @bharathv in #16706
• Fixes a race between compaction and Raft recovery for compacted topics that could result in aborted transactional data batches being visible. by @andrwng in #16295
• Fixes an an improper initialization of metrics related to controller snapshot uploads. by @andrwng in #16070
• Fixes an issue where using the CPU profiler with running Data Transforms could cause the process to deadlock. by @rockwotj in #17877
• Fixes issue that causes the connection to hang when an unsupported compression type is passed via an incremental_alter_configs request by @graphcareful in #16399
• Fixes lock starvation during transform offset commits. by @bharathv in #17402
• Have fetch handler ensure rack awareness is enabled before performing follower fetching by @michael-redpanda in #15883
• Prevent an assertion from being triggered when Wasm VMs fail immediately. by @rockwotj in #15933
• Prevent detecting leader epoch advancement when state is not up to date by @mmaslankaprv in #16560
• Prevent reactor stalls querying leadership information for large clusters by @rockwotj in #17473
• Protect against a very rare scenario where after node restart, some of the partition replicas hosted on that node could not take part in leader elections. by @ztlpn in #16068
• Redpanda used to accept an empty string in redpanda.rack in node config. This would cause issues in Kafka operations. Redpanda will now error on startup if redpanda.rack is set to an empty string. by @michael-redpanda in #15835
• Redpanda will now correctly handle an empty rack ID provided in a fetch request by @michael-redpanda in #15846
• Reduces maximum log line size from 1MiB to 128KiB to reduce occurrences of memory allocation failures by @michael-redpanda in #17922
• Report runtime public metrics by task queue for all cores, not just core 0 by @rockwotj in #16154
• Return a HTTP 400 error code when deploying a transform to a topic that doesn't exist instead of a 500 by @rockwotj in #17011
• Schema Registry: Deleted schemas no longer reappear after certain compaction patterns on the _schemas topic. by @BenPope in #17091
• #15042 Fixes a bug in the tiered storage time-based query implementation that could result in a consumer hang when consuming very old data. by @andrwng in #16645
• #15201 Fix assertion triggered by interleaving of log flush and log truncation followed by append by @Lazin in #16105
• #15603 cluster config aliases are accepted while reading from yaml by @andijcr in #15605
• #15674 Fix an issue where new configs would continually revert to legacy defaults after an upgrade. by @oleiman in #15761
• #15722 #7946 Fix an issue where create topics responses would show incorrect partition count and replication factor by @oleiman in #16410* #15811 Several additional metrics will have their "partition" label aggregated away (i.e., into a single series per remaining label set with no partition label, whose value is the sum of all input series with the same label set and different partition labels). This is already the default behavior for most metrics, but this change extends it to almost all remaining metrics. by @travisdowns in #15966
• #15839 safer handle unknown properties in local state by @andijcr in #15840
• #15909 Prevent oversized allocs when group fetching from many partitions. by @ rockwotj in #15918
• #16129 Fixes a bug in SASL user deletion and update where usernames with a + symbol in the username were prevented from being deleted by @pgellert in #16694
• #16251 rpk: fixed a bug where the --password flag could not be used along with the new configuration flag -X pass in clusters where basic authentication was enabled. by @r-vasquez in #16278* #16259 Fixes a bug that would previously cause read replicas to report the wrong value for the redpand_kafka_max_offset metric. by @andrwng in #16263
• #16320 Prevent oversized allocation with large amounts of controller metadata by @ rockwotj in #16381
• #16322 Fix graceful shutdown of the TS archive area retention procedure. by @Lazin in #16382
• #16402 Fix invalidate iterator access when yielding control to the scheduler in data transforms debug endpoint. by @rockwotj in #16421
• #16479 Fix timequery error that triggered full partition scan by @Lazin in #16503
• #16521 Avoid a large contiguous allocation when creating thousands of topics in a single CreateTopics request. by @travisdowns in #16529
• #16552 #16064 rpk tune --output-script: Add a missing new line in the ballast file tuner when using the --output-script flag by @r-vasquez in #16576
• #16578 Fixes a bug in CreateTopicsResponse to now return all the configs of the topic, not just the topic-specific override configs. by @pgellert in #16922
• #16612 fixes small inconsistency between Kafka and Redpanda when trying to query end_offset of an empty log by @mmaslankaprv in #17789
• #16643 Fixed deleting Data Transforms with names that had URL unsafe characters by @rockwotj in #16858
• #16840 Fixes a bug with TLS metrics where expiration timestamps would not advance on certificate reload by @oleiman in #17233
• #17086 fixed enabling cloud storage in existing clusters by @mmaslankaprv in #17112
• #17731 Fix incorrect log truncations caused by delayed replication requests. by @ ztlpn in #17895
• #17788 Fix problem in Tiered-Storage that could potentially cause consumers to get stuck by @Lazin in #17805
• #17827 fix a race between eviction and producer registration that results in an invalid transaction state. by @bharathv in #17880
• #2225 Fix reported config source for cleanup.policy by reporting DEFAULT_CONFIG instead of DYNAMIC_TOPIC_CONFIG for the default value. by @pgellert in #17456
• ext4 is no longer incorrectly detected as ext2 (all of ext2, 3 and 4 are assumed to be ext4). by @travisdowns in #13496
• fixed a problem leading to UAF error while calculating cloud stage usage by @mmaslankaprv in #17975
• fixed incorrect fetch offset validation by @mmaslankaprv in #16146
• fixes a bug where compacting away last aborted data batch in a segment may cause readers to be stuck because of a gap in the segment ranges thus blocking raft recovery by @andrwng in #16295
• prevents partial consumer group recovery by @mmaslankaprv in #17673
• rpk: prevent a segfault when creating a profile from a cloud that is not in ready state. by @r-vasquez in #17553

Improvements

• Add new upload management mechanism by @Lazin in #16684
• Add an internal debugging endpoint to see the committed progress of data transforms. by @rockwotj in #16185
• Add Prometheus metrics for data transforms logging by @oleiman in
• Introduce "trust_file_crc32c" metric to export a checksum for each trust file in the system. by @oleiman in #17539
  #16566
• Marks the fetch_read_strategy cluster configuration property as deprecated by @ballard26 in #17191
• Record result of running microbenchmarks in a file using JSON format by @mfleming in #15912
• #14069 Adds wait_ms parameter to CPU profiler admin API. The API will wait for wait_ms milliseconds then return the profile samples collected during that period of time. by @ballard26 in #14468
• spillover manifests are enabled by default for clusters that did not explicit set a value or null by @andijcr in #16172
• new metric vectorized_ntp_archiver_compacted_replaced_bytes by @andijcr in #17627
• Removes plan_and_execute_latency_us histograms from internal metrics by @ballard26 in #17191* A clearer error message when the data directory does not exist or is the wrong type by @travisdowns in #15821
• Add a dedicated CPU scheduling policy for Data Transforms by @rockwotj in #16114
• Add error messages when an unsupported Schema Registry client is used in Data Transforms by @rockwotj in #17591
• Adds UpdateRoleMembership function to rpk admin API client. by @bojand in #17589
• Adds a new cluster configuration property fetch_read_strategy. This property determines which fetch execution strategy Redpanda will use to fulfill a fetch request. The newly introduced non_polling execution strategy is the default for this property with the polling strategy being included to make backporting possible. by @ballard26 in #15328
• Adds a new public metric redpanda_raft_recovery_partition_movement_consumed_bandwidth that tracks how much bandwidth is currently in use for raft recovery. This helps tune raft_learner_recovery_rate. by @bharathv in #16842
• Adds metric for number of quorum ack requests with write caching. by @bharathv in #17179
• Adds new cluster configuration parameter minimum_topic_replications that permits Redpanda administrators to set a minimum permitted replication factor for newly created topics. by @michael-redpanda in #15777
• Adds observability into producer evictions in each shard. by @bharathv in #16724
• Caches the connections local address preventing the need to make a system calls to grab this value when auditing events. by @graphcareful in #15931
• Changes what the kafka_latency_fetch_latency metric measures to be the time the first fetch_ntps_in_parallel takes. by @ballard26 in #17720
• Data Transforms written in Golang now use a non-buffered write mechanism by @rockwotj in #15922
• Data Transforms written in Rust now use a non-buffered write mechanism by @rockwotj in #15923
• Fix large wasm module deployments by @rockwotj in #16722
• Handle missing data transform logs topic in rpk transform logs by @rockwotj in #17830
• Improve concurrency control and metadata consistency in Tiered-Storage. by @Lazin in #16774
• Improve log segment upload mechanism which avoid reading files from disk directly. by @Lazin in #16999
• Improved handling of follower fetching offset validation when used with relaxed consistency by @mmaslankaprv in #16475
• Improves observability by allowing Redpanda to detect that some internal processes are stuck. by @Lazin in #16466
• Increase data_transforms_logging_buffer_capacity_bytes from 100KiB to 500KiB by @oleiman in #16972
• Introduces a new non-polling fetch execution strategy that decreases CPU utilization of fetch requests and fetch request latency. by @ballard26 in #15328
• Large allocations are now logged by default (similar to reactor stalls) by @StephanDollberg in #16828
• Making the Redpanda README.md more beautiful and functional! by @WillemKauf in #17822
• Prevent metadata oversized allocations when requesting many topics. by @rockwotj in #15751
• Publish total reclaimable space to avoid stuck decommission scenario. by @dotnwat in #16354
• Reduces the number of allocations performed by the auditing subsystem by @graphcareful in #16056
• SIMD instructions are generated by default for WebAssembly binaries when building with rpk. by @rockwotj in #16296
• Schema Registry: Improve retry logic for delete_config and delete_subject_permanent by @BenPope in #17905
• Schema Registry: Improve tombstoning when deleting a subject by @BenPope in #17905
• Support changing the timeout for WebAssembly functions by @rockwotj in #15967
• Support dynamically changing the limit for WebAssembly binary size by @rockwotj in #15967
• The full_raft_configuration_recovery_pattern cluster config property is now deprecated and will be ignored by Redpanda. by @andrwng in #16890
• This PR partially reverts the change such that strict retention remains enabled after upgrade unless it had been explicitly disabled before the upgrade. by @dotnwat in #16077
• Transform SDKs for Schema Registry now publish their ABI version for compatibility checks with the broker. by @rockwotj in #17593* Transforms now start at a record relative to the deploy time instead of the time at which the VM starts.
  This allows for easier testing of transforms, as one does not have to wait for the VM to boot before producing. by @rockwotj in #17590
• Validate transform code at deploy time to ensure the correct SDK is used. by @rockwotj in #16290
• #13023 rpk container now prints the container logs in case of failures during startup. by @r-vasquez in #17780
• #14116 SR/PP will now reply with a 500 error if an internal service semaphore has been completely exhausted by @michael-redpanda in #15977
• #14814 Added new metric to provide Follower Fetching feature observability by @ mmaslankaprv in #15799
• #15818 rpk cluster health: now --exit-when-healthy enables --watch when provided. by @r-vasquez in #16106
• #15900 Internal kafka client now uses asynchronous compression (when possible) to reduce possibility of oversized allocations and reactor stalls by @michael-redpanda in #15920
• #16162 get_cluster_uuid returns a correctly formatted string by @andijcr in #16183
• #16552 #16064 rpk tune --output-script: rpk now creates a file for you if the provided file does not exist. by @r-vasquez in #16576
• #16758 cluster: Avoid oversize allocs for topic creation and configuration by @BenPope in #16982
• #16771 Data Transform builds in rpk now uses tinygo v0.31.1 by @rockwotj in #16856
• #16795 Added ability to change transactional manage topic properties by @mmaslankaprv in #16797
• #16901 Added EHOSTUNREACH to retry-able error code list by @michael-redpanda in #16902
• #17197 more accurate node status reporting by @mmaslankaprv in #17625
• #17368 Improves error feedback when Redpanda is given an invalid number of partitions during either topic creation or when the partition count for a topic is increased. by @michael-redpanda in #17369
• #17925 Allocate and rebalance partition replicas in random order to prevent an undesirable pattern when many partitions have the same replica set. by @ztlpn in #17962
• #8809 Node-wide throughput throttling is now fair an responsive. by @BenPope in #16441
• [rpk] more informative error message display on create topic failure by @michael-redpanda in #15837
• rpk acl user has been moved to rpk security security. Old command is soft deprecated. by @r-vasquez in #17664
• rpk acl has been moved to rpk security acl. Old command is soft deprecated. by @r-vasquez in #17664
• rpk profile has been reworked in an attempt to be simpler; see PR #17038 for more detail by @twmb in #17038
• rpk redpanda start know knows about io characteristics on EC2 instance types i4i, is4gen and im4gn in addition to i3en and i3, allowing more effective IO scheduling even if rpk iotune has not been run. by @travisdowns in #17220
• rpk transform deploy --file now supports https:// URLs by @rockwotj in #16050
• rpk transform deploy takes a --file flag to deploy a compiled WebAssembly binary. by @rockwotj in #15932
• verbose_logging_timeout_max node config, to prevent setting TRACE and DEBUG log level indefinitely by @oleiman in #15798
• better ERROR|WARN messages for candidate_creation_error by @andijcr in #17090
• better control of memory usage in storage layer. by @mmaslankaprv in #16846
• ensure only positive values are accepted for log_segment_ms_min/max by @andijcr in #16513
• largely reduced number of health report copies by @mmaslankaprv in #17863
• less overhead of health report collection by @mmaslankaprv in #17158
• minimising possibility of data loss when using relaxed conistency replication by @mmaslankaprv in #17915
• new metric redpanda_cloud_storage_cloud_log_size reports size in bytes of the log in cloud storage by @andijcr in #17445
• optimized updating leadership metadata with health reports by @mmaslankaprv in #16512
• preventing large allocation in partition balancer code by @mmaslankaprv in #16917
• rpk: Remove 10s timeout in rpk profile create by @r-vasquez in #16836
• rpk: change hash for backup filenames to use sha256 by @andrewhsu in #17568
• rpk: tune script uses sha256sum instead of md5sum by @andrewhsu in #17595
• rpk: use go module version of common proto definitions. by @bojand in #17628
• skipping overhead of collecting node health report for each node separately. by @mmaslankaprv in #17715
• smaller memory footprint when using with large number of topics with small partition count by @mmaslankaprv in #16247

None

No release notes explicitly specified.

• PR #13532 Add small batches nightly regression test by @ballard26
• PR #13927 archival: Add offset_range_size method to the storage::log by @Lazin
• PR #14599 archival: Start housekeeping jobs after STM sync by @Lazin
• PR #14745 storage: assert when appending empty batch by @andrwng
• PR #15199 rptest: ignore test_cloud_cache_thrash by @nvartolomei
• PR #15271 Fix CI failure in test_concurrent_append_flush by @travisdowns
• PR #15287 rptest/scale_test: enable test_many_partitions by @nvartolomei
• PR #15407 c/topic_table: do not log duplicated lifecycle marker command by @ mmaslankaprv
• PR #15424 Manage STM creation outside of cluster::partition by @mmaslankaprv
• PR #15594 removes kics by @wreet
• PR #15616 Add at: in Top-N alloc site output by @travisdowns
• PR #15669 ci: Add bucket cleanup scripts by @savex
• PR #15672 tests: repro for zombie txs after compaction by @andrwng
• PR #15677 c/archival_stm: do not reset _last_replicate on timeout by @nvartolomei
• PR #15678 c/topics_dispatcher: do not guesstimate leader ids by @mmaslankaprv
• PR #15697 cloud_storage: hold gate in hydration by @andrwng
• PR #15701 rptest: add test_add_and_decommission() by @andrewhsu
• PR #15718 kafka: fix holding of attached_partition by @andrwng
• PR #15724 archival: remove outdated codedoc by @nvartolomei
• PR #15725 cluster/config_manager: use property's main name in store_delta by @andijcr
• PR #15726 transform-sdk/rust: fix publishing by @rockwotj
• PR #15730 rptest: fix an always true assertion by @nvartolomei
• PR #15731 c/log_eviction_stm: do not take the snapshot if start offset progressed by @mmaslankaprv
• PR #15735 security: update supported version by @rockwotj
• PR #15736 transform-sdk/go: fix releases by @rockwotj
• PR #15737 Fix bad argument passed to kgo-repeater constructor by @graphcareful
• PR #15738 c/members_member: don't ignore update_broker_client future by @rockwotj
• PR #15739 Increase audit buffer sizes for audit scale test by @graphcareful
• PR #15740 cluster_recovery: direct requests to controller leader by @andrwng
• PR #15743 cloud_storage_clients: use appropriate beast error for end of stream by @ nvartolomei
• PR #15748 cloud_metadata: upload consumer groups in batches by @andrwng
• PR #15754 rptest: clean up cluster metadata in CDT by @andrwng
• PR #15755 rptest: log error on failure to delete bucket by @andrwng
• PR #15765 follow-up: config/config_store_test: ignored_keys test by @andijcr
• PR #15778 Fix 128K iobuf zero-copy by @travisdowns
• PR #15779 e2e: Basic producer and consumer using Table API by @savex
• PR #15804 c/archival_stm: downgrade timeout log to warn by @nvartolomei
• PR #15809 rptest: update cloud test suite to subset by @andrewhsu
• PR #15813 archival: disable scrubber by default by @nvartolomei
• PR #15814 rptest: increase grace period for finalize by @nvartolomei
• PR #15815 cluster/config_manager: ensure cleanup of old aliases by @andijcr
• PR #15817 audit: Renamed audit log topic by @michael-redpanda
• PR #15820 Skip assertion in audit log tests if results beat the baseline by @ graphcareful
• PR #15822 test: remove source of false positive failure by @dotnwat
• PR #15836 gha: remove invalid permission by @rockwotj
• PR #15844 Disable cloud metadata upload and cluster restore in recovery mode by @ztlpn
• PR #15852 t/kgo: upgrade kgo to do a full run after /last_pass by @nvartolomei
• PR #15853 rptest: allow the new version of xfs/ext4 fs error msg by @nvartolomei
• PR #15862 c/s/leader_balancer: prevent oversized alloc by @rockwotj
• PR #15870 gh/workflow: add build message in promote trigger by @gousteris
• PR #15882 gha: s/git.ref_name/github.ref_name by @rockwotj
• PR #15889 utils: Move cstore column and frame implementation to utils by @Lazin
• PR #15890 gha: improve backport reviewers by @rockwotj
• PR #15903 fix grammar in README by @bachmanity1
• PR #15910 cmake: Fix minor typos in doc string by @mfleming
• PR #15919 r/offset_translator: remove unsafe bootstrap code by @ztlpn
• PR #15924 rm_stm: remove dead code by @bharathv
• PR #15935 gha: create stale issue workflow by @rockwotj
• PR #15942 gha: mark ci-failures as stale by @rockwotj
• PR #15945 c/log_eviction_stm: do not request snapshot if already progressed by @ mmaslankaprv
• PR #15948 cloud_storage: Improve scrubber by @Lazin
• PR #15951 dt-deps: Bump openjdk and Keycloak by @oleiman
• PR #15953 tx_migration: avoid ping pong of requests between brokers by @bharathv
• PR #15955 rptest: Basic transaction produce test by @savex
• PR #15957 topic_recovery_test: adjust wait condition for finished uploads by @andrwng
• PR #15962 dt: Fixed flaky schemas test by @michael-redpanda
• PR #15963 tests/si_utils: error message for quiesce_uploads timeouts by @andijcr
• PR #15968 build: introduce foundational base library by @dotnwat
• PR #15971 rpc: pass requests by value by @rockwotj
• PR #15981 k/metadata: guesstimate leader when information is not yet present by @ mmaslankaprv
• PR #15982 c/partition_balancer: use full partition move when disk is full by @ mmaslankaprv
• PR #15983 utils: allow memory data source to take tmpbuf by @dotnwat
• PR #15986 securit/:OIDC: Enable licence check and telemetry by @BenPope
• PR #15990 tests/si_utils: assert message for verify_file_layout() by @andijcr
• PR #15993 rptest: more typing 1 by @travisdowns
• PR #15997 build: expose submodule public headers by @dotnwat
• PR #16000 tests: Use multiple topics in OMBValidationTest.test_max_partitions by @ StephanDollberg
• PR #16025 tests: delete run script by @rockwotj
• PR #16028 rptest: use 10 nodes in OMB validation tests by @travisdowns
• PR #16029 features: Update to version 12 by @StephanDollberg
• PR #16049 rptest: clean cloudv2 cluster topics in tearDown() by @andrewhsu
• PR #16053 build: make random a foundational library by @dotnwat
• PR #16055 Controller backend: reconcile each ntp in individual fibers by @ztlpn
• PR #16067 Introduced partition shutdown watchdog timer by @mmaslankaprv
• PR #16071 archival: avoid division by 0 when computing slow down rate by @nvartolomei
• PR #16076 gha: use latest action for stale by @andrewhsu
• PR #16078 Fix for CI Failure (Failed to make list_offsets request, unknown broker) in DeleteRecordsTest.test_delete_records_segment_deletion by @graphcareful
• PR #16083 Add oversized allocation to bad log lines by @ballard26
• PR #16086 tests: Make the OMB validator use result.json by @StephanDollberg
• PR #16095 transform-sdk: decouple releases from broker by @rockwotj
• PR #16097 rptest: Additional logging audit_log_test infra by @graphcareful
• PR #16108 Rename max_client_count to max_connection_count by @travisdowns
• PR #16119 rptest: Implement flink service memory autosizing by @savex
• PR #16121 hashing: remove unused hashing code by @dotnwat
• PR #16123 build: move bytes headers into public include path by @dotnwat
• PR #16125 rptest/scale_test: Remove p50 check in audit test by @graphcareful
• PR #16126 rptest: bump multiplier to 2.0 for test_max_partitions() by @andrewhsu
• PR #16127 rptest: Wait for the group to be in a stable state before starting tests by @graphcareful
• PR #16132 rptest: bad field use in test_consume_miss_cache by @travisdowns
• PR #16133 rptest: invalid comparison in test_max_connections by @travisdowns
• PR #16134 rptest: more typing 2 by @travisdowns
• PR #16135 ci: Fix flink install on ARM64 by @savex
• PR #16136 OMBSampleConfigurations predicates are static by @travisdowns
• PR #16137 tests: update email in setup.py by @andrewhsu
• PR #16140 gha: run build-redpanda workflow once per day by @ivotron
• PR #16144 build: add container and strings library by @dotnwat
• PR #16148 rptest: Transaction workload with scaling in mind by @savex
• PR #16152 build: move frag_vec to container library by @dotnwat
• PR #16153 Introduce wasm::logger by @oleiman
• PR #16157 rptest: reduce HighThroughputTests.msg_size by @andrewhsu
• PR #16158 Change metrics to match generate_charts.py by @ballard26
• PR #16159 Stabilize OMB regression tests by @ballard26
• PR #16181 rptest: log kubectl errors by @travisdowns
• PR #16184 rptest: expose health check reason by @travisdowns
• PR #16189 archival: apply PR feedback by @dotnwat
• PR #16190 Transform logging data model by @oleiman
• PR #16210 transform-sdk/rust: borrow output record by @rockwotj
• PR #16211 cloud_storage: Fix unit-test timeout on ARM CI by @Lazin
• PR #16212 tests/tiered_storage_model_test: temp ok_to_fail by @andijcr
• PR #16215 io: add fault injection helpers to persistence layer by @dotnwat
• PR #16226 tests/rptest/tests/tiered_storage_model_test: ignore by @andijcr
• PR #16229 transform-sdk: experimental C++ support by @rockwotj
• PR #16232 Change cloud test hierarchy, introduce cloud test base by @travisdowns
• PR #16233 admin api: skip partition info in /brokers end point by @bharathv
• PR #16234 ducktape: fix flink URL 404 by @bharathv
• PR #16242 tests: updated flink to 1.18.1 by @mmaslankaprv
• PR #16243 Node UUID admin by @mmaslankaprv
• PR #16244 Make the request context fixture code work with produce requests by @ mfleming
• PR #16248 "enable by default spillover manifest" testing followups by @andijcr
• PR #16249 ci: Mark test ok_to_fail untill fix released by @savex
• PR #16250 rptest: Refactor consumer validation and tune table idle settings by @savex
• PR #16253 wasm: transform callback by @rockwotj
• PR #16257 container: introduce chunked vector by @rockwotj
• PR #16258 gha: don't mark backports as drafts by @rockwotj
• PR #16260 lint: fix linter error by @rockwotj
• PR #16277 rptest: simplify metric_sum filtering by @andrwng
• PR #16283 dt/deps: Use setuptools for confluent-kafka source build by @oleiman
• PR #16284 cloud_storage_compaction_test: a couple test fixes by @andrwng
• PR #16285 rm_stm: remove mem_state::last_end_tx by @bharathv
• PR #16286 rm_stm: remove log_state::prepared by @bharathv
• PR #16287 Fixed large allocation in kafka::wait_for_leaders by @mmaslankaprv
• PR #16293 labeler: apply wasm label for Redpanda subsystems by @rockwotj
• PR #16301 Introduce transform::logging::manager by @oleiman
• PR #16302 demote ERROR message to DEBUG for timequerys at the edge of spillover retention by @andijcr
• PR #16303 tests: replaced no_timeout with default timeout in raft test by @ mmaslankaprv
• PR #16304 Wrapped logging with vlog macro in places that missed it by @mmaslankaprv
• PR #16311 resource_mgmt: disable flaky test by @rockwotj
• PR #16314 transform-sdk/rust: update rustdocs by @rockwotj
• PR #16319 rptest: add ability to restart pods in RedpandaServiceCloud by @andrewhsu
• PR #16337 rptest: fix cluster healthy check by @andrewhsu
• PR #16347 rptest: Bump audit logging scale test run time by @graphcareful
• PR #16353 CDT: fix rpk tune list test in GCP. by @r-vasquez
• PR #16359 rptest: clean teleport data dir before use by @andrewhsu
• PR #16361 rptest: Fix race condition in CloudRetentionTest by @Lazin
• PR #16375 rptest: Create multi-node test with 1k topics and >10M events by @savex
• PR #16383 rptest: Add error message to log allow list by @Lazin
• PR #16384 rptest: Fix throttling e2e test by @Lazin
• PR #16385 cloud_storage_clients/client_pool: handle broken _self_config_barrier by @ andijcr
• PR #16386 tests: Add produce microbenchmark by @mfleming
• PR #16400 coding-style: propose documentation recommendations by @rockwotj
• PR #16427 slash-cmd: fix /backport command by @gousteris
• PR #16428 rptest: Fix s3.copy_object when running on GCP by @savex
• PR #16429 metrics-mock: initial commit by @travisdowns
• PR #16437 io: misc updates and test helpers by @dotnwat
• PR #16440 transform: data path for multiple output topics by @rockwotj
• PR #16443 config/config_store_test: unit test for property::operator= + watchers by @ andijcr
• PR #16454 io: add io queue manager by @dotnwat
• PR #16472 rptest: tolerate missing topic metrics after start by @nvartolomei
• PR #16478 transform/logging/test: deflake TransformLogManagerTest.LargeBuffer by @ rockwotj
• PR #16489 rptest: exclude OMBValidationTest.test_retention by @andrewhsu
• PR #16495 tests: disable snapshots when testing partial controller deletion by @ mmaslankaprv
• PR #16496 rptest: Ignore error logs generated by anomaly injection by @abhijat
• PR #16497 rptest: enable test_add_and_decommission by @andrewhsu
• PR #16499 rptest: Fix instance type getter for GCP by @savex
• PR #16502 container: chunked_vector improvements by @rockwotj
• PR #16505 rptest: use okta auth for teleport by @andrewhsu
• PR #16506 Fix missing comma in kubectl.py by @StephanDollberg
• PR #16525 rpc: Add config flag to enable/disable compression for replies by @ StephanDollberg
• PR #16526 transform: gc committed offsets by @rockwotj
• PR #16527 Increase default value of rpc_client_connections_per_peer to 32 by @ ballard26
• PR #16538 Introduced cluster::vcluster_id type underlied by XID by @mmaslankaprv
• PR #16539 c/namespaced_cache: added implementation of namespaced cache by @ mmaslankaprv
• PR #16542 rptest: force cleaning of teleport data dir by @andrewhsu
• PR #16548 rpc: Disable compression for internal rpc replies by @StephanDollberg
• PR #16554 Rethrow on unknown exceptions in fetch handler by @ballard26
• PR #16571 xform/service: Remove exception handling from _log_manager->start by @ oleiman
• PR #16577 Set KAFKA_OPTS to empty in OMB by @travisdowns
• PR #16580 cloud_storage: update upload_object api by @abhijat
• PR #16611 cloud_storage/inventory: Add inventory create-config API for AWS by @abhijat
• PR #16614 cloud_storage: Add download_object method for downloading small objects by @abhijat
• PR #16617 Flex partition assignment stage 1 by @ztlpn
• PR #16633 io: add scheduler by @dotnwat
• PR #16636 cst/inv: Create helpers to conditionally create inventory config by @abhijat
• PR #16637 Added timeout to gtest_raft_rpunit by @mmaslankaprv
• PR #16646 rptest: remove config_file credential source support by @nvartolomei
• PR #16653 Increase allowed p75 e2e latency for ts_write OMB test by @ballard26
• PR #16656 serde: Add node_health_report ser/de benchmark by @StephanDollberg
• PR #16658 Virtual connections in Kafka server by @mmaslankaprv
• PR #16662 Implement async_for_each by @travisdowns
• PR #16664 vint_bench by @travisdowns
• PR #16667 ci: Bump client-swarm to latest commit by @savex
• PR #16668 PESDLC-109 rptest: add metrics gathering methods to RedpandaServiceCloud by @andrewhsu
• PR #16669 test: retry on readyReplicas test by @dotnwat
• PR #16671 test: check for non-empty samples from any core by @dotnwat
• PR #16674 Revert "rptest: remove config_file credential source support" by @ nvartolomei
• PR #16675 r/tests: removed infinite loop from raft test by @mmaslankaprv
• PR #16676 Add forward iterator to async_for_each by @travisdowns
• PR #16680 test: wait for cluster to become healthy after setup by @dotnwat
• PR #16693 rpk: update help text of decommission-status by @daisukebe
• PR #16704 rptest/scale_tests: compute expected time for non-ts workload by @ nvartolomei
• PR #16705 http/client: add request interface that takes an iobuf by @dotnwat
• PR #16710 tests: Fix missing include in redpanda_cloud_test by @StephanDollberg
• PR #16711 c/p_leaders: check if leaders table was modified after async iteration by @ mmaslankaprv
• PR #16714 Added redpanda.virtual.cluster.id topic property by @mmaslankaprv
• PR #16715 cloud-tests: use a time-based calculation of load by @travisdowns
• PR #16718 http/client: add post() method helper by @dotnwat
• PR #16720 io: add input stream equality test helper by @dotnwat
• PR #16721 storage: move ghost batch generation into log reader by @andrwng
• PR #16745 u/named_type: use fmt to format named type by @mmaslankaprv
• PR #16750 PESDLC-901 Tests to create 10k+ topics and do checks with low throughput by @savex
• PR #16751 http: remove demo by @dotnwat
• PR #16763 Introduce RBAC data model by @oleiman
• PR #16764 rptest: fix hwm computation in topic_recovery_test by @andrwng
• PR #16765 rptest: better health check by @travisdowns
• PR #16768 Introduce principal_type::role by @oleiman
• PR #16775 tests: refactor consumer group balancing test by @mmaslankaprv
• PR #16776 c/leaders: trigger leadership notification when term changes by @ mmaslankaprv
• PR #16780 Cloud Storage: Ignore abort_requested_exception on shutdown by @jcipar
• PR #16781 RBAC Admin API Swagger and handler stubs by @oleiman
• PR #16796 Add support for client_swarm's metrics api to produce_swarm by @ballard26
• PR #16798 c/topic_table_probe: use btree_map in topic table probe by @mmaslankaprv
• PR #16799 tests: dropped old upgrade tests which validated v22.2 version by @ mmaslankaprv
• PR #16817 PESDLC-890 Use timeout when creating topics by @savex
• PR #16824 PESDLC-895 Split produce/consume methods by @savex
• PR #16825 Crypto lib by @michael-redpanda
• PR #16826 rpk: add debug logs to BYOC by @r-vasquez
• PR #16835 rpk: replace web UI url for cloud clusters by @r-vasquez
• PR #16841 rpk: bump deps and toolchain to 1.22 by @r-vasquez
• PR #16843 PESDLC-919 Constant data generation workload by @savex
• PR #16845 Remove v23.1 from the PR template. by @piyushredpanda
• PR #16847 Raft vote process unification by @mmaslankaprv
• PR #16850 Fixed background apply fiber race condition in raft:: state_machine_manager by @mmaslankaprv
• PR #16853 cloud_storage: Improve stale_reader test by @Lazin
• PR #16857 cmake: upgrade tinygo compiler by @rockwotj
• PR #16883 docs: update version in README by @rockwotj
• PR #16886 cloud_storage: various non-functional changes by @andrwng
• PR #16892 storage: adopt offset_translator by @andrwng
• PR #16900 r/tests: added context to visible offset validating test assertion by @ mmaslankaprv
• PR #16905 crypto: Updated jwt to use crypto lib by @michael-redpanda
• PR #16906 security/license: Use crypto library by @michael-redpanda
• PR #16907 Revert "rm_stm/idempotency: fix the producer lock scope" by @bharathv
• PR #16909 Add support for cryptolib RNG by @michael-redpanda
• PR #16910 Use cryptolib instead of gnutls for HMAC and secure digests by @michael- redpanda
• PR #16911 tests: add test for GTEST_SKIP_CORO by @rockwotj
• PR #16919 c/topic_table: replaced partition metadata map with chunked_vector by @ mmaslankaprv
• PR #16920 k/group_manager: used chunked_vector when cleaning groups by @mmaslankaprv
• PR #16927 Add OpenSSL Library Context Service by @michael-redpanda
• PR #16929 storage: add missing flush in a test by @nvartolomei
• PR #16931 fix for cluster_config_test.py::test_aliasing by @andijcr
• PR #16933 gha: allow manually dispatching the open source build by @rockwotj
• PR #16936 crypto/cmake: Fixed exact versioning by @michael-redpanda
• PR #16943 rptest: Fix cloud_retention_test by @Lazin
• PR #16953 wasm: add out_handle by @rockwotj
• PR #16954 storage: support dirty entries in batch cache by @nvartolomei
• PR #16958 Ensure fragment_vector fragments are always <= 128KiB by @ballard26
• PR #16962 cst/inv: locate latest report by @abhijat
• PR #16969 tests: Tweak OMBValidationTest.test_max_connections by @StephanDollberg
• PR #16970 Prepare leader_balancer for node-local core assignment by @ztlpn
• PR #16971 dependencies: Added missing OpenSSL dependencies by @michael-redpanda
• PR #16973 rptest: improve data transform tests by @rockwotj
• PR #16975 Name all util/mutexes by @travisdowns
• PR #16976 increased a timeout for cluster replicas by @rpdevmp
• PR #16980 Update client-swarm dependency by @ballard26
• PR #16986 storage: space management o11y by @dotnwat
• PR #16987 crypto: Move special members after impl definition by @BenPope
• PR #16988 cloud: Address some warnings by @BenPope
• PR #16989 raft|cluster: Address some warnings by @BenPope
• PR #16990 treewide: Address some warnings by @BenPope
• PR #16991 dt: Fix flaky TLS metrics test by @oleiman
• PR #16992 Shard producer state manager with v_cluster_id by @mmaslankaprv
• PR #16994 transform-sdks: introduce integration tests by @rockwotj
• PR #17000 ducktape: temp fix for erlang 26.2.3 arm by @andrewhsu
• PR #17002 treewide: Reduce header dependencies by @BenPope
• PR #17003 CORE-6: Pre-fetching algorithms by @michael-redpanda
• PR #17010 Check cloud type and do not run if FMC by @rpdevmp
• PR #17021 PESDLC-953 ConsumerSwarm fixes by @savex
• PR #17024 ducktape: lint ocsf-server by @andrewhsu
• PR #17025 Role based authorization by @oleiman
• PR #17026 CORE-2459 Fix UBSAN diagnostics by @nvartolomei
• PR #17027 tools/cmake_test: remove accidental env override by @nvartolomei
• PR #17028 crypto: Cleanup includes by @BenPope
• PR #17029 Add /microbench slash command for running microbenchmarks by @mfleming
• PR #17032 storage: assert no inflight writes when closing segment by @nvartolomei
• PR #17037 Commented out this log info since it is AWS specific and affects GCP by @ rpdevmp
• PR #17039 storage: ensure monotonic stable offset updates by @nvartolomei
• PR #17084 io: add pager for append-only file abstraction by @dotnwat
• PR #17085 rptest: fix _lifecycle_test naming by @travisdowns
• PR #17087 container: add interval_set by @andrwng
• PR #17089 admin: implement create role API by @pgellert
• PR #17092 Using contiguous_range_map in partition_leaders_table by @mmaslankaprv
• PR #17101 cloud_storage_clients: classify request_timeout as retriable by @nvartolomei
• PR #17102 PESDLC-978 Add status check when conducting a health assertion of the cluster by @savex
• PR #17103 added topic parameter by @rpdevmp
• PR #17109 rpk: bump deps and toolchain to 1.22.1 by @r-vasquez
• PR #17110 test: Don't run MTT in CDT by @StephanDollberg
• PR #17117 rpk profile: a few more fixes by @twmb
• PR #17118 PESDLC-983 Trim errors and add Broker exception by @savex
• PR #17119 CORE-20: Use a venv to install and use ducktape by @michael-redpanda
• PR #17145 ducktape: deflake test_topic_aware_rebalance in debug mode by @bharathv
• PR #17152 acl_store: Return prefix matches as a view by @oleiman
• PR #17156 compression: Allocate memory for LZ4_compressEnd by @abhijat
• PR #17161 c/types: fix typo in operator<< for topic_properties by @nvartolomei
• PR #17163 rbac: Add Telemetry and Licence nag by @BenPope
• PR #17164 test: Add n2d-standard-32 machinetype by @StephanDollberg
• PR #17165 tests: Update OMB version on CDT by @StephanDollberg
• PR #17166 admin: implement RBAC List/Get/Rename/Delete APIs by @pgellert
• PR #17169 Fixed typo in comment in ntp_archiver by @WillemKauf
• PR #17170 rpk profile double pointer fixes by @twmb
• PR #17172 PESDLC-986 Introduce randomized name checking mechanism by @savex
• PR #17178 PESDLC-985 Raise crash errors on pod restarts by @savex
• PR #17181 CORE-7 Add MD5 Support in FIPS mode by @michael-redpanda
• PR #17182 container: Introduce chunked_hash_map by @StephanDollberg
• PR #17183 Add a way to skip the test based on cloud_type GH:1169 Jira:PESDLC-971 by @ rpdevmp
• PR #17187 git: ignore top level data/ dir by @rockwotj
• PR #17193 Lint includes by @dotnwat
• PR #17194 shard_placement_table: stress test and fixes by @ztlpn
• PR #17195 clang-tidy: remove deprecated option by @rockwotj
• PR #17196 io/tests: fix bugprone-empty-catch by @rockwotj
• PR #17208 wasm: logging tests by @rockwotj
• PR #17212 clang-tidy: disable misc-include-cleaner by @rockwotj
• PR #17214 CORE-33: Hooked OpenSSL context service into app by @michael-redpanda
• PR #17215 write caching - raft - follow up fixes by @bharathv
• PR #17219 RBAC End to End Integration Test by @oleiman
• PR #17228 rptest: remove vestigial CloudTierName.DOCKER by @travisdowns
• PR #17229 tests: allow node ready timeout to be adjusted by @dotnwat
• PR #17230 goreleaser: adjust goreleaser to dynamically link rpk on linux by @ andrewhsu
• PR #17231 cluster: Documentation for security_frontend.h by @oleiman
• PR #17232 PESDL-942 concurrent pods restart by @rpdevmp
• PR #17236 CORE-2177: rptest: e2e write caching tests with failure injection by @ nvartolomei
• PR #17242 feature_table: remove dependency on cluster/types.h by @dotnwat
• PR #17243 CORE-31 dt: Use version of ducktape with updated paramiko by @michael- redpanda
• PR #17245 kafka: chunked_vector for config responses by @pgellert
• PR #17249 transform-sdk/cpp: add string_view conversion from bytes_view by @rockwotj
• PR #17250 PESDLC-995 Create means to search logs in cloud pods by @savex
• PR #17260 k/group: recover leader epoch on leader change by @nvartolomei
• PR #17262 features: rename node_local_core_assignment by @ztlpn
• PR #17351 PESDLC-1009 Bump latency for e2e p99 by @savex
• PR #17352 Remove global dependencies from test_utils/random by @dotnwat
• PR #17357 storage: coalesce and rename record batch disk serialization methods by @ andrwng
• PR #17358 storage/mvlog: introduce batch collection to be used in new log impl by @ andrwng
• PR #17359 security: Wire up RBAC role to OCSF user::groups by @BenPope
• PR #17372 http: make http module new-style! by @dotnwat
• PR #17376 transform/processor test: prevent batching flakes by @rockwotj
• PR #17379 cluster/tests: add test case for eviction of the tx coordinator by @andrwng
• PR #17381 Offset tracker invariants by @WillemKauf
• PR #17382 storage: adopt offset_translator in namespace by @andrwng
• PR #17384 storage: add log_reader option to translate offsets by @andrwng
• PR #17385 CORE-1722: compression: Use preallocated decompression buffers for lz4 by @ abhijat
• PR #17388 serde/rw/sstring: fix header self inclusion by @andijcr
• PR #17389 cluster/types: removed redundant operator<< declaration by @andijcr
• PR #17391 CORE-14 disable tests that fail in fips by @michael-redpanda
• PR #17392 c/test: fix shard_placement_table_test by @rockwotj
• PR #17396 licenses: Add unordered_dense license by @StephanDollberg
• PR #17399 admin/api-doc: Remove role_member_update def by @oleiman
• PR #17401 treewide: address newer clang-tidy warnings by @rockwotj
• PR #17403 json: minor clean-ups by @dotnwat
• PR #17405 Feat/topic manifest serde by @andijcr
• PR #17413 c/partition_balancer: don't copy topic metadata on each partition visit by @ztlpn
• PR #17414 RBAC: Fix use-after-free in security::acl_store by @oleiman
• PR #17421 c/frag_vector: added get_allocator() method to fragmented vector by @ mmaslankaprv
• PR #17423 Fixed node_hash_map caused oversized allocations in cluster module by @ mmaslankaprv
• PR #17425 use chunked vector as batches cache in raft::replicate_batcher by @ mmaslankaprv
• PR #17427 authorizer: add parens by @travisdowns
• PR #17433 tx: fix param ordering in log statement by @nvartolomei
• PR #17434 container/fragmented_vector: add comment about resize by @rockwotj
• PR #17436 PESDLC-1036 Wait for metrics at startup of producer_swarm by @savex
• PR #17437 RBAC: Add ability to delete role ACLs when deleting the role itself in the Admin API by @oleiman
• PR #17444 PESDLC-1021 Update cloud cleanup with options by @savex
• PR #17446 RBAC: remove update role and friends by @oleiman
• PR #17447 security/role: Minor cleanups by @oleiman
• PR #17450 io: clang tidy fixes by @dotnwat
• PR #17455 cloud_storage: Fix download throttling ducktape test by @Lazin
• PR #17457 tests: do not run offsets for leader epoch test in debug mode by @ mmaslankaprv
• PR #17463 security/rbac: Add metrics by @BenPope
• PR #17464 c/partition: cleaned up files included in partition.h by @mmaslankaprv
• PR #17465 PESDLC-1021 Use bucket policy for cleanup by default by @savex
• PR #17467 security: Various minor cleanups by @oleiman
• PR #17472 rm_stm: do not hold producer lock for the duration of the barrier by @ bharathv
• PR #17474 rptest: Fix throttling test by @dotnwat
• PR #17475 s/authorizer: Comments about allow_empty_matches by @oleiman
• PR #17476 dt/rbac_test: Add RolePersistenceTest by @oleiman
• PR #17477 c/replication_monitor: majority replication optimization by @bharathv
• PR #17494 rptest: handle errors and retry them in list offsets request by @nvartolomei
• PR #17498 Improved validation of Fetch requests when reading from follower by @ mmaslankaprv
• PR #17499 archival/tests: add include for archival_metadata_stm by @bharathv
• PR #17500 gh: Added workflows to sync GitHub with Jira by @michael-redpanda
• PR #17501 Update metrics_reporter to show the number of rbac roles by @oleiman
• PR #17512 gha: update lint-golang by @andrewhsu
• PR #17515 k/topics: make write caching configs settable at creation by @bharathv
• PR #17519 cloud_roles: convert to new-style module by @dotnwat
• PR #17520 deps: Add lz4-static for fedora by @abhijat
• PR #17525 PESDLC-1021 Add optional 8h delay to bucket cleaning by @savex
• PR #17527 Update approach to kafka security for clients by @travisdowns
• PR #17528 rptest: delete vestigial datapolicy stuff by @travisdowns
• PR #17529 ducktape: improve ubsan violation handling in dt tests by @bharathv
• PR #17534 io: fix clang-tidy warning by @rockwotj
• PR #17537 serde: support variant by @rockwotj
• PR #17544 c/topics_frontend: break the loop when dispatching to current leader by @ mmaslankaprv
• PR #17545 CORE-1643 rptest: Increase backoff interval for GCS by @Lazin
• PR #17546 Revert "Add a way to skip the test based on cloud_type GH:1169 by @rpdevmp
• PR #17549 Convert modules to new-style: ssx, io, syschecks by @dotnwat
• PR #17551 Removed adl support for cluster rpc types by @mmaslankaprv
• PR #17552 rptest: Improve RRR end to end test by @Lazin
• PR #17555 rptest: add ubsan suppressions by @rockwotj
• PR #17557 iwyu: add pragma to always keep by @rockwotj
• PR #17559 remote: fix exception checking in delete_objects_sequentially by @andrwng
• PR #17562 RBAC: Enable the role store to survive a controller snapshot by @oleiman
• PR #17563 cluster: avoid throwing on duplicate recovery by @andrwng
• PR #17580 gha: lock to goreleaser v1.24.0 by @andrewhsu
• PR #17588 rptest: upgrade kgo by @nvartolomei
• PR #17592 gh: Updates to Jira issue sync by @michael-redpanda
• PR #17594 rptest/transform: add multiple output topics test by @rockwotj
• PR #17596 CORE-2055: net: allow transport logger to be customized by @dotnwat
• PR #17615 PESDLC-1063 Refactor test_consume_miss_cache by @savex
• PR #17617 Add ducktape tests for rpk security role by @r-vasquez
• PR #17621 c/controller_snapshot: Add serde envelope for named_role by @oleiman
• PR #17624 cluster: create default "Users" role when RBAC is activated by @pgellert
• PR #17626 transform: convert to new-style module by @rockwotj
• PR #17629 PESDLC-1064 Enable cloud network clean by @savex
• PR #17665 PESDLC-1069 Disable decomission tests by @savex
• PR #17666 s3_client: add keys to log messages by @andrwng
• PR #17671 rptest: write caching tests with failure injection by @nvartolomei
• PR #17672 gh: Improve workflow script by @michael-redpanda
• PR #17676 goreleaser: remove GOEXPERIMENT=systemcrypto by @andrewhsu
• PR #17678 CORE-2056: net: allow conn_quota logger to be configured by @dotnwat
• PR #17680 goreleaser: default to CGO_ENABLED=0 allow override by @andrewhsu
• PR #17681 transform-sdk/cpp: add string conversion operators by @rockwotj
• PR #17685 PESDLC-1084 Add copyright to topic_operations.py by @savex
• PR #17686 transform-sdk/tests: add schema registry test by @rockwotj
• PR #17687 rpk: use fork of avro by @andrewhsu
• PR #17689 rpk: doc updates and small QOL improvements by @r-vasquez
• PR #17695 wasm: reset call context on failure by @rockwotj
• PR #17700 PESDLC-1095 Fix and check len limits when creating shell by @savex
• PR #17701 cmake: upgrade wasmtime by @rockwotj
• PR #17702 transform/rpc: prevent cross shard memory writes by @rockwotj
• PR #17703 CORE-2061: net: remove rpc logger from server connection by @dotnwat
• PR #17704 Topic-aware replica placement in partition_allocator:: reallocate_partition by @ztlpn
• PR #17705 CORE-1705 rptest: filter abort strings from GCP headers by @andrwng
• PR #17708 tests: wait for partition leaders before querying hwm by @mmaslankaprv
• PR #17709 gtest: set death_test_style=threadsafe by @ztlpn
• PR #17714 Introduce RedpandaMixedTest by @travisdowns
• PR #17716 CORE-1614 storage: Initialize timestamps in the compaction_placeholder by @ Lazin
• PR #17717 CORE-2059: net: separate rpc metrics from generic transport by @dotnwat
• PR #17721 gha: rpk improvements by @andrewhsu
• PR #17725 test/shadow_indexing_compacted_topic_test: factored out rpk.describe_topic by @andijcr
• PR #17726 kafka: fix write caching validation during topic creation by @nvartolomei
• PR #17729 storage: document semantics of the read interface by @nvartolomei
• PR #17730 CORE-2212: test: do not start seastar for --gtest_list_tests by @dotnwat
• PR #17773 verifier/tx: explicit flush before aborting transaction by @bharathv
• PR #17785 CORE-1752: cst: improved logging by @abhijat
• PR #17786 test/compaction_e2e_test: strict check by @andijcr
• PR #17795 rptest: mix write caching with disabled batch cache by @nvartolomei
• PR #17798 tests: do not upgrade more than one feature version at a time by @ mmaslankaprv
• PR #17800 rbac_migrator: suppress error on role_exists by @pgellert
• PR #17801 slash-cmds: add /publish-to-install-pack command by @gousteris
• PR #17802 c/partition_allocator: fix bench test by @ztlpn
• PR #17807 dt: allow cleanup.policy config diff in backport by @pgellert
• PR #17811 cluster: add number of transform to metrics reporter by @rockwotj
• PR #17820 PESDLC-1121 Use test start time to capture proper logs by @savex
• PR #17823 write caching follow ups by @bharathv
• PR #17833 CORE-1743 rptest: fix flaky metric check by disabling leader balancer by @ nvartolomei
• PR #17836 ducktape: more robust tinygo downloads by @rockwotj
• PR #17838 rpk: bump docker version by @r-vasquez
• PR #17839 CORE-2243 archival: Start housekeeping jobs only on a leader by @Lazin
• PR #17840 CORE-2369: Fix/azure managed identities hns detection by @andijcr
• PR #17841 CORE-2347: net: remove dependence on rpc parsing exception type by @dotnwat
• PR #17859 transform-sdk: Introduce QuickJS based JS SDK by @rockwotj
• PR #17868 CORE-2348 s3_client: fix typo in log statement format specifier by @ nvartolomei
• PR #17870 rptest: propagate log level to verifiable services by @nvartolomei
• PR #17879 CORE-2365: storage: increase size of offset key map fragment size by @ dotnwat
• PR #17890 ducktape: retry tinygo by @rockwotj
• PR #17892 CORE-2378 id_allocator: do not forward requests beyond first hop by @ nvartolomei
• PR #17893 dt: Updated tinygo to work on Ubuntu 20.04 by @michael-redpanda
• PR #17894 tests: Marking test_drain_audit_disabled as ok_to_fail by @graphcareful
• PR #17896 Fix KafkaCliTools.produce in cloud tests by @travisdowns
• PR #17901 gh: add rpunit issue template by @rockwotj
• PR #17906 [CORE-2400] kafka/server: Disable quota balancer by @BenPope
• PR #17909 ducktape: add write caching to random_node_ops test by @bharathv
• PR #17910 log_manager: gate orphan file cleanup by @bharathv
• PR #17912 c/hm_backend: cache the collected report by @mmaslankaprv
• PR #17913 CORE-2388 rptest: remove node from started list after kill by @nvartolomei
• PR #17919 gh: Add docs team as CODEOWNERS for configs by @michael-redpanda
• PR #17926 c/controller_backend: don't require cur_operation after reconciliation step by @ztlpn
• PR #17932 rptest: fix test_exceed_broker_limit flake by @travisdowns
• PR #17949 cleanup: clang 18 fixes by @rockwotj
• PR #17953 CORE-2452 followup for azure managed identities #17840 by @andijcr
• PR #17963 ducktape: Update client-swarm deps by @StephanDollberg
• PR #17965 transform: start loading VMs after logs are started by @rockwotj
• PR #17967 fetch: Avoid formatting client ip and port for each partition by @ StephanDollberg
• PR #17968 cloud_roles: fix includes by @rockwotj
• PR #17969 CORE-2451 cloud_storage_clients/abs_clients: fix iterator check by @andijcr
• PR #17970 wasm: add a README explaining the architecture of the Wasm subsystem by @ rockwotj
• PR #17978 treewide: fix oss build with clang-18 by @rockwotj
• PR #15996 storage: enable space management by default by @dotnwat
• PR #16925 Add an option to specify topic name for nodes_report_cloud_segments by @ rpdevmp
• PR #16926 Cluster integration for role_store and related structures. by @oleiman
• PR #17387 (fix): Handle EDITOR with options by @kameshsampath
• PR #17723 Followup/feat/cloud storage compaction metric by @andijcr
• PR #18025 [v24.1.x] CORE-2490 cloud_storage_clients/client_pool: error->warn by @ vbotbuildovich
• PR #18041 [v24.1.x] c/controller_backend: try to force-abort reconfiguration only on leaders by @vbotbuildovich
• PR #18042 [v24.1.x] CORE-2627 fix AZURE_AUTHORITY_HOST parsing in azure_ask_refresh_impl by @vbotbuildovich
• PR #18068 [v24.1.x] storage: add additional logging for deep log_reader recursion by @vbotbuildovich
• PR #18070 [v24.1.x] cmake: upgrade wasmtime by @vbotbuildovich
• PR #18108 [v24.1.x] wasm/parser: fix global parsing by @vbotbuildovich
• PR #18110 [v24.1.x] gh: fix lint-cpp for ubuntu noble by @vbotbuildovich
• PR #18117 [v24.1.x] wasm/parser: better global support by @vbotbuildovich
• PR #18132 [v24.1.x] cmake: Add support for sccache by @vbotbuildovich
• PR #18137 [v24.1.x] tests: adjust time between failures in relaxed consistency test by @vbotbuildovich
• PR #18148 [v24.1.x] rpk container: offer port selection on start by @vbotbuildovich

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.1...v24.1.1

Bug Fixes

• fixed a problem leading to UAF error while calculating cloud stage usage by @mmaslankaprv in [#17980](https://github.com/ redpanda-data/redpanda/pull/17980)
• prevents partial consumer group recovery by @mmaslankaprv in #18016

Improvements

• Changes what the kafka_latency_fetch_latency metric measures to be the time the first fetch_ntps_in_parallel takes. by @ ballard26 in #17977
• largely reduced number of health report copies by @mmaslankaprv in #18017## None
  No release notes explicitly specified.
• PR #17917 [v23.3.x] archival: Start housekeeping jobs only on a leader by @Lazin
• PR #17927 [v23.3.x] c/hm_backend: cache the collected report by @ mmaslankaprv
• PR #17960 [v23.3.x] rptest: fix test_exceed_broker_limit flake by @ travisdowns
• PR #17979 [v23.3.x] [CORE-2400] kafka/server: Disable quota balancer by @BenPope
• PR #18019 [v23.3.x] CORE-1752: cst: Downgrade error logs to debug by @ abhijat
• PR #18023 [v23.3.x] Backport of #16243 by @mmaslankaprv
• PR #18039 [v23.3.x] c/controller_backend: try to force-abort reconfiguration only on leaders by @ztlpn
• PR #18053 [v23.3.x] Address oversized allocs across kafka API and schema registry by @oleiman

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.12...v23.3.13

Bug Fixes

• Fix a race condition between suffix truncation / delete records and adjacent segment compaction that can lead to crashes and data-loss. by @ nvartolomei in #17254
• Fix initial_leader_epoch/KIP-320 handling in fetch requests. It was ignored until now which prevented consumers to correctly detect suffix truncation. For Redpanda (and Raft), this is a minor problem since suffix truncation is a very improbable event. by @nvartolomei in #17727
• Fixes a bug of config_frontend methods getting called on shards other than the controller shard. by @pgellert in [#17211](https://github.com/ redpanda-data/redpanda/pull/17211)
• Prevent detecting leader epoch advancement when state is not up to date by @mmaslankaprv in [#17882](https://github.com/redpanda-data/ redpanda/pull/17882)
• Reduces maximum log line size from 1MiB to 128KiB to reduce occurrences of memory allocation failures by @michael-redpanda in #17924
• #16612 fixes small inconsistency between Kafka and Redpanda when trying to query end_offset of an empty log by @mmaslankaprv in #17881
• #17238 Fixes a bug in CreateTopicsResponse to now return all the configs of the topic, not just the topic-specific override configs. by @pgellert in #17241
• #17790 Fix a bug that could lead to raft log inconsistencies when 2 out of 3 nodes in a configuration are changed. by @ztlpn in #17797
• prevents partial consumer group recovery by @mmaslankaprv in #17882
• PR #17160 [v23.2.x] compression: Allocate memory for LZ4_compressEnd by @abhijat
• PR #17826 [v23.2.x] CORE-1722: compression: Use preallocated decompression buffers for lz4 by @abhijat
• PR #17881 [v23.2.x] k/replicated_partition: fixed querying end offset of an empty log by @nvartolomei
• PR #17882 Backport of #17673 #17498 #16560 by @mmaslankaprv

Improvements

• Adds a new public metric redpanda_raft_recovery_partition_movement_consumed_bandwidth that tracks how much bandwidth is currently in use for raft recovery. This helps tune raft_learner_recovery_rate. by @bharathv in #17217

• PR #17397 [v23.2.x] k/group: recover leader epoch on leader change by @nvartolomei * PR #17448 [v23.2.x] tx: fix param ordering in log statement by @nvartolomei

• PR #17577 [v23.2.x] c/topics_frontend: break the loop when dispatching to current leader by @mmaslankaprv

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.2.27...v23.2.28

Bug Fixes

• Fix an issue where rpk transform logs waits for records without the --follow flag specified. by @rockwotj in #17837
• Fixes rpk transform init --install-deps so that an explicit true value is not needed. by @rockwotj in [#17867](https:// github.com/redpanda-data/redpanda/pull/17867)
• Fixes a crash when data transforms error and restart by @rockwotj in [#17696](https://github.com/redpanda-data/redpanda/pull/ 17696)
• Reduces maximum log line size from 1MiB to 128KiB to reduce occurrences of memory allocation failures by @michael-redpanda in #17923
• #16612 fixes small inconsistency between Kafka and Redpanda when trying to query end_offset of an empty log by @mmaslankaprv in #17809
• #17718 Fix reported config source for cleanup.policy by reporting DEFAULT_CONFIG instead of DYNAMIC_TOPIC_CONFIG for the default value. by @pgellert in [#17719](https://github.com/redpanda-data/ redpanda/pull/17719)
• #17791 Fix a bug that could lead to raft log inconsistencies when 2 out of 3 nodes in a configuration are changed. by @ztlpn in #17796
• #17817 Fix problem in Tiered-Storage that could potentially cause consumers to get stuck by @Lazin in #17818
• #17891 fix a race between eviction and producer registration that results in an invalid transaction state. by @bharathv in #17900

Improvements

• Handle missing data transform logs topic in rpk transform logs by @rockwotj in [#17835](https://github.com/redpanda-data/ redpanda/pull/17835)
• #17197 more accurate node status reporting by @mmaslankaprv in #17698
• skipping overhead of collecting node health report for each node separately. by @mmaslankaprv in [#17864](https://github.com/ redpanda-data/redpanda/pull/17864)
• PR #17756 [v23.3.x] kafka: chunked_vector for config responses by @ pgellert
• PR #17792 [v23.3.x] CORE-1752: cst: improved logging by @abhijat
• PR #17825 [v23.3.x] CORE-1722: compression: Use preallocated decompression buffers for lz4 by @abhijat
• PR #17888 [v23.3.x] CORE-2365: storage: increase size of offset key map fragment size by @dotnwat

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.11...v23.3.12

Features

• Introduce "trust_file_crc32c" metric to export a checksum for each trust file in the system. by @oleiman in #17587

Bug Fixes

• #16650 Fix oversized allocation in storage. by @Lazin in #17541
• #17459 Fixes a bug with TLS metrics where expiration timestamps would not advance on certificate reload by @oleiman in #17460
• rpk: prevent a segfault when creating a profile from a cloud that is not in ready state. by @r-vasquez in #17585
• PR #17435 [v23.3.x] c/frag_vector: added get_allocator() method to fragmented vector by @mmaslankaprv
• PR #17449 [v23.3.x] tx: fix param ordering in log statement by @ nvartolomei
• PR #17572 [v23.3.x] Fixed node_hash_map caused oversized allocations in cluster module by @mmaslankaprv
• PR #17573 [v23.3.x] use chunked vector as batches cache in raft:: replicate_batcher by @mmaslankaprv
• PR #17576 [v23.3.x] c/topics_frontend: break the loop when dispatching to current leader by @mmaslankaprv
• PR #17578 [v23.3.x] rm_stm: do not hold producer lock for the duration of the barrier by @bharathv
• PR #17584 [v23.3.x] k/group_manager: used chunked_vector when cleaning groups by @mmaslankaprv

Improvements

• #17428 Improves error feedback when Redpanda is given an invalid number of partitions during either topic creation or when the partition count for a topic is increased. by @michael-redpanda in #17431
• PR #17574 [v23.3.x] Improved validation of Fetch requests when reading from follower by @mmaslankaprv

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.10...v23.3.11

Features

• #16075 Data Transform's Rust SDK now supports a Schema Registry Client. by @rockwotj in #17174
• #17235 rpk: support running rpk transform against Redpanda cloud clusters. by @r-vasquez in #17252

Bug Fixes

• Fix a bug where logging in a transform could cause the transform to not make progress. by @rockwotj in #17192
• Fix a race condition between suffix truncation / delete records and adjacent segment compaction that can lead to crashes and data-loss. by @nvartolomei in #17253
• Fixes a bug of config_frontend methods getting called on shards other than the controller shard. by @pgellert in #17184
• Fixes a bug that would prevent the transaction coordinator topic from reclaiming disk space. by @andrwng in #17380
• Fixes a concurrency issue in transform offset commits pertaining to taking/applying snapshots. by @bharathv in #17400
• #17237 Fixes a bug in CreateTopicsResponse to now return all the configs of the topic, not just the topic-specific override configs. by @pgellert in #17239
• #17419 fixed enabling cloud storage in existing clusters by @mmaslankaprv in #17420

Improvements

• Adds a new public metric redpanda_raft_recovery_partition_movement_consumed_bandwidth that tracks how much bandwidth is currently in use for raft recovery. This helps tune raft_learner_recovery_rate. by @bharathv in #17216
• less overhead of health report collection by @mmaslankaprv in #17360

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.9...v23.3.10

Features

• Introduce rpk transform logs NAME to view logs for a transform by @rockwotj in #17148

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.8...v23.3.9

Bug Fixes

• Schema Registry: Deleted schemas no longer reappear after certain compaction patterns on the _schemas topic. by @BenPope in #17095
• #16129 Fixes a bug in SASL user deletion and update where usernames with a + symbol in the username were prevented from being deleted by @pgellert in #16813
• #16716 Fixed a few oversized allocations for some admin server endpoints. by @rockwotj in #16747

Improvements

• #16995 Added EHOSTUNREACH to retry-able error code list by @michael-redpanda in #16996

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.2.26...v23.2.27

Features

• #16941 rpk redpanda config bootstrap now support bootstrapping your advertised addresses configuration. by @r-vasquez in #16942

Bug Fixes

• Fix a crash that happened when a cluster that was partially in recovery mode tried to upload consumer offsets to cloud storage. by @ ztlpn in #17022
• Return a HTTP 400 error code when deploying a transform to a topic that doesn't exist instead of a 500 by @rockwotj in #17018
• Schema Registry: Deleted schemas no longer reappear after certain compaction patterns on the _schemas topic. by @BenPope in #17094
• #16679 Retains control batches from transactions to preserve transaction boundaries. This prevents some (very unlikely) scenarios where aborted data is read. by @bharathv in [#17100](https://github.com/ redpanda-data/redpanda/pull/17100)
• PR #17093 [v23.3.x] c/topic_table: replaced partition metadata map with chunked_vector by @mmaslankaprv
• PR #17099 [v23.3.x] storage: ensure monotonic stable offset updates by @ nvartolomei
• PR #17111 [v23.3.x] cloud_storage_clients: classify request_timeout as retriable by @nvartolomei

Improvements

• #16815 Node-wide throughput throttling is now fair an responsive. by @ BenPope in #16848
• #16993 cluster: Avoid oversize allocs for topic creation and configuration by @BenPope in #17012
• #17107 rpk profile has been reworked in an attempt to be simpler; see PR #17038 for more detail by @twmb in #17108
• PR #17115 [v23.3.x] Using contiguous_range_map in partition_leaders_table by @mmaslankaprv
• PR #17120 [v23.3.x] rpk profile: a few more fixes by @twmb

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.7...v23.3.8