redpanda

Bug Fixes

• Avoids a crash when attempting to create a read replica topic while cloud storage is not configured. by @andrwng in #11808
• Fix for OffsetForLeaderEpoch returning a value for when the requestedEpoch is larger then the highest known. by @graphcareful in #12086
• Fix for OffsetForLeaderEpoch returning the current leaders epoch (term) instead of the requested. by @graphcareful in #12086
• Fixed a potential invalid memory access when iterating through segments with timestamps in the future. by @andrwng in #11969
• Redpanda will now report upload housekeping metrics. by @andrwng in #11955
• #11542 fixes rare situation in which consumer may stuck due to incorrect truncation point by @mmaslankaprv in #11543
• #11604 net: Fix a rare crash during shutdown of a failed connection with outstanding requests by @BenPope in #11605
• #11717 Memory consumption for housekeeping on compacted topics is reduced by @jcsp in #11718
• #12014 Make tiered-storage metadata handling more strict during rolling upgrades by @Lazin in #12055
• #12415 #12417 schema_registry: Strip redundant namespaces in Avro to improve schema lookup by @BenPope in #12425
• rpk cluster logdirs no longer panics if there is an error getting a response from Redpanda by @twmb in #11926
• rpk group offset-delete no longer tries to delete offsets for all topics if no empty topics are specified by @twmb in #11926
• rpk group offset-delete no longer tries to delete offsets for empty-name topics by @twmb in #11926
• fixes log segments being evicted too early by @mmaslankaprv in #12190

Improvements

• Lower the TCP keepalive timeout to reap dead/idle connections faster and claim back resources by @StephanDollberg in #11774
• Reduced latency impact from storing and retrieving metadata in certain scenarios where the number of partitions per shard is high by @BenPope in #11927
• Reduced latency impact from storing metadata in certain scenarios where the number of partitions per shard is high by @BenPope in #11798
• #11262 #11302 If kafka_max_bytes_per_fetch is not configured properly, redpanda is now more robust against huge Fetch requests by controlling memory consumption dynamically. by @dlex in #11858
• #11262 #11302 Memory control is improved in Fetch request handler, covering the cases when a client tries to fetch too many partitions lead by the same shard. Some of the request partitions will not be fetched if the broker does not have enough memory for that, or if that would violate constraints set by kafka_max_bytes_per_fetch and fetch_max_bytes. by @dlex in #11858
• #11643 Improved efficiency in encoding tag values in kafka wire protocol by @michael-redpanda in #11645
• #11853 Avoid large allocation when storing metadata in certain scenarios where the number of partitions per shard is high by @BenPope in #11854
• #11924 admin_server/get_partition: Avoid oversized allocation by @BenPope in #11928
• #12034 Disable use of fetch scheduling group by default. We found a few cases that were negatively affected. by @StephanDollberg in #12035
• #12051 Redpanda will now gracefully handle badly formatted SCRAM authentication messages by @michael-redpanda in #12052
• #12106 Avoid large allocations with zstd compacted topics. by @BenPope in #12107
• #12409 schema_registry: Improve the ordering of protobuf files by @BenPope in #12411
• rpk: support SASL flags in rpk redpanda admin commands. by @r-vasquez in #11650

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.1.13...v23.1.14

Bug Fixes

• #12183 rpk: fixes a bug where using quotes in flags values inside redpanda.yaml's rpk.additional_start_flags would make redpanda to fail on start. by @r-vasquez in #12184
• #12416 schema_registry: Strip redundant namespaces in Avro to improve schema lookup by @BenPope in #12418
• #12489 Fixes a race condition in log-eviction-stm. by @andrwng in #12489

Improvements

• #12405 schema_registry: Improve the ordering of protobuf files by @BenPope in #12406

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.2.2...v23.2.3

Bug Fixes

• rpk cloud byoc now properly adds the --redpanda-id flag when handing off to the cloud plugin. by @twmb

Full Changelog:

https://github.com/redpanda-data/redpanda/compare/v23.2.1...v23.2.2

New Features

• Added support for follower fetching #9780
• Server-side validation of Schema IDs encoded into the header of a record encoded by a serde client library #10650
• Tiered Storage now uses a fine-grained local disk cache using segment chunks as opposed to full segments #9426
• Improve scalability of Tiered Storage by offloading metadata to cloud storage and fetching it when its needed #11150, #10560
• Redpanda now loads the controller log from a snapshot on startup. This significantly improves startup times of nodes in long-running Redpanda clusters. Enabled on new clusters, disabled on existing clusters. To enable in 23.2.1 contact Redpanda Support #8285
• Use multiple partitions in the transaction manager topic. This significantly improves throughput when using the Kafka Transactions API. It is configurable with transaction_coordinator_partitions option and can be set only on cluster creation #9362
• Adds a built-in CPU profiler configured with the cpu_profiler_enabled and cpu_profiler_sample_rate_ms cluster properties #10708
• Adds support for the deleteRecords Kafka API #10061
• Adds support for rpk profile to manage multiple rpk configuration profiles #10528
• Schema Registry: Support for Protobuf ‘well known types’ #7431
• Adds a ‘force reconfiguration’ API to uncleanly reconfigure a raft group into a smaller replica set. Meant to be used as an escape hatch or equivalent high level abstractions to recover partitions from unrecoverable state. #9096
• Kubernetes Operator: Support mTLS with customer provided client CA at Panda Proxy and Schema Registry #9863
• Redpanda OperatorV2 (BETA *Note: Helm-based deployment is still recommended at this point An upcoming patch release will deliver a GA version of the OperatorV2. #10317

RPK CLI improvements

• adds rpk topic trim-prefix, rpk support for DeleteRecords Kafka API #11781
• rpk: adds rpk topic describe-storage which describe the cloud storage status of a given topic #9500, #8894
• Introduce SSO to rpk cloud login. #9864
• Adds support for rpk cloud auth to manage multiple rpk cloud auths #10528
• Automatically populates a new cloud profile when you run rpk cloud login #10528
• You can now generate a sample starter application to produce and consume using rpk generate app #12037
• request. Now you can trim records to a specific offset in the given topic/partition
• rpk now generate a password automatically when running rpk acl user create with no password #11235
• Add the ability to pass rpk TLS configuration flags to the legacy dashboard. #5623
• rpk now downloads the Grafana dashboard from our repo https://github.com/redpanda-data/observability #9662
• rpk now support for a -X flag. This will eventually replace --brokers, --user, --password, etc. -- i.e., all flags that configure how rpk talks to brokers. -X supports -X list for a short list of available configuration parameters, -X help for a full list, and autocomplete. #9981
• rpk now supports --rack to specify the rack to consume from, which also opts into follower fetching #11105
• adds --print-commits / -c to rpk group describe to opt into printing only the commits & lag section #12113
• now you can update user credentials with rpk acl user update. #11289
• You can now generate a sample starter application to produce and consume using rpk generate app #12037

Other Notable Improvements

• The Redpanda ‘Admin API’ is now documented publicly
• A new admin API endpoint is added at /v1/cloud_storage/manifest/{topic}/{partition} which allows for retrieving the in-memory manifest for a partition in JSON format. #9745
• Cloud storage connections are balanced between the shards according to workload #9410
• pandaproxy: Support users with SCRAM-SHA-512 for `authentication: #11425
• Responses to Kafka DescribeLogDirs requests now include remote tiered storage space utilization. Remote space utilization is reported as a special directory called "remote://{your_bucket_name}". #8145
• Remove ghost Redpanda Node IDs #9750
• adds usage metrics to each Kafka request handler #10623
• Add admin API command that changes --blocked-reactor-notify-ms parameter on the fly. #9331
• Adds a large_allocation_warning_threshold node config option to enable warnings to be logged on large allocations. #8596
• Add mechanism that can be used to truncate tiered-storage using kafka offset. #9994
• Hierarchical hard/soft partition allocation constraints #9767
• Log retention rules applied eagerly in low disk space scenarios. #10658
• The vectorized_memory_available_memory metric is now available on the /metrics endpoint in addition to /public_metrics. #10018

Bug Fixes

• A bug is fixed where tiered storage topics might delay uploading manifests when partition leadership changes shortly after segments are uploaded https://github.com/redpanda-data/redpanda/pull/9154
• A spurious "Leaving S3 objects behind" log message is no longer printed when deleting non-tiered-storage partitions. https://github.com/redpanda-data/redpanda/pull/9405
• Always reconcile Node ID annotation in Redpanda POD https://github.com/redpanda-data/redpanda/pull/9749
• Avoids a crash when attempting to create a read replica topic while cloud storage is not configured. https://github.com/redpanda-data/redpanda/pull/11792
• Changes to enable deployment of new operator, not by default. https://github.com/redpanda-data/redpanda/pull/11651
• Convert tm_snapshot to fragmented_vector to reduce memory pressure. https://github.com/redpanda-data/redpanda/pull/10035
• Decommission during scale down uses correct Redpanda Node ID https://github.com/redpanda-data/redpanda/pull/11312
• Disable adjacent segment merging when the partition is not a leader https://github.com/redpanda-data/redpanda/pull/10538
• Do not remove the controller broker on not_controller error from kafka::client::create_topic( https://github.com/redpanda-data/redpanda/pull/9830
• Fix a bug in the re-uploading of compacted segment that could lead to consumers getting blocked. https://github.com/redpanda-data/redpanda/pull/9404
• Fix a bug where recovered topics would use the retention settings specified at creation time (ignoring updates after that point. https://github.com/redpanda-data/redpanda/pull/10297
• Fix a rare race between segment rolling and the readers cache which caused spurious, temporary dead-locks https://github.com/redpanda-data/redpanda/pull/9676
• Fix an issue with compacted segment reupload that could cause self-concatenated segments to be reuploaded. https://github.com/redpanda-data/redpanda/pull/9576
• Fix cases where sending very large batches to compacted topics using LZ4 or gzip compression could result in bad_alloc errors. https://github.com/redpanda-data/redpanda/pull/9598
• Fix for OffsetForLeaderEpoch returning a value for when the requestedEpoch is larger then the highest known. https://github.com/redpanda-data/redpanda/pull/12063
• Fix memory leak upon shutdown which can occur when using tiered storage with ABS and TLS. https://github.com/redpanda-data/redpanda/pull/9998
• Fix offset translation errors after topic recovery https://github.com/redpanda-data/redpanda/pull/9919
• Fix offset translation failure that could be triggered after topic recovery https://github.com/redpanda-data/redpanda/pull/9865
• Fix race between compaction and application of retention to the local log. This occurred when compaction happened below the start offset of the log. It did not have a noticeable impact upon user workloads. https://github.com/redpanda-data/redpanda/pull/9483
• Fix race condition during forced segment roll https://github.com/redpanda-data/redpanda/pull/10506
• Fix rare histogram metric reporting bug which could lead to Redpanda crashing or terminating the HTTP connection on which it was serving the metric request. This only occurred when metrics aggregation was enabled. https://github.com/redpanda-data/redpanda/pull/9192
• Fix upload housekeeping errors on log truncation https://github.com/redpanda-data/redpanda/pull/9409
• Fixed a bug in which the segment merging could run from read replicas. https://github.com/redpanda-data/redpanda/pull/9685
• Fixed a bug that would cause Redpanda to return an invalid offset when consuming from a term that falls below the beginning of the local log when reading from tiered storage is disabled. https://github.com/redpanda-data/redpanda/pull/9264
• Fixed a bug that would previously prevent read replicas from progressing if cloud_storage_enable_remote_write is false. https://github.com/redpanda-data/redpanda/pull/9684
• Fixed a potential invalid memory access when iterating through segments with timestamps in the future. https://github.com/redpanda-data/redpanda/pull/11956
• Fixed an issue where sq or sq-split mode may be chosen for NIC IRQ affinity in the redpanda tuner, resulting in core/shard 0 becoming a bottleneck for some high packet-per-second loads. https://github.com/redpanda-data/redpanda/pull/10568
• Fixed another bug where multiple configuration updates would cause an inconsistency upon next reload of usage state https://github.com/redpanda-data/redpanda/pull/9917
• Fixed not being able to elect a leader in situation when only voter is in maintenance mode https://github.com/redpanda-data/redpanda/pull/10800
• Fixes a bug in our http client that may crash redpanda in exceptional cases https://github.com/redpanda-data/redpanda/pull/9903
• Fixes a bug where usage_manger would report incorrect cloud storage metrics when queried on the leader node. https://github.com/redpanda-data/redpanda/pull/9917
• Fixes an issue with the archival upload path that could contribute to data loss when archival metadata updates took a long time to be replicated and applied. https://github.com/redpanda-data/redpanda/pull/9633
• Limits on tiered storage read concurrency are enforced more strictly, to improve stability when clients issue very large numbers of concurrent fetches, or do time queries on topics with large numbers of partitions. https://github.com/redpanda-data/redpanda/pull/11914
• Make tiered-storage metadata handling more strict during rolling upgrades https://github.com/redpanda-data/redpanda/pull/12001
• Makes starting Redpanda with long controller log containing redundant topic properties update faster https://github.com/redpanda-data/redpanda/pull/9739
• Prevent possible data loss in situation when the same segment is added twice to the manifest https://github.com/redpanda-data/redpanda/pull/9597
• Regenerate the console configmap after RP cluster spec is changed so that the console works properly. https://github.com/redpanda-data/redpanda/pull/11211
• Schema Registry: Fix a bug in GET /subjects/<subject>/versions/latest that would previously not find the latest non-deleted version. https://github.com/redpanda-data/redpanda/pull/10543
• The PR resolves, if requireClientAuth is enabled and clientCACertRef is NOT set for panda proxy or schema registry TLS listener, the Redpanda pods will not come up due to the error below. https://github.com/redpanda-data/redpanda/pull/10546
• The redpanda_cloud_storage_cache_op_miss metric was not showing the right value. https://github.com/redpanda-data/redpanda/pull/9333
• A bug is fixed where if a PUT to object storage was in flight during shutdown, Redpanda might incorrectly record a failed upload as successful. https://github.com/redpanda-data/redpanda/pull/10107
• A stability issue is fixed where many concurrent Produce requests using very large compressed batches could exhaust memory. https://github.com/redpanda-data/redpanda/pull/10235
• A stability issue is fixed where very large ZSTD-compressed batches could exhaust memory https://github.com/redpanda-data/redpanda/pull/10235
• Fix post-recovery Raft boostrap https://github.com/redpanda-data/redpanda/pull/10535
• Prevent allocation failures with many idempotent producers https://github.com/redpanda-data/redpanda/pull/10250
• Fixes an assertion during archival when a leadership change occurs https://github.com/redpanda-data/redpanda/pull/10244
• Fixed an issue where the metric redpanda_kafka_consumer_group_consumers was reporting double the real count of consumers. https://github.com/redpanda-data/redpanda/pull/10320
• Fix over allocation in metadata dissemination leadership update https://github.com/redpanda-data/redpanda/pull/10585
• bugfix: add correct file metadata to files in the debug bundle generated by rpk debug bundle https://github.com/redpanda-data/redpanda/pull/10705
• Fix bug caused by unexpected exception on the remote read path which caused Kafka fetch requests to time-out and leave lingering connections. https://github.com/redpanda-data/redpanda/pull/10828
• rpk disk_irq tuner now provides a warning for a known issue introduced in kernel 5.17 where instances utilizing MSI IRQ may encounter an empty IRQs list in sysfs that caused hwloc segmentation faults and results in a tuner failure. https://github.com/redpanda-data/redpanda/pull/10864
• An issue is fixed where clients might see a lower partition high watermark than expected if querying very soon after a new raft leader is elected for the partition. https://github.com/redpanda-data/redpanda/pull/10921
• Fixes inspecting Docker networks on macOS when using podman. https://github.com/redpanda-data/redpanda/pull/10902
• k8s: allow console to be deleted even if cluster is not configured https://github.com/redpanda-data/redpanda/pull/11026
• Fixed a bug where ephemeral credentials were stored in controller snapshots https://github.com/redpanda-data/redpanda/pull/11563
• Schema Registry: Return references for GET /schemas/ids/<id>. https://github.com/redpanda-data/redpanda/pull/11216
• fixes rare situation in which consumer may stuck due to incorrect truncation point https://github.com/redpanda-data/redpanda/pull/11450
• Fix a possible dangling pointer issue in the storage layer https://github.com/redpanda-data/redpanda/pull/11436
• rpk: change required kernel version from 4.9 to 3.9 in rpk redpanda check and rpk redpanda start https://github.com/redpanda-data/redpanda/pull/11502
• Memory consumption for housekeeping on compacted topics is reduced https://github.com/redpanda-data/redpanda/pull/11681
• fixed not being able to clear consumer backlog when using incremental fetch requests and follower fetching https://github.com/redpanda-data/redpanda/pull/11748
• fixed returning read replica that may be unavailable from replica selector https://github.com/redpanda-data/redpanda/pull/11748
• Prevent uploaded segment from having incorrect archiver term https://github.com/redpanda-data/redpanda/pull/11892
• Pandaproxy will no longer assert if attempting to consume compressed message batches https://github.com/redpanda-data/redpanda/pull/10117
• Schema Registry will no longer assert if the _schemas topic contains compressed batches https://github.com/redpanda-data/redpanda/pull/10117
• Fix bug in calculate_unevenness_error when allocated_replicas on node goes below zero https://github.com/redpanda-data/redpanda/pull/9870
• Fixed issue where Redpanda will assert if data being consumed by Pandaproxy is not JSON serializable https://github.com/redpanda-data/redpanda/pull/9867
• The consumer used in our internal kafka client will automatically find the new consumer group coordinator on not_coordinator errors. https://github.com/redpanda-data/redpanda/pull/9967
• rpk will not modify redpanda.rpc_server_tls property when decoding the redpanda.yaml, which means that it will leave the field as a list or as an element depending on what the user has configured before executing rpk commands. https://github.com/redpanda-data/redpanda/pull/7719
• Fixed issue with offset translation in rm_stm on snapshot hydrate https://github.com/redpanda-data/redpanda/pull/10232
• Fix race condition in tm_stm take_snapshot. https://github.com/redpanda-data/redpanda/pull/9575
• Do not validate console if it's being removed https://github.com/redpanda-data/redpanda/pull/9164
• Fix a memory leak in the operator https://github.com/redpanda-data/redpanda/pull/9201
• Fixed an issue that could prevent cloud storage truncation following leadership changes. https://github.com/redpanda-data/redpanda/pull/9494
• Fixed excessive kvstore writes that could lead to memory fragmentation issues during heavy produce load. https://github.com/redpanda-data/redpanda/pull/9827
• fixes an assertion that may happen when Raft snapshot is delivered to recover follower having any of rm tx or id_allocator stms https://github.com/redpanda-data/redpanda/pull/9656
• https://github.com/redpanda-data/redpanda/issues/9507 Fixed automatic rebalancing of replicas stopping to early before achieving even distribution of replicas. https://github.com/redpanda-data/redpanda/pull/9515
• Fixed a bug that would result in read replicas reporting a high watermark that was too high. https://github.com/redpanda-data/redpanda/pull/9493
• Fixed memory allocation errors when using very large batches with LZ4 compression on compacted topics https://github.com/redpanda-data/redpanda/pull/9563
• This change fixes an out of memory in the replicate batcher by ensuring that only one flush task is outstanding at any time. https://github.com/redpanda-data/redpanda/pull/9966
• Fix race between re-uploaded segment compaction and local truncation that prevented the re-upload from succeeding. https://github.com/redpanda-data/redpanda/pull/9566
• Fixes violation of the atomicity of the consumer - transform - produce loop https://github.com/redpanda-data/redpanda/pull/9573
• https://github.com/redpanda-data/redpanda/issues/9602 Made Join and configuration change validation logic consistent https://github.com/redpanda-data/redpanda/pull/10325
• Improve precision of the adjacent segment merger to avoid unnecessary segment reuploads https://github.com/redpanda-data/redpanda/pull/9657
• An issue is fixed where time queries on tiered storage partitions could return -1 incorrectly if the queried timestamp was earlier than the start of the log. https://github.com/redpanda-data/redpanda/pull/9815
• An issue is fixed where time queries on tiered storage partitions using offsets close to the end of the local raft log could sometimes return offsets slightly ahead of the correct offset, if the segment containing the correct offset had already been offloaded to object storage. https://github.com/redpanda-data/redpanda/pull/9815
• fix consumer group kafka incompatibilities (update offset_fetch to return committed_leader_epoch https://github.com/redpanda-data/redpanda/pull/9851
• redpanda_cluster_partitions is incremented per partition, not per replica, on a topic creation https://github.com/redpanda-data/redpanda/pull/9393
• rpk cluster logdirs no longer panics if there is an error getting a response from Redpanda https://github.com/redpanda-data/redpanda/pull/11900
• rpk group offset-delete no longer tries to delete offsets for all topics if no empty topics are specified https://github.com/redpanda-data/redpanda/pull/11900
• rpk group offset-delete no longer tries to delete offsets for empty-name topics https://github.com/redpanda-data/redpanda/pull/11900
• close Redpanda admin connection after reconciling the Console user and ACL. https://github.com/redpanda-data/redpanda/pull/11016
• fixed rm_stm resource leak https://github.com/redpanda-data/redpanda/pull/11597
• fixed a bug in rpk cluster storage recovery which rendered the status command unusable. https://github.com/redpanda-data/redpanda/pull/9795
• fixed stms interoperation with relaxed consistency semantics https://github.com/redpanda-data/redpanda/pull/11840
• fixes rare issue that may arise when new node id is assigned to a node with the same set of ip addresses https://github.com/redpanda-data/redpanda/pull/9254
• k8s: Status.Version is not updated until upgrade to that version is finished https://github.com/redpanda-data/redpanda/pull/10877
• net: Fix a rare crash during shutdown of a failed connection with outstanding requests https://github.com/redpanda-data/redpanda/pull/11586
• operator: fixed an issue that caused continue reconciliations on the custom resource https://github.com/redpanda-data/redpanda/pull/9950
• rpk bugfix: now rpk debug bundle in k8s works for brokers with TLS enabled. https://github.com/redpanda-data/redpanda/pull/9320
• rpk now properly handles -o @:-1h to consume from the start to one hour ago https://github.com/redpanda-data/redpanda/pull/9667
• rpk: Fix a panic when using rpk container with Podman https://github.com/redpanda-data/redpanda/pull/9133
• rpk: In k8s rpk debug bundle --since flag no longer supports non-standard durations (such as day, week, years, only Go standard durations will be accepted. https://github.com/redpanda-data/redpanda/pull/10002
• rpk: fixes a bug that prevented shell autocompletion to work in zsh and improve the help text for Mac users. https://github.com/redpanda-data/redpanda/pull/10170

Full Changelog:

https://github.com/redpanda-data/redpanda/compare/v23.1.13...v23.2.1

Improvements

• by default operator allows scale down by @RafalKorepta in #11583

Bug Fixes

• Decommission during scale down uses correct Redpanda Node ID by @RafalKorepta in #11583
• Fixed not being able to elect a leader in situation when only voter is in maintenance mode by @mmaslankaprv in #11544
• pandaproxy: Support users with SCRAM-SHA-512 for authentication: http_basic. by @BenPope in #11429
• regenerating console configmap if Redpanda cluster spec changed by @paulzhang97 in #11327
• #11309 [v23.1.x] Fix aborted txs being visible by @rystsov
• #11333 [v23.1.x] k/proto: use log-level warn for missmatch crc by @NyaliaLui
• #11400 [v23.1.x] k/proto: use log-level warn for invalid crc by @NyaliaLui
• #11417 [v23.1.x] ducktape: use pgrep to find the redpanda process by @rockwotj
• #11446 [v23.1.x] ducktape: Change condition for upload check to pass when all segments are uploaded by @abhijat
• PR #11453 [v23.1.x] throughput control groups: fixed zero byte dumped to the log by noname group by @dlex

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.1.12...v23.1.13

Bug Fixes

• #11238 Schema Registry: Return references for GET /schemas/ids/<id>. by @BenPope in #11241
• pandaproxy: Support users with SCRAM-SHA-512 for authentication: http_basic. by @BenPope in #11430
• #10729 [v22.3.x] Use chunk_cache as source of truth for chunk size by @ballard26
• #11128 [v22.3.x] Check for perfect replicas distribution in ScalingUpTest.test_adding_nodes_to_cluster by @dlex

Improvements

• Choose more CPU friendly TLS ciphers to reduce TLS overhead by @StephanDollberg in #11217
• #11144 Schema Registry: Canonicalize AVRO schema so that syntactic differences don't fail lookups by schema contents. by @BenPope in #11147
• #10861 [v22.3.x] cloud_storage: improved debug on partition manifest parse error by @jcsp
• #11059 [v22.3.x] Breakdown installation of ducktape dependencies in multiple bash scripts by @gousteris
• #11334 [v22.3.x] k/proto: use log-level warn for missmatch crc by @NyaliaLui
• #11398 [v22.3.x] k/proto: use log-level warn for invalid crc by @NyaliaLui

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v22.3.21...v22.3.22

Bug Fixes

• Fix post-recovery Raft boostrap by @VladLazar in #11214
• #11073 k8s: allow console to be deleted even if cluster is not configured by @alenkacz in #11074
• #11237 Schema Registry: Return references for GET /schemas/ids/<id>. by @BenPope
• #11240 close Redpanda admin connection after reconciling the Console user and ACL. by @joejulian
• #11051 operator: close kadm connections created by console controller. by @joejulian
• #10999 [v23.1.x] add pre-exit hook that calls the vtools one by @gousteris
• #11025 [v23.1.x] Using chunked_fifo in heartbeat_manager by @mmaslankaprv
• #11083 [v23.1.x] Use chunked_fifo to group shard heartbeats by @mmaslankaprv
• #11124 [v23.1.x] Check for perfect replicas distribution in ScalingUpTest.test_adding_nodes_to_cluster by @dlex
• #11236 [v23.1.x] cloud_storage_clients/abs: fix arg count for log by @VladLazar
• #11301 [v23.1.x] k8s: revert remove --platform flag from operator images' dockerfile by @nk-87

Improvements

• #11188 A client_id, or a group of client_ids specified with a regex, can now be excluded from node wide throughput control. The new cluster property kafka_throughput_control can be used to define throughput control groups for which Kafka traffic will not be limited by the values specified by kafka_throughput_limit_node_*_bps. by @dlex in #11213
• Choose more CPU friendly TLS ciphers to reduce TLS overhead by @StephanDollberg in #11218
• #11111 rpk debug bundle now can be used in clusters with SASL enabled to gather Admin API information. by @r-vasquez in #11108
• #11142 Schema Registry: Canonicalize AVRO schema so that syntactic differences don't fail lookups by schema contents. by @BenPope in #11143
• #11162 Compaction: Reduce large allocations by @BenPope in #11163
• #11195 Self-Test: Avoid large allocations during diskcheck by @BenPope in #11196
• #10830 [v23.1.x] cloud_storage: escape hatch to reset metadata partition manifest by @andrwng
• #10924 [v23.1.x] Change node_health_report::topics to be ss::chunked_fifo by @rockwotj
• #11058 [v23.1.x] Breakdown installation of ducktape dependencies in multiple bash scripts by @gousteris
• #11071 [v23.1.x] Treat RpkException as retryable in _node_leadership_evacuated by @dlex

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.1.11...v23.1.12

Bug Fixes

• Fix bug caused by unexpected exception on the remote read path which caused Kafka fetch requests to time-out and leave lingering connections. by @VladLazar in #10916
• #10964 rpk disk_irq tuner now provides a warning for a known issue introduced in kernel 5.17 where instances utilizing MSI IRQ may encounter an empty IRQs list in sysfs that caused hwloc segmentation faults and results in a tuner failure. by @r-vasquez in #10975
• close Redpanda admin connection after reconciling the Console user and ACL. by @joejulian in #11052
• PR #10869 [v22.3.x] cloud_storage: fix empty segment check in reader by @jcsp
• PR #10966 [v22.3.x] rpc: Log broken semaphore errors as timeout by @abhijat
• PR #11013 [v22.3.x] k8s: parse configuration arrays correctly by @alejandroEsc

Improvements

• net: Handle GNUTLS_E_INVALID_SESSION by reconnecting. by @BenPope in #11030
• PR #10858 [v22.3.x] Cache observability improvements by @jcsp
• PR #10880 [v22.3.x] cluster: include ntp in timequery logs by @jcsp
• PR #11034 k8s: Fix redundant imports by @RafalKorepta

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v22.3.20...v22.3.21

Bug Fixes

• Fixed a bug caused by unexpected exception on the remote read path which caused Kafka fetch
  requests to time-out and leave lingering connections. by @VladLazar in #10917
• #10886 rpk disk_irq tuner now provides a warning for a known issue introduced in kernel 5.17 where instances utilizing MSI IRQ may encounter an empty IRQs list in sysfs that caused hwloc segmentation faults and results in a tuner failure. by @r-vasquez in #10904
• #10989 An issue is fixed where clients might see a lower partition high watermark than expected if querying very soon after a new raft leader is elected for the partition. by @jcsp in #10990
• k8s: Status.Version is not updated until upgrade to that version is finished by @alenkacz in #10988
• PR #10866 [v23.1.x] cloud_storage: fix empty segment check in reader by @jcsp
• PR #10883 [v23.1.x] Change topic_status partitions to be a chunked_fifo by @rockwotj
• PR #11004 [v23.1.x] archival: fix segment merging running when disabled by @jcsp
• PR #11008 [v23.1.x] tests: use timeouts on http requests to avoid potential runner hangs by @jcsp
• PR #11012 [v23.1.x] k8s: parse configuration arrays correctly by @alejandroEsc

Improvements

• Fixed increased produce latency related with deleting segments by @mmaslankaprv in #10973
• #11031 net: Handle GNUTLS_E_INVALID_SESSION by reconnecting. by @BenPope in #11032
• PR #10675 [v23.1.x] tests: disable many_partitions_tiered_storage by @andrwng
• PR #10859 [v23.1.x] Cache observability improvements by @jcsp
• PR #10881 [v23.1.x] cluster: include ntp in timequery logs by @jcsp
• PR #10969 [v23.1.x] rpc: Log broken semaphore errors as timeout by @abhijat
• PR #10987 [v23.1.x] operator: run goimports to normalize import formatting by @joejulian
• PR #11009 [v23.1.x] cloud_storage: improved debug on partition manifest parse error by @jcsp
• PR #11020 k8s: Remove nolinter linter comment by @RafalKorepta

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.1.10...v23.1.11

Bug Fixes

• PR #10625 [v23.1.x] tests: wait for recovery rate only on active brokers by @bharathv
• PR #10688 cloud_storage: Handle overlapping segments by @abhijat
• PR #10714 [backport v23.1.x] cloud_storage: Adjust conditions to reset stuck reader by @abhijat
• PR #10735 [v23.1.x] async-clear: avoid repeated calls to begin() in the loop. by @bharathv
• PR #10766 [v23.1.x] k8s: Rename e2e test K8S namespace by @RafalKorepta
• PR #10817 [v23.1.x] tests: use pagination on releases API by @jcsp
• PR #10866 [v23.1.x] cloud_storage: fix empty segment check in reader by @jcsp
• PR #9414 c/topic_table: Fix topic de-allocations with canceled updates by @bharathv

Improvements

• Improve support for IPv6 listeners. by @BenPope in #10678
• PR #10700 [v23.1.x] Soft assert on different sizes of shard_fetch collections by @dlex
• PR #8895 More flexible constraints solver interface by @mmaslankaprv
• PR #9251 Make it easier to expressed chained constraints by @mmaslankaprv
• PR #9413 c/partition_allocator: remove unused method by @ztlpn
• PR #10408 Serialization for seastar::chunked_fifo by @mmaslankaprv
• PR #10416 Removed large contiguous allocations from the topic creation path by @mmaslankaprv

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.1.9...v23.1.10

Features

• PR #10185 Add cloud storage escape hatch to reset manifset by @andrwng

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v22.3.19...v22.3.20

Bug Fixes

• #10694 map replicated_entry_truncated error to 503 http error code by @ZeDRoman in #10695
• PR #10701 [v22.3.x] Soft assert on different sizes of shard_fetch collections by @dlex
• PR #10715 [backport v22.3.x] cloud_storage: Adjust conditions to reset stuck reader by @abhijat

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v22.3.18...v22.3.19

Bug Fixes

• Schema Registry: Fix a bug in GET /subjects/<subject>/versions/latest that would previously not find the latest non-deleted version. by @BenPope in #10555
• #10694 map replicated_entry_truncated error to 503 http error code by @ZeDRoman in #10695
• PR #10414 [v22.3.x] storage: switch to node_hash_map for kvstore by @jcsp
• PR #10443 [v22.3.x] cluster: Switch partition_leaders_table to use absl::node_hash_map by @StephanDollberg
• PR #10461 [v22.3.x] Set promise on exceptions in dispatch_method_once by @ballard26
• PR #10472 [v22.3.x] Remove duplicate initialization of chunk_cache by @ballard26
• PR #10485 [v22.3.x] Buffer record_batches using fragmented vector by @rockwotj
• PR #10570 [v22.3.x] c/backend: s/parallel_for_each/max_concurrent_for_each by @bharathv
• PR #10617 [v22.3.x] Do not recommission a node when it is demoted to learner by @mmaslankaprv
• PR #10618 [v22.3.x] c/topic_table: switch _pending_deltas to fragmented_vector by @ztlpn
• PR #10638 [v22.3.x] Validate raft response node by @bharathv
• PR #10640 [v22.3.x] ducktape: NodeOpsExecutor ensure node not empty before DE_RECOMMISSION by @ZeDRoman
• PR #10693 [v22.3.x] cloud_storage: Handle overlapping segments by @VladLazar

Improvements

• Pandaproxy: The client identifier is now pandaproxy_client by @BenPope in #10565
• Schema Registry: The client identifier is now schema_registry_client by @BenPope in #10565
• one faulty raft group will not influence others by @mmaslankaprv in #10647
• PR #10296 [v22.3.x] tests: permit OMB log error by @jcsp
• PR #10433 [v22.3.x]tests/setup: updated confluent_kafka version by @mmaslankaprv
• PR #10451 [v22.3.x] config: validate that strings are non-empty by @jcsp
• PR #10627 [v22.3.x] storage: drop clean_segment record on truncate by @jcsp

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v22.3.17...v22.3.18

Bug Fixes

• Disable adjacent segment merging when the partition is not a leader by @Lazin in #10540
• Fix race condition during forced segment roll by @Lazin in #10533
• Schema Registry: Fix a bug in GET /subjects/<subject>/versions/latest that would previously not find the latest non-deleted version. by @BenPope in #10556
• The PR resolves, if requireClientAuth is enabled and clientCACertRef is NOT set for panda proxy or schema registry TLS listener, the Redpanda pods will not come up due to the error below. by @paulzhang97 in #10600
• #10085 Fixed a lock ordering inversion that could lead the upload loop to occasionally timeout. by @andrwng in #10415
• #10426 Made Join and configuration change validation logic consistent by @mmaslankaprv in #10427
• #10596 Fixed an issue where sq or sq-split mode may be chosen for NIC IRQ affinity in the redpanda tuner, resulting in core/shard 0 becoming a bottleneck for some high packet-per-second loads. by @travisdowns in #10606
• #10656 Fixed oversized allocation in cloud_storage::walk_accumulator::visit by @michael-redpanda
• #10619 Fix overallocation in cluster::get_leadership_reply by @michael-redpanda

Improvements

• Console supports the annotation operator.redpanda.com/finalizers-timeout to indicate a timeout for finalizers to be removed by @nicolaferraro in #10599
• Pandaproxy: The client identifier is now pandaproxy_client by @BenPope in #10563
• Schema Registry: The client identifier is now schema_registry_client by @BenPope in #10563
• #10494 Better recovery bandwidth utilization and faster reconfigurations by @bharathv in #10505
• one faulty raft group will not influence others by @mmaslankaprv in #10646

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.1.8...v23.1.9

Features

• Schema Registry: Support for Protobuf known types by @BenPope in #10167
• Support mTLS with customer provided client CA at Panda Proxy and Schema Registry. by @paulzhang97 in #10148

Bug Fixes

• Convert tm_snapshot to fragmented_vector to reduce memory pressure. by @BenPope in #10056
• Fix a bug in AlterConfigs which caused cleanup.policy to be un-set unless explicitly overriden. The creation time default is cleanup.policy=delete and AlterConfigs now enforces it. by @VladLazar in #10380
• Fixed issue with offset translation in rm_stm on snapshot hydrate by @mmaslankaprv in #10300
• #10059 This change fixes an out of memory in the replicate batcher by ensuring that only one flush task is outstanding at any time. by @travisdowns in #10060
• #10113 A bug is fixed where if a PUT to object storage was in flight during shutdown, Redpanda might incorrectly record a failed upload as successful. by @jcsp in #10114
• #10175 Pandaproxy will no longer assert if attempting to consume compressed message batches by @michael-redpanda in #10176
• #10175 Schema Registry will no longer assert if the _schemas topic contains compressed batches by @michael-redpanda in #10176
• #10254 A stability issue is fixed where many concurrent Produce requests using very large compressed batches could exhaust memory. by @jcsp in #10255
• #10254 A stability issue is fixed where very large ZSTD-compressed batches could exhaust memory by @jcsp in #10255
• #10333 Fixed an issue where the metric redpanda_kafka_consumer_group_consumers was reporting double the real count of consumers. by @NyaliaLui in #10334
• #10355 The consumer used in our internal kafka client will automatically find the new consumer group coordinator on not_coordinator errors. by @NyaliaLui in #10356

Improvements

• Added storage_ignore_timestamps_in_future_sec cluster configuration property (default null). If set to non-null, then timestamps more than this many seconds in the future will be ignored by Redpanda when considering whether a segment is old enough to garbage collect. by @jcsp in #10236
• Console is automatically reconciled when the linked Redpanda cluster changes state by @nicolaferraro in #10128
• Reduced memory consumption on configurations with very large numbers of small segments. by @jcsp in #10044
• Shutting down a server with many hydrated tiered storage segments and partitions is now significantly faster. by @andrwng in #9239
• The shutdown sequence for partitions that use tiered storage is now faster in clusters with heavy read traffic that hydrates readers from object storage. by @andrwng in #9933
• #10292 vectorized_internal_rpc_requests_pending now has better description by @daisukebe in #10293

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v22.3.16...v22.3.17

Features

• Schema Registry: Support for Protobuf known types by @BenPope in #10166
• Support mTLS with customer provided client CA at Panda Proxy and Schema Registry. by @paulzhang97 in #10147

Bug Fixes

• Convert tm_snapshot to fragmented_vector to reduce memory pressure. by @BenPope in #10055
• Fix a bug in AlterConfigs which caused cleanup.policy to be un-set unless explicitly overriden. The creation time default is cleanup.policy=delete and AlterConfigs now enforces it. by @VladLazar in #10384
• Fix a bug where recovered topics would use the retention settings specified at creation time (ignoring updates after that point). by @VladLazar in #10384
• Fix offset translation errors after topic recovery by @Lazin in #10230
• #10071 Fixes violation of the atomicity of the consumer - transform - produce loop by @rystsov in #10072
• #10115 A bug is fixed where if a PUT to object storage was in flight during shutdown, Redpanda might incorrectly record a failed upload as successful. by @jcsp in #10116
• #10173 Pandaproxy will no longer assert if attempting to consume compressed message batches by @michael-redpanda in #10174
• #10173 Schema Registry will no longer assert if the _schemas topic contains compressed batches by @michael-redpanda in #10174
• #10252 A stability issue is fixed where many concurrent Produce requests using very large compressed batches could exhaust memory. by @jcsp in #10253
• #10252 A stability issue is fixed where very large ZSTD-compressed batches could exhaust memory by @jcsp in #10253
• #10273 Fixed issue with offset translation in rm_stm on snapshot hydrate by @mmaslankaprv in #10274
• #10275 Prevent allocation failures with many idempotent producers by @rockwotj in #10276
• #10331 Fixed an issue where the metric redpanda_kafka_consumer_group_consumers was reporting double the real count of consumers. by @NyaliaLui in #10332
• #10353 The consumer used in our internal kafka client will automatically find the new consumer group coordinator on not_coordinator errors. by @NyaliaLui in #10354

Improvements

• Console is automatically reconciled when the linked Redpanda cluster changes state by @nicolaferraro in #10127
• Fix reactor stalls when serializing/deserializing large snapshots. by @bharathv in #10321
• #10045 Reduced "partial message" error logging when object storage backends uncleanly terminate connections. by @jcsp in #10046
• #10290 vectorized_internal_rpc_requests_pending now has better description by @daisukebe in #10291
• #10395 Some Kafka API requests are now exempt from the node-wide throughput limiting, so that functions that are not directly related to the data path can stay responsive even in heavily throttled configurations. by @dlex in #10396
• #10410 Improved CPU balance between fetch requests and other system activity when there is a high fetch load. Previously, fetchers reading from the tip of the log could partly starve out other types of work (which produce a smaller number of tasks) causing overall poor performance, but now fetch requests have less impact on other tasks running on the CPU. by @travisdowns in #10410

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.1.7...v23.1.8

Bug Fixes

• Fix cases where sending very large batches to compacted topics using LZ4 or gzip compression could result in bad_alloc errors. by @jcsp in #9721
• Fix offset translation failure that could be triggered after topic recovery by @Lazin in #9871
• Fixed a bug in which the segment merging could run from read replicas. by @andrwng in #9757
• Fixed another bug where multiple configuration updates would cause an inconsistency upon next reload of usage state by @graphcareful in #9965
• Fixed excessive kvstore writes that could lead to memory fragmentation issues during heavy produce load. by @ztlpn in #9960
• Fixes a bug in our http client that may crash redpanda in exceptional cases by @graphcareful in #9922
• Fixes a bug where usage_manger would report incorrect cloud storage metrics when queried on the leader node. by @graphcareful in #9965
• #10036 Fixed issue where Redpanda will assert if data being consumed by Pandaproxy is not JSON serializable by @michael-redpanda in #10038
• #10068 The vectorized_memory_available_memory metric is now available on the /metrics endpoint in addition to /public_metrics. by @travisdowns in #10069
• #9651 Improve precision of the adjacent segment merger to avoid unnecessary segment reuploads by @Lazin in #9845
• #9859 fix consumer group kafka incompatibilities (update offset_fetch to return committed_leader_epoch) by @rystsov in #9860
• #9873 An issue is fixed where time queries on tiered storage partitions could return -1 incorrectly if the queried timestamp was earlier than the start of the log. by @jcsp in #9874
• #9873 An issue is fixed where time queries on tiered storage partitions using offsets close to the end of the local raft log could sometimes return offsets slightly ahead of the correct offset, if the segment containing the correct offset had already been offloaded to object storage. by @jcsp in #9874
• operator: fixed an issue that caused continue reconciliations on the custom resource by @nicolaferraro in #9979

Improvements

• Adds a timeout parameter to transfer_leadership_request to change the recovery timeout. by @ballard26 in #10073
• Admn API: GET /v1/security/users excludes ephemeral users unless specified otherwise. by @BenPope in #9952
• Duplicate PUT/POST/DELETE admin API requests for user management are now handled more efficiently. by @jcsp in #9855
• Maximum throttle time for Kafka API requests reduced to 30s to avoid heartbeat timeouts in default client configurations. by @dlex in #9985
• Operator allows referencing secrets that do not yet exist as certificate holders for listeners by @nicolaferraro in #9898
• Reduced memory consumption on configurations with very large numbers of small segments. by @jcsp in #10043
• Spurious end_of_stream log messages under normal operation are reduced. by @jcsp in #9972
• String configuration properties which ought not to be set to an empty string are now validated as such. This reduces risk of issues from tools that might treat "" equivalently to not setting a property. by @jcsp in #9805
• #9810 The /v1/brokers admin API endpoint now includes the broker's internal_rpc_address and internal_rpc_port. by @jcsp in #9811
• #9825 schema_registry: Allow a schema id and version to be supplied by @BenPope in #9826

Features

• Adds a configurable option to have topic-aware leadership balancer. by @ballard26 in [#7997] (https://github.com/redpanda-data/redpanda/pull/7997)

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.1.6...v23.1.7

Bug Fixes

• Fix a rare race between segment rolling and the readers cache which caused spurious, temporary dead-locks by @VladLazar in #9708
• Fix cases where sending very large batches to compacted topics using LZ4 or gzip compression could result in bad_alloc errors. by @jcsp in #9722
• Fix offset translation failure that could be triggered after topic recovery by @Lazin in #9872
• Fix race between compaction and application of retention to the local log. This occurred when compaction
  happened below the start offset of the log. It did not have a noticeable impact upon user workloads. by @VladLazar in #9612
• Fixed excessive kvstore writes that could lead to memory fragmentation issues during heavy produce load. by @ztlpn in #9962
• Fixes a bug in our http client that may crash redpanda in exceptional cases by @graphcareful in #9924
• #10037 Fixed issue where Redpanda will assert if data being consumed by Pandaproxy is not JSON serializable by @michael-redpanda in #10039
• #8987 An issue is fixed where a cluster might not activate new functionality after an upgrade if a controller leadership change happened at a particular point during upgrade by @jcsp in #8988
• #9513 Fixed a bug that would result in read replicas reporting a high watermark that was too high. by @andrwng in #9727
• #9680 fixes an assertion that may happen when Raft snapshot is delivered to recover follower having any of rm tx or id_allocator stms by @mmaslankaprv in #9681
• #9861 fix consumer group kafka incompatibilities (update offset_fetch to return committed_leader_epoch) by @rystsov in #9862
• #9875 An issue is fixed where time queries on tiered storage partitions could return -1 incorrectly if the queried timestamp was earlier than the start of the log. by @jcsp in #9876
• #9875 An issue is fixed where time queries on tiered storage partitions using offsets close to the end of the local raft log could sometimes return offsets slightly ahead of the correct offset, if the segment containing the correct offset had already been offloaded to object storage. by @jcsp in #9876

Improvements

• Admn API: GET /v1/security/users excludes ephemeral users unless specified otherwise. by @BenPope in #9957
• Duplicate PUT/POST/DELETE admin API requests for user management are now handled more efficiently. by @jcsp in #9856
• Improves memory management with transactions and idempotency by avoiding large contiguous allocations. by @bharathv in #9632
• Partition movements will not automatically make the partition under replicated by @mmaslankaprv in #9510
• Reduced memory consumption on configurations with very large numbers of small segments. by @jcsp in #10044
• #9825 schema_registry: Allow a schema id and version to be supplied by @BenPope in #9828
• rpk: decommission command fails with a friendly message when a node is dead by @daisukebe in #9699

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v22.3.15...v22.3.16

Features

• Remove ghost Redpanda Node IDs by @joejulian in #9548
• Validation for mounted Redpanda data dir in init container by @joejulian in #9548

Bug Fixes

• Always reconcile Node IDs in Pod annotation by @joejulian in #9548
• Fix an issue with compacted segment reupload that could cause self-concatenated segments to
  be reuploaded. by @VladLazar in #9793
• Remove maintenance mode when the broker comes back up during rolling upgrade by @joejulian in #9548
• #9513 Fixed a bug that would result in read replicas reporting a high watermark that was too high. by @andrwng in #9770
• fixed a bug in rpk cluster storage recovery which rendered the status command unusable. by @r-vasquez in #9801

Improvements

• Bump cert-manager dependency version by @joejulian in #9548
• The shutdown sequence for partitions that use tiered storage is now faster in clusters with heavy read traffic that hydrates readers from object storage. by @andrwng in #9758

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.1.4...v23.1.6

Bug Fixes

• Fix a rare race between segment rolling and the readers cache which caused spurious, temporary dead-locks by @VladLazar in #9706
• Fixed a bug that would previously prevent read replicas from progressing if cloud_storage_enable_remote_write is false. by @andrwng in #9704
• Makes starting Redpanda with long controller log containing redundant topic properties update faster by @mmaslankaprv in #9743
• Makes starting Redpanda with long controller log containing redundant topic properties update faster by @mmaslankaprv in #9744
• #9678 fixes an assertion that may happen when Raft snapshot is delivered to recover follower having any of rm tx or id_allocator stms by @mmaslankaprv in #9679
• #9762 Fixed automatic rebalancing of replicas stopping to early before achieving even distribution of replicas. by @mmaslankaprv in #9763
• fixes rare issue that may arise when new node id is assigned to a node with the same set of ip addresses by @mmaslankaprv in #9650

Improvements

• Adds a new metering API for fine grained tracking of kafka metrics by @graphcareful in #9666
• Adds cloud storage metrics to usage reporting by @vbotbuildovich in #9670
• Improves memory management with transactions and idempotency by avoiding large contiguous allocations. by @bharathv in #9626
• Partition movements will not automatically make the partition under replicated by @mmaslankaprv in #9511
• #9481 all topics are evenly distributed after nodes are added to the cluster by @mmaslankaprv in #9482
• #9658 Changes the path for restarting a Redpanda Service via the Admin API. Previously users would issue PUT /v1//redpanda-services/restart?service=schema-registry, now users issue PUT /v1/debug/restart_service?service=schema-registry by @NyaliaLui in #9659
• rpk: decommission command fails with a friendly message when a node is dead by @daisukebe in #9698
• #9611 storage: do not compact below log start offset by @VladLazar

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.1.3...v23.1.4