redpanda

Bug fixes

• https://github.com/redpanda-data/redpanda/issues/2619 k8s: After cluster config is changed in CR, the redpanda pods will get restarted for the config to get used. Beware that updating to this version of operator will always trigger one restart of your managed redpanda clusters after it's updated. by @alenkacz in https://github.com/redpanda-data/redpanda/pull/3723
• https://github.com/redpanda-data/redpanda/issues/3277 https://github.com/redpanda-data/redpanda/issues/3328 Fixes for node decommissioning in face of failures. by @mmaslankaprv in https://github.com/redpanda-data/redpanda/pull/3631
• https://github.com/redpanda-data/redpanda/issues/3316 Fix shadow indexing error that occurred when the partition was moved between the nodes. by @Lazin in https://github.com/redpanda-data/redpanda/pull/3632
• https://github.com/redpanda-data/redpanda/issues/3429 https://github.com/redpanda-data/redpanda/issues/3561 Better stack traces in case of crash on low memory. by @LenaAn in https://github.com/redpanda-data/redpanda/pull/3672
• https://github.com/redpanda-data/redpanda/issues/3486 Fix an issue where nodes may have stale leadership metadata for a short period after a node restarts. by @mmaslankaprv in https://github.com/redpanda-data/redpanda/pull/3631
• https://github.com/redpanda-data/redpanda/issues/3548 Shadow indexing memory utilization was optimised. by @Lazin in https://github.com/redpanda-data/redpanda/pull/3660
• https://github.com/redpanda-data/redpanda/issues/3639 k8s operator: the webhook now guards against decrease of the number of assigned cores. by @nicolaferraro in https://github.com/redpanda-data/redpanda/pull/3724
• https://github.com/redpanda-data/redpanda/issues/3649 Pandaproxy: An inactive consumer is now timed out after pandaproxy.consumer_instance_timeout_ms. by @BenPope in https://github.com/redpanda-data/redpanda/pull/3673
• https://github.com/redpanda-data/redpanda/issues/3649 Pandaproxy: Creating a consumer with an existing name now fails with {"error_code": 40902}. by @BenPope in https://github.com/redpanda-data/redpanda/pull/3673
• https://github.com/redpanda-data/redpanda/issues/3649 Pandaproxy: Improve header validation and Swagger API. by @BenPope in https://github.com/redpanda-data/redpanda/pull/3673
• https://github.com/redpanda-data/redpanda/issues/3650 Schema Registry: Fix a crash when publishing multiple protobuf schema. by @BenPope in https://github.com/redpanda-data/redpanda/pull/3673
• https://github.com/redpanda-data/redpanda/issues/3650 Schema Registry: Support protobuf encoded protobuf schema for compatibility with Protobuf Serializers. by @BenPope in https://github.com/redpanda-data/redpanda/pull/3673
• Fixed cleanup policy application when using cloud storage or transactions. by @mmaslankaprv in https://github.com/redpanda-data/redpanda/pull/3758
• Fixed error preventing redpanda from starting if the only partition movement operation involved x-core move. by @mmaslankaprv in https://github.com/redpanda-data/redpanda/pull/3674
• Fixed metadata dissemination in newly created clusters. by @mmaslankaprv in https://github.com/redpanda-data/redpanda/pull/3674
• Fixes a panic in rpk group seek if there was an error during the offset commit. by @twmb in https://github.com/redpanda-data/redpanda/pull/3759
• Not forcing consumers to rejoin group when transferring group partition leadership. by @mmaslankaprv in https://github.com/redpanda-data/redpanda/pull/3758

Features

• https://github.com/redpanda-data/redpanda/issues/2707 Metrics reporter. by @mmaslankaprv in https://github.com/redpanda-data/redpanda/pull/3727

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v21.11.3...v21.11.4

Bug fixes

• Fix a consistency issue with transactions. by @rystsov in https://github.com/vectorizedio/redpanda/pull/3247
• https://github.com/vectorizedio/redpanda/issues/1275 Fix kafka server send group topic partition offset metric to prometheus. by @ZeDRoman in https://github.com/vectorizedio/redpanda/pull/3440
• https://github.com/vectorizedio/redpanda/issues/3263 k8s: fix bug reconciling clusters with fixed nodeport. by @alenkacz in https://github.com/vectorizedio/redpanda/pull/3281
• https://github.com/vectorizedio/redpanda/issues/3383 k8s: fix bug in external port binding when port is specified in configuration. by @alenkacz in https://github.com/vectorizedio/redpanda/pull/3387
• https://github.com/vectorizedio/redpanda/issues/3211 Schema Registry: Fix a crash during compatibility checks. by @BenPope in https://github.com/vectorizedio/redpanda/pull/3222
• https://github.com/vectorizedio/redpanda/issues/3305 Fix archival upload data corruption. by @Lazin in https://github.com/vectorizedio/redpanda/pull/3309
• Fix incorrect handling of failed snapshot delivery that may lead to situation in which snapshot is being redelivered in tight loop. by @mmaslankaprv in https://github.com/vectorizedio/redpanda/pull/3286
• https://github.com/vectorizedio/redpanda/issues/2142 https://github.com/vectorizedio/redpanda/issues/2568 Better handling of consumer group related errors. by @mmaslankaprv in https://github.com/vectorizedio/redpanda/pull/3286
• https://github.com/vectorizedio/redpanda/issues/3323 Fix rare crash that could happen when log segments eviction happened concurrently with fetching near the start of the log. by @ztlpn in https://github.com/vectorizedio/redpanda/pull/3541
• https://github.com/vectorizedio/redpanda/issues/3324 https://github.com/vectorizedio/redpanda/issues/3334 Fix disk space issues. by @Lazin in https://github.com/vectorizedio/redpanda/pull/3373
• https://github.com/vectorizedio/redpanda/issues/3378 cloud_storage: ensure readers only destroyed in remote_partition. by @jcsp in https://github.com/vectorizedio/redpanda/pull/3448
• https://github.com/vectorizedio/redpanda/issues/3334 cloud_storage: fix file descriptor leak. by @jcsp in https://github.com/vectorizedio/redpanda/pull/3437
• archival: fix bug in logging on success. by @LenaAn in https://github.com/vectorizedio/redpanda/pull/3512
• https://github.com/vectorizedio/redpanda/issues/3494 k8s: fix how schema reg node cert is mounted. by @simon0191 in https://github.com/vectorizedio/redpanda/pull/3568
• https://github.com/vectorizedio/redpanda/issues/3528 fix possible deadlock of raft groups. by @mmaslankaprv in https://github.com/vectorizedio/redpanda/pull/3566
• fix regression that caused an assertion to be triggered during Fetch request handling. by @mmaslankaprv in https://github.com/vectorizedio/redpanda/pull/3597
• Shadow Indexing (Tech Preview): Kubernetes operator: All Redpanda nodes created through the Kubernetes operator will have a default cloud storage maximum upload interval (cloud_storage_segment_max_upload_interval_sec) of 30 minutes. by @RafalKorepta in https://github.com/vectorizedio/redpanda/pull/3241
• https://github.com/vectorizedio/redpanda/issues/3272 Shadow Indexing (Tech Preview): Fix data loss in shadow indexing archived data that could occur after quick partition leadership transfer back and forth between two nodes. Compatibility note: previous redpanda versions won't be able to read shadow indexing data archived by newer versions. by @ztlpn in https://github.com/vectorizedio/redpanda/pull/3434

Features

• schema_registry: Support GET /subjects/{subject}/versions/{version}/referencedBy. by @BenPope in https://github.com/vectorizedio/redpanda/pull/3402
• Cluster level configuration parameter cloud_storage_enable_remote_read can be used to enable shadow indexing fetch for all topics (disabled by default). Cluster level configuration parameter cloud_storage_enable_remote_write can be used to enable archival uploads for all topics (disabled by default). by @Lazin in https://github.com/vectorizedio/redpanda/pull/3248

Improvements

• https://github.com/vectorizedio/redpanda/issues/3269 redpanda/cluster: improve logging for leader_balancer. by @BenPope in https://github.com/vectorizedio/redpanda/pull/3284
• https://github.com/vectorizedio/redpanda/issues/3304 Gracefully handle a situation when segment in S3 is truncated. Before client would be stuck in a cycle trying to proceed, not client will get an error. by @LenaAn in https://github.com/vectorizedio/redpanda/pull/3343
• https://github.com/vectorizedio/redpanda/issues/3428 Improved stability when doing large kafka fetch requests under low memory conditions. by @jcsp in https://github.com/vectorizedio/redpanda/pull/3472
• Consuming from a very large number of partitions at once is now subject to a per-request size limit, reducing the risk of exhausting memory if a client specifies a partition count and per-partition size limit that is greater than available RAM. The size limit per-fetch is set via the kafka_max_bytes_per_fetch configuration property, default 64MiB. by @jcsp in https://github.com/vectorizedio/redpanda/pull/3438
• https://github.com/vectorizedio/redpanda/issues/3412 Memory utilization on systems with large number of partitions can now be tweaked using configuration properties storage_read_buffer_size (default 128KiB) and storage_read_readahead_count (default 10). These properties may be decreased to more conservative values such as buffer_size=16KiB, readahead_count=1 to reduce the per-partition memory overhead and improve stability when the number of partitions is large (e.g. more than 10000). by @jcsp in https://github.com/vectorizedio/redpanda/pull/3436
• https://github.com/vectorizedio/redpanda/issues/3269 redpanda/cluster: improve logging for leader_balancer. by @Lazin in https://github.com/vectorizedio/redpanda/pull/3364
• https://github.com/vectorizedio/redpanda/issues/3333 Fine-tune logging. by @LenaAn in https://github.com/vectorizedio/redpanda/pull/3357
• https://github.com/vectorizedio/redpanda/issues/3322 cloud_storage: suppress warning on successfull topic creation by @LenaAn in https://github.com/vectorizedio/redpanda/pull/3484
• https://github.com/vectorizedio/redpanda/issues/3376 archival: make upload loop more resilient. by @Lazin in https://github.com/vectorizedio/redpanda/pull/3441
• cloud_storage: limit remote partition concurrency. by @Lazin in https://github.com/vectorizedio/redpanda/pull/3510
• https://github.com/vectorizedio/redpanda/issues/3310 archival: warnings generated when topic leadership changes. by @LenaAn in https://github.com/vectorizedio/redpanda/pull/3379

Bug fixes:

• Shadow indexing: Add per-topic configurations into topic describe
• Kafka API: Fix Sarama control record compatibility issue

Bug fixes:

• Idempotent producers: Add deduplication window
• Compatibility: Fix backward compatibility issues that may cause assertions
• Kafka API: Fix timequery ListOffsets API from returning an error in some cases

Features

• Shadow Indexing (Tech Preview): If cloud storage is enabled, start offset for the partition will take cloud data into account and fetches for data that was already deleted locally will go to cloud storage. A cluster with this feature enabled is not yet supported for usage.
• Schema Registry: Support protobuf schema.
• Cluster health monitor: Ability to query for cluster runtime parameters such as node versions, node availability, node resources and raft group status
• API: The v1/cluster_config/schema admin API endpoint is added for discovering available cluster configuration properties.
• API: Added support for manual partition assignment in CreateTopics API. Currently custom partition assignment is only used during initial partition allocation. Nodes decommissioning and automatic partition re-balancing may change the partition assignment
• rpk: The new rpk group command can be used to list groups, delete groups, describe groups, and modify group offsets. rpk cluster offsets has been deprecated.
• rpk: new topic add-partitions command to add partitions to existing topics

Improvements

• Admin API: For requests that must be submitted to the controller leader node, or to the leader for a particular partition: other nodes will now respond with an HTTP 307 redirect to the leader rather than an HTTP 400 error.
• configuration: describe_config now supports key specification in configuration_keys field.
• install: Add Archlinux support for installDependencies.sh
• k8s: External ports can now be specified in the Kafka API external listener configuration. When this port is specified, and if it's not available on all the nodes, redpanda will fail to schedule.
• k8s: Operator now by default creates PodDisruptionBudget that defaults to MaxUnavailable=1 and can be configured via CRD
• k8s: rpk-status sidecar is now disabled by default
• k8s: rpk-status sidecar resource limits were increased from 10M to 30M
• k8s: Operator depenencies bumped to 1.21.
• k8s: Operator can now create ingresses for Kubernetes clusters by default.
• raft: Initial leadership information for newly created topics is set more uniformly across the cluster, avoiding some client delays when accessing a newly created topic that has not gone through a leadership election yet.
• rpk: The fstrim tuner, which enables or installs the fstrim.service and fstrim.timer, is no longer enabled by default when running rpk redpanda mode production

Bug fixes

• ACL: Fix wrong ACL user create arguments
• Idempotent producers: Fix bug affecting idempotent producers, reducing the rate of fatal errors
• kafka/client: Fix retry handling for several operations
• kafka/client: Fix concurrency bug with retry mitigation function
• kafka/server: fix double counting bytes sent metric
• schema_registry: Prevent deleting referenced schema
• schema_registry/store: Use correct shard in get_subject_schema

Note that downgrades from 21.11.1 to earlier releases are not supported.

v21.10.2 contains fixes for the following issues and areas:

• Reconfiguration availability
• Violation of idempotent produces
• Panda proxy client retires
• Bug that prevented deleting ACLs by transactional ID
• Bug that prevented rpk from creating deny ACLs

Release notes for 21.10.1

• Upgrading franz-go in rpk for various bug fixes
• Improvements to ACL handling in rpk
• Fixes in schema registry for correct shard
• Archival metadata STM
• rpk now uses franz-go instead of sarama
• General improvements to log messages
• Community build now works
• Normalised RPK commands descriptions, so all descriptions are consistent - starting with upper letter and ending with dot.
• Lots of improvements to testing framework and tests
• An issue is fixed where partition leadership transfers would sometimes fail to complete when the partition was under load.
• Fixes for snappy producers that lead to OOM
• General bug fixes in Raft
• rpk topic delete now accepts the -r flag to delete any topics completely matching regular expressions
• rpk topic list now accepts specific topics for listing, as well as the -r flag, so that the command can be used as a dry run for delete
• rpk container: This change fixes a bug that fails to correctly set the schema registry port sPort when creating the port bindings.
• raft: fix for one of the replicas not being created when moved cross cores #2642
• coproc: support for most native topic operations on materialized topics
• rpk: add the ability to print offsets for empty groups
• Admin API now has more validation for inputs
• rpk acl user create now accepts the user as an argument, and --new-password has been renamed to --password. These changes are backwards compatible; using the old format will continue to work for the time being.
• rpk acl user delete now accepts the user to delete as an argument. Similar to the user create change, the --delete-username flag has been soft deprecated and will continue working for now.
• rpk acl user create supports a new flag --mechanism which allows for using SCRAM-SHA-512.
• Fix for a seg fault using transactions
• k8s:fix values being parsed as scientific notation
• Fixes for consumer group early startup race leading to serving request on old versions of group data
• A bug is fixed where leadership transfers on partitions using acks=1 producers could lead to partitions becoming unwritable until the node restarts.
• Plugins will now be named with a .rpk- prefix.
• rpk topic produce and rpk topic consume now support expansive formatting options
• rpk topic produce by default now requires a trailing newline, and strips that newline before producing
• Feature: Topic recovery from archival storage
• Fixes for partition movement
• Consumer group state is now GCd from memory when possible
• Fixes for return codes that result in better behaving clients while partitions are moved
• schema registry: Release note: Avro name sanitization is now namespace aware.
• New feature: Fully functional create partitions support.
• cluster: A bug is fixed where redpanda could stop unexpectedly when topic properties were modified at the same time as a partition within the topic was being moved between nodes.
• Fix for rare off-by-one kafka offset being exposed to client
• Logging of IP connection information for client kafka connection errors (e.g. auth) to amke it easier to debug deployments that have lots of clients
• Redpanda now logs the system file handle limit on startup, to help diagnose issues on systems where the limit may be too small for the number of log segments.
• raft: In some situations, leadership of a topic's raft group could change unexpectedly when a non-leader node restarts. After this change, leadership should remain stable through follower node restarts, notwithstanding any intentional leadership changes orchestrated by the new leadership balancing feature.
• Improvement deriving core counts when using dockerized redpanda
• raft: fixed critical bug related to log truncation and join consensus
• archival: ability to limit upload idle time
• Provide user reliable way for querying partition movement status.
• storage: A new configuration setting append_chunk_size enables tuning the disk write size (default is 16kB, same as what was previously hardcoded). No change to behavior unless the setting is actively changed by the operator.

Bug fixes

• Ensure that the last applied offset is persisted only when it is flushed, fixing a potential stall that might happen during Redpanda startup after a crash
• Fix segmentation fault when consumer was active while Redpanda was shutting down
• Fix allocating to a large contiguous buffer when decompressing Snappy-compressed batches

Bugfix release:

• fixed problem leading to unrecoverable state of consumer groups coordinator, after group deletion
• fixed issue leading to crash after disabling transactions
• fixed rendering numeric properties in K8S operator

Bug fixes:

• Partition movement fixes (#2454)
• Added group state cleanup when leadership is lost (#2529)
• Fixed shutting down log (#2473)
• ParamsFromConfig: return early on FlagConfig (#2536)

Bug fixes

• Schema registry: Avro name sanitization is now namespace aware #2480
• Idempotency: Make id_allocator_stm support handle_eviction #2481
• Fully functional create partitions support #2486

Improvements

• Networking: add TCP send/recv buffer size settings
• Raft: introduced a couple of properties to control concurrency

Bug fixes

• Partition allocation: ensure constaints are applied to all partitions that are allocated
• Operator: multiple bug fixes to improve certificate handling

Features

• Automatic leadership rebalancing: Automatically rebalance Raft leadership of partitions across nodes in the cluster when a node becomes unavailable or available again.
• Schema registry: Support a schema registry, with support for the Avro format.
• Transactions (Tech Preview): Support the Kafka Transactions API

Improvements

• Operator: On k8s, by default we set transaction_coordinator_replication and id_allocator_replication to 3 when having at least 3 replicas.
• Operator: The Kubernetes operator now runs only one process in the main redpanda container which should prevent OOMs
• Raft: A new configuration option raft_heartbeat_disconnect_failures is added, defaulting to three missed heartbeats before connections are closed (i.e. reconnected). This should not need manual adjustment unless running on extremely overloaded systems which are emitting many Closed unresponsive connection log messages.

Bug fixes

• Automatic partition rebalancing: Bug fixes for corner cases
• Node decommissioning: Bug fixes for corner cases
• Seastar: Bug fix for hanging I/Os under heavy load
• Schema Registry:
  • Support enum defaults for avro compatibility checks
  • Allow deserializing 3rdparty record format
  • Introduce POST /subjects/{subject}
  • Introduce GET /schemas/ids/{id}/versions
  • Soft-deleting a subject no longer returns soft-deleted versions
  • Set the default compatibility to BACKWARDS
  • Allow set_compatibility on non-existent subject
  • Add tests for union
  • Weaken AVRO resolve check to support integer promotion on e.g., arrays
  • Fix error_code and message for subject_version_not_found
  • Fix error_code for schema_id_not_found
  • Remove delete_subject_value::version - this is a breaking change, but it hasn't been released yet
  • Permanently deleting a subject_version twice should return subject_version_not_found, instead of subject_not_found
  • Validate schema on insert to prevent invalid schemas reaching the topic

Releasing v21.7.6:

• Added configuration option for controlling if auto partition rebalance
  occurs when nodes are added. This option is defaulted to off after
  observing some poor performance after upgrading.

• Fixes for rpk's support of ipv6