nats-server

Changelog

Go Version

• 1.20.14

Improved

JetStream

• In lame duck mode, shutdown JetStream at the start to signal transfer of leadership if the leader (backported from #4579)
• Prevent processing of consumer assignments after JetStream shutdown occurs (backported from #4625)

Fixed

JetStream

• Fix possible stream assignment race condition (backported from #4589)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.23...v2.9.24

📣 Notice

This is the first preview release of 2.11.0! The motivation is to release new features and changes during the development lifecycle. This release is not feature complete for what the final 2.11.0 release will contain. This should only be used in development environments and not production. Additional preview releases will become available in subsequent weeks. Please check it out and give us feedback!

Changelog

Go Version

• 1.22.0 (#5133)

Added

Core

• Distributed pub/sub message tracing including OpenTelemetry support (#5014, #5057, #5078, #5121)
  • Refer to the ADR for more details

JetStream

• Support for batched direct-gets requests from streams (#4981, #5036)
  • Refer to the ADR for more details
• Add support for pausing/resuming consumers (#5066)
• Support for batched direct multi-get requests from streams (#5107)
  • Refer to the (in-progress) ADR for more details

Changed

MQTT

• Change retained messages to use headers for metadata rather and JSON-encoding the message (#5050)
  • Note this is currently a backwards incompatible change with 2.10.x servers.

Improved

General

• Track recovery from slow consumer events in routes (#5101)

OCSP

• Make OCSP request with POST method (#5109)

JetStream

• Prevent potential degradation of publish latency over time (#5116) Thanks to @stefanLeo for contributing the issue and the improvement!

Fixed

JetStream

• Add explicit structs for the account claims update response payload (#4942)
• Acking a redelivered msg with more pending outstanding that should move the ack floor (#5008)

MQTT

• Header length calculation for stored retained messages (#5125)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.11..v2.11.0-preview.1

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.7

Dependencies

• github.com/nats-io/nats.go v1.33.0

Improved

JetStream

• Optimize replaying messages when they are at the end of the stream (#5083) Thanks to @david-wakeo for the report!
• Optimize replaying messages with start sequence by time when there are large gaps between the time sequence and first sequence (#5088) Thanks again to @david-wakeo for the report!
• Reduce possible contention for NRG step-down (#4990)
• Reduce log-level to debug for non-actionable reallocations warnings (#5085)
• Improved placement of streams in larger clusters when created in rapid succession (#5079) Thanks to @kohlisid for the report!
• Improved filtered consumer retrieval time for large subject space streams with wildcards (#5089)

Fixed

JetStream

• Fixed a bug that could cause consumers to not match filtered subjects with wildcards correctly (#5080)
• Fixed a bug that could not allow keys/subjects to be found after restarts after extended downtime (#5054)
• Fixed a bug that would not properly update all consumer state when a filtered purge of the parent stream was narrower then the consumer’s filtered subject space (#5075)

MQTT

• Fixed an issue with retained messages not working properly if the server name had a ‘.’ (#5048)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.10...v2.10.11

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.6

Dependencies

• github.com/klauspost/compress v1.17.5
• github.com/nats-io/nats.go v1.32.0

Added

• Add ping_interval cluster route option to configure differently than client connections (#5029)

Improved

JetStream

• Avoid a linear scan of filestore blocks when loading messages for the first time after restart or long period of inactivity (#4969)
• NumPending calculations and subject index memory in filestore and memstore (#4960, #4983)
• Log healthy Raft group status when no longer falling behind (#4995)
• Optimize writing full filestore state where there is high subject cardinality (#5004)
• Share higher fidelity client info in JetStream advisory messages (#5019, #5026)
• More thorough gating of blocking disk I/O operations (#5022, #5027)
• Improved fallback cleanup of consumer Raft directory (#5028)

MQTT

• Use different unique names for internal consumers (#5017)

Fixed

Auth callout

• Ensure the server properly binds scoped users (#5013) Thanks to @dpotapov for the report!

Logging

• Fix missing variable in consumer skew log line (#4967)

JetStream

• Fix data race with stream stop and removal (#4963)
• Don't unconditionally send consumer snapshot when becoming leader (#4965)
• Fix tracking holes in the message block index when they are at the end of the block, improving filestore cache efficiency (#4988)
• Wait for recovery to complete before sending snapshot on stream scale-up from R1 (#5001)
• Acking a redelivered msg with more pending should move the ack floor (#5008)
• Fix for a panic calculating record length for secure erase followed by compaction (#5009)
• Reduce memory growth on interest stream recovery when consumers are ahead of the recovered stream state (#5011)
• Raft node responds to VoteRequest with outdated Term (#5021)
• Avoid sending meta leader snapshot as normal entry on leader change when not up-to-date (#5024)
• Make sure to not restart streams or consumers when they are deleted or immediately after being created (#5032)

Shutdown

• Avoid panic when reloading during lame duck mode (#5012)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.9...v2.10.10

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.6

Fixed

JetStream

• Guard against potential panic when finding starting sequence for stream sources from 2.9.x servers (#4950)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.8...v2.10.9

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.6

Dependencies

• golang.org/x/crypto v0.18.0
• golang.org/x/sys v0.16.0
• github.com/nats-io/nkeys v0.4.7

Added

TLS

• Add 'certs' option to TLS block for multi-cert support (#4889)

Improved

General

• Random number generation now uses a faster lock-free algorithm (#4901)
• no_auth_user is now allowed to be an nkey (#4938)

JetStream

• Improve matching efficiency of filter subjects in consumer (#4864) Thanks to @svenfoo for the contribution!
• Optimize JetStream metalayer snapshots by reducing allocations and simplifying marshaling (#4925)
• Micro-optimization where subject tokenization occurs (#4880) Thanks to @svenfoo for the contribution!
• Prevent backing up internal JS API requests in large-scale source and mirror setups (#4884)
• Optimize catchups for replicas and mirrors where there are a significant number of interior deletes (#4929)
• Reduce lock contention on the stream lock for some operations that could block routes & gateways (#4933)
• Do not load all blocks for NumPending when delivery is LastPerSubject (#4885)
• Call stream update only if the config has changed (#4898)
• Prevent large memory buildup in the apply queue for NRGs during startup (#4895)
• Finding the last sourced message for each source of a stream is now much faster (#4935)

MQTT

• Retained messages will now be fetched concurrently for a new subscription (#4835)

Fixed

Accounts

• Guard account random number generator with mutex (#4894) Thanks to @igorrius for the report!

JetStream

• Fix accounting for replicas and tier limits (#4868, #4909)
• Ensure all filter subjects across consumers are accounted for when purging a stream (#4873) Thanks to @svenfoo for the contribution!
• Detect corrupt psim subjects during recovery of index.db (#4890)
• Don’t allow writing snapshots to disk before recovery has completed (#4927)
• Reduce memory usage during purge operations by flushing cache (#4905)
• Return an “Account not enabled” error when trying to access JetStream via the system account (#4910)
• Reduce the number of blocks loaded into memory when doing linear scans (#4916)

Leafnodes

• Mapping updates on reload for the global account are now propagated to leafnodes correctly (#4937)
• Leafnode authorization now supports nkeys (#4940)

MQTT

• Fixed an out-of-date error message on unsupported characters in MQTT topics (#4903)

OCSP

• Default to Unknown status instead of Good for unknown status assertions (#4917)
• Fixed OCSP Stapling not resuming for gateways on reload after certs change (#4943)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.7...v2.10.8

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.5

Dependencies

• github.com/klauspost/compress v1.17.4
• golang.org/x/crypto v0.16.0
• golang.org/x/sys v0.15.0
• golang.org/x/time v0.5.0

Improved

JetStream

• Increase minimum interval for full index.db state writes to reduce contention for high-speed ingest in large streams (#4858)

Fixed

JetStream

• Corruption of per-subject tracking on recover of bad or missing index.db (#4851) Thanks to @oscarwcl for the report!
• Prevent GetSeqFromTime panic in memstore if the last sequence was deleted (#4853) Thanks to @andreib1 for the report!
• Protect against bad consumer state and high-memory usage during rollback (#4857)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.6...v2.10.7

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.4

Dependencies

• github.com/klauspost/compress v1.17.3

Improved

JetStream

• Added in internal filestore state checks on write and recover (#4804)
• Reduce memory usage for streams with a large subject space (#4806)
• Only gather subject filters if we need them (#4820) Thanks to @svenfoo for the contribution!
• Add pre-check when expected-last-subject-sequence header is present (#4827)
• Avoid resetting WAL in RAFT layer if we already processed the message (#4830)

Monitoring

• Remove ocsp_peer_cache from varz response when not applicable (#4829)

Fixed

JetStream

• Only drop firstSeq under DiscardOld policy (#4802) Thanks to @davidmcote for the report and contribution!
• Do not allow consumers to be updated if they have been deleted (#4818) Thanks to @matevzmihalic for the report!
• Fix potential race when starting the consumer monitor (#4828)
• Fix race condition in debug print (#4833)

MQTT

• Fix typo in README (#4791) Thanks to @testwill for the contribution!
• Improved large number of MQTT clients on reconnect with retain messages and larger scoped subscriptions (#4810)

WebSockets

• Fix potential data race in overlapping re-use of buffers (#4811) Thanks to @oscarwcl for the report!

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.5...v2.10.6

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.4

Dependencies

• golang.org/x/crypto v0.15.0
• golang.org/x/sys v0.14.0
• golang.org/x/time v0.4.0
• github.com/nats-io/jwt/v2 v2.5.3

Improved

General

• Remove places where using time.After could cause GC pressure (#4756)

JetStream

• Remove unused Observer const, add unit test to check observer applies (#4727)
• Throttle writeFullState from separate goroutine (#4731)
• Reduce memory usage with lots of subjects in filestore (#4742)
• Resiliency when doing lots of conditional updates to a KV and restarting servers (#4764)
• ​​General stability and consistency improvements for clustered streams with failure offsets during server restarts (#4777)
• Improve code comments for Raft subsystem (#4724)
• Optimize linear scan when looking by comparing the first seq in a block (#4780)
• Move filestore cleanup to separate goroutine to make non-blocking (#4782)
• Move deletion of filestore files to separate goroutine to make non-blocking (#4783)

Monitoring

• Better check for standalone mode when determining to send statsz (#4757)

MQTT

• Add "clean" flag in trace message (#4740)

WebSocket

• ​​Check for /leafnode suffix path on leaf WebSocket connection (#4774)

Fixed

Accounts

• Fix panic in JWT permissions template handling (#4730)

Leafnode

• Fix subpath concatenation used for WebSocket remote connect URL (#4770) Thanks to @yoadey for the report!

JetStream

• Remove the state check in the runAs loops except for runAsLeader (#4725)
• Make sure to properly remove meta files for filestore after conversion from 2.9.x to 2.10.x (#4733)
• Make sure we check limits when scaling up a stream (#4738)
• Improve estimation on full state allocations to avoid reallocations in filestore (#4743)
• Make access to message block first and last sequence consistently use atomics (#4744)
• Fix DiscardNew exceed bytes calculation (#4772) Thanks to @MauriceVanVeen for the contribution! Thanks to @davidmcote for the report!
• Fix data race and possible panic when compacting (#4773, #4776)
• Fix panic in fileStore.Stop() (#4779)

MQTT

• Rapid load-balanced (re-)CONNECT to cluster causes races (#4734)
• Potential deadlock between JS API and mqttDeliverMsgCbQoS0 (#4760)

WebSocket

• Partial writes may lead to disconnect (#4755)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.4...v2.10.5

Changelog

Go Version

• 1.20.11

Improved

JetStream

• Stricter management of Raft state, which should improve recovery from a leaderless state (#4684 backport via #4737)
• Remove unused Observer const, add unit test to check observer applies (#4727 backport via #4737)

Fixed

Accounts

• Fix panic in JWT permissions template handling (#4730 backport via #4759)

WebSocket

• Partial writes may lead to disconnect (#4755 backport via #4759)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.23...v2.9.24

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

CVEs

• CVE-2023-46129 - nkeys: xkeys seal encryption used fixed key for all encryption

Go Version

• 1.21.3

Dependencies

• github.com/nats-io/nats.go v1.31.0
• github.com/nats-io/nkeys v0.4.6
• github.com/klauspost/compress v1.17.2
• golang.org/x/crypto v0.14.0
• golang.org/x/sys v0.13.0

Added

JetStream

• Report Raft group name in stream and consumer info responses (#4661)

MQTT

• Add config options to disable QoS 2 support (#4705)

TLS

• Add opt-in TLS handshake first for client connections (#4642)

Improved

Dependencies

• Remove unnecessary constraints dependency for ordered constraint (#4709) Thanks to @misterpickypants for the contribution!

JetStream

• Add internal pprof labels as metadata to the stream config for improved debuggability (#4662)
• Stricter management of Raft state, which should improve recovery from a leaderless state (#4684)
• Avoid unnecessary reallocations when writing the full filestore state to disk (#4687)
• Improve recovery of blocks that are being updated midway (#4692)
• Recycle filestore buffers on rebuild and write out full state prior to snapshotting (#4699)
• Extend AckTerm advisory event to support a reason (#4697)
• Improve time to select skip list and starting sequence number for deliver last by subject (#4712, #4713) Thanks to @StanEgo for the report!
• Optimize loading messages on last by subject if max messages per subject is one (#4714)

MQTT

• No longer require a server name to be set for a standalone server (#4679)

Routes

• Remove unnecessary account lookups for pinned accounts (#4686)
• Upgrade non-solicited routes if present in config (#4701, #4708)

Systemd

• Use correct network target to prevent host-dependent race conditions when establishing external connections (#4676)

Fixed

Configuration

• Fix possible panic during configuration reload during a server shutdown (#4666)

Exports/imports

• Prevent service import from duplicating MSG as HMSG with a remapped subject (#4678) Thanks to @izwerg for the report!

JetStream

• Fix panic if store error occurs when requesting consumer info (#4669)
• Fix incorrect calculation of num pending with a filtered subject (#4693) Thanks to @a-h for the report!
• Prevent purge of entire stream when targeting a sequence of 1 (#4698) Thanks to @john-bagatta for the report!
• Ensure there is a valid messages queue prior to processing within a mirror (#4700)
• Avoid concurrent consumer setLeader calls resulting in chance of multiple leaders (#4703)

MQTT

• Fix memory leak for retained messages (#4665) Thanks to @pricelessrabbit for the contribution!

Windows

• Ensure signal handler is stopped when shutting down on Windows to prevent goroutine leak (#4690)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.3...v2.10.4

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.3

Fixed

JetStream

• Reclaim more space with streams having many interior deletes during compaction with compression enabled (#4645)
• Fixed updating a non unique consumer on workqueue stream not returning an error. Thanks to @mdawar for the contribution (#4654)
• Stream / KV lookups fail after decreasing history size (#4656)
• Only mark fs as dirty vs full write on mb compaction (#4657)

MQTT

• Fix crash in MQTT layer with outgoing PUBREL header (#4646)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.2...v2.10.3

Changelog

Go Version

• 1.20.10

Fixed

Accounts

• Prevent bypassing authorization block when enabling system account access in accounts block (#4605). Backport from v2.10.2

Leafnodes

• Prevent a leafnode cluster from receiving a message multiple times in a queue subscription (#4578). Backport from v2.10.2

JetStream

• Hold lock when calculating the first message for subject in a message block (#4531). Backport from v2.10.0
• Add self-healing mechanism to detect and delete orphaned Raft groups (#4647). Backport from v2.10.0
• Prevent forward proposals in consumers after scaling down a stream (#4647). Backport from v2.10.0
• Fix race condition during leader failover scenarios resulting in potential duplicate messages being sourced (#4592). Backport from v2.10.2

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.22...v2.9.23

Changelog

Downgrade compatibility note

2.10.x brings on-disk storage changes which bring significant performance improvements. Upgrade existing server versions will handle the new storage format transparently. However, if a downgrade from 2.10.x occurs, the old version will not understand the format on disk with the exception 2.9.22 and any subsequent patch releases for 2.9. So if you upgrade from 2.9.x to 2.10.0 and then need to downgrade for some reason, it must be back to 2.9.22+ to ensure the stream data can be read correctly.

Go Version

• 1.21.2

Dependencies

• github.com/nats-io/nats.go v1.30.2

Added

Profiling

• Add prof_block_rate config option for configuring the block profile (#4587)
• Add more pprof labels to consumers, sources, and mirrors (#4609)

Improved

Core

• Reduce contention when pattern matching subjects when the sublist cache is disabled (#4586)
• Various service import reply optimizations (#4591)
• Remove unnecessary lock on subscription list if cache is disabled (#4594)

Docs

• Fix links in various repo markdown files (#4590) Thanks to @jdhenke for the contribution!

Leafnodes

• Set S2 writer concurrency to 1 rather than the default of GOMAXPROCS to improve performance (#4570)

JetStream

• Make install snapshot errors rate limited when catching up (#4574)
• Log a warning on reset if bad stream state is detected (#4583)
• Change some contended locks to atomic swap operations (#4585)
• Log a warning if filestore recovery fails on the happy path (#4599)
• Ensure concurrent stream of the same stream does not return not found (#4600)
• Add additional markers for indicating unflushed state (#4601)
• Log a warning when subject skew is detected in the filestore (#4606)
• Reduce contention for a high number of connections in JetStream enabled account (#4613)
• Reduce contention in the consumer info API (#4615)
• Reduce contention and increase throughput of replica synchronization (#4621)

Systemd

• Update systemd scripts to use SIGUSR2 (lame duck model) for shutdown (#4603)

WebSocket

• Minimize memory growth for compressed WebSocket connections (#4620)
• Significantly reduce allocations in WebSocket interface (#4623)

Fixed

Accounts

• Fix inversion of lock on startup when setting up the account resolver (#4588)
• Prevent bypassing authorization block when enabling system account access in accounts block (#4605) Thanks to @alexherington for the report!

Leafnodes

• Prevent a leafnode cluster from receiving a message multiple times in a queue subscription (#4578) Thanks to @pcsegal for the report!

JetStream

• Fix possible panic due to message block unlock occurring prematurely (#4571)
• Guard against an accounting error resulting in a negative message count (#4575)
• Skip enabling direct gets if no commits (#4576)
• In lame duck mode, shutdown JetStream at the start to signal transfer of leadership if the leader (#4579)
• Fix possible stream assignment race condition (#4589)
• Fix race condition during leader failover scenarios resulting in potential duplicate messages being sourced (#4592)
• Respond with “not found” for consumer info if consumer is closed (#4610)
• Prevent processing of consumer assignments after JetStream shutdown occurs (#4625)
• Fix possibly lookup misses when MaxMsgsPerSubject=1 leading to excess messages in stream (#4631)

MQTT

• Fix PUBREL header incompatibility (#4616)

Routes

• Fix potential of pinned accounts not establishing a route on connect (#4602)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.1...v2.10.2

Changelog

Downgrade compatibility note

2.10.x brings on-disk storage changes which bring significant performance improvements. Upgrade existing server versions will handle the new storage format transparently. However, if a downgrade from 2.10.x occurs, the old version will not understand the format on disk with the exception 2.9.22 and any subsequent patch releases for 2.9. So if you upgrade from 2.9.x to 2.10.0 and then need to downgrade for some reason, it must be back to 2.9.22+ to ensure the stream data can be read correctly.

Go Version

• 1.21.1

Fixed

Leafnode

• Fix TLS handshake being prevented if remote (leaf) does not have a TLS block configured (#4565)

JetStream

• Ensure a single filter in new consumer SubjectFilters or stream SubjectTransforms block uses the extended consumer subject format as it did with SubjectFilter (#4564)
• Ensure stream-specified consumer limits are correctly applied in combination with the explicit ack policy (#4567)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.10.0...v2.10.1

Changelog

Downgrade compatibility note

2.10.0 brings on-disk storage changes which bring significant performance improvements. Upgrade existing server versions will handle the new storage format transparently. However, if a downgrade from 2.10.0 occurs, the old version will not understand the format on disk with the exception 2.9.22 and any subsequent patch releases for 2.9. So if you upgrade from 2.9.x to 2.10.0 and then need to downgrade for some reason, it must be back to 2.9.22+ to ensure the stream data can be read correctly.

Go Version

• 1.21.1

Dependencies

• github.com/nats-io/nats.go v1.29.0
• github.com/nats-io/jwt/v2 v2.5.2
• github.com/nats-io/nkeys v0.4.5
• github.com/klauspost/compress v1.17.0
• golang.org/x/crypto v0.13.0

Added

Accounts

• Add $SYS.REQ.USER.INFO NATS endpoint for user info (#3671)

Auth

• Authorization callout extension for delegating to external auth providers (#3719, #3784, #3799, #3864, #3987, #4501, #4544)

Builds

• Add early build support for NetBSD (#3526) Thanks to @MatthiasPetermann for the contribution!
• Add early build support for IBM z/OS (#4209) Thanks to @v1gnesh for the contribution!

Cluster

• Multiple routes and ability to have per-account routes to reduce head-of-line blocking in clustered setups (#4001, #4183, #4414)
• Support for S2 compression of traffic over route connections (#4115, #4137)

Config

• Reload server config by sending a message in the system account to $SYS.REQ.SERVER.{server-id}.RELOAD (#4307)

Embedded

• Add ConnectionDeadline field to User to force server disconnect after deadline (#3580, #3674)

Leafnode

• Add TLSHandshakeFirst option to perform a TLS handshake before sending connection info (#4119)
• Support S2 compression of traffic over leafnode connections where the default now is s2_auto to compress relative to the RTT of the hub (#4167, #4230)
• Allow remotes from same server binding to same hub account (#4259)

Logging

• Add logfile_max_num server config field to auto-rotate files (#4548)

JetStream

• Add stream subject transforms (#3814, #3823, #3827, #4035, #4354, #4400, #4403, #4512)
• Add freeform metadata field to stream and consumer configs (#3797)
• Add support for consumers filtering on multiple subjects (#3500, #3865, #4008, #4129, #4188)
• Add original timestamp as header to republished message (#3933) Thanks to @amorey for the contribution!
• Allow republish for mirroring/sourcing streams (#4010)
• Add optional S2 stream compression for file store-backed streams (#4004, #4072)
• Add file store ability to re-encrypt with new encryption keys (#4296)
• Add embedded option to disable JetStream ASCII art at startup (#4261) Thanks to @renevo for the contribution!
• Add ability to configure first_seq when creating streams (#4322, #4345)
• Add sync_internal option to JetStream config (#4483)

Monitoring

• Add unique_tag field in /jsz and /varz endpoints (#3617)
• Add $SYS.REQ.SERVER.PING.IDZ NATS endpoint for basic server info (#3663)
• Add $SYS.REQ.SERVER.<id>.PROFILEZ NATS endpoint for requesting debugging profiles (#3774)
• Add subscription count to /statz endpoint (#3875)
• Add Raft query parameter to /jsz to include Raft group info (#3914)
• Add slow_consumer_stats to the /varz endpoint (#4330)

MQTT

• Add support for QoS2 exactly-once delivery (#4349, #4440)

Reload

• Match --signal PIDs with globular-style expression (#4370) Thanks to @jevolk for the contribution!

Subject Mapping

• Add ability to remove wildcard tokens in subject transforms (#4152)
• Allows cluster filtering in account subject mapping (#4175)

System Services

• Add $SYS.REQ.SERVER.<id>.KICK NATS endpoint to disconnect a client by id or by name from the target server (#4298)
• Add $SYS.REQ.SERVER.<id>.LDM NATS endpoint that sends a “lame duck mode” message to a client by id or name on the target server (#4298)

Windows

• Add NATS_STARTUP_DELAY env for configurable startup time (#3743) Thanks to @Alberic-Hardis for the contribution!

Improved

Leafnodes

• Add jitter to leafnode reconnections (#4398)

Logging

• Add account, stream and consumer name to consumer alignment cleanup warning (#3666) Thanks to @ch629 for the contribution!

JetStream

• Significant optimisations and reduced memory impact for replicated streams with a large number of interior deletes (common in large KVs), considerably reducing the amount of CPU and memory required to create stream snapshots and smoothing out publish latencies (#4070, #4071, #4075, #4284, #4520, #4553)
• Improve signaling mechanism for consumers to improve performance and reduce latency (#3706)
• Allow edit of Stream RePublish (#3811)
• Add batch completed status to pull consumers (#3822)
• Improve behavior of stream source consumer creation or config updates on leadership change (#4009)
• Record the stream and consumer info timestamps (#4133)
• Allow switching between limits and interest retention policies (#4361)
• Improve performance of deleting blocks (#4371)
• Update the way meta indexing is handled for filestore, significantly reducing time to recover streams at startup (#4450, #4481)
• Add self-healing mechanism to detect and delete orphaned Raft groups (#4510)
• Improve monitoring of consumers that need to be cleaned up (#4536)

MQTT

• Optimize retained messages by using KV semantics instead of holding retained messages in memory (#4199, #4228)
• Support for topics with . character (#4243) Thanks to @petedavis and @telemac for the reports!
• Set the RETAIN flag when delivering to new subscriptions and clear the flag in all other conditions (#4443)

Profiling

• Annotate CPU and goroutine profiles with additional asset information to assist with debugging (#4204)
• Remove unused block profile rate (#4402)

Subject Mapping

• Subject transform validation and error reporting (#4202)

Fixed

Accounts

• Fix data race when updating accounts (#4435, #4550, #4561) Thanks to @hspaay for the report!

Clients

• Check if client connection name was already set when storing it (#3824)

Leafnode

• Data race during validation and setup (#4194)

JetStream

• Check for invalid stream name in sources (#4222)
• Stream config update idempotency (#4292)
• Seqset encode bug that could cause bad stream state snapshots (#4348)
• Ensure stream assignment is set when checking replica count and updating retention (#4391)
• Hold lock when enforcing message limit on startup (#4469)
• Fix filestore data race on hash during snapshots (#4470)
• Use write lock for memory store filtered state (#4498)
• Fix data race on stream’s clustered filestore sequence (#4508)
• Fix possible panic when recalculating the first sequence of a subject (#4530) Thanks to @aldiesel for the report!
• Fix leaking timers in stream sources resulting in runaway CPU usage (#4532)
• Fix possible panic when consumer is not closed (#4541)
• Fix data race when accessing consumer assignment (#4547)
• Fix data race when changing stream retention policy (#4551)
• Fix data race when loading the next message in memory-based streams (#4552)
• Prevent forward proposals in consumers after scaling down a stream (#4556)

OSCP

• Fixed local issuer determination for OCSP Staple (#4355)

Routes

• Update LastActivity on connect for routes (#4415)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.22...v2.10.0

Changelog

Go Version

• 1.20.8 (updated out-of-cycle since Go 1.19 is now EOL)

Dependencies

• github.com/nats-io/jwt/v2 v2.5.0
• golang.org/x/crypto v0.12.0
• golang.org/x/sys v0.11.0

Improved

Monitoring

• CORS Allow-Origin passthrough for monitoring server (#4423) Thanks to @mdawar for the contribution!

JetStream

• Improve consumer scaling reliability with filters and cluster restart (#4404)
• Send event on lame duck mode (LDM) to avoid placing assets on shutting down nodes (#4405)
• Skip filestore tombstones if downgrade from 2.10 occurs (#4452)
• Adjust delivered and waiting count when consumer message delivery fails (#4472)

Fixed

Config

• Allow empty configs and fix JSON compatibility (#4394, #4418)
• Remove TLS OCSP debug log on reload (#4453)

Monitoring

• Fix Content-Type header when /healthz is not 200 OK (#4437) Thanks to @mdawar for the contribution!
• Fix server /connz idle time sorting (#4463) Thanks to @mdawar for the contribution!
• Interface conversion bug which could cause a panic when calling /ipqueuesz endpoint (#4477)

Leafnode

• Fix race condition which could affect propagating interest over leafnode connections (#4464)

JetStream

• Fix possible deadlock in checking for drift in the usage reporting when storing a message (#4411)
• Durable pull consumers could get cleaned up incorrectly on leader change (#4412)
• Moving an R1 stream could sometimes lose all messages (#4413)
• Prevent peer-remove of an R1 stream which could result in the stream becoming orphaned (#4420)
• Ensure consumer ack pending is less than max ack pending on state restore (#4427)
• Ensure to reset election timer when catching up (#4428) Thanks to @yuzhou-nj for the report!
• Auto step-down Raft leader if an entry is missing on a catchup request (#4432)
• Fix PurgeEx with keep having deletes in blocks (#4431)
• Update global subject index when message blocks expire (#4439)
• Ensure max messages per subject is respected after update (#4446) Thanks to @anthonyjacques20 for the report!
• Ignore and remove empty message blocks on rebuild (#4447)
• Fix possible accounting discrepancy on message write (#4455)
• Fix potential message duplication from stream sources when downgrading from 2.10 (#4454)
• Check for checksum violations for all records before sequence processing (#4465)
• Fix message block accounting (#4473)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.21...v2.9.22

Changelog

Go Version

• 1.19.12

Dependencies

• github.com/klauspost/compress v1.16.7
• github.com/nats-io/nats.go v1.28.0
• go.uber.org/automaxprocs v1.5.3
• golang.org/x/crypto v0.11.0
• golang.org/x/sys v0.10.0

Added

OCSP

• Add fetch, cache, and verification of client CA's OCSP Response for NATS, WebSocket, and MQTT client mTLS connections (#4362, backported from 2.10)
• Add bi-directional fetch, cache, and verification of CA OCSP Response for LEAF connections (#4362, backported from 2.10)

See ADR-38 OCSP Peer Verification

General

• Add UTC log timestamp option (#4331, backported from 2.10)

Improved

JetStream

• Don't error to server logs if message was deleted for consumer (#4328)
• Improve publish performance for zero-interest subjects (#4359) Thanks to @antlad for reporting the issue!
• Sync and reset message rejected count to ensure replicas don’t incorrectly discard messages (#4365, #4366)

Fixed

General

• Leaking memory on usage of getHash() (#4329) Thanks to @VuongUranus for reporting the issue!
• Server reload with highly active accounts and service imports could cause panic or dataloss (#4327)
• Fix detection of an unusable configuration file (#4358)
  • NOTE: as a side effect of this fix, the server will no longer startup with an empty config file
• Fix a few system service imports going missing after configuration reload (#4360)

OCSP

• Fix local-determination of issuer CA at startup (#4362)
• Remove constraint that all (super)cluster node peers must be issued by the same CA (#4362)

Embedded

• Don't require TLS for in-process client connection (#4323)

JetStream

• Fix serializability guarantee for concurrent publish when using expected-last-subject-sequence (#4319)
• Report correct consumer count in paged list response (#4339)
• Fix not validating single token filtered consumer (#4338)
• Fix stream recovery of message block with sequence gaps (#4344)
• Fix panic when re-calculating first sequence of SimpleState info (#4346)
• Fix stream store accounting drift (#4357)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.20...v2.9.21

Changelog

Go Version

• 1.19.11

Added

Windows

• Backport 2.10 support for native Windows certificate store (#4268)

Improved

Accounts

• Allow advisories to be exported/imported across accounts (#4302)

JetStream

• Optimize consumer create time on streams with a large number of blocks (#4269)

Fixed

Gateways

• Protect possible data race when reloading accounts (#4274)

Leafnodes

• Prevent zombie subscriptions which could lead to silent data loss when using queue subscriptions (#4299)

WebSocket

• Prevent reporting tls_required when tls_available is not set (#4264)

JetStream

• Prevent corrupting streams actively being restored during health check (#4277) Thank you @vitush93 for the report!
• Prevent encrypted data attempting to be decrypted with an empty key (#4301)

MQTT

• Ensure republished messages from streams are received by MQTT subscriptions (#4303)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.19...v2.9.20

Changelog

Go Version

• 1.19.10

Improved

JetStream

• Improve resource utilization when creating mirrors on very high-sequence streams (#4249)

Fixed

WebSocket

• Ensure INFO properties are populated based on the WebSocket listener when enabled (#4255) Thanks to @Envek for reporting the issue!

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.18...v2.9.19