nats-server

Changelog

Go Version

• 1.19.10

Dependency Updates

• golang.org/x/crypto v0.9.0 (#4236)
• golang.org/x/sys v0.8.0 (#4236)
• github.com/nats-io/nats.go v1.27.0 (#4239)

Improved

Monitoring

• Optimize /statsz locking and sending in standalone mode (#4235)

JetStream

• Apply ack floor check only for interest-based streams (#4206)
• Improved efficiency and reduced CPU usage of the consumer ack floor check, particularly when the stream first sequence is a large number (#4226)
• Improve clean-up phase of R1 consumers on server restart for name reuse (#4216)
• Optimize “last message lookups” by subject (KV get operations) for small messages (#4232) Thanks to @jjthiessen for reporting the issue!
• Only enable JetStream account updates in clustered mode (#4233) Thanks to @tpihl for reporting the issue!

Fixed

General

• Fix a variety of potential panic scenarios (#4214) Thanks to @Zamony and @artemseleznev for the contribution!

Leadnode

• Daisy chained leafnodes could have unreliable interest propagation (#4207)
• Properly distribute requests to queue groups across leafnodes (#4231)

JetStream

• Killed server on restart could render encrypted stream unrecoverable (#4210) Thanks to @BhatheyBalaji for the report!
• Fix a few data races detected internal testing (#4211)
• Process extended purge operations correctly when being replayed (#4212, #4213) Thanks to @MauriceVanVeen for the report and contribution!

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.17...v2.9.18

Changelog

Go Version

• 1.19.9

Dependency Updates

• github.com/klauspost/compress v1.16.5 (#4088)

Improved

Core

• Additional optimizations to outbound queues, reducing memory footprint (#4084, #4093, #4139)
• Use faster flate compression library for WebSocket transport compression (#4087)

Leafnodes

• Optimize subscription interest propagation for large leafnode fleet (#4117, #4135)

Monitoring

• Support sorting by RTT for /connz (#4157)

Resolver

• Improve signaling for missing account lookups (#4151)

JetStream

• Optimized determining if a stream snapshot is required (#4074)
• Run periodic check for consumer “ack floor” drift on leader (#4086)
• Optimize leadership transfer during a stream migration (#4104)
• Improve how clustered consumer state is hydrated on startup (#4107)
• Add operation type to panic messages for improved debugging (#4108)
• Improve health check to repair stalled assets periodically (#4116, #4172)
• Remove unnecessary filestore lock to improve I/O performance (#4123)
• Various Raft leadership improvements (#4126, #4142, #4143, #4145)
• Improve accuracy of account usage (#4131)
• Clean up old Raft groups when streams are reset (#4177)

Fixed

General

• Fix various names in comments (#4099) Thanks to @cuishuang for the contribution!
• Fix various typos in comments (#4169) Thanks to @savion1024 for the contribution!
• Update tests to reflect the server.Start() call no longer blocks (#4111) Thanks to @lheiskan for reporting the issue!
• Fix race condition in config reload with gateway sublist check (#4127)
• Track all remote servers in a NATS system with different domains (#4159)

Core

• Fix premature closing in WebSocket transport due to outbound queue changes (#4084)
• Fix subscription interest for config-based accounts during config reload (#4130)
• Use monotonic time for measuring durations internally (#4132, #4154, #4163)

Monitoring

• Service import reporting for /accountz when mapping to local subjects (#4158)

JetStream

• Fix formatting of Raft debug log (#4090)
• Prevent failure of /healthz in single server mode on failed snapshot restore (#4100)
• Ensure a stream Raft node has fully stopped and resources freed (#4118)
• Fix case where R1 streams are orphaned and can’t scale up (#4146)
• Protect against out of bounds access on usage updates (#4164)
• Fix state rebuild where the first block is truncated and missing index info (#4166)
• Avoid stale KV reads on server restarted for replicated stores (#4171) Thanks to @yixinin for reporting the issue!
• Prevent deadlock with usage report for accounts (#4176)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.16...v2.9.17

Changelog

Go Version

• 1.19.8

Dependency Updates

• github.com/klauspost/compress v1.16.4
• github.com/nats-io/jwt/v2 v2.4.1
• github.com/nats-io/nkeys v0.4.4
• golang.org/x/crypto v0.8.0
• golang.org/x/sys v0.7.0

Added

Build

• Nightly build of the “main” branch as a Docker image: synadia/nats-server:nightly-main (#3961, #3962, #3963, #3972, #4019, #4063)
• Version control SHA in the Goreleaser build of the server (#3993). Thanks for the report @jzhoucliqr!
  Monitoring
• Add server name and route remote server name to /routez (#4054)

Resolver

• Add “hard_delete” option for stored JWTs (#3783). Thanks for the contribution @JulienVdG!

Improved

JetStream

• Storage and Raft layer improvements to decrease p99 latency variance and memory pressure in high load environments (#3956, #3952, #3965, #3981, #3999, #4018, #4020, #4021, #4022, #4026, #4027, #4028, #4029, #4030, #4038, #4045, #4050, #4053)
• Don’t show Raft warning for node that is closed (#3968)
• Use pooled buffer for flushing encrypted message blocks (#3975)
• Remove snapshotting of cores and maxprocs (#3979)
• Improvements to interest-based streams to optimize when messages are deleted (#4006, #4007, #4012)
• Better handling of concurrent stream and consumer creation of the same name (#4013)
• Finer-grain locking during asset checking to reduce contention in the /healthz endpoint (#4031)
• Encrypted file stores will now limit block sizes to improve performance (#4046)
• Improve performance on storing messages with varying subjects and limits are imposed (#4048, #4057). Thanks for the report @kung-foo!

Fixed

Subjects

• Ensure subjects containing percent (%) are escaped (#4040)

Accounts

• Fix data race when setting up service import subscriptions (#4068)

Leaf

• Fix leaf client connection failing on OCSP setups (#3964)
• Fix case when allow/deny permissions on leaf connection could block legitimate interest (#4032)

Cluster

• Route disconnect not detected by ping/pong (#4016). Thanks for the contribution @sandykellagher!

JetStream

• Pull consumer not sending timeout error to clients for expired requests (#3942)
• Prevent meta leader deadlock during deletion of orphaned streams during server startup (#3945)
• Clear ack’ed messages when scaling workqueue or interest-based streams (#3960) Thanks for the report @Kaarel!
• Remove messages from interest-based stream on consumer snapshot (#3970)
• Fix potential panic in message block buffer pool (#3978)
• Fixed an issue with consumer states growing and causing instability (#3980)
• Improve handling of out-of-storage condition (#3985)
• Address memory leak of unreachable Raft groups when JetStream becomes disabled (#3986)
• Prevent Raft leader from being placed on server in lame-duck mode (#4002)
• Remove potential race condition on sysRequest (#4017)
• Fix FirstSeq not being updated with filestore when purging subject (#4041). Thanks for the contribution @MauriceVanVeen!
• Fix Raft log debug reloading (#4047)
• Ensure consumer recovers fully on restart before being eligible for leader (#4049)
• Fix incorrect check between stream source/mirror with external streams (#4052)
• Fix various conditions during Raft group recovery (#4056, #4058)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.15...v2.9.16

Changelog

Go Version

• 1.19.6: Both the release executables and Docker images are built with this Go release

Added

• Monitoring
  • Add raft query parameter to /jsz to include group info (#3915)
  • Update /leafz to include leaf node remove server name and “spoke” flag (#3923, #3925)

Changed

• Lower default value of jetstream.max_outstanding_catchup to prevent slow consumers between routes (#3922)
  • Note: The new value is now 64MB from 128MB. This is better optimized for 1 Gbit links, however if your links are 10 Gbit or higher, this value can be set back to 128MB if slow consumers were not previously observed.

Improved

• Refactor intra-process queue to reduce allocations (#3894)
• JetStream
  • Better system stability and recovery from corrupt metadata due to hard forced restarts (#3934)
  • Optimize on-disk, per-subject info update (#3867)
  • Limit concurrent blocking IO to improve stability under heavy IO loads (#3867)
  • Improve message expiry calculation for large numbers of messages (#3867)
  • Optimize when and how consumer num pending is calculated, significantly speeding up consumer info requests (#3877)
  • Improve parallel consumer creation to prevent dropped messages (#3880)
  • Properly warn on consumer state update state failures (#3892)
  • Performance of consumer creation for certain configurations (#3901)
  • Send current snapshot to followers when becoming meta-leader (#3904)
  • Ensure preferred peer during stepdown is healthy (#3905)
  • Optimized various store calls on stream state (#3910)
  • Various performance and stability under heavy IO loads (#3922) (Thank you @matkam and @davidzhao for the report and the test harness!)

Fixed

• Fix stack overflow panic in reverse entry check when inbox ends with wildcard (#3862)
• Check if client connection name was already set when storing, preventing recursive memory growth (#3886)
• Fix check for count of wildcard tokens in “partition” subject transform (#3887) (Thank you @MauriceVanVeen for the contribution!)
• Fix panic if service export is nil (#3917) (Thank you @MauriceVanVeen for the report!)
• JetStream
  • Ensure per-subject info is updated when doing stream compact (#3860)
  • Ensure account usage is updated in the filestore when extended version purge occurs (#3876)
  • Prevent consumer deletes on restart, with non-fatal errors (#3881)
  • Do not warn if consumer replicas is zero since it will be inherited from the stream (#3882)
  • Named push consumers with inactive thresholds deleted when still active (#3884)
  • Prevent spurious “Error storing entry to WAL” log messages (#3893, #3891)
  • Clean up consumer redelivery state on stream purge (#3892)
  • Clean up consumer ack pending if below stream ack floor (#3892)
  • Update ack floors on messages being expired (#3892)
  • Fix lost ack pending consumer state on partial stream purge (#3892)
  • Snapshot and compact the consumer RAFT WAL, even when state changes do not occur, to prevent excessive disk usage (#3898)
  • Fix KV accounting errors under heavy concurrent usage (#3900)
  • Ensure new replicas respect MaxAge when a stream is scaled up (#3861)
  • Snapshots would not compact after being applied (#3907)
  • Fix filtered pending state calculation (#3910)
  • Recover from a failed truncate on raft WAL (#3911)
  • Fix JWT claims update if headers are passed (#3918)
• MQTT
  • Prevent use of wildcard characters with topics beginning with $, per the MQTT spec violation 4.7.2-1 (#3926) (Thank you @dominikh for the report!)

Dependency Updates

• klauspost/compress - v1.16.0
• nats-io/nats.go - v1.24.0
• golang.org/x/crypto - v0.6.0
• golang.org/x/sys - v0.5.0

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.14...v2.9.15

Changelog

Go Version

• 1.19.5: Both the release executables and Docker images are built with this Go release

Fixed

• JetStream
  • Fix circumstance when an empty snapshot could be written (#3844)
  • Fix possible panic and deadlock during a consumer filter subject update (#3845)
  • Fix consumer snapshot logic (#3846)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.12...v2.9.14

Changelog

NOTE: regressions were found in this release. Please skip this and go directly to the v2.9.14 release.

Go Version

• 1.19.5: Both the release executables and Docker images are built with this Go release

Added

• OS/Arch
  • Add support for dragonfly BSD (#3804)

Improved

• JetStream
  • Use highwayhash to optimize difference tracking for stream, consumer, and cluster snapshots (#3780)
  • Add small tolerance in lag for stream and consumer health checks (#3794)
  • Various optimizations related to snapshots and memory usage (#3828, #3831, #3837) Thanks to @MauriceVanVeen for the collaboration on this issue.

Fixed

• JetStream
  • Update numCores and maxProcs if altered by container limits (#3796)
  • Fix filtered state for all subjects when the first sequences are deleted (#3801)
  • Updating a stream to direct gets would fail direct gets (#3806)
  • Force consumer replicas to match for interest-based policy streams (#3817)
  • Assign signal subscription to consumer when created (#3808)
  • Properly process updates on a stream on restart (#3818)
  • Select consumer peer(s) from active/online peers only on creation (#3821)
  • Sourced streams that do not overlap subjects were incorrectly reported as a cycle (#3825)
  • Fix for isGroupLeaderless when JS not available due to shutdown (#3830)
  • Deadlock on data loss when holding mb lock (#3832)
  • Fix consumer not getting messages after filter update (#3829)
  • Fix current consumers not getting messages after purge (#3838) Thanks to @pcsegal for the report!

Updated Dependencies

• github.com/klauspost/compress - v1.15.15
• github.com/nats-io/nats.go - v1.23.0
• golang.org/x/time - v0.3.0

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.11...v2.9.12

Changelog

Go Version

• 1.19.4: Both the release executables and Docker images are built with this Go release

Improved

• Remove duplicate subscription point to reduce memory usage (#3768)

Updated

• Update golang.org/x/crypto dependency to v0.5.0 (#3761). Thanks to @friedrichwilken for the contribution.

Fixed

• JetStream
  • Allow recovery of stream meta file if not properly flushed prior to restart (#3752)
  • Ensure max bytes are honored when message block is not properly flushed (#3757). Thanks to @containerpope for the report.
  • Ensure new consumers assigned to an offline server are properly restored from the metaleader's snapshot (#3760)
  • Set and clear observer state for servers with disconnected leaf nodes (#3763)
• Leaf node
  • Fix duplicate messages when leaf node binds an account with service imports (#3751). Thanks to @asambres-form3 and @maxarndt for the original reports.
• Monitoring
  • Add back existing HealthzOptions.JSEnabled field to restore backwards compatibility (#3744)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.10...v2.9.11

Changelog

Go Version

• 1.19.4: Both the release executables and Docker images are built with this Go release

Improved

• JetStream
  • Improve performance and latency with large number of sparse consumers (#3710)
  • Simplify locking for consumer info requests (#3714)

Fixed

• JetStream

  • Ensure previously assigned nodes are removed from stream assignment (#3707, #3709)
  • Bad first sequence and expiration stopped working after server restart (#3711)
  • Fix no-quorum issue if leader is restarted with a memory-based clustered stream (#3713)
  • Fix dangling message block on compaction/cleanup which could prevent a stream from restoring (#3717) thanks to @k15r for the report
  • Prevent overlapping stream subjects on stream update in non-clustered JetStream (#3715)

• Monitoring

  • /healthz?js-enabled=true behavior now properly asserts whether JetStream is enabled if it is expected to be and does not check for health of the assets. (#3704)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.9...v2.9.10

Changelog

Go Version

• 1.19.4: Both the release executables and Docker images are built with this Go release

Improved

• JetStream

  • Avoid full state snapshot for streams with many deleted items (#3680)
  • Refactor to make stream removal from server consistent (#3691)

• Help/Usage

  • Display two additional --reload signal options, ldm and term (#3683)

Fixed

• Authorization

  • Prevent returning no_auth_required when a client tries to connect in operator mode (#3667)
    • Any client (Java) that suppressed sending credentials before of this flag could be affected.
    • This only affects the 2.9.8 server version

• JetStream

  • Tag policies not honored during stream replica reassignment after a peer is removed (#3678)
  • Address issues when concurrent "create" requests for the same stream are issued (#3679)
  • Server panic when consumer state was not decoded correctly (#3688)
  • Ensure consumers that are deleted on startup are removed from the system (#3689)
  • Fixed JetStream remained disabled for reactivated JWT accounts (#3690, thank you @JulienVdG)

• Leafnodes

  • Do not delay PINGs for leaf connections (#3692, thank you @sandykellagher)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.8...v2.9.9

Changelog

Go Version

• 1.19.3: Both release executables and Docker images are built with this Go release.

Monitoring:

• JetStream:
  • Server might crash if a pull consumer with an activity threshold is deleted immediately after a message ack (#3658). Thanks to @jdhenke for the report.
• Server Info:
  • If no_auth_user is set, clear the auth required flag in the server info presented to the client (#3659)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.7...v2.9.8

Changelog

Go Version

• 1.19.3: Both release executables and Docker images are built with this Go release.

Improved:

• JetStream:
  • Processing of acknowledgments (#3624)
  • Stream ingest performance with large numbers of consumers (#3633)
  • StreamDetail now includes stream creation for /healthz reporting. Thanks to @raypinto for the contribution (#3646)

Fixed

• JetStream:
  • For "Interest" or "WorkQueue" policy streams, when a consumer was deleted, a stream view could cause high CPU or memory usage. Thanks to @Kaarel for the report (#3610)
  • Make sure to enforce HA asset limits during peer processing as well as assignment (#3614)
  • Account max streams/consumers not always honored (#3615)
  • Reduce some warning log messages "append entries" (#3619)
  • Logic bug that would prevent some stream messages from being deleted. Thanks to @Kaarel for the report (#3620, #3625)
  • Account removal leaks subscriptions. Thanks to @JulienVdG for the report (#3627)
  • Server fails to start on concurrent map read/write (#3629, #3634)
  • Possible panic on stream info when leader is not yet or elected or loses quorum. Thanks to @apuckey for the report (#3631)
  • Allow any type of ack to suppress auto-cleanup of consumer (#3635)
  • Make mirror consumers use filtered version of consumer create (#3638)
  • WorkQueue was not correctly rejecting overlapping consumers in some cases. Thanks to @Kilio22 for the report (#3640)
  • Make sure header keys do not have additional prefixes. Thanks to @osmanovv for the report (#3649)
• Routing:
  • TLS connections to discovered server may fail (#3611, #3613)
• Weighted subject mappings updates not applied (#3618)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.6...v2.9.7

Changelog

Go Version

• 1.19.3: Both release executables and Docker images are built with this Go release.

Fixed

• JetStream:
  • Possible panic on some rare cases where a clustered consumer monitor go routine started while the consumer was deleted or scaled down to an R1. Some tracing could have caused a panic (#3599)
  • On stream proposal failures, the server would incorrectly warn about high stream lag. Thanks to @kino71 for the report (#3601)
  • Stream sources with OptStartTime would get redelivered messages on server restart. Thanks to @DavidCockerill for the report (#3606)
  • Scaling down a replicated stream (R>1) to R1 while it has no quorum, for instance due to a network partition, would leave the stream in a bad state and there would be a constant report of "No quorum, stalled" for this stream, even after the network partition is resolved (#3608)
• LeafNode:
  • Possible duplicate messages in complex setup. Thanks to @chenchunping for the report (#3604)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.5...v2.9.6

Changelog

Go Version

• 1.19.3: Both release executables and Docker images are built with this Go release.

Fixed

• JetStream:
  • Honor MaxMsgsPerSubject on stream update (#3595)
  • Errors such as "wrong sequence for skipped msg" or "expected sequence does not match store " while processing snapshots of streams with expired messages. Thanks to @MauriceVanVeen and @sylrfor the reports (#3589, #3596)
• Sublist's cache hit rate would be wrong in presence of multiple accounts. Thanks to @aopetrov86 for the report and contribution (#3591)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.4...v2.9.5

Changelog

Go Version

• 1.19.2: Both release executables and Docker images are built with this Go release.

Fixed

• Configuration Reload:
  • The option allow_non_tls would be ignored after a configuration reload. Thanks to @JulienVdG for the report (#3583)
• JetStream:
  • Possible deadlock. Thanks to @ashumkin for the report and @neilalexander for the contribution (#3555)
  • Possible panic in disk full situations. Thanks to @fantashley for the contribution (#3560, #3568)
  • "First sequence mismatch" after a restart/deployment with streams that have message TTLs. Thanks to @MauriceVanVeen for the report (#3567)
  • Update of an R1 consumer would not get a response. The update was accepted by the server, but the client library or NATS CLI would timeout waiting for the response (#3574)
  • Update of a consumer's InactiveThreshold would not always take effect. Thanks to @neilalexander for the contribution (#3575)
  • A consumer may not be removed based on InactiveThreshold in presence of gateways (#3575)
  • Migration of ephemerals on server shutdown was not working and could create "ghost" consumers on servers restart, that is, consumers that would be listed by the meta leader, but getting information about this consumer would fail. Migration will no longer occur, instead, all R1 pull consumers will be notified that the server is shutting down, invalidating the pending requests (#3576)
  • Consumers on a Limits policy stream could have their replicas changed to R1 (#3576)
  • Ensure that RAFT communication is properly stopped when needed, which otherwise could cause server memory usage increase (#3577)
  • Adding a warning when the inbound of messages causes a lag with the storage layer. In future release, the producers may be notified through a PubAck failure that the message cannot be accepted (#3578)
  • Added pending messages/bytes to pull request errors and status: when the server responds to the client library that a request has timed-out, or server is shutdown, etc..., the response will now include the request pending messages and bytes (#3572, #3587)
  • More messages than the max_msgs_per_subject value could be recovered on server restart following an abnormal server exit (#3579, #3582)
• Leafnode:
  • Existing subscriptions would be sent to leafnodes even though they violated permissions. The publish side would be doing the right thing by not sending the messages over, but the subscription interest was still sent (#3585)
• MQTT:
  • Subjects mapping were not working. Thanks to @ozon2 for the report (#3552)
• Routing:
  • An implicit route may not reconnect, regardless of the ConnectRetries setting. This can happen in configurations where the routes[] block contains only the seed (and not as a name that could resolve to each IP of the cluster). If a route to a discovered server is disconnected, it may not try to reconnect due to the implicit nature of that connection. Thanks to @wubumihuo for the report (#3573)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.3...v2.9.4

Changelog

Go Version

• 1.19.2: Both release executables and Docker images are built with this Go release.

Fixed

• JetStream:
  • Unresponsiveness (health check failures, routes being blocked) while creating a RAFT group when disk performance if very slow. Thanks to @TomasVojacek for the report (#3519)
  • Purge with additional options may leave some messages in the stream (#3529)
  • Prevent stack overflow when an account imports its own export. CVE-2022-42709 (#3538)
  • Prevent panic on negative replicas count. CVE-2022-42708 (#3539)
  • User-provided ephemeral consumer name was not used in cluster mode, that is, the server would still pick a random name (#3537)
• Added missing command line options in the -help section. Thanks to @ariel-zilber for the contributions (#3523, #3527)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.2...v2.9.3

Changelog

Go Version

• 1.19.1: Both release executables and Docker images are built with this Go release.

Improved:

• Fan-out performance degraded between v2.8.4 and v2.9.0. This was mainly due to addition of message count/size accounting per-account. Some code refactoring restored or even increased the performance compared to v2.8.4 (#3504)

Fixed

• JetStream:
  • Prevent panic processing a consumer assignment. This could happen in rare situations where a stream would catchup and start processing consumer assignments while the stream itself was "stopped", for instance during a cluster-reset event following a sequence mismatch detection, etc... (#3498)
  • The FileStore implementation could have held into memory of message blocks for longer (about 5 seconds) than needed. In situations where the stream was filling up quickly, the amount of blocks held in memory could grow enough that even 5 seconds expiration could cause the memory growth to be noticeable, even more so if the garbage collection has no reason to trigger if the overall memory usage is below the host's limit (#3501)
  • Scaling a consumer down to a R=1 would work but not send a response to the CLI/application requesting that change. That is, if an application would call js.UpdateConsumer() (using the Golang client library) with a replica of 1, the call would timeout but the operation would actually succeed (#3502)
  • The consumer names paging had issues and could have returned only the API limit, which for this request is 1024 (#3510)
  • Some streams may fail to be recovered if a meta-layer snapshot was done but an upstream source or mirror changed it subjects (#3508)
  • Redeliveries for consumers with the "LastPerSubject" delivery policy were not honored. Thanks to @brentd for the report (#3511)
• Possible exit of NATS Server running as a Windows service without logging enabled when server would report invalid protocols. The workaround is to enable logging (specify a log file as opposed to have the logging directed to the Windows Event Logs). Thanks to @BentTranberg for the report (#3497)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.1...v2.9.2

Changelog

Go Version

• 1.19.1: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Ability to remove a server by peer ID instead of server name (#3473)
  • Peer ID in the meta_cluster of statsz or jsz monitoring endpoint (#3476)
  • Ability to apply a discard new policy per subject. A new JSON field in the stream configuration called discard_new_per_subject can now be set (along with discard new policy and max messages per subject > 0) (#3487)

Improved

• JetStream:
  • Optimize acknowledgment handling. Thanks to @neilalexander for the report and contribution (#3468, #3472)

Updated

• Dependencies (#3491)

Changed

• JetStream:
  • When filtering a source stream, use the new consumer create API subject (#3478)

Fixed

• JetStream:
  • Peer randomization when creating consumers group for replicas of 1. Thanks to @goku321 for the contribution (#3470)
  • Added an error if consumer's Name and Durable are not equal when sending to the new $JS.API.CONSUMER.CREATE.%s.%s.%s subject (#3471)
  • Server was not sending a 409 to the client library when a pull request was closed after sending at least a message but could not send more if that would exceed the max_bytes pull request limit (#3474)
  • Possible panic on peer remove on server shutdown (#3477)
  • Filtered consumers may also receive messages on other subjects from the stream. Thanks to @perestoronin for the report (#3486)
• LeafNode:
  • A server that accepts a leaf connection on the websocket port, and the websocket{} block had a no_auth_user defined, this user was not being used for the account binding for that leaf node connection (#3489)
• Edge condition handling in {{Split()}} subject mapping function (#3463)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.9.0...v2.9.1

Changelog

IMPORTANT NOTE: Leafnode connections will now be closed if the cluster name is detected to be the same on the "hub" and "spoke", and reconnect attempts will be delayed by 30 seconds. If you are unaware of this misconfiguration you may experience a split network for longer than expected during the upgrade process. See the CHANGED section below and more details in pull request #3232.

Go Version

• 1.19.1: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • A new stream configuration field AllowDirect allows capable client libraries to have a new API that retrieves a message from any member of the group (leader or replicas). Note that this can lead to non-coherent reads after write since a replica may respond to the request although that replica does not yet have the latest write. The configuration field MirrorDirect allow the mirror to be part of the origin's group (#3158, #3221, #3238, #3247, #3252, #3325, #3329, #3358, #3380, #3392, #3441)
  • Support for InactiveThreshold for durable consumers. If the inactivity threshold is specified, a durable that is offline for more than this duration will be removed (#3190)
  • Ability for an operator to move streams, and support for tags reload (#3217, #3236, #3234, #3270, #3354, #3376, #3419)
  • Support for filter subject in a mirror configuration (#3227)
  • Support for consumer replica change. Thanks to @goku321 for the report (#3293)
  • Support for account purge operation by sending a request to $JS.API.ACCOUNT.PURGE.<account name>. Thanks to @goku321 and @sourabhaggrawal for the report (#3319, #3378)
  • Support for AES-GCM cipher encryption for FileStore (#3371)
  • Ability to override the default server limit for stream catchup. For instance jetstream: { max_outstanding_catchup: 32MB }. This limit is how many bytes in total may be inflight during streams catchup. This can help lower network bandwidth saturation (#3418)
  • Pagination for StreamInfo requests. Note that not all clients may have the ability to provide the offset at the time of the server release (#3454)
• LeafNodes:
  • Support for a SignatureHandler in remote configurations. This is applicable to applications embedding the NATS Server. Thanks to @kung-foo for the suggestion (#3335)
• Monitoring:
  • Account specific in/out messages/bytes and slow consumers statistics in $SYS.ACCOUNT.%s.SERVER.CONNS message response (#3187)
  • New endpoint /accstatz to get specific account statistics (such as number of connections, messages/bytes in/out, etc...) (#3250, #3382)
  • The /healthz endpoint is now also available via the system account under the $SYS.REQ.SERVER.PING.HEALTHZ subject (#3250)
  • New options for the /healthz endpoint: /healthz?js-enabled=true to return an error if JetStream is disabled, and /healthz?js-server-only=true to skip the check of JetStream accounts, streams and consumers. Thanks to @mfaizanse for the contribution (#3326)
  • The /connz?auth=1 endpoint now includes a tls_peer_certs array with subject, subject public key and raw certificate sha256. Thanks to @RedShift1 for the suggestion (#3387)
  • The /jsz?accounts=true endpoint will now show non 0 values for reserved_memory and reserved_storage when applicable (#3435)
• MQTT:
  • A new configuration option consumer_inactive_threshold allow new QoS1 consumers to be removed if they are offline for more than the specified duration (based on #3190) (#3193)
• Use of a library that automatically sets GOMAXPROCS to match Linux container CPU quota. Thanks to @1995parham for the contribution (#3218, #3224, #3237, #3406)
• A new server configuration option DontListen that triggers the server to accept only "in memory" client connections. This is for embedded use-cases only and is paired with changes made in the client library nats.go. Thanks to @neilalexander for the contribution (#2360, #3225)
• Support for JWT account option DisallowBearer (#3127)
• Stubs for WebAssembly. This allows NATS Server to be built under the js/wasm target. Thanks to @neilalexander for the contribution (#2363)
• Symlink for the deb/rpm packages. Since v2.7.4, the server is installed under /usr/bin instead of /usr/local/bin as it used to. We now have added symlink to /usr/local/sbin. Thanks to @ismail0352 for the report (#3242)
• Templates to scoped signing key user permissions (#3367, #3373, #3390)
• New subject mapping functions: SplitFromLeft, SplitFromRight, SliceFromLeft, SliceFromRight and Split (#3305)
• Building of executable, deb and rpm packages for the s390x architecture (#3458)

Changed

• Gateway:
  • Phasing out of the "optimistic" mode whereby a server could send messages to the remote cluster without knowing if there was an interest or not. The remote cluster would reply with a "no interest" protocol. As of v2.9.0, servers that creates a gateway connection to a server of that version (and above) will no longer send messages in optimistic mode since it is assumed that all accounts will be switched to interest-only mode (where the subscription interest map is sent over) (#3383)
• JetStream:
  • Stream's RePublish configuration field is now a new RePublish object (and SubjectMapping has been removed) that allows for a new boolean field called HeadersOnly. This could be useful for large messages and having republish just be a signaling mechanism (#3157)
  • When a pull request exceeds the maximum bytes, the error returned will be a "409" now instead of a "408" (#3172)
  • Make pull consumers FIFO per message, not per request (#3241)
  • Accept Nats-Expected-Last-Sequence with a 0 value. The server used to ignore if the sequence was 0, but now it will treat it as a requirement that the stream be empty if the header is present with a value of 0. Thanks to @bruth for the suggestion (#3038)
  • A consumer "Maximum Deliveries" count can now be updated. Thanks to @abegaj for the contribution (#3265)
  • Encryption of meta and RAFT stores (#3308)
  • Now return an error if there is an overlap between a source/mirror filter subject and the existing origin stream's subjects (#3356)
  • Compression in RAFT and stream catchup traffic when nodes are v2.9.0+. This can reduce network bandwidth (#3419)
• LeafNodes:
  • On establishment, the connection will now be closed when the same cluster name is detected on the "hub" and "spoke" side (#3232)
• Default to essential client information. We did default to full sharing for an export from the system account, the main one being JetStream (#3220)
• The queue group named _sys_ is now reserved and an application attempting to use it would get a permission violation (#3246)

Improved

• JetStream:
  • Server under heavy load and low on resources like file descriptors (#3168)
  • Performance of wildcard filtered consumer with stream with many subjects (#3184)
  • Better distribution in placement of streams in a cluster when no "max bytes" is set (#3194)
  • When the user adds a stream that already exists with a different configuration, the error description returned to the application was "stream name already in use", it will now be "stream name already in use with a different configuration". Also, when the user tries to restore a stream, if the stream was already present, the operation would fail with "stream name already in use", it will now be "stream name already in use, cannot restore". Thanks to @AndrewDK for the suggestion (#3280)
  • Speed of storing new keys with a large number of pre-existing keys (#3320)
  • Replicas ordering and information regarding unknown replica in stream information API response (#3347)
  • Catchup logic (#3348)
  • Catchup aborted on requester failure and better flow control (#3349)
  • FileStore for large KeyValue streams (#3351, #3353, #3366, #3401, #3413, #3416)
  • Usage of the FileStore implementation for the RAFT logs (#3377, #3394)
  • General improvements to clustered streams during server restart and KV/Compare-And-Swap use cases (#3392)
  • Ability to secure the creation of consumer for libraries sending consumer creation requests to the new subject $JS.API.CONSUMER.CREATE.<stream>.<subject>.<filter> (#3409)
  • Better accounting for max-bytes for pull consumers (#3456)
  • Better error description when an asset placement fails (#3342, #3459)
• Server banner for Trusted Operators now displays the Expires date as Never instead of the epoch time. Thanks @mschneider82 for the contribution (#3452)

Updated

• Dependencies (#3153, #3263, #3286)
• JetStream:
  • Allow consumer filter subjects to be updated (#3216)
  • AckAll and NoAck are now allowed on pull consumers (#3442)

Fixed

• Configuration Reload:
  • Configuration reload would fail if a leafnodes block would contain an authorization{} block, even if no modification was done there. Thanks to @cleaton for the report (#3204)
  • Some data races that could also lead to a server panic when processing a subscription and trying to send it to routes (#3222)
• Gateway:
  • Possible panic if monitor endpoint is inspected too soon on startup (#3374)
  • Possible routing issues with System messages and JetStream when internal subscriptions are created and internal requests are sent through gateways. The replies may sometimes be missed (#3427)
• JetStream:
  • Suppress consumer and stream advisories on server restart and any direct stream get message (#3156, #3160, #3162)
  • Possibly fail to retrieve a newly stored message. This would happen when stores and load using "last for subject" were concurrent (#3159)
  • When using Republish feature, republish on the republish subject and place original subject in a Nate-Subject header - similar to stream direct get messages (#3169)
  • Data race with account exported services. Thanks to @Davincible for the report (#3189)
  • Path separators in consumer or stream names prevented restoring the stream. Thanks to @daudcanugerah for the report (#3205)
  • Pull consumer may be incorrectly removed after the InactiveThreshold interval (#3229)
  • When a pull consumer is stalled for MaxAckPending, expire all pull requests that had at least 1 delivered message (#3241)
  • Possible bad stream placement when increasing the stream replica's value (scale up) and using unique_tag configuration (#3248)
  • Stalled stream catchup in endless cycle on EOF error trying to retrieve a catchup message (#3249)
  • Servers may be reported as orphaned (considering the node offline) (#3258)
  • Updates to a stream mirror configuration were not rejected in standalone mode, as they are in cluster mode (#3269)
  • Possible panic when removing a mirror configuration from a stream, which was possible because of the issue above. Thanks to @chris13524 for the report (#3269)
  • A stream with RePublish configuration would republish any incoming message, regardless of the RePublish's source subject (#3271)
  • Raft issue that could cause a follower's log index to be ahead of the leader's by 1 (#3277)
  • Possible panic in cluster mode. Thanks to @vishaltripathi24 for the report (#3279)
  • Scale down of a stream was not always waiting for the scale down of its consumers (#3282)
  • Short index write could lead to loss of stream sequence for an empty stream (#3283)
  • Losing stream sequence on multiple restarts and leader changes (#3284)
  • In clustering mode, the "stream names" API could return more than the JS API limit, and "stream infos" list would return that the total of stream is the JS API limit of 256 channels, when in reality it could be more (#3287)
  • Scaling up and some RAFT issues (#3288)
  • Reject update of a pull consumer MaxWaiting configuration since it is not currently supported (#3302)
  • Catching up a RAFT follower that involved snapshots (#3307)
  • Instability with encrypted systems (#3314)
  • Reject stream update with changes to the RePublish configuration since it is not currently supported (#3328)
  • Internal consumer restart on source filter update (#3355)
  • The proper error regarding "subject overlap" was not returned in standalone mode (was correct in clustering mode). Thanks to @swDevTX for the report (#3363)
  • Messages that had reached the max delivery count may be redelivered after a server or cluster restart (#3365)
  • Issue if the source consumer changes the subjects filter (#3364)
  • Possible race during an asset scale down that could cause a node to prematurely being considered caught up (#3381)
  • Consumer subject validation on recovery which caused delivery of messages to a downstream stream that should not have been delivered (#3389)
  • Issues with concurrent use of Nats-Expected headers, such as concurrent uses of kv.Create()/kv.Delete() (#3400)
  • Stream information numbers may be 0 after a cluster restart (#3411)
  • Wrong streams returned in some conditions involving wildcard in subjects and filters (#3423)
  • Expired/removed accounts were counted toward limits. Thanks to @JulienVdG for the contribution (#3421, #3428)
  • Don't allow subjects overlap between multiple subjects for a stream (for instance configuration that had subjects "foo.*", "foo.bar" would create duplicate messages if produced on foo.bar and will now be reported as an invalid configuration (#3429)
  • Some nodes in a super cluster may never be reported as offline in some situations (#3433)
  • Restarted queue subscriptions may not receive messages. Thanks to @kyle8615 for the report (#3440)
  • Durables with replicas of 1 but different from the stream's replicas could possibly be incorrectly migrated on server shutdown (#3451)
• MQTT:
  • Possible panic when clients repeatedly connect with a client ID while that is already connected. Thanks to @gebv for the report (#3315)
• Delivery of internal system messages to queue subscribers across routes. Thanks to @rebelf for the report (#3185)
• Memory leak when unsubscribing the last queue subscription on a given queue group (#3338)
• Issue with services import/export cycles that could cause stack-overflow. Thanks to @apis for the report (#3407)
• Strings that started with a number and had some of the "bytes" suffix (such as K/G/etc..) but not as suffixes were not parsed properly. Thanks to @julian-dolce-form3 for the report (#3434, #3436)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.8.4...v2.9.0

Changelog

Go Version

• 1.17.10: Both release executables and Docker images are built with this Go release.

Fixed

• JetStream:
  • Consumer pending count was not correct when a stream had max messages per subject greater than 1 and a consumer that filtered out part of the stream was created (#3148)
  • Spurious 408 status messages sent to pull consumers. Also made some modifications to reduce lock contention between processing of acknowledgments and delivery of messages (#3149)
• MQTT:
  • Connections with same client ID but on different domains may have been detected as duplicates (#3150)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.8.3...v2.8.4

Changelog

Go Version

• 1.17.10: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Consumer configuration has now MaxRequestMaxBytes as a int to limit a pull request to that many bytes (#3126)
  • Allow explicit consumer configuration of replica count and memory storage (#3128, #3131). Note: This feature is not compatible or tested in conjunction with modifying the associated stream's replication factor once created (2.7.3+).
  • EXPERIMENTAL FEATURE: Stream configuration has now RePublish which is a SubjectMapping with a Source and Destination. This enables lightweight distribution of messages to very large number of NATS subscribers (#3129, #3138)
• MQTT:
  • New configuration parameters stream_replicas, consumer_replicas and consumer_memory_storage, which allow the override of the replica count for MQTT streams (which are normally auto-determined base on the cluster size, but capped at 3) and also override the consumer replica count and possibly underlying storage (based on feature added in #3128). Thanks to @ianzhang1988 for the feedback (#3130)

Improved

• JetStream:
  • Tweak RAFT election timeouts and heartbeats and also reduce lock contention (#3132, #3135)
• Monitoring:
  • A route information in /routez will now include start, last_activity, uptime and idle fields (#3133)

Fixed

• JetStream:
  • Panic due to non loaded message cache during a file compaction (#3110)
  • Downstream sourced retention policy streams during server restart have redelivered messages (#3109, #3122)
  • Possible lockup due to a missing mutex unlock in some conditions (#3115)
  • Panic when processing a consumer create in clustered mode. This was introduced in v2.8.2 (#3117)
  • Some data races (#3120, #3134)
  • Panic runtime error: makeslice: cap out of range on server restart. Thanks to @ajlane for the report (#3121)
  • A purge with a given subject and a sequence of 1 for a stream with a single message would not work (#3121)
  • Some JS API requests handled through routes or gateways may be dropped when there were too many of them at once (#3142)
  • Consumer with "deliver new" delivery mode could skip messages in some conditions after a consumer step down or server(s) restart (#3143)
• MQTT:
  • Errors deleting consumers will now prevent deletion of the session record, which would otherwise leave orphan consumers. Thanks to @ianzhang1988 for the report (#3144)
• Accounts Export/Import isolation with overlap subjects. Thanks to @JH7 for the report (#3112)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.8.2...v2.8.3