nats-server

Changelog

Go Version

• 1.17.9: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • A JSConsumerDeliveryNakAdvisory when a message is nak'ed. Thanks to @Coffeeri for the contribution (#3074)

Improved:

• JetStream:
  • KeyValue memory store performance for history of 1 (#3081)
  • A check was done twice (per message) for max_msg_per_subject and discard_new policy, which is the case for KV stores (#3101)
  • Reduce the number of RAFT warnings to 1 per second (for exact same log line) (#3100)
  • Bumped the default file store block size (#3102)

Changed

• JetStream:
  • When reaching the max number of pull requests, the server would evict existing pending requests (with a 408 Request Canceled) instead of sending a 409 Exceeded MaxWaiting to the new request (#3099)

Fixed

• JetStream:
  • Step-down timing for consumers or streams. When becoming leader a consumer's stream sequence number could be reset (#3079)
  • Not able to recover a consumer if a source based stream which housed the consumer filter subject was removed. Also made updates to the /healthz behavior (#3080)
  • File store compaction would not work well for blocks with small messages and lots of interior deletes. The index file was also written too often (#3087)
  • Subject transforms and delivery subjects (#3088)
  • Lock inversion, but no known situations that would have triggered it (#3092)
  • Possible panic when checking for RAFT group leaderless status (#3094)
  • A pull consumer with a MaxWaiting of 1 would not send the notification that the request timed-out (#3099)
  • RAFT and clustering issues that may have contributed to repeated warnings about inability to store an entry to WAL and expected first catchup entry to be a snapshot and peer state (#3100)
• Permissions:
  • Queue subscription's deny subjects were not always enforced. CVE-2022-29946 (#3090)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.8.1...v2.8.2

Changelog

Go Version

• 1.17.9: Both release executables and Docker images are built with this Go release.

Changed:

• JetStream:
  • Enforce a minimum of 100ms for "max age" and "duplicates window" settings (#3056)

Updated:

• Dependencies:
  • golang.org/x/crypto due to a CVE scan. However, since this affects crypto/ssh that the server is not using, the vulnerability does not impact the NATS Server. Thank you to @pgvishnuram for the contribution (#3065)

Fixed

• JetStream:
  • Assets migration may fail due to an aggressive internal polling mechanism (#3059)
  • Mirror streams would fail to be recovered from an earlier NATS Server versions with an error regarding duplicate window. Thanks to @yadvlz and @BenChen-PKI for the reports (#3060, #3067)
  • Mirror and Sources issues, especially in a mixed-mode super cluster layout (#3061, #3066)
  • A stream with a source may be canceled when the stream was updated. Thanks to @sargikk for the report (#3061)
• LeafNode:
  • Failed to propagate subscription interest after a configuration reload. Thanks to @LLLLimbo for the report (#3058)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.8.0...v2.8.1

Changelog

Go Version

• 1.17.9: Both release executables and Docker images are built with this Go release.

Added:

• LeafNode:
  • Support for a min_version in the leafnodes{} that would reject servers with a lower version. Note that this would work only for servers that are v2.8.0 and above (#3013)
• Monitoring:
  • Server version in monitoring landing page (#2928)
  • Logging to /healthz endpoint when failure occurs. Thanks to @samuel-form3 for the contribution (#2976)
  • MQTT and Websocket blocks in the /varz endpoint (#2996)
• JetStream:
  • Consumer check added to healthz endpoint (#2927)
  • Max stream bytes checks (#2970)
  • Ability to limit a consumer's MaxAckPending value (#2982)
  • Allow streams and consumers to migrate between clusters. This feature is considered "beta" (#3001, #3036, #3041, #3043, #3047)
  • New unique_tag option in jetstream{} configuration block to prevent placing a stream in the same availability zone twice (#3011)
  • Stream Alternates field in StreamInfo response. They provide a priority list of mirrors and the source in relation to where the request originated (#3023)
• Deterministic subject tokens to partition mapping (#2890)

Changed:

• Gateway:
  • Duplicate server names are now detected across a super-cluster. Server names ought to be unique, which is critical when JetStream is used (#2923)
• JetStream:
  • Processing of consumers acknowledgments are now done outside of the socket read go routine. This helps reducing occurrences of "Readloop" warnings for route connections when there was a lot of pending acknowledgements to process per consumer. The redeliveries are also possibly postponed (by chunk of 100ms) to favor processing of acknowledgements (#2898)
  • Lower default consumer's "Maximum Ack Pending" from 20,000 to 1,000. This affects only applications/NATS CLI that do not set an explicit value. If you notice a performance degradation and your system can handle very well a higher value, then the parameter should be explicitly configured to a higher value when creating the consumer/subscription (#2972)
• Duplicate user names in authorization{} and accounts{} blocks are now detected and will fail the start of the server. Thanks to @smlx for the report (#2943)

Improved

• Configuration:
  • Skip exact duplicate URLS for routes, gateways or leaf nodes. Duplicate route URLs could cause issues for JetStream cases since it may prevent electing a leader (#2930)
• Logging:
  • Limiting rate of some identical warnings (#2994)
• JetStream:
  • Behavior of "list" and "delete" operations for offline streams has been improved (responses no longer hanging or failing) (#2925)
  • When a consumer had a backoff duration list, the server could check for redeliveries more frequently than it should have. The redelivery timing was still honored though (#2948)
  • Ensures the cluster information in /jsz monitoring endpoint is sent from the leader only (#2932, #2983, #2984)
  • Memory pooling and management (#2960)
  • Consumer snapshot logic and disk usage in clustered mode. Thanks to @phho for the report (#2973)
  • Performance of ordered consumers (and stream catchup) with longer RTTs (#2975)
  • Performance for streams containing multiple subjects and consumer with a filter. Thanks to @samuel-form3 for the report (#3008)
  • Reduction of unnecessary leader elections (#3035)
  • On recovery, the server will now print the filenames for which bad checksums were detected (#3053)

Fixed

• JetStream:
  • Consumer state handling, for instance a consumer with "DeliverNew" deliver policy could receive old messages after a server restart in some cases (#2927)
  • Removal of an external source stream was not working properly. Thanks to @kylebernhardy for the report (#2938)
  • Possible panic on leadership change notices (#2939)
  • Possible deadlock during consumer leadership change. Thanks to @wchajl for the report (#2951)
  • Scaling up a stream was not replicating existing messages (#2958)
  • Heavy contention in file store could result in underflow and panic. Thanks to @whynowy for the report (#2959)
  • Consumer sample frequency not updated during a consumer update. Thanks to @boris-ilijic for the report and contribution (#2966)
  • Some limit issues on update (#2945)
  • Memory based replicated consumers could possibly stop working after snapshots and server restart. The $SYS folder could also being seen as growing in size. Thanks to @phho and @MilkyWay-core for the reports (#2973)
  • Possible panic (due to data races) when processing pull consumer requests. Thanks to @phho for the report (#2977)
  • Account stream imports were not removed on configuration reload. Thanks to @alirezaDavid for the report (#2978)
  • Sealed streams would not recover on server restart (#2991)
  • Possible panic on server shutdown trying to migrate ephemeral consumers (#2999)
  • A "next message" request for a pull consumer, when going over a gateway but down to a Leafnode could fail (#3016)
  • Consumer deliver subject incorrect when imported and crossing route or gateway (#3017, #3025)
  • RAFT layer for stability and leader election (#3020)
  • Memory stream potentially delivering duplicates during a node restart. Thanks to @aksdb for the report (#3020)
  • A stream could become leader when it should not, causing messages to be lost (#3029)
  • A stream catchup could stall because the server sending data could fail during the process but still send an indication that the other server that catchup did complete (#3029, #3040)
  • Route could be blocked when processing an API call while an asset was recovering a big file (#3035)
  • Assets (streams or consumers) state could be removed if they had been recreated after being initially removed (#3039)
  • When running on mixed-mode, a JetStream export could be removed on JWT update (#3044)
  • Possible panic on cluster step-down of a consumer (#3045)
  • Some limit enforcement issues and prevent loops in cyclic source stream configurations (#3046)
  • Some stream source issues, including missing messages and possible stall (#3052)
  • On configuration reload, JetStream would be disabled if it was enabled only from the command line options. Thanks to @xieyuschen for the contribution (#3050)
• Leafnode:
  • Interest propagation issue when crossing accounts and the leaf connection is recreated. This could also manifest with JetStream since internally there are subscriptions that do cross accounts. Thanks to @LLLLimbo and @JH7 for the report (#3031)
• Monitoring:
  • reserved_memory and/or reserved_storage in the jetstream{} block of the /varz endpoint could show incorrect huge number due to a unint64 underflow (#2907)
  • verify_and_map in the tls{} block would prevent inspecting the monitoring page when using the secure https port. Thanks to @rsoberano-ld for the report (#2981)
  • Possible deadlock when inspecting the /jsz endpoint (#3029)
• Miscellaneous:
  • Client connection occasionally incorrectly assigned to the global account. This happened when the configuration was incorrectly referencing the same user in authorization{} and accounts{}. Thanks to @smlx for the report (#2943)
  • The NATS account resolver, while synchronizing all JWTs, would not validate the nkey(s) or jwt(s) received via the system account (CVE-2022-28357) (#2985)
  • Reject messages from application that have an invalid reply subject (contains the $JS.ACK prefix) (#3026)
  • Allow server to run as system user on Windows. Thanks to @LaurensVergote for the contribution (#3022)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.7.4...v2.8.0

Changelog

Note about .deb/.rpm packages

We use Goreleaser to build our assets, and it seems that it changed the default install from /usr/local/bin to /usr/bin. See Goreleaser's change here. We are sorry for the inconvenience this may cause.

Go Version

• 1.17.8: Both release executables and Docker images are built with this Go release.

Improved

• JetStream:
  • Better startup logging to help debug RAFT log directories to streams/consumers (#2897)

Fixed

• JetStream:
  • The consumers count when getting stream information could be wrong in clustered mode (#2896)
  • Never used clustered and filtered consumers consume storage under $SYS. Thanks to @nayanparegi and @aksdb for the reports (#2899, #2914)
  • Stream not recovered on restart with "deleted message", "checksum" or "no message cache" errors (#2900)
  • Schema violations in the NATS CLI tool caused by large number overflow when "active" field for Sources and Mirrors was computed and there had been no contact yet (#2903)
  • Some Stream advisories were missing (#2887)
  • Inconsistent durable consumer state after stream peer removal (#2904)
  • Scaling up and down for streams and consumers (#2905)
  • Validate files' path when restoring stream from a snapshot/backup (CVE-2022-26652). Thanks to Yiming Xiang for the report (#2917)
• Monitoring:
  • Panic on non 64-bit platforms due to an unaligned 64-bit atomic operation. Thanks to @mlorenz-tug for the report (#2915)
• LeafNode:
  • Queue subscription interest could be suppressed in complex situations causing messages to not flow from a LeafNode server to the rest of the (super)cluster (#2901)
• Fixed some lock inversions (#2911)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.7.3...v2.7.4

Changelog

Go Version

• 1.17.7: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Allow replica updates during stream update (#2867)
  • Allow stream placement by tags (#2871)

Updated

• Dependencies:
  • github.com/klauspost/compress: v1.13.4 -> v1.14.4
  • github.com/minio/highwayhash : v1.0.1 -> v1.0.2

Improved

• JetStream:
  • Sparse consumers replay time. This is when a stream has multiple subjects and a consumer filters the stream to a small and spread out list of messages (#2848, #2863)
  • Small improvements to send performance when sending to a full stream (#2877)

Fixed

• Gateway:
  • Connection could fail with 'Authorization Violation' and parser error due to an initial PING possibly sent prior to the CONNECT protocol. Also, the server accepting a connection was not starting the authentication timer, which responsibility is to close the connection if the CONNECT protocol is not received within the gateway's authentication timeout. It can be configured with authorization{ timeout: 5.0 } in the gateway{} block, but if not set will default to 2 seconds. Thanks to @jimenj1970 for the report (#2881)
• JetStream:
  • Flow control stall under specific conditions of messages size (#2847)
  • A stream name is tied to its identity and cannot be changed on restore (#2857)
  • The consumer information's state in response to a consumer create request could return inaccurate information, specially regarding number of pending messages (#2858)
  • Remove "fss" files from a snapshot when a block is removed. Thanks to @nekufa for the report (#2859)
  • Prevent a panic when stream restore fails (#2864
  • Interest policy and staggered filtered consumers could fail to remove messages. Thanks to @lukasGemela for the report (#2875)
  • Remove "key" files when removing message blocks and encryption is used. Thanks to @exename for the report (#2878)
  • Reduce logging for internal message exchange (#2879)
  • Rare possible re-use of internal RAFT inbox that could cause a node to receive a message on the wrong inbox, possibly leading to a panic. Servers would likely have to be started at the exact same time for that to happen (#2879)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.7.2...v2.7.3

Changelog

Go Version

• 1.17.6: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Ability to get per-subject details via StreamInfo responses. The request needs to include a subjects_filter:<subject> for the server to include the list. The StreamInfo structure now has NumSubjects that will always be set, but optionally Subjects which contains the list of distincts subjects present in the stream with the count of messages per subject (#2833)

Removed

• Dynamic account behaviors. The server option AllowNewAccounts and function NewAccountsAllowed() have been removed. Note that the option could only be set for applications embedding the NATS Server since configuration parsing was not parsing this option (#2840)

Fixed

• JetStream:
  • Adding streams may fail with "insufficient resources" in some cases (#2824)
  • Possible panic when attempting to update a push consumer by removing the deliver subject (#2829)
  • Consumer updates were not updated on disk and would be reverted after a server restart. Thanks to @oliverpool for the report (#2830)
  • Behavior of a stream when MaxMsgsPerSubject is set along with DiscardNew discard policy. Unless the stream is reaching a limit, old messages will be removed on a per-subject basis (#2831)
  • A pull request no_wait:true without expiration was not considering redeliveries (#2832)
  • BackOff redeliveries would always use the first delay from the list if the consumer's sequence was not matching the stream sequence (#2834)
  • Under certain scenarios, the number of pending messages (unprocessed messages reported by the NATS CLI) for a consumer could appear to get stuck at 0 (#2835)
  • When a consumer had no filtered subject and was attached to an interest policy retention stream, the server could incorrectly drop messages (#2838)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.7.1...v2.7.2

Changelog

Go Version

• 1.17.6: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Support for a delay when Nak'ing a message (#2812)
  • Support for a backoff list of times in the consumer configuration (#2812)
• Monitoring:
  • Replication lag metrics added to the jsz endpoint. Thanks to @mattstep for the contribution (#2791)
  • /healthz endpoint which will return 200 with {status: ok} if and only if all configured ports are opened and, if JetStream is configured, there is contact with the meta leader and is current and all streams are up to date, otherwise returns 503 with { "status": "unavailable", "error": "DESCRIPTION" } body (#2815)

Fixed

• JetStream:
  • Pull consumers have been reworked (#2813)
  • A stream's state number of deleted messages could be negative (#2814)
  • A message's "in progress" request may not be honored in case of consumer's leadership change (#2812)
• Git commit hash was not present in v2.7.0 (#2790)
• Possible deadlock during Client or Leaf JWT authentication (#2803)
• Do not emit warning about plaintext password when only using accounts with NKey and setting system_account (#2811)
• JWT based user/activation token revocation and granularity. Thanks to @shurya-kumar for the report (#2816)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.7.0...v2.7.1

Changelog

Notice for JetStream Users

See important note if using LeafNode regarding domains.

Go Version

• 1.17.6: Both release executables and Docker images are built with this Go release.

Added

• Configuration:
  • Ability to configure account limits (max_connections, max_subscriptions, max_payload, max_leafnodes) in server configuration file (#2755)
• JetStream:
  • Overflow placement for streams. A stream can now be placed in the closest cluster from the origin request if it can be placed there (#2771, #2779)
  • Support for ephemeral Pull consumers (client libraries will need to be updated to allow those) (#2776)
  • New consumer configuration options (#2776):
    • For Pull Consumers:
      • MaxRequestBatch to limit the batch size any client can request
      • MaxRequestExpires to limit the expiration any client can request
    • For ephemeral consumers :
      • InactiveThreshold duration that instructs the server to cleanup ephemeral consumers that are inactive for that long
  • Ability to configure max_file_store and max_memory_store in the jetstream{} block as strings with the following suffixes K, M, G and T, for instance: max_file_store: "256M". Thanks to @hooksie1 for the contribution (#2777)
  • Support for the JWT field MaxBytesRequired, which defines a per-account maximum bytes for assets (#2779)
• MQTT:
  • Support for websocket protocol. MQTT clients must connect to the opened websocket port and add /mqtt to the URL path. Thanks to @Etran-H for the suggestion (#2735)
• TLS:
  • Ability to rate-limit the clients connections by adding the connection_rate_limit: <number of connections per seconds> in the tls{} top-level block. Thanks to @julius-welink for the contribution (#2573)

Improved

• JetStream:
  • MemStore is improved in the presence of a lot of interior delete messages, which could happen for instance with the use of KV store (#2752)
  • Pull consumers behavior: more resilient, better at waiting request management, and the ability to determine and watch interest cross account (#2776)
  • In clustering mode, with streams with replication factor greater than 1, a server will now use much less memory. For instance: 1,000 streams, with 10 consumers each would use about 15GB of memory, with this change memory usage would go down to about 1.5GB (#2775)
• Monitoring:
  • Make the HTTP(s) monitoring endpoints available before starting JetStream to be able to monitor whether it is available during startup and recovery of streams. This could help with health probes (#2782)
• Websocket:
  • Added client IP from X-Forwarded-For header. Thanks to @byazrail for the suggestion (#2734, #2769)

Changed

• JetStream:
  • JetStream API traffic will now be denied across leaf nodes, unless the system account is shared and the availability domain is identical (#2693)

Updated

• Some dependencies (#2727)

Fixed

• JetStream:
  • Possible panic "could not decode consumer snapshot". Thanks to @raoptimus for the report (#2738)
  • Added missing entries to stream and consumer list requests (#2739)
  • Interest across gateways that would prevent a push consumer from receiving messages. Thanks to @ZMing316 for the report (#2750)
  • Stream could "stall" after printing warnings such as "AppendEntry failed to be placed on internal channel" (#2751)
  • Stream (with WorkQueue retention policy) could have its first/last sequences reset to 0 after a non-clean server restart, causing pull subscriptions to fail getting newly published messages (#2753)
  • Stall of consumers after getting a "partial cache" error message in the server log. Thanks to @rino-pupkin, @raoptimus, @broken-ufa and @abishai for the report (#2761)
  • Large number of ephemeral consumers could exhaust Go runtime's maximum threads. Thanks to @rh2048, @OpenGuidou for the report (#2764)
  • Restarting a cluster with lots of streams/consumers could cause routes to break with "write deadline" in some situations (#2780)
  • Changes to inter-process communications could solve some issues caused by inability to send messages to internal Go channels (#2775, #2780)
  • A consumer's number of pending messages (or "Unprocessed Messages" in NATS cli) could be wrong when the consumer was created with inflight messages. Stepping down of the consumer leader would solve the discrepancy (#2776)
  • Possible file store deadlock in when trying to rebuild some state (#2781)
• Monitoring:
  • Possible panic when inspecting the /jsz endpoint. Thanks to @rh2048 for the report (#2743)
• TLS:
  • When setting verify_and_map to true, if a connection connects with a certificate with an ID matching an existing user, but that user's allowed_connection_types is specified and does not have the type of the connection trying to connect, the server would panic (#2747)
• Check for no_auth_user should be done only when no authentication at all is provided by the connection (#2744)
• System account issue where the wrong structure was updated (#2757)
• Broken link to monitoring documentation. Thanks to @kfabryczny for the contribution (#2766)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.6.6...v2.7.0

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.10: Both release executables and Docker images are built with this Go release.

Changed

• Profiler:
  • Moved the start of the profiler earlier in the start sequence so it is possible to get some insight should a start take a long time (for instance recovering JetStream datastore) (#2723)
• Added LEAFNODE_WS in allowed_connection_types to allow or prevent Leafnode over websocket. Previously, if allowed_connection_types was specified and contained LEAFNODE, this user could be used to create a Leafnode connection over websocket. The existing WEBSOCKET connection type is used for regular client applications (#2707)

Fixed

• JetStream:
  • Slowness with large interior deletes (#2711)
  • Possible panic when removing a source from a stream. Thanks to @kylebernhardy for the report (#2712)
  • Reject invalid subjects in stream configuration, and attempt to fix them on recovery (#2717, #2721)
  • Prevent stream update to add subjects to mirror and remove them on recovery (#2718, #2721)
  • Application may receive corrupted headers when receiving from a consumer configured with "MetaOnly" option (#2719)
  • Aligned internal timeout to 4 seconds for stream and consumer list requests (#2722)
  • Incomplete error when consumer list in cluster mode failed (#2725)
• Leafnode:
  • When using allowed_connection_types, it was not possible to restrict a Leafnode connection over websocket (#2707)
  • Reaching the account connections limit left an outgoing Leafnode connection in a bad state (#2715)
• Monitoring:
  • TLS configuration changes would not take effect after a configuration reload (#2714)
• Documentation:
  • Links in the README. Thanks to @Shoothzj for the contribution (#2724)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.6.5...v2.6.6

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.10: Both release executables and Docker images are built with this Go release.

Added

• Server option AlwaysEnableNonce to force the server to send a nonce in the INFO block, regardless if the client connects with a NKey or not. Since its primarily useful to embedded scenarios there is no corresponding option in configuration file (#2696, #2699)

Fixed

• JetStream:
  • The stream list API request was not using the optional "subject" filter (#2695)
  • Followers may keep trying to reset streams, due to either snapshots not containing any actionable data or when stream had lot of failures by design (such as KV abstraction) (#2702)
  • Memstore: setting max messages per subject to 1 would not be properly enforced (#2704)
• OCSP:
  • Allow intermediates, or designated responders, to sign responses on behalf of the CA (#2692)
• A slow consumer could cause the publisher to block. Thanks to @cpiotr, @GuanYin108 and @lukedodd for the reports (#2684)
• Possible panic due to latency tracking when more than one replies are sent (#2688)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.6.4...v2.6.5

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.10: Both release executables and Docker images are built with this Go release.

Improved

• JetStream:
  • Allow system account to respond with "JetStream not enabled", which makes interactions with NATS cli better (#2671)
  • Allow certain consumer attributes to be updated. Thanks to @andreib1 for the suggestion (#2674)
• Websocket:
  • Add client IP in websocket upgrade error messages (#2660)

Fixed

• JetStream:
  • Messages not expiring properly after server restart (#2665)
  • Conditional failures for stream messages could cause stream resets (#2668)
  • Duplicate stream create returned wrong response type (#2669)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.6.3...v2.6.4

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.9: Both release executables and Docker images are built with this Go release.

Fixed

• Gateways:
  • Each server within a cluster gossips its gateway URL, however, when two servers create route to each other and one route is dropped as a duplicate, a server may lose the peer's gateway URL and therefore would not be able to gossip it to inbound gateway connections (#2653)
• JetStream:
  • Purging would, in some cases, not honor the amount of messages to keep after purge (#2623)
  • Panic caused by the Unlock of unlocked RWMutex, which would happen through non client connections when the limit of JetStream API in-flight requests was reached. Thanks to @minight for the report (#2645)
  • Cluster becomes inconsistent with constant "catchup for stream stalled" warnings. Thanks to @aksdb for the report (#2648)
  • Ensure that stream's sealed, deny delete, deny purge and allow rollup booleans in stream config are always set (for languages that may not set to false when decoding JSON when absent) and set the deny delete and purge to true and allow rollup to false when a stream is sealed (#2650)
  • Redelivered state of consumers was not properly decoded from file (#2652)
• LeafNode:
  • Apparent memory leak on "Authorization Violation" errors. Thanks to @DavidSimner for the contribution (#2637)
• Monitoring:
  • The /varz endpoint may still show gateway URLs of servers that have been removed from the remote cluster (#2647)
  • The /varz endpoint would not show the configured URLs of a remote gateway block if that remote gateway was the one of the server being inspected (#2653)
• OCSP:
  • Only last of LeafNode and Gateway remotes were being updated. Thanks to @guodongli-google for the report (#2632)
• TLS:
  • If a connection is closed before the specified TLS timeout, the connection object would be retained in the go runtime due to the TLS timeout timer not being stopped, which could be seen as a memory leak, until that timer fired (#2638)
• Redact URLs before logging. Thanks to @fowlerp-qlik for the report (#2643)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.6.2...v2.6.3

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.9: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Ability to seal a stream through a stream update (#2584)
  • Ability to do roll ups (#2585, #2593, #2600, #2601)
  • Allow consumers to request only headers to be delivered with HeadersOnly consumer configuration (#2596)
  • New stream configuration properties to limit delete/purge/rollups (#2595)

Updated

• JWT library dependency to v2.1.0

Improved

• MQTT:
  • The client ID will now be part of the client log statements. Thanks to @imranrazakhan for the report (#2598)

Fixed

• JetStream:
  • Adjust cluster size on cold start in mixed mode (#2579)
  • Issue with the "get message" audit event (#2583)
  • Some data races (#2590, #2619)
  • More robust checking for decoding append entries (#2606)
  • Issues with RAFT around snapshots and recovery (#2615)
  • Starting a consumer with "deliver last per subject" with a filtered subject same as the stream's subject would possibly return wrong NumPending value (#2616)
  • Purging filtered stream resets all stream consumer offsets. Thanks to @minight for the report (#2616, #2617)
  • Honor max memory and max file settings of 0. Thanks to @biskit for the report (#2618)
• Websocket:
  • Issue with compression and Safari browsers using the experimental feature NSURLSession WebSocket. Thanks to @byazrail for the report (#2613)
• Handling of message headers in some system requests (#2580)
• Go doc fixes and some code cleanup. Thanks to @dtest11 for the contributions (#2610, #2614)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.6.1...v2.6.2

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.8: Both release executables and Docker images are built with this Go release.

Fixed

• JetStream:
  • Deadlock when using stream mirrors with source with non limit retention, which could manifest with increase memory usage (#2561)
  • Possible panic due when getting a slice of a message block cache buffer (#2565)
  • Internal "direct consumers" used for mirroring were incorrectly subject to the "max consumers" limit (#2564)
  • Storage limit of 0 (which means disabled) was not always enforced (#2563)
  • Handling when exceeding account resources (#2569)
  • Internal connections were counted toward the account's max connections (client or leafnode) (#2568)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.6.0...v2.6.1

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See important note if upgrading from a version prior to v2.5.0.

Go Version

• 1.16.8: Both release executables and Docker images are built with this Go release.

Added

• Monitoring:
  • JetStream's reserved memory and memory used from accounts with reservations in /jsz and /varz endpoints (#2539, #2555)
• Hardened systemd service. Thanks to @c0deaddict for the contribution (#2318)

Changed

• JetStream:
  • The server store message headers with a uint 16, which limits to 64KB, however, it was not rejecting incoming messages with headers bigger than that but then was failing its recovery (#2526)
  • The server was accepting consumer creation requests with flow control enabled but without heartbeats, which could lead to the consumer stalling if the library missed the flow control message. The server will now reject such consumer creation request (#2533)
  • Trying to add a stream with a configuration that matched exactly the one of an existing stream would report an error in cluster mode, not in standalone mode. The cluster mode will now behave as standalone mode and treat the stream creation as idempotent (#2535)
  • The server will now reject negative duplicate window values (#2546)
• The server will now fail to start if a token is specified in cluster{authorization{}} or gateway{authorization{}}. Routes and Gateways have never supported token authentication, however, it was silently ignored. Thanks to @antong for the report (#2518)

Fixed

• JetStream:
  • Filestore compaction issue that could to message lookup to fail, possibly stalling consumers (#2522)
  • Reject messages with headers greater than 64KB (#2526)
  • Condition where assets could be created in multiple leafnode domains (#2529)
  • Reject consumer creation requests with a flow control but no heartbeat (#2533)
  • Stream creation request was idempotent in standalone mode, but not in clustered mode (#2535)
  • In clustered mode, getting consumer information or a just created consumer could sometimes return a "consumer not found" error (#2537)
  • Make large pull subscribe batch requests expire more efficiently (#2538)
  • Stabilize filestore implementation. Thanks to @izarraga for the report (#2545)
  • Reject duplicate window with negative value (#2546)
  • Memory store memory leak when running in cluster mode. Thanks to @anjmao for the report (#2550)
  • Some issues with RAFT WAL repair (#2549)
  • A consumer on mirror stream would fail to be recovered if its stream was recovered before the mirrored stream. Thanks to @tbeets for the report (#2554)
• Websocket:
  • Possible panic when decoding CLOSE frames. Thanks to @byazrail for the report (#2519)
• Memory leak when using Queue Subscriptions. Thanks to @andrey-shalamov for the report and fix! (#2517)
• Fail configuration that would have token specified in cluster{} or gateway{}'s authorization{} blocks. Authentication token were never supported there but the misconfiguration was silently ignored. Thanks to @antong for the report (#2518)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.6.0...v2.5.0

Changelog

Notice for JetStream Users

See important note if upgrading from a version prior to v2.4.0.

Notice for MQTT Users

See note in the Improved and Changed section.

Go Version

• 1.16.8: Both release executables and Docker images are built with this Go release.

Added

• MQTT/Monitoring:
  • MQTTClient in the /connz connections report and system events CONNECT and DISCONNECT. Ability to select on mqtt_client. Thanks to @carr123 and @imranrazakhan for the suggestions (#2507)

Improved

• MQTT:
  • Sessions are now all stored inside a single stream, as opposed to individual streams, reducing resources usage (#2501)

Changed

• JetStream:
  • Using Nats-Expected-Last-Subject-Sequence header with a value of 0 now means that the server will reject the store command if there were messages on this subject (#2506)
• MQTT:
  • Due to improvement described above, when an MQTT client connects for the first time after an upgrade to this server version, the server will migrate all individual $MQTT_sess_<xxxx> streams to a new $MQTT_sess stream for the user's account (#2501)

Fixed

• JetStream:
  • Possible deadlock due to lock inversion (#2479)
  • Possible consumer stall. Thanks to @carr123 for the report (#2480)
  • Don't send 408 status when pull request expires. This has no visible impact for users, but library implementers may want to be aware of it (#2482)
  • During peer removal, try to remap any stream or consumer assets (#2493)
  • Issues with remapping of stream and/or consumer assets during peer removal (#2493)
  • Issue that could lead to perceived message loss (#2490)
  • Message cleanup for interest stream and AckNone consumers in clustered mode (#2499)
  • Suppress duplicates on JS deny all for system account (#2502)
  • Consumers stopped receiving messages. Thanks to @anjmao, @izarraga and @tigrato for the report (#2505)
• Handle SIGTERM on Windows platform (#2481)
• Account resolver TLS connection may fail with x509: certificate signed by unknown authority errors. Thanks to @Ryner51, @ronaldslc for the report (#2483)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.4.0...v2.5.0

Changelog

Notice for JetStream Users

With the latest release of the NATS server we have fixed bugs around queue subscriptions and have restricted undesired behavior that could be confusing or introduce data loss by unintended/undefined behavior of client applications. If you are using queue subscriptions on a JetStream Push Consumer or have created multiple push subscriptions on the same consumer, you may be affected and need to upgrade your client version along with the server version. We’ve detailed the behavior with different client versions below.

With a NATS Server prior to v2.4.0 and client libraries prior to these versions: NATS C client v3.1.0, Go client v1.12.0, Java client 2.12.0-SNAPSHOT, NATS.js v2.2.0, NATS.ws v1.3.0, NATS.deno v1.2.0, NATS .NET 0.14.0-pre2:

• It was possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is not correct since each instance will receive the same copy of a message and acknowledgment is therefore meaningless since the first instance to acknowledge the message will prevent other instances to control if/when a message should be acknowledged.
• Similar to the first issue, it was possible to create many different queue groups for one single JetStream consumer.
• For queue subscriptions, if no consumer nor durable name was provided, the libraries would create ephemeral JetStream consumers, which meant that each member of the same group would receive the same message than the other members, which was not the expected behavior. Users assumed that 2 members subscribing to “foo” with the queue group named “bar” would load-balance the consumption of messages from the stream/consumer.
• It was possible to create a queue subscription on a JetStream consumer configured with heartbeat and/or flow control. This does not make sense because by definition, queue members would receive some (randomly distributed) messages, so the library would think that heartbeat are missed, and flow control would also be disrupted.

If above client libraries are not updated to the latest but the NATS server is upgraded to v2.4.0:

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. Since the check is performed by the library (with the help of a new field called PushBound in the consumer information object set by the server), this mis-behavior is still possible.
• Queue subscriptions will not receive any message. This is because the server now has a new field DeliverGroup in the consumer configuration, which won’t be set for existing JetStream consumers and by the older libraries, and detects interest (and starts delivering) only when a subscription on the deliver subject for a queue subscription matching the “deliver group” name is found. Since the JetStream consumer is thought to be a non-deliver-group consumer, the opposite happens: the server detects an core NATS queue subscription on the “deliver subject”, therefore does not trigger delivery on the JetStream consumer’s “deliver subject”.

The 2 other issues are still present because those checks are done in the updated libraries.

If the above client libraries are update to the latest version, but the NATS Server is still to version prior to v2.4.0 (that is, up to v2.3.4):

• It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is because the JetStream consumer’s information retrieved by the library will not have the PushBound boolean set by the server, therefore will not be able to alert the user that they are trying to create multiple subscription instances for the same JetStream consumer.
• Queue subscriptions will fail because the consumer information returned will not contain the DeliverGroup field. The error will be likely to the effect that the user tries to create a queue subscription to a non-queue JetStream consumer. Note that if the application creates a queue subscription for a non-yet created JetStream consumer, then this call will succeed, however, adding new members or restarting the application with the now existing JetStream consumer will fail.
• Creating queue subscriptions without a named consumer/durable will now result in the library using the queue name as the durable name.
• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

For completeness, using the latest client libraries and NATS Server v2.4.0:

• Trying to start multiple non-queue subscriptions instances for the same JetStream consumer will now return an error to the effect that the user is trying to create a “duplicate subscription”. That is, there is already an active subscription on that JetStream consumer.
  It is now only possible to create a queue group for a JetStream consumer created for that group. The DeliverGroup field will be set by the library or need to be provided when creating the consumer externally.
• Trying to create a queue subscription without a durable nor consumer name results in the library creating/using the queue group as the JetStream consumer’s durable name.
• Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message.

Note that if the server v2.4.0 recovers existing JetStream consumers that were created prior to v2.4.0 (and with older libraries), none of them will have a DeliverGroup, so none of them can be used for queue subscriptions. They will have to be recreated.

Go Version

• 1.16.7: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Domain to the content of a PubAck protocol (#2432, #2434)
  • PushBound boolean in ConsumerInfo to indicate that a push consumer is already bound to an active subscription (#2438)
  • DeliverGroup string in ConsumerConfig to specify which deliver group (or queue group name) the consumer is created for (#2438)
  • Warning log statement in situations where catchup for a stream resulted in an error (#2444)
• Monitoring:
  • Ability for normal accounts to access scoped connz information (#2437)
• Operator option resolver_pinned_accounts to ensure user are signed by certain accounts (#2461)

Changed

• JetStream:
  • ConsumerInfo's Delivered and AckFloor are now SequenceInfo instead of SequencePair. SequenceInfo contains Last which represents the last active time (in UTC) (#2462)
  • Delivery of messages for consumers will now be subject to proper group information. Older clients may fail to receive messages on a queue subscription if the library has not been updated to specify the DeliverGroup (#2438)

Improved

• Make error message actionable when adding operator and leaf nodes. Thanks to @alsuren for the contribution (#2449)
• JetStream:
  • File utilization when using a Jetstream stream as a KeyValue store (#2456)
  • Encryption at rest with message expiration on server restart (#2467)
  • Enable global account on non JetStream servers in mixed mode (#2473)

Fixed

• JetStream:
  • Stream delete can fail for non empty directory (#2418)
  • Possible panic for concurrent stream remove and consumer create (#2419)
  • Simplified flow control and avoid stalls due to message loss (#2425)
  • Consumer info max_msgs_per_subject defaults to 0, but should be -1 (#2426)
  • Creating a consumer with a max_waiting value and a deliver subject was returning the wrong error message (#2427)
  • Assign default to max_ack_pending when AckExplicit or AckAll (#2428)
  • Subscriptions for internal clients for JetStream consumers were not properly removed, resulting in possible subscriptions leak/high memory usage (#2439)
  • Expiration of messages during a server restart could lead to clients reporting errors after reconnect when trying to send new messages (#2452)
  • Added additional checks for failures during filestore encryption (#2453)
  • Processing of a publish ACK could cause a server panic (#2460)
  • Consumer's number of pending messages for multiple matches and merging (#2464)
  • Deadlock due to lock inversion when creating a RAFT group. Thanks to @lokiwins for the report (#2471)
• Leafnode:
  • User authorization issue when JetStream is involved. Thanks to @wchajl for the report (#2430)
  • A remote websocket connection with wss:// scheme but no tls{} configuration block would be attempted as non TLS connection, resulting on an "invalid websocket connection" in the log of the server attempting to create the remote connection (#2442)
  • Wrong permission checks prevented messages flow. Thanks to @rbboulton for the report (#2455, #2470)
  • Daisy chained subject propagation issue. Thanks to @KimonHoffmann for the report (#2468)
• Possible subscription leak with the use of "AutoUnsubscribe" (#2421)
• Prevent JWT claim updates from removing system imports (#2450, #2451)
• Error print when adding back existing system imports (#2466)
• Build on OpenBSD-6.9. Thanks to @miraculli for the contribution (#2472)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.3.4...v2.4.0

Changelog

Go Version

• 1.16.6: Both release executables and Docker images are built with this Go release.

Changed:

• Server will now reject a max_payload that is set higher than max_pending since no message would be able to be delivered to subscriptions. The server will also warn if the value is set above 8MB (and may enforce this limit in the future) (#2407, #2413)

Fixed

• JetStream:
  • A deadlock could happen when the server was removing messages. Thanks to @rwrz for the report (#2404)
  • Various fixes and improvements to clustered Filestore consumer stores (#2406)
  • Leafnodes with same domain and shared system account should behave like flat Jetstream network (#2410)
  • With stream with multi-subject or wildcard and max_msgs_per_subject set, if a consumer had deliver_last_per_subject set, the initial pending would be 1 higher than it actually was (#2412)
  • Memory store would sometimes incorrectly select the proper starting sequence (#2412)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.3.3...v2.3.4

Changelog

Go Version

• 1.16.6: Both release executables and Docker images are built with this Go release.

Added

• ReloadOptions API to support configuration reload without use of configuration file for embedded cases. Thanks to @taigrr for the contribution (#2341)
• Kind and ClientType to account CONNECT/DISCONNECT events. Thanks to @mullerch for the report (#2351)
• JetStream:
  • Streams and consumers now have a Description property (#2377)
  • New DeliverLastPerSubject delivery policy (#2381, #2390)

Changed

• Default account fetch timeout to be smaller than client timeout to increase probability of getting the Authorization Violation error instead of a i/o timeout error when credentials of an account has not yet been pushed to the account server (#2365)
• Executable symbol table no longer stripped. Thanks to @yzhao1012 and @justicezyx for the contribution (#2383)

Improved

• TLS timeout in configuration file parsing now accept units, such as "2s" for 2 seconds (#2364)
• JetStream:
  • Server restart time with many expired messages (#2387)

Fixed

• JetStream:
  • A data race on JetStream shutdown (#2353)
  • In clustered mode, the maximum consumers limit was not always applied for ephemeral consumers (#2354)
  • Consumer's NumPending may be stuck at 1 (#2357)
  • Removed a stack print that may appear in some cases since v2.3.1 (#2362)
  • Possible subscription leak when processing service imports and processing of pull subscribers (#2373)
  • Unique server name requirement across domains (#2378)
  • A clustered consumer on an interest retention policy could cause the server to panic when the consumer was being deleted (#2382)
  • Allow non-JS leafnode(s) to access a HUB transparently (#2393)
  • A stream with un-acknowledged messages would not redeliver new un-acknowledged messages following a purge. Thanks to @sloveridge for the report (#2394)
• Subscription on a subject that is not a subset of a wildcard import. Thanks to @DamianoChini for the report (#2369)
• OCSP issue in embedded cases when the TLS configuration did not set the certificate Leaf (#2376)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.3.2...v2.3.3

Changelog

Go Version

• 1.16.5: Both release executables and Docker images are built with this Go release.

Added

• JetStream:
  • Error codes for consumers creation errors (#2345)

Changed

• JetStream:
  • Creating an ephemeral consumer if there is not yet interest will no longer fail, that is, a JetStream consumer can be created prior to low level NATS subscription on the delivery subject (#2347)

Fixed

• JetStream:
  • Updates to a multi-subject tacked stream (#2334)
  • Possible publish timeout due to server sending messages to consumers on a slow connection (#2337)
  • Possible message corruption (#2344)
  • Peer info reports had a large last active values
• Headers handling in system services (#2338, #2348)

Complete Changes

https://github.com/nats-io/nats-server/compare/v2.3.1...v2.3.2