vernemq

Note:

VerneMQ software distribution packages and Docker images are now covered by the VerneMQ EULA. See the blog post describing the change:
https://vernemq.com/blog/2019/11/26/vernemq-end-user-license-agreement.html.

Changelog:

• Fix bug where websocket MQTT clients that didn't provide a
  sec-websocket-protocol header to cause a crash to be logged. This case is
  now handled and the connection is terminated gracefully (#1317).
• Fix bug which caused clients MQTT bridges using protocol versions (131 and
  132) to be unable to connect (#1306).
• Upgrade the hackney so VerneMQ is compatible with a TLS change in Erlang/OTP
  22.1.
• Fix bug preventing MQTT 5 publish in/out counts from being shown on the HTTP
  status page.
• Fix lager issue on Raspberry Pi preventing VerneMQ from starting (#1305).
• Upgrade epgsql dependency to version 4.3.0 to get better error messages
  (#1336).
• Add new max_last_will_delay value client which means the value from the
  client is used instead of being overriden by the broker. This client value
  is now the default. This setting only applies to MQTT 5.0 clients.
• Add hidden option response_timeout to vmq_webhooks endpoints. With this
  the time vmq_webhooks waits for a response from the remote endpoint can be
  configured. Default is 5000 milliseconds.
• Add new on_deliver/6 and on_deliver_m5/7 hooks which adds QoS and retained
  information to the parameters. The old variants are deprecated and will be
  removed in the next major version.
• Add new on_session_expired/1 hook which is called when an offline session
  expires and the state is deleted.
• Handle CRL PEM entries with certificates containing empty revocation lists
  (#1337).
• Fix typo in vmq-admin listener start command (#1348).
• Optimize subscription performance when many retained messages are stored on
  the broker and the subscription contains wildcards.
• Fix bug where MQTT 5 publish expiry interval was lost when writing the message
  to the message store.
• Fix bug where a retried MQTT publish after resuming an offline session used
  the wrong message id.
• Fix MQTT 5 shared subscription bug where writing the message to the message
  store resulted in a corrupt message record once a offline session was resumed.
• Refactor built-in message storage to enable multiple storage engines as well as
  to streamline the implementation of alternative message storage plugins.
• Fix metric description string in vmq_bridge.
• Fix regression in vmq_bridge where the retain bit wasn't handled properly for
  incoming publishes.
• Enable to define the MQTT protocol version to be used by vmq_bridge (4 or 3).
• Fix vmq_bridge dynamic configuration handling.
• Fix multiple minor issues in vmq_swc.
• Fix HTTP handlers to support parameterized media types.
• Upgrade lager, poolboy, jsx, luerl dependency.

• Fix bug causing idle websocket connections to be closed after 60 seconds
  (#1292).
• Fix MQTT 5.0 bug causing LWT not being sent when client disconnected with
  Disconnect with Will Message (0x04) (#1291).
• Ensure MQTT 5.0 subscription identifers are added to messages delivered via
  shared subscriptions (#1294).
• Fix protocol version check bug which would allow MQTT 5.0 clients to connect
  even if the MTT 5.0 protocol was not specified for the listener. This happened
  for clients which connected with an empty client-id.
• Fix bug in vmq_diversity and vmq_webhooks where the plugin hooks would be
  registered out of order, even though the plugins themselves would be started
  in the correct order (#1295).

• Update cuttlefish to fix parse issue with lines in the vernemq.conf file
  consisting solely of white-space (#1208).
• Fix bug in the vmq_webhooks auth_on_subscribe_m5 and on_subscribe_m5
  hooks (#1280).
• Fix issue where errors would be logged when the /health endpoint was called
  (#1281).
• Ensure MQTT 5.0 subscription identifiers are added to retained messages
  delivered to the client when the subscription is made (#1287).
• Fix bug in vmq_swc which could prevent VerneMQ from starting up.

• Ensure mountpoints specified at the protocol level are inherited on the
  specific listeners.
• Fix missing output from vernemq version (#1190).
• Ensure errors from the parser variable decode are handled correctly and logged
  at the debug level instead of causing a crash in the socket process.
• Handle list of sec-websocket-protocol correctly (#1149) in the websocket
  implementation.
• Ensure that plumtree application isn't started when metadata plugin SWC is used.
• Fix bug preventing restarting Lua states properly in case of a crashing script.
• Fix warnings due to deprecated lager configuration (#1209).
• Upgrade lager to version 3.7.0 (In preparation for OTP 22 support).
• Make VerneMQ run on Erlang/OTP 22.
• Add new webhooks CLI register flag --no_payload=true which, if enabled, will
  remove the MQTT payload from the JSON object send via the auth_on_publish
  and auth_on_publish_m5 payloads. The flag can also be set in the config file
  using vmq_webhooks.hookname.no_payload=on.
• Add metric client_keepalive_expired which tracks clients that failed to
  communicate within the keepalive time.
• Expose the crypto:hash/1 function in LUA, for example to do a 256bit sha3
  hash one can call crypto.hash("sha3_256", data). The hashing algorithms
  supported are exactly the ones supported by the crypto:hash/1 function and
  may change depending on the Erlang/OTP version used to build Erlang. See the
  complete list of supported hashes here:
  http://erlang.org/doc/man/crypto.html#Digests%20and%20hash. If passed an
  unknown hashing algorithm an error is raised.
• Fix prefix handling in the bridge plugin (vmq_bridge).
• Strengthen parameter validation in the bcrypt.hashpw/2 LUA function in
  vmq_diversity.
• Pass arguments correctly to vmq-admin when called via sudo.
• Handle rate metric labels correctly in the example grafana dashboard.
• Upgrade cowboy to version 2.6.3 as well as cowlib and ranch to versions 2.7.3
  and 1.7.1 respectively. This update also means the PROXY protocol code was
  removed and the PROXY support from cowboy is used instead.
• Handle retained flag in the bridge plugin (vmq_bridge).
• Ensure systemd doesn't terminate VerneMQ if it is slow to start.
• Implement bridge protocol handling (protocol version 131) for bridges
  connecting to VerneMQ such that the retained bit is kept on messages routed to
  the bridge (Retained as Publish) and messages coming from the bridge are not
  forwarded to the bridge if it has a matching description (No Local).
• Fix message store startup consistency checking routine which resulted in
  potentially deleting wrong messages in cases where an inconsistency was detected.
  This fix also increases message storage performance during startup as well as
  during normal operation. However, the nature of the bug and the provided solution
  don't enable a simple downgrade path and introduces a backward incompatibility if
  a VerneMQ message store containing offline messages has to be downgraded to 1.8.0
  or earlier versions.
• Upgrade dependency cuttlefish to version 2.2.0.
• Improve large queue initialization performance by reducing algorithmic
  complexity from O(n^2) to O(nlogn) where n is the number of offline messages.
• Add the ability to modify the username on auth_on_register and auth_on_register_m5
  hooks. Supports both vmq_diversity and vmq_webhooks.
• Upgrade the vmq_diversity redis driver eredis to version 1.2.0.
• Fix vmq_diversity supervisor restart shutdown issue (#1241).

• Cleanup cluster state information on a node which is being gracefully removed
  from the cluster so if it is restarted it comes back up as a stand-alone node
  and won't try to reconnect to the remaining nodes.
• The allow_multiple_sessions option is deprecated and will be removed in
  VerneMQ 2.0.
• Add new coordinated_registrations config option which allows for faster
  uncoordinated registrations (replaces side-effect of
  allow_multiple_sessions).
• Upgraded dependency swc. Improves memory and cpu usage by fixing node clock
  update bug.
• Set queue expiration (persistent_client_expiration) timer on queues after a
  broker restart (#1071).
• Fix status page to show all cluster nodes (#1066).
• Fix Lua json.decode() when decoding empty Json objects (#1080).
• Fix vmq_diversity cache invalidation timing issue when a client reconnects
  right after a disconnect (#996).
• Fix retry of MQTT publish frame in vmq_bridge (#1084).
• Fix outgoing qos2 message id bug (#1045).
• VerneMQ now requires Erlang/OTP 21.2 or newer to build.
• Use atomics and persistent_term to handle metrics instead of using the
  mzmetrics library which has been removed.
• Change vmq_reg:publish/4 to return {ok, {NumLocalMatches, NumRemoteMatchs}}
  instead of ok. This introduces a breaking change to the unofficial publish
  API provided by the vmq_reg:direct_publish_exports/1.
• Add the router_matches_local and router_matches_remote counter metrics,
  counting the total number of matched local and remote subscriptions.
• Add a routing score to every cluster node listed on the VerneMQ status page.
  The routing score consists of two percentages similar to 75 / 25 indicating
  that 75% of the received MQTT publish frames were routed to local subscribers
  and 25% were forwarded to subscribers on different cluster nodes. The routing
  score can be used to detect imbalances in a VerneMQ cluster.
• The MQTT 5.0 plugin hook definitions have been updated as follows:
  • The Payload and Properties arguments to the on_deliver_m5 hook have
    been swapped to be consistent with all the other _m5 hooks.
  • property modifiers are now using the same names as the properties in the
    incoming properties, for example the user_property is now
    p_user_property.
  • Property modifiers have been moved into a properties modifier map. So for
    example instead of returning {ok, #{p_user_property => ...}} one now
    returns {ok, #{property => #{p_user_property => ...}}}. This change makes
    it possible to drop properties from a plugin hook.
  • In the vmq_webhooks plugin property modifiers have been moved into a
    separate JSON object with the key properties inside the modifiers JSON
    object. The vmq_webhooks.
  • The {ok, binary()} return clause has been removed from the on_deliver_m5
    hook, use the modifier map ({ok, Modifiers}) instead.
  • The {ok, binary()} return clause has been removed from the
    auth_on_publish_m5 hook, use the modifier map ({ok, Modifiers}) instead.
  • Note, MQTT 5.0 support in VerneMQ is still in Beta.
• Fix error when tracing clients connecting with a LWT.
• Add file validation to check if files in the vernemq.conf exist and are
  readable.
• Fix that disconnects due to client migrations were logged as warnings. They
  are now treated as the normal termination case.
• Improve MQTT 5.0 support in the vmq_webhooks plugin: support more properties
  and improve the tests.
• Instead of using the node_package dependency to build binary packages, the
  external Ruby command line tool fpm is used. This enables cleaning up the
  Makefile as well as the upgrade of the rebar3 build tool.
• Improve MQTT 5.0 support in the vmq_diversity plugin: support all MQTT 5.0
  hooks in Lua and more properties.
• Support TLS encrypted connections to PostgreSQL when using
  vmq_diversity. Fixes #811.
• Support TLS encrypted connections to MongoDB when using vmq_diversity.
• Add CockroachDB support to the vmq_diversity plugin.
• Make it possible to configure how many concurrent bcrypt operations are
  possible via the vmq_bcrypt.pool_size config.
• Fix incorrect format string in shared subscription debug log.
• Upgrade hackney to version 1.15.1 (dependencies of hackney were updated as
  well).
• Fix bug which could happen while repairing dead queues in case the sync
  request fails.
• It is now possible to configure TCP send and receive buffer and user-level
  buffer sizes directly on the MQTT listeners or protocol levels (currently only
  TCP and TLS listeners).
• Add the socket_close_timeout metric. The metric counts the number of times
  VerneMQ hasn't received the MQTT CONNECT frame on time and therefore forcefully
  closed the socket.
• Fix the potential case of late arriving close_timeout messages triggered
  by the vmq_mqtt_pre_init.
• Change log level from debug to warning when VerneMQ forcefully terminates a
  MQTT session FSM because of an unexpected message.
• Fix vmq_diversity ACL rule hash collision issue (#1164).
• Improve performance of shared subscriptions when using many subscribers.
• Add the vmq_pulse plugin. This plugin periodically sends cluster information to
  a pulse backend via HTTP(S). The plugin is NOT enabled by default, and once
  enabled it has to be setup with vmq-admin pulse setup. This enables a support
  crew to analyze the cluster performance without having direct access to the VerneMQ
  nodes. Such a 'Pulse' contains the following information:
  • Cluster status, similar to vmq-admin cluster show
  • Plugins, similar to vmq-admin plugin show --internal
  • Metrics, similar to vmq-admin metrics show
  • Names and versions of loaded OTP applications
  • Erlang System Version
  • OS Kernel information, output of uname -a
  • CPU, RAM, and disk usage
  • Pulse Version
    Note: The vmq_pulse plugin is in Beta.
• Fix the systemd service unit file to set LimitNOFILE=infinity. This enables VerneMQ
  to request as many file descriptors as configured for the vernemq user.

• Cleanup cluster state information on a node which is being gracefully removed
  from the cluster so if it is restarted it comes back up as a stand-alone node
  and won't try to reconnect to the remaining nodes.
• Upgraded dependency swc. Improves memory and cpu usage by fixing node clock
  update bug.
• Set queue expiration (persistent_client_expiration) timer on queues after a
  broker restart (#1071).
• Fix status page to show all cluster nodes (#1066).
• Fix Lua json.decode() when decoding empty Json objects (#1080).
• Fix vmq_diversity cache invalidation timing issue when a client reconnects
  right after a disconnect (#996).
• Fix retry of MQTT publish frame in vmq_bridge (#1084).
• Fix outgoing qos2 message id bug (#1045).

• Fix vmq_webhooks issue where MQTTv5 hooks where not configurable in the
  vernemq.conf file.
• Support shared subscriptions in vmq_bridge.
• Fix bug in Prometheus output (#923).
• Fix max_message_rate to include MQTTv5 sessions.
• Support new throttle modifier for the auth_on_publish and
  auth_on_publish_m5 hooks which will throttle the publishing client by
  waiting for a given number of milliseconds before reading new data from the
  client socket. Note, backpressure is not supported for websocket
  connections. This feature was kindly sponsored by Arduino SA
  (https://www.arduino.cc/).
• Fix issue with long-running vmq-admin commands (#644).
• Added a new HTTP module vmq_health_http exposing a /health endpoint that
  returns 200 when VerneMQ is accepting connections and joined the cluster
  (for clustered setups) or returns 503 in case any of the two conditions
  are not met (#889).
• Fix issue where a QoS2 message would be republished if a client would resend
  the PUBLISH packet with the same message id within a non-finished QoS2 flow
  (#944).
• Make VerneMQ build on FreeBSD (11.2-RELEASE) with gmake rel.
• Fix issue with blocking socket commands in inter-node communication.
• Improve error messages when parsing invalid CONNECT packets.
• Log IP and port information on authentication failures to make it easy to
  block the client with tools like Fail2ban (#931).
• Fix issue which would crash the session if a client would resend a (valid)
  pubrec during a Qos2 flow (#926).
• Fix vmq_diversity error message if invalid or unknown modifiers are returned
  from a lua hook implementation.
• Fix issue preventing MQTT 5.0 properties from being modifiable in
  auth_on_publish_m5 (#964).
• Fix issue which causes a crash if not enough data is available while parsing
  the CONNECT frame in vmq_mqtt_pre_init (#950, #962).
• Fix issue which could cause vmq-admin session show to crash when using
  --limit=X to limit the number of returned results (#902).
• Handle edge case in the websocket implementation which could cause warninigs
  when the session was terminating.
• Expose vernemq_dev_api:disconnect_by_subscriber_id/2 in lua
  vmq_api.disconnect_by_subscriber_id/2, an example looks like:
  vmq_api.disconnect_by_subscriber_id({mountpoint = "", client_id = "client-id"}, {do_cleanup = true}).
• Improve vmq_webhooks errors so the error from the endpoint is shown as the
  error reason where relevant.
• Optimization: optimize away read operation during client registration by
  reusing already available data.
• Enable plugins to expose metrics via the available metric providers.
• Add histogram metric type.
• Fix incorrect debug log message in QoS 2 flow.
• Expose bridge metric for messages dropped in case the outgoing queue is full.
• Log (debug level) when a LWT is suppressed on session takeover.
• Optimize subscribe operation by refactoring away one read from the metadata
  store.
• Add protection mechanism for the plumtree metadata store from subscription
  floods by dropping and not acknowledging i_have messages when the mailbox
  size grows above a certain threshold. This threshold is configurable via the
  plumtree.drop_i_have_threshold hidden option. The default is 1000000. This
  work was kindly contributed by ADB (https://www.adbglobal.com).
• Add new vmq-admin retain commands to inspect the retained message store.
• Support for different MySQL password hashing methods in vmq_diversity,
  ensuring compatibility with MySQL 8.0.11+. The method is configurable via the
  vmq_diversity.mysql.password_hash_method option which allows:
  password (default for compatibility), md5, sha1 or sha256.
• Fix the HTTP /status.json endpoint to have a valid JSON output (#786).
• Fix bug where internal application plugins where shown as normal plugins.
• Fix crash in bridge when calling vmq-admin session show by fixing the
  vmq_ql row initializer to handle plugin sessions (the bridge starts a local
  plugin session).
• Add improvements to VerneMQ status page to have nicer UI and be readable on smaller devices.
• Upgraded dependency sext to 1.5.0 (required for better OSX compilation).
• Do not accept new client connections while the broker is shutting down as this
  could cause errors to be reported in the log (#1004).
• Fix vmq_diversity PostgreSQL reconnect issue (#1008).
• Fix vmq_webhooks so peer port is not considered when caching the
  auth_on_register to avoid cache misses.
• Multiple bug fixes and improvements in vmq_swc.
• Add the vmq_swc metadata plugin (using the already existing LevelDB backend) to
  the default VerneMQ release. To use vmq_swc instead of vmq_plumtree set
  metadata_plugin = vmq_swc in vernemq.conf. vmq_swc is still in Beta.
• Add missing increments of the mqtt_connack_sent metric for CONNACK
  success(0) for MQTT 3.1.1.
• Handle edge case with unknown task completion messages in vmq_reg_sync after
  a restart.
• Fix bug which could cause a queue cleanup to block indefinitely and cause the
  vmq_in_order_delivery_SUITE tests to fail.
• Add a new metric (queue_initialized_from_storage) to better monitor queue
  initialization process after a node restart.
• Fix edge case where an extra queue process could be started when metadata
  events arrive late. Now local queue processes are only started when triggered
  via a new local MQTT session.
• Reimplement dead queue repair mechanism.
• Add feature to reauthorize existing client subscriptions by reapplying current auth_on_subscribe
  and auth_on_subscribe_m5 hooks. This feature is exposed as a developer API in vernemq_dev, via
  the vmq-admin session reauthorize CLI command, and as a API for vmq_diversity Lua scripts. This
  work was kindly sponsored by AppModule AG (http://www.appmodule.net).
• Improve planned cluster leave queue migration speed significantly (#766).
• Start metrics server before setting up queues to ensure queue metric counts
  are correct when restarting VerneMQ.
• Let Travis CI build the VerneMQ release packages.
• Add new metric system_process_count which is a gauge representing the
  current number of Erlang processes.
• Add queue_started_at and session_started_at information to the vmq-admin session show command. Times are POSIX time in milliseconds and local to the
  node where the session or queue was started.

• Fix max_message_rate to include MQTTv5 sessions.
• Fix issue with long-running vmq-admin commands (#644).
• Fix issue where a QoS2 message would be republished if a client would resend
  the PUBLISH packet with the same message id within a non-finished QoS2 flow
  (#944).
• Improve error messages when parsing invalid CONNECT packets.
• Fix issue which would crash the session if a client would resend a (valid)
  pubrec during a Qos2 flow (#926).
• Fix vmq_diversity error message if invalid or unknown modifiers are returned
  from a lua hook implementation.
• Fix issue preventing MQTT 5.0 properties from being modifiable in
  auth_on_publish_m5 (#964).
• Fix issue which causes a crash if not enough data is available while parsing
  the CONNECT frame in vmq_mqtt_pre_init (#950, #962).
• Fix issue which could cause vmq-admin session show to crash when using
  --limit=X to limit the number of returned results (#902).
• Handle edge case in the websocket implementation which could cause warninigs
  when the session was terminating.

• Fix vmq_webhooks issue where MQTTv5 hooks where not configurable in the
  vernemq.conf file.
• Fix bug in Prometheus output (#923).

• Fix issue when calling a function in a Lua script that requires more time to complete than the default gen_server timeout (#589).

• Silently drop messages published by the client that use a routing key starting with '$'.

• Full MQTTv5 support

  With this release VerneMQ officially supports MQTT protocol version
  5.0. The MQTTv5 support is currently marked as in BETA and we may
  still have to change some things.

  VerneMQ has full support for the complete MQTTv5 spec, but to list a
  few of the new features:

  • Support for enhanced (re)authentication

  • User properties

  • Message Expiration

  • Last Will and Testament delay

  • Shared subscriptions

  • Retained messages

  • Request/Response flows

  • Topic aliases

    VerneMQ supports topic aliases from both the client to the broker and from
    the broker to the client.

    When a client connects to the broker, the broker will inform the client of
    the maximum allowed topic alias using the topic alias max property on the
    CONNACK packet (if it has been set to a non-zero value). The topic alias
    maximum property can be configured through the topic_alias_max
    configuration variable or overriden in a plugin in the auth_on_register
    hook. The broker will then handle topic aliases from the client as per the
    MQTTv5 spec.

  • Flow control

  • Subscription flags Retain as Published, No Local and Retain Handling.

  • Subscriber ids

  By default MQTTv5 is disabled, but can be enabled on a listener basis, for
  example listener.tcp.allowed_protocol_versions=3,4,5 would enable MQTT
  version 3.1, 3.1.1 and 5.0 on the TCP listener.

  MQTTv5 support has been added to the vmq_passwd, vmq_acl, vmq_webhooks
  plugins as well as the parts of the vmq_diversity plugin to support the
  MySQL, PostgreSQL, Redis and MongoDB authentication and authorization
  mechanisms. The Lua scripting language provided by the vmq_diversity plugin
  does not yet expose all MQTT 5.0 plugin hooks.

  The vmq_bridge plugin currently has no support for MQTTv5.

  !! Note !! that all MQTTv5 related features and plugins are in BETA and may
  still change if needed.

• The metrics:

  mqtt_connack_not_authorized_sent
  mqtt_connack_bad_credentials_sent
  mqtt_connack_server_unavailable_sent
  mqtt_connack_identifier_rejected_sent
  mqtt_connack_unacceptable_protocol_sent
  mqtt_connack_accepted_sent

  Have been merged into a single metric mqtt_connack_sent and the various MQTT
  3.1.1 return codes have been mapped into labels. So for example
  mqtt_connack_sent with the label return_code=success replaces
  mqtt_connack_accepted_sent.

• Added on_message_drop hook that is called for every message dropped due to
  exceeding the MQTTv5 max_packet_size property, hitting the message expiry,
  or when load shedding when enqueing messages.

• Fix ordering bug in webhook subscribe topic authentication and topic rewrites
  (#823)

• Fix issue when terminating the vmq_server application (#828).

• Make VerneMQ build on SmartOS / Illumos / Solaris.

• Ensure strings passed from Lua to the logger are escaped (#864).

• Handle Mongo Date / ISODate datatype properly in Lua / vmq_diversity
  (#857).

• Optimize subscribe/unsubscribe operations for large fanout cases.

• Allow non-standard MQTT version 131 (MQTT 3.1 bridge) by default (this was
  accidentally changed in VerneMQ 1.4.0).

• Improve error messages returned by the vmq_diversity plugin so it's easier
  to understand and debug authentication and authorisation issues.

• Improve performance for looking up queue processes.

• Make VerneMQ run on Erlang/OTP 21. To do this we had to upgrade rebar3 to
  version 3.6.0 and lager to version 3.6.3. We also removed the eper
  dependency and suppressed some new warnings due to erlang:get_stacktrace/0
  having been deprecated in Erlang/OTP 21.

• The bridge plugin (vmq_bridge) now has an option to queue outgoing message
  in case the network connection disappears. This is configurable with the
  max_outgoing_buffered_messages setting on a bridge. In addition the bridge
  now has a simple cli interface under vmq-admin bridge show. This work was
  kindly sponsored by Diacare-Soft(http://diacare-soft.ru).

• Fix multiple message retry issues in the MQTT client used by vmq_bridge.

• Fix issue where the message ordering wasn't preserved after a client reconnect.

• Add experimental vmq_swc plugin that provides an alternative to the existing
  vmq_plumtree for metadata storage and replication. One must compile VerneMQ
  with make swc to generate a release containing the vmq_swc plugin.

• Remove unused vmq_bridge ssl capath config option. This was never used
  internally.

• Fix issue in the bridge preventing it from reconnecting after a connection
  timeout (#726).
• Fix issue with self referential cluster leaves and cluster readiness probing
  in general (#717).
• Add preparations for MQTTv5:
  • Make it possible to extend the cluster internal message format when adding
    new features while ensuring back- and forwards compatibility.
  • Refactor the leveldb message store code to support versioned indexes as well
    as messages.
  • Refactor the retained message store to support versions.
  • Add support for subscriber data with subscription options.
• Fix issue when validating a shared subscription topic (#756).
• Fix issue with retried PUBREC frames (#750).
• Make it possible to suppress the Last Will and Testament message if an
  existing session already exists for the connecting client. This is
  configurable via the hidden setting suppress_lwt_on_session_takeover in the
  VerneMQ configuration file. The default is off.
• Fix issue with PUBREL frames retried after client reconnects (#762).
• Refactor and cleanup retry mechanism.
• Warn about deprecated setting message_size_limit only when it has been defined.
• Change build script so nightly packages will have names on the form
  vernemq_1.4.0-10-gde1b1f5-1_amd64.deb, where de1b1f5 refers to the commit
  hash from which the package was built and 10 is the number of commits since
  the latest tag. This makes it much easier to reason about nightly builds and
  the features they contain.
• Refactor and generalize the metrics systems to allow labelling metrics coming
  from different sources in order to differentiate them in the various exporters
  and the command line. Labels are added to the Prometheus exporter by
  default. To enable generating additional metrics from the labels in the
  graphite exporter the hidden setting graphite_include_labels has to be set
  to on in the vernemq.conf file. Labels are not exposed in the $SYS
  metrics.
• Ensure the vmq_bridge is properly restarted after a crash (#785).
• Fix issue where not calling http.body(ref) in a Lua script would not return
  the underlyning HTTP connection to the connection pool. This now happens
  automatically, irregardless of the user calling http.body(ref) or not
  (#588).

• Fix issue with PUBREL frames retried after client reconnects (#762).
• Refactor and cleanup retry mechanism.
• Ensure the vmq_bridge is properly restarted after a crash (#785).
• Fix issue where not calling http.body(ref) in a Lua script would not return
  the underlyning HTTP connection to the connection pool. This now happens
  automatically, irregardless of the user calling http.body(ref) or not
  (#588).

• Fix issue in the bridge preventing it from reconnecting after a connection timeout (#726).
• Fix issue with self referential cluster leaves and cluster readiness probing in general (#717).
• Fix issue when validating a shared subscription topic (#756).
• Fix issue with retried PUBREC frames (#750).

• Fix for OSX compilation issue on vmq_passwd due to openssl headers not found.
• Refactoring to further abstract and generalize the Plumtree metadata layer.
• Small refactoring moving the calling of plugin hooks into the fsm code. This
  is a preparation for MQTTv5. Note, this change has an impact on the
  vmq_reg:direct_plugin_exports/1 function. Even though this function is
  internal and as such not guaranteed to be stable, we know some plugin
  developers use it and we therefore wanted to describe the changes. It works as
  before, except that the hooks auth_on_subscribe, on_subscribe or
  on_unsuscribe hooks will no longer be called when using the exported
  functions.
• Corrected a few spelling errors in the vmq-admin help commands.
• Added two new hidden configuration parameters for vmq_webhooks. vmq_webhooks uses
  the hackney HTTP client for HTTP requests.
  It starts its own hackney pool with a default connection pool of 100, and a
  connection timeout of 60000 milliseconds. This connection pool con now be
  configurable using the settings vmq_webhooks.pool_max_connections and
  vmq_webhooks.pool_timeout (#621).
• Add a first version of a VerneMQ status page showing node and cluster
  information. This web-page is by default available on
  http://localhost:8888/status. The status page is implemented using jQuery,
  Twitter Bootstrap und mustache.js. As this is the first version the page
  should be considered experimental and will likely be changed in future
  releases.
• Bugfix: Handle return values correctly when enqueuing messages to offline
  shared subscribers (#625).
• Bugfix: Fix issue preventing messages delivered to a subscriber group from
  being delivered to the online group members first before attempting delivery
  to offline queues (#618).
• Fix some Dialyzer issues.
• Reduce replication load during a netsplit by making sure data is not attempted
  to be replicated to unreachable nodes.
• Bugfix: Fix issue causing some session show options (peer_host,
  peer_port) to not work for websocket clients (#542).
• Bugfix: Fix routing table initialization issue after restarting a node
  preventing shared subscriptions on a remote node from being included in the
  routing table (#595).
• Bugfix: Fix race condition when fetching data from the internal query subsystem.
• Fix build issue on Raspberry PI (make rpi-32).
• Make it possible to specify which protocol versions are allowed on an MQTT
  listener. By default the protocol versions allowed are versions 3 and 4 (MQTT
  v3.1 and v3.1.1 respectively). To set the allowed protocol versions one can
  use listener.tcp.allowed_protocol_versions = [3,4] on the transport level or
  for a specific listener using
  listener.tcp.specific_listener.allowed_protocol_versions.
• Fix bug causing an exception to be thrown when vmq-admin cluster leave is
  used with a timout value less than 5 seconds (#642).
• Small refactoring enabling to store versioned message store values. This is a
  preparation for MQTTv5.
• Bugfix: Fix a bug that prevented user plugins with an explicit path to be
  loaded.
• Fix issue with new rebar3 upstream plugin version (the port-compiler) which
  made builds fail by pegging it to an older version (1.8.0).
• Add xmerl and inets from the Erlang standard library to the release in order
  to allow plugin developers to have these libraries available.
• Bugfix: Fix typo (graphie_api_key -> graphite_api_key) preventing the
  graphite api key from being set in the vernemq.conf file.
• Bugfix: WebHooks Plugin. Close the Hackney CRef so that the socket is given
  back to the Hackney pool, for the case of non-200 HTTP OK status codes.
• Bugfix: fix bug where queries with mountpoint and client-ids would return no
  entries causing disconnecting a client with a specific mountpoint to
  fail. Also ensure to use the default mountpoint if no mountpoint was
  passed. (#714).
• Upgraded vernemq_dev to include the disconnect_by_subscriber_id/2 API.

• Fix bug causing an exception to be thrown when vmq-admin cluster leave is
  used with a timout value less than 5 seconds (#642).
• Fix for OSX compilation issue on vmq_passwd due to openssl headers not found.
• Bugfix: Fix a bug that prevented user plugins with an explicit path to be
  loaded
• Fix build issue on Raspberry PI (make rpi-32).
• Bugfix: Fix race condition when fetching data from the internal query subsystem.

• Bugfix: Handle return values correctly when enqueuing messages to offline
  shared subscribers (#625).
• Bugfix: Fix issue preventing messages delivered to a subscriber group from
  being delivered to the online group members first before attempting delivery
  to offline queues (#618).
• Fix some Dialyzer issues.
• Reduce replication load during a netsplit by making sure data is not attempted
  to be replicated to unreachable nodes.
• Bugfix: Fix issue causing some session show options (peer_host,
  peer_port) to not work for websocket clients (#542).
• Bugfix: Fix routing table initialization issue after restarting a node
  preventing shared subscriptions on a remote node from being included in the
  routing table (#595).

• Add proxy_protocol_use_cn_as_username feature which for proxy_protocol
  enabled listeners enable or disable using the common name forwarded by the
  PROXY protocol instead of the MQTT username. To ensure backward compatibility
  this setting is enabled by default. This feature was kindly contributed by
  SoftAtHome (https://softathome.com/).

• New vmq-admin command to forcefully disconnect and cleanup sessions.

• Fix issue preventing ssl settings being inheritable on the listener level
  (#583).

• Fix issue where enqueuing data to a queue on a remote cluster node could cause
  the calling process to be blocked for a long time in case of the remote
  cluster node being overloaded or if a net-split has occurred.

  This issue can occur while delivering a shared subscriber message to a remote
  subscriber in the cluster (blocking the publishing client) or when migrating
  queue data to to another node in the cluster. In the case of shared
  subscribers a new (hidden) configuration parameter
  (shared_subscription_timeout_action) has been added which decides which
  action to take if enqueuing on a remote note times out waiting for the
  receiving node to acknowledge the message. The possibilities are to either
  ignore the timeout or requeue the message. Ignoring the timeout can
  potentially lead to losing the message if the message was still in flight
  between the two nodes and the connection was lost due to a
  net-split. Requeueing may lead to the same message being delivered twice if
  the original client received the message, but the acknowledgement was lost.

• Fix typo in configuration name plumtree.outstandind_limit should be
  plumtree.outstanding_limit.

• Make the vmq_cluster_node processes able to handle system messages to make
  it easier inspect the process behaviour at run-time.

• Fix bug preventing use_identity_as_username from working on WSS sockets
  (#563).

• The shared subscriber feature is no longer marked as in beta.
• Strengthen check verifying if a remote node is available or not. The new check
  verifies that a data connection to the remote node has been established, in
  addition to the current check which verifies that a specific process is running
  on the remote node. This new check will make it more visible if, for instance,
  the IP an port configured via listener.vmq.clustering is not reachable or a
  listener was not able to be started.
• Fix issue preventing the proxy_protocol setting (listener.tcp.proxy_protocol = on) being inherited by specific listeners (#516).
• Remove shared subscriptions backwards compatibility code. Introducing shared
  subscriptions changed some internals and code ensuring backwards compatibility
  had to be written. This code has existed since VerneMQ 1.0.0rc1 and anyone
  needing to upgrade a running cluster from older versions than 1.0.0rc1 of
  VerneMQ will have to first do an intermediate upgrade to a version containing
  the compatibility code.
• Remove time and randomness related dependencies for backwards compatibility
  for OTP 17. These are no longer required as OTP 17 support was removed before
  VerneMQ 1.0.
• Minor optimizations.
• Fix issue in the queue initialization introduced in VerneMQ 1.2.2 which meant
  offline messages were not being read into the queue process after a node
  restart. Added tests to prevent this issue from occurring again.
• Fix crash in cleanup of the vmq_webhooks plugin when the plugin is being
  stopped or the broker is shutting down (#556).
• Fix incorrect format strings in vmq_reg_sup log statements.
• Add missing argument to HTTP module configuration log statement.
• Do not resolve host names when including peer host in an vmq-admin session show query as this can lead to a timeout.
• Remove superfluous warnings for ip and port when using the proxy protocol.

• Fixed a number issues when filtering vmq-admin session show results. Note msg_ref filtering is currently broken.
• Fix node startup routine to cleanup stale subscriber data.
• Do not log getting a client peername failed because the client already disconnected.

• Upgrade MongoDB driver.
• Prevent unbounded memory growth in a netsplit situation by putting an explicit
  limit (250K) on the number of outstanding plumtree-msgs in plumtree. If this
  limit is exceeded, delayed replication will be handled by active entropy.
• Handle uncaught error type in the vmq_ql_query handler.
• Make sure the peer_host can always be retrieved via the HTTP API. It was
  returned as an erlang tuple which caused the conversion to JSON to fail.
• Fix issue causing too few results to be returned from vmq-admin session show
  when used with filter options. This could happen when terms included in the
  filters also existed as erlang atoms.
• Plugin workflow improvements: move plugin development specific things into
  vernemq_dev.
• Fix error in the HTTP API interface. The alias /api/v1/sessions mapped to
  session list which is deprecated for session show.
• Fix max-age parse issue in vmq_webhooks (#527).