New Feature: Device SDK automatic and custom retries

A large proportion of issues reported with the device SDK are connectivity issues. with this in mind we set out a few months ago to rewrite the transports layers to be more stable and introduced code that enabled better state and error management for each protocol - in this release, we were finally able to remove the over-arching client state machine that had become useless and was causing issues. We have now introduced per-operation retries on top of that.

This means that when a user calls any of the device client API (subscribe to receive C2D messages, send telemetry, etc) the client doesn't have to know what the transport should do: just ask the transport to do it and report. the transport can then bubble up either a success result or an error. It's then up to the client to ask the transport to retry or send the error back to the user.

By default, the client will retry using "an exponential backoff with jitter" strategy, for up to 4 minutes. This is done without any changes to the API surface. your code will automatically benefit from this.

It is possible for a user to disable retries alltogether, or to define a custom retry logic and feed this to the client using the Client.setRetryPolicy() API.

For example:

var Client = require('azure-iot-device').Client;
var Amqp = require('azure-iot-device').Amqp;
var NoRetry = require('azure-iot-common').NoRetry;

var client = Client.fromConnectionString('<connectionString>', Amqp);
// Disable Retries
client.setRetryPolicy(new NoRetry());

// Custom retry policy
var myRetryPolicy = {
  shouldRetry: function (err) { 
    // decide depending on err if you would like to retry or not - should return true or false.
  },
  nextRetryTimeout: function (retryCount, throttled) {
    // should return an integer that is the number of milliseconds to wait before the next retry
    // based on the current count of retries (retryCount) 
    // and if the IoT Hub is asking clients to throttle their calls (throttled, boolean)
  }
}

client.setRetryPolicy(myRetryPolicy);

caveat
There are a few issues that this won't solve and that we will tackle next:

• There are a few errors coming from the MQTT library that we are not catching because they are thrown before we can attach an event handler on the error event - we are looking for ways to circumvent this
• The Twin feature requires heavy refactoring to support a fully-fledged retry logic - right now twin operations are not being retried automatically.

External Contributions:

• #118 Added Twin type to azure-iothub exports. Thanks @nebrius !!

This release includes upgrades to the inner workings of the transport packages (AMQP, MQTT and HTTP) to prepare for the introduction of retries in what will be the SDK version 1.2.0

• Introduction of state machines and connection state management in all transports
• Remove the receiver objects where pertinent and have a more consistent client experience across transports
• Moving some objects around (especially RestApiClient) so that more code can be shared between packages.

No API or behavioral changes should be apparent yet (except maybe more stable/more consistent connections)

All the changes in this release are internal refactoring, as we get ready to introduce the much awaited retry logic.

Fix for #76 - Fix Content-Length on UTF-8 2 bytes characters. Special thanks to @Kaaml for this!

Bugs fixed

• #49 (protobuf support)
• #60 (putToken causing crash when renewing the shared access signature)

PR accepted

• #50 (protobuf support), thanks to @willin !

Improvements

• Support for CA signed certs with MQTT
• Support for Twin operations over AMQP

• Updates to the LTS documentation
• Fixes to whitespace for typescript
• Fixes a typo in a variable name for the results interface

• Allow slashes in the blob name for file uploads
• Added information about end to end testing to the documentation
• Updated the documentation for the SDK development environment

• Update end to end test error handling
• Enable throttling tests and improve their speed
• Improve detection of link, session and connection failure in AMQP transport

• Fix #43
• Better handling of some AMQP errors associated with session and link failures

• Fix for #39
• Fix for #41 (and potentially #40 (to be confirmed))
• Additional handling of AMQP protocol errors

Bugfixes

azure-iot-device-http

• messageId and other system properties were not set correctly in D2C messages over HTTP

azure-iothub

• Device Method parameters were still using the timeoutInSeconds parameter instead of connectTimeoutInSeconds and responseTimeoutInSeconds.

Important note

We've switched to typescript during this release. while we expect no breaking changes and have decent end-to-end test coverage, it is not impossible that we missed something, especially as we turn from manual to automatic generation of d.ts file. please report any problem you find!

Bugfixes:

• Fix for #34: Twin timeout has been increased to support slower connections
• Set MQTT keepalive ping to 3 minutes, and no rescheduling, to make sure we detect MQTT connections dropping

New features

• CBS Support for Service SDK Client (uses AMQP)

Bugfixes

• Fix for #28 (could not getTwin after disconnect/reconnect)
• Fix for #25 (Node-red: Allow multiple nodes with different connection strings in the same flow)

External contributions:

• #24: Fix typos in docs (Thanks @efreiberg)

New features:

• CBS support for AMQP protocol
  CBS is a type of authentication that relies on Shared Access Keys, like SASL-PLAIN. Instead of sending the shared access signature token in the URL used to connect to IoT Hub (which means the client has to disconnect and reconnect to update this token), the connection is opened anonymously and the SAS token is sent to a specific endpoint separately and can be renewed without disconnecting the client. This is a big improvement for reliability of the AMQP protocol stack.

Bugfixes:

• A couple of typescript definition bugs were found while writing end-to-end tests and were fixed (f550328b014ea94e4867ef4a8de8c0fca4e6cf72)
• Links are now properly detached before disconnection in AMQP (15dc3a38aaf4b598e6389223517d991fe59c5f86)
• Fix an error type when failing to connect using MQTT over Websockets (68cf52df63c8edcb22d058328d2c7e9dd89fa2c0)

Service SDK: [email protected]

• Added Bulk device management APIs:
  • Registry.addDevices(deviceArray, callback)
  • Registry.updateDevices(deviceArray, callback)
  • Registry.removeDevices(deviceArray, callback)

See the code sample for details

Issues/PRs:

Pulled #16 and #17 to improve docs.

Other items:

Added a few internal APIs to azure-iot-common and azure-iot-amqp-base to enable future work.

Service Client

• Fix a typo on the methodName property in device_method_params.d.ts
• Add optional continuationToken argument to query.next*()
• Add getRegistryStatistics API to the Service SDK

Device Client

• Properly set correlationId property in AMQP
• Properly handle some errors when attaching with AMQP

Update the readme on how to run samples.

Various changes/fixes to the end to end tests.

Service SDK

• Fix AMQP over Websockets transport for the Client (Connection URL was being overwritten)

Device SDK

• Enforce UTF-8 decoding of PEM files in tests and samples (node 0.10 compatibility)
• Add ConnectionString.createWithSharedAccessKey() and ConnectionString.createWithX509Certificate() APIs
• Add E2E tests for authentication failure

Thanks to @jsturtevant for helping with the iothub-explorer debugging instructions!

Fixes for the following issues:

• https://github.com/Azure/azure-iot-sdk-node/issues/10
• https://github.com/Azure/azure-iot-sdk-node/issues/11
• https://github.com/Azure/azure-iot-sdk-node/issues/12
• https://github.com/Azure/iothub-explorer/issues/7

• Added support for message properties with the MQTT Transport (#9)
• Fixed a bug that prevented the proper error messages from showing in the console

Special thanks to: @stewartadam for his contributions to the docs and good feedback in #9 that led us to 784f5dc, 7806b43 and b780708

Device SDK

• Delay loading of azure-storage until actually needed
• Update samples to use MQTT by default
• Fix Twin class export in typescript definition files
• Throws correct NotImplementedError when a feature is not implemented by a transport

Service SDK

• Fixed wrong timeout property name in device method parameters
• Update scheduled jobs samples

iothub-diagnostics

• removed deprecated amqp-ws dependency