plumber

This release includes a few fixes for the plumber batch ... commands which can be used to perform actions on your collections via PlumbeR CLI:

• Additional flags for creating collections via plumber batch create collection:
  • --envelope-type for specifying deep or shallow protobuf decoding
  • --envelope-root-message for specifying protobuf envelope message name
  • --payload-field-message for specifying protobuf shallow envelope message name
  • --payload-field-id for specifying shallow envelope field ID
• Updated plumber batch list replay output to support new API response format
• Removed deprecated Root Type column from plumber batch list schema

When performing a replay to a tunnel, you can now select the Include Original Attributes option, and plumber will replay the original Kafka message headers along with the message

Thrift decoding can now use your .thrift IDL files to decode the payload and output field names and num values instead of IDs.
Two new flags have been added to support this:

• --thrift-dirs: Path to the .thrift IDL files containing your struct definitions
• --thrift-struct: namespace and name of the struct to decode the message with. Ex: com.mycompany.schema.Account, where com.mycompany.schema is the namespace and Account is the name of the struct that the message is encoded with

These flags are optional, and using --decode-type thrift without these flags will retain the previous behavior

See example located in the docs: https://github.com/batchcorp/plumber/blob/master/docs/examples.md#thrift-decoding-with-idl-files

Overview

This release has a number of changes (like updating RabbitMQ libs, support for auth-n in NATS) and various small bug fixes but the "biggest" update is the addition of telemetry.

We know a bunch of folks are using plumber - but we don't know what features or what backends they use, so we have a hard time figuring out where we should put our energy towards to next.

This release adds telemetry support to plumber. It's primary function is to figure out usage info - things like what actions you perform the most (read/write/tunnel/etc), what backends you use and of course, what OS and arch you're on.

Here are some facts about the telemetry:

• The telemetry is opt-in, as in, telemetry is off by default
• You will be asked whether you want to participate in telemetry on the initial launch of plumber
• We do NOT collect any potentially confidential information or PII

You can read more about plumber telemetry here: https://docs.batch.sh/plumber/telemetry

Install

Reminder: you can install the latest client by downloading the arch/os appropriate release below or using brew:

$ brew tap batchcorp/public
$ brew install plumber

Thank you for using plumber!

• Added new flag --nkey flag to nats and nats-jetstream backends to support nkey authentication for #303

All backend Read() methods now respect contexts properly and cleanly exit. This fixes a bug introduced in v1.10.3

• Nats consumers will now be cleaned up properly on plumber exit. Thank you @edarha for the fix!

• Fix: Use correct subject when reading via Nats Jetstream with a durable consumer

Thanks @edarha for the fix!

Fixes:

• When writing to Nats JetStream, --stream flag has been changed to the more correct --subject flag

Improvements:
New options for consuming from Nats JetStream have been added:

      --consumer-name=STRING              Consumer name (default: random consumer name; has no effect if
                                          create_durable_consumer or existing_durable_consumer is not set)
                                          ($PLUMBER_RELAY_NATS_JETSTREAM_CONSUMER_NAME)
      --create-durable-consumer           Create a new durable consumer (default: use ephemeral consumer)
                                          ($PLUMBER_RELAY_NATS_JETSTREAM_CREATE_DURABLE_CONSUMER)
      --existing-durable-consumer         Use an existing durable consumer (default: use ephemeral consumer)
                                          ($PLUMBER_RELAY_NATS_JETSTREAM_EXISTING_DURABLE_CONSUMER)
      --keep-consumer                     Do not delete consumer after operation is done (default: delete on exit)
                                          ($PLUMBER_RELAY_NATS_JETSTREAM_KEEP_CONSUMER)
      --consumer-start-sequence=INT-64    Where to start read in stream (default: from start)
                                          ($PLUMBER_RELAY_NATS_JETSTREAM_CONSUMER_START_SEQUENCE)
      --consumer-start-time=STRING        At what time to start stream (RFC3339 format, Ex: 2016-06-20T12:41:45.14Z)
                                          ($PLUMBER_RELAY_NATS_JETSTREAM_CONSUMER_START_TIME)
      --consumer-filter-subject=STRING    Only receive a subset of messages from the Stream based on the subject
                                          ($PLUMBER_RELAY_NATS_JETSTREAM_CONSUMER_FILTER_SUBJECT)

By default, plumber will use an ephemeral consumer (which does not track stream sequence bits). To track sequence numbers/offsets, use create-durable-consumer or create one by hand and specify it via --existing-durable-consumer and --consumer-name.

NOTE: If you specify consumer-start-sequence - this applies to the consumer config so once the consumer consumes from sequence XYZ, a follow-up consume won't produce the same results. If you want to always consume from a given position - do create-durable-consumer every time.

• Improvement: Implement UpdateRelay() gRPC method in server mode
• Fix: Tunnel/Relay startup will no longer block server startup
• Fix: Update relay examples in examples.md for newer CLI syntax

• Added support for google.protobuf.Any field definitions
• Added flag --protobuf-descriptor-set flag which accepts a path to a .protoset/.fds file. This will allow for more reliable protobuf encoding/decoding when dealing with complicated protobuf include paths

• Fixed incorrect error messages that were outputting old TLS flag names
• Fixed MQTT TLS config
• Deduplicated some TLS code

• Fix multiple dropped errors. Thanks @alrs for the fixes!
• Deleting a tunnel from a plumber server now instantly removes it from your https://console.batch.sh destinations list

Minor fix to plumber manage delete relay subcommand in.

• Adding --use-tls flag to NATS backends
• Adding examples for NATS JetStream to examples.md
• Documenting NATS relay envars

You can now manage a running Plumber server instance via the CLI!
See the docs for instructions https://docs.batch.sh/plumber/server-mode/server-management/manage-via-cli