plumber

Added support for reading/writing/relaying messages from KubeMQ queues.
Thank you @kubemq for the work on this feature!

Thank you to @Gsantomaggio for these improvements!

Support for the newly released RabbitMQ Streams has been added. Thank you to @Gsantomaggio for his assistance with this!

Read example: https://github.com/batchcorp/plumber/blob/master/EXAMPLES.md#rabbitmq-streams
Writing example: https://github.com/batchcorp/plumber/blob/master/EXAMPLES.md#rabbitmq-streams-1

If your message bus contains Thrift IDL encoded messages, you can now specify the --thrift flag, and plumber will decode the messages

You can now view Kafka consumer group lag for a topic by running

$ plumber lag kafka --topic yourtopic --group-id yourconsumergroupname

You can also pass the --lag flag when reading from Kafka and the output will display the latest offset.

$ plumber read kafka --topic yourtopic --lag --group-id yourconsumergroupname

Thank you to @ei09010 for his work on this!

• You can now pipe data into a Plumber write command. See https://github.com/batchcorp/plumber/blob/master/README.md#write-messages-via-pipe for examples
• Prometheus metrics are now exposed in relay mode. See https://github.com/batchcorp/plumber/blob/master/EXAMPLES.md#prometheus-metrics for details

When using relay mode, plumber will now ensure all relay messages are sent to Batch before exiting the application

Plumber's new dynamic destination mode allows it to act as a replay destination for your batch collections!
See https://docs.batch.sh/what-are/what-are-destinations/plumber-as-a-destination for more information

• Support for reading/writing messages for NSQ
• Support for relaying NSQ messages to your batch collections

Plumber release 0.26.0

Lots of good stuff here.

1. Ability to specify headers when writing to Kafka
2. Dramatically improved read output
3. Added --json flag which will attempt to format + colorize output that is valid JSON
4. Kafka relay now includes headers
5. Protobuf root message type now must include the full name that includes pkgs (ie. events.Message instead of Message)
6. --line-numbers has been removed - count is now always showing.

This is what a read now looks like:

• Listing of replays has been added plumber batch list replay
• plumber batch list * commands now support --output json flag to display output as JSON instead of a table

It is now possible to read (and relay) from multiple topics at the same time for Kafka.

To read/relay for multiple topics, specify multiple --topic flags (or separate topic names with comma, if using env vars).

Also:

• Fixed a relay recovery bug that can occur if Kafka broker temporarily goes away
• Fixed stats display bug (was not including the relay type in output)

• Added support for Apache Pulsar
• You can now relay MQTT messages to a Batch.sh collection

• Kafka read/write: multiple kafka brokers can now be specified by repeating the --address flag for each broker
• Kafka relay: multiple kafka brokers can now be specified by separating them by a comma. Example: PLUMBER_RELAY_KAFKA_ADDRESS="broker1.domain.com:9092,broker2.domain.com:9092"