plumber

• Added --app-id flag for RabbitMQ write mode
• Added --consumer-tag flag for RabbitMQ read mode

• Added Avro support for all backends
• Avro schema can be used by setting the --avro-schema ./yourschema.avsc

• Relay mode now supports RabbitMQ!
• RabbitMQ now gracefully handles network disconnects and will reattempt connections every 60s
• Introduced new library batchcorp/rabbit to plumber
• Bug fixes for SQS
• README.md cleanup
• Added --remote-account-id flag for SQS
• Added support for specifying multiple protobuf dirs

Big release - relay support in plumber!

It is now possible to run plumber in "relay" mode. In this mode, plumber will attach to a target messaging system and relay all of the messages/events it sees to Batch via gRPC.

You can run plumber in either foreground mode:

$ plumber relay aws-sqs --queue-name PlumberTestQueue --token 48b30466-e3cb-4a58-9905-123456781234 --auto-delete

Or you can run it via Docker:

docker run --name plumber -p 8080:8080 \
    -e PLUMBER_RELAY_TYPE=aws-sqs \
    -e PLUMBER_RELAY_TOKEN=48b30466-e3cb-4a58-9905-123456781234 \
    -e PLUMBER_RELAY_SQS_QUEUE_NAME=PlumberTestQueue \
    -e PLUMBER_RELAY_SQS_AUTO_DELETE=true \
    -d batchcorp/plumber:latest

Support for both reading and writing to/from AWS SQS:

# Writing plaintext
$ plumber write message aws-sqs --queue-name PlumberTestQueue --input-data "hello world"

# Writing protobuf
$ plumber write message aws-sqs --queue-name PlumberTestQueue --input-file ./test-assets/messages/sample-outbound.json --input-type jsonpb --output-type protobuf --protobuf-dir ./test-assets/protos/ --protobuf-root-message Outbound

# Reading plaintext
$ plumber read message aws-sqs --queue-name PlumberTestQueue --auto-delete
INFO[0000] Listening for message(s) ...                  pkg=awssqs/read.go
hello world

# Reading protobuf
$ plumber read message aws-sqs --queue-name PlumberTestQueue --protobuf-dir ./test-assets/protos --protobuf-root-message Outbound --output-type protobuf
INFO[0000] Listening for message(s) ...                  pkg=awssqs/read.go
{
  "replayId": "30ddb850-1aca-4ee5-870c-1bb7b339ee5d",
  "blob": "eyJoZWxsbyI6ImRhbiJ9Cg=="
}

For full info: plumber read message aws-sqs --help and plumber write message aws-sqs --help

• Support for MQTT

  # Read (forever) example
  $ plumber read message mqtt --topic dantest  -d -f --line-numbers --address tcp://test.mosquitto.org:1883
  
  # Write example
  $ plumber write message mqtt --topic dantest --input-data testing --address tcp://test.mosquitto.org:1883
  
  # Read (forever) with protobuf
  $ plumber read message mqtt --topic dantest --address tcp://test.mosquitto.org:1883 -d -f --line-numbers --output-type protobuf --protobuf-dir ./test-assets/protos --protobuf-root-message Outbound
  
  # Write protobuf
  $ plumber write message mqtt --topic dantest --address tcp://test.mosquitto.org:1883 --input-file ./test-assets/messages/sample-outbound.json --input-type jsonpb --output-type protobuf --protobuf-dir ./test-assets/protos --protobuf-root-message Outbound
  

• Cleaned up debug info from output

Fairly significant cleanup release; significant refactor (moved to a diff lib); cleanup of some flags.

Added support for GCP PubSub.

• Added support to gzip decompress output on reads (from kafka and rabbit)
• Removed --exchange-name as required arg (forgot that rabbit has default exchange support)

This is the initial release of the port of our internal "plumber" tool that we use for consuming and publishing messages on our event bus.

It is nowhere near feature complete, but it does have the most important part implemented (in our opinion):

It is able to read and write protobuf based messages to either Kafka or RabbitMQ. We think that's pretty damn sweet.

We hope it's useful for you!