plumber

Now you can manage common Batch.sh account tasks directly from Plumber!

This release adds support for relaying Redis pubsub channels to Batch.

To run plumber relay with redis in Docker:

docker run -d --name plumber-redis -p 8080:8080 \
    -e PLUMBER_RELAY_REDIS_ADDRESS=localhost:6379 \
    -e PLUMBER_RELAY_REDIS_CHANNELS=channel1,channel2 \
    -e PLUMBER_RELAY_TYPE=redis \
    -e PLUMBER_RELAY_TOKEN=$YOUR-BATCHSH-TOKEN-HERE \
    batchcorp/plumber

Dockerhub: docker pull batchcorp/plumber:a934947

Breaking changes:

• remove --output-type for reading; remove --input-type for writing. Both are unnecessary and the intent can be gathered from other flags.

Features:

• Kafka: Ability to specify an offset to begin reading from
• GCP: Environment variable PUBSUB_EMULATOR_HOST can now be specified to connect to a pubsub emulator instead of GCP
• Retries for GRPC calls in relay mode

Bug fixes:

• --queue-declare now actually declares the queue for rabbit
• Increasing max record size for relay payloads

This is a fairly large (and awesome) release.

In this release:

• New --stats flag (and PLUMBER_STATS env var) which will cause plumber to periodically dump consume/produce speed stats.
• Lots of kafka improvements to improve read and relay throughput
  • Several new env vars that can be tweaked to further improve throughput and/or reliability
• Batching support for relay
• Pretty config printing - all relay modes will now print out the config they're using on console
  • plumber now has a lot of options - some messaging platforms (such as kafka) have 20+ flags/options - it is nice to be able to see what options are actually being used at runtime.

Enjoy!

1. Support for relaying Azure Service Bus to Batch.sh
2. Improved Kafka read and relay speeds; works with Confluent cloud!
   1. Kafka read and relay is highly configurable for improved throughput
3. Improved docs (including env var docs!)

Docker: https://hub.docker.com/r/batchcorp/plumber/

• Added --app-id flag for RabbitMQ write mode
• Added --consumer-tag flag for RabbitMQ read mode

• Added Avro support for all backends
• Avro schema can be used by setting the --avro-schema ./yourschema.avsc

• Relay mode now supports RabbitMQ!
• RabbitMQ now gracefully handles network disconnects and will reattempt connections every 60s
• Introduced new library batchcorp/rabbit to plumber
• Bug fixes for SQS
• README.md cleanup
• Added --remote-account-id flag for SQS
• Added support for specifying multiple protobuf dirs

Big release - relay support in plumber!

It is now possible to run plumber in "relay" mode. In this mode, plumber will attach to a target messaging system and relay all of the messages/events it sees to Batch via gRPC.

You can run plumber in either foreground mode:

$ plumber relay aws-sqs --queue-name PlumberTestQueue --token 48b30466-e3cb-4a58-9905-123456781234 --auto-delete

Or you can run it via Docker:

docker run --name plumber -p 8080:8080 \
    -e PLUMBER_RELAY_TYPE=aws-sqs \
    -e PLUMBER_RELAY_TOKEN=48b30466-e3cb-4a58-9905-123456781234 \
    -e PLUMBER_RELAY_SQS_QUEUE_NAME=PlumberTestQueue \
    -e PLUMBER_RELAY_SQS_AUTO_DELETE=true \
    -d batchcorp/plumber:latest

Support for both reading and writing to/from AWS SQS:

# Writing plaintext
$ plumber write message aws-sqs --queue-name PlumberTestQueue --input-data "hello world"

# Writing protobuf
$ plumber write message aws-sqs --queue-name PlumberTestQueue --input-file ./test-assets/messages/sample-outbound.json --input-type jsonpb --output-type protobuf --protobuf-dir ./test-assets/protos/ --protobuf-root-message Outbound

# Reading plaintext
$ plumber read message aws-sqs --queue-name PlumberTestQueue --auto-delete
INFO[0000] Listening for message(s) ...                  pkg=awssqs/read.go
hello world

# Reading protobuf
$ plumber read message aws-sqs --queue-name PlumberTestQueue --protobuf-dir ./test-assets/protos --protobuf-root-message Outbound --output-type protobuf
INFO[0000] Listening for message(s) ...                  pkg=awssqs/read.go
{
  "replayId": "30ddb850-1aca-4ee5-870c-1bb7b339ee5d",
  "blob": "eyJoZWxsbyI6ImRhbiJ9Cg=="
}

For full info: plumber read message aws-sqs --help and plumber write message aws-sqs --help