Improvements:

• A new record is shipped for every change for cdc-postgres, instead of grouping by commit
• Reduced cdc-postgres code deduplication

Fixes:

• Adding missing stats.Incr() calls for cdc-postgres and cdc-mongo
• You may now listen for changes to your entire mongo deployment by omitting the database and collection flags

This release adds two new relay commands: cdc-postgres and cdc-mongo to allow relaying of CDC data from Postgres' logical replication streams and MongoDB's change stream events.

See https://docs.batch.sh/event-ingestion/change-data-capture/mongodb and https://docs.batch.sh/event-ingestion/change-data-capture/postgresql for documentation on how to stream these events into a Batch.sh collection

--all - Delivers all available messages
--since - Delivers messages from an interval ago. Ex: 1m, 24h
--last - Delivers last received message
--seq - Delivers message from a given sequence ID

• plumber read azure-eventhub and plumber write azure-eventhub to publish and consume from an azure event hub
• plumber read nats-streaming and plumber write nats-streaming to publish and consume from Nats jetstream
• Added --message-group-id' and --message-deduplication-idflags toplumber write aws-sqs` to support FIFO queus

Now you can manage common Batch.sh account tasks directly from Plumber!

Minor relay fixes:

• Fix redis-streams stats display (wasn't increasing stats or muting on error)
• Fix bug where redis-streams relay incorrectly exits on error (should run forever and just log error)
• Kafka relay stats now get muted when read results in an error (+ retry in 5s)
• Add RetryReadInterval on error for gcp pubsub relay
• Add RetryReadInterval on error for aws-sqs relay

• Added read, write and relay support for redis streams
• Renamed redis to redis-pubsub (to follow existing conventions such as aws-sqs and aws-sns)
• write redis-pubsub now writes to all specified channels (originally was just the first)
• use semantic versioning in --version .. turns out, having a short sha is not terribly intuitive 😄

Install

Download the appropriate release binary or install via brew:

brew tap batchcorp/public
brew install plumber

This release adds support for relaying Redis pubsub channels to Batch.

To run plumber relay with redis in Docker:

docker run -d --name plumber-redis -p 8080:8080 \
    -e PLUMBER_RELAY_REDIS_ADDRESS=localhost:6379 \
    -e PLUMBER_RELAY_REDIS_CHANNELS=channel1,channel2 \
    -e PLUMBER_RELAY_TYPE=redis \
    -e PLUMBER_RELAY_TOKEN=$YOUR-BATCHSH-TOKEN-HERE \
    batchcorp/plumber

Dockerhub: docker pull batchcorp/plumber:a934947

Breaking changes:

• remove --output-type for reading; remove --input-type for writing. Both are unnecessary and the intent can be gathered from other flags.

Features:

• Kafka: Ability to specify an offset to begin reading from
• GCP: Environment variable PUBSUB_EMULATOR_HOST can now be specified to connect to a pubsub emulator instead of GCP
• Retries for GRPC calls in relay mode

Bug fixes:

• --queue-declare now actually declares the queue for rabbit
• Increasing max record size for relay payloads

This is a fairly large (and awesome) release.

In this release:

• New --stats flag (and PLUMBER_STATS env var) which will cause plumber to periodically dump consume/produce speed stats.
• Lots of kafka improvements to improve read and relay throughput
  • Several new env vars that can be tweaked to further improve throughput and/or reliability
• Batching support for relay
• Pretty config printing - all relay modes will now print out the config they're using on console
  • plumber now has a lot of options - some messaging platforms (such as kafka) have 20+ flags/options - it is nice to be able to see what options are actually being used at runtime.

Enjoy!

1. Support for relaying Azure Service Bus to Batch.sh
2. Improved Kafka read and relay speeds; works with Confluent cloud!
   1. Kafka read and relay is highly configurable for improved throughput
3. Improved docs (including env var docs!)

Docker: https://hub.docker.com/r/batchcorp/plumber/