plumber
A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.
MIT License
Bot releases are visible (Hide)
Published by github-actions[bot] over 2 years ago
- Fix to ensure all relay workers finish with their work and exit before allowing CLI relay mode to exit
Published by github-actions[bot] over 2 years ago
Plumber can now run in server mode, allowing you to have long running plumber instances on your infrastructure.
See details at: https://docs.batch.sh/plumber/server-mode
You can communicate to Plumber server using gRPC https://docs.batch.sh/plumber/server-mode/grpc-api-quickstart
Or you can use our terraform provider https://registry.terraform.io/providers/batchcorp/plumber/latest to manage connections/relays/tunnels
Published by github-actions[bot] over 2 years ago
Published by github-actions[bot] over 2 years ago
Published by github-actions[bot] over 2 years ago
Published by github-actions[bot] over 2 years ago
- You can now relay messages from your NATS, NATS Streaming, or NATS JetStream instance to your Batch.sh collections!
Published by github-actions[bot] almost 3 years ago
Published by github-actions[bot] almost 3 years ago
Published by github-actions[bot] almost 3 years ago
- Added ability to read/write to AWS Kinesis
Published by github-actions[bot] almost 3 years ago
- Fixing version injection in build process
Published by github-actions[bot] almost 3 years ago
Features:
- Support for NATS JetStream
-
Shallow Envelope Protobuf reading/writing
- For more information about shallow envelope protobufs, see https://www.confluent.io/blog/spring-kafka-protobuf-part-1-event-data-modeling/#shallow-envelope
Published by github-actions[bot] almost 3 years ago
Example:
plumber read rabbit --exchange-name events --queue-name myqueue --binding-key \# --queue-durable --queue-arg x-dead-letter-exchange=mydeadletter
Published by github-actions[bot] almost 3 years ago
Published by github-actions[bot] almost 3 years ago
Addressed #206 (--json output), #208 (plumber not buildable from src).
Published by github-actions[bot] almost 3 years ago
v1.0.0 Release Notes
This release has a number of breaking changes, be sure to read these release notes fully.
- Improvements
- Major code architecture overhaul
- Improved Kafka lag monitoring
- Improved read output
- Improved protobuf flags to remove verbosity
- Improved CLI flag parsing
- Rearchitected message bus interface for easier addition of new messaging systems in the future
- Features
- Kafka Sampling
New Feature: Kafka Sampling
We've added two new flags for kafka --sample-rate and --sample-interval-seconds
With these flags, you can now sample messages over a time period from a high-traffic kafka topic rather trying to read through a firehose of messages like it's the Matrix!
Command changes
-
plumber azure ...is nowplumber azure-service-bus ... -
plumber azure-eventhub ...is nowplumber azure-event-hub ... -
plumber cdc-mongo ...is nowplumber mongo ... -
plumber cdc-posgres ...is nowplumber postgres ...
CLI Flag Changes
Plumber has undergone a major CLI refactor and some flags have changed.
Changed flags
| Message Bus | Old Flag | New Flag | Notes |
|---|---|---|---|
--json |
--pretty |
||
--follow |
--continuous |
||
--avro-schema |
--avro-schema-file |
||
--protobuf-dir |
--protobuf-dirs |
||
--convert |
--convert-output |
||
--json-array |
--input-as-json-array |
||
--skip-verify |
--tls-skip-verify |
||
--insecure-tls |
--tls-skip-verify |
||
| Azure Service Bus | --subscription |
--subscription-name |
|
| GCP PubSub | --ack |
--ack-messages |
|
| GCP PubSub | --sub-id |
--subscription-id |
|
| Kafka | --username |
--sasl-username |
|
| Kafka | --password |
--sasl-Password |
|
| Kafka | --auth-type |
--sasl-type |
|
| Kafka | --rebalance-timeout |
--rebalance-timeout-seconds |
|
| Kafka | --commit-interval |
--commit-interval-seconds |
|
| Kafka | --group-id |
--consumer-group-name |
|
| MQTT | --tls-ca-file |
-tls-ca-cert |
Now accepts file path or certificate as a string |
| MQTT | --tls-client-cert-file |
--tls-client-cert |
Now accepts file path or certificate as a string |
| MQTT | --tls-client-key-file |
-tls-client-key |
Now accepts file path or certificate as a string |
| MQTT | --qos |
--qos-level |
|
| MQTT | --timeout |
--con-timeout-seconds |
|
| Nats | --creds-file |
--user-credentials |
|
| Nats | --tls-ca-file |
--tls-ca-cert |
Now accepts file path or certificate as a string |
| Nats | --tls-cert-file |
--tls-client-cert |
Now accepts file path or certificate as a string |
| Nats | --tls-key-file |
--tls-client-key |
Now accepts file path or certificate as a string |
| Nats | --address |
--dsn |
|
| Nats Streaming | --creds-file |
--user-credentials |
|
| Nats Streaming | --last |
--read-last-available |
|
| Nats Streaming | --all |
--read-all |
|
| Nats Streaming | --seq |
--read-sequence-number |
|
| Nats Streaming | --since |
--read-since |
|
| NSQ | --tls-ca-file |
--tls-ca-cert |
Now accepts file path or certificate as a string |
| NSQ | --tls-cert-file |
--tls-client-cert |
Now accepts file path or certificate as a string |
| NSQ | --tls-key-file |
--tls-client-key |
Now accepts file path or certificate as a string |
| Postgres CDC | --slot |
--replication-slot |
|
| Postgres CDC | --publisher |
--publisher-name |
|
| Pulsar | --name |
--subscription-name |
|
| Pulsar | --type |
--subscription-type |
|
| Pulsar | --auth-certificate-file |
--tls-client-cert |
|
| Pulsar | --auth-key-file |
--tls-client-key |
|
| RabbitMQ | --exchange |
--exchange-name |
|
| RabbitMQ | --queue-auto-delete |
--queue-delete |
|
| RabbitMQ | --routing-key |
--binding-key |
Read mode only. Write mode still uses correct nomenclature of --routing-key
|
| Redis Streams | --id |
--write-id |
|
| Redis Streams | --start-id |
--ofset-start |
Options: oldest, latest
|
Removed Flags
| Message Bus | Flag | Notes |
|---|---|---|
--thrift |
Moved to an option under --decode-type
|
New Flags
| Message Bus | Flag | Description |
|---|---|---|
--decode-type |
Options: protobuf, avro, thrift
|
|
| RabbitMQ | --exchange-type |
Exchange type to use when declaring a new exchange. Ex: topic, direct, fanout
|
| RabbitMQ | --exchange-delete |
Whether to remove the exchange at disconnect |
| RabbitMQ | --exhcnage-declare |
Whether to declare a new exchange when connecting |
| RabbitMQ | --exchange-durable |
Create exchange as durable when declaring a new exchange |
| Kafka | --include-offset-info |
Displays consumer offset stats when reading |
| Kafka | --lag-consumer-group |
What consumer group to lookup lag for (required if --lag is set) |
Published by github-actions[bot] almost 3 years ago
Published by github-actions[bot] almost 3 years ago
Published by github-actions[bot] almost 3 years ago
Bumping gjson version to v1.10.2 to address CVE
Published by github-actions[bot] about 3 years ago
Published by github-actions[bot] about 3 years ago
Adding --credentials-json flag for GCP-PubSub so that credentials can be passed in via a string
