peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

OTHER License

Stars
1.8K
Committers
9
View Code on GitHub Visit Website
Ecosystems: PostgreSQL, Rust, BigQuery

Bot releases are hidden (Show)

peerdb - v0.13.3 Latest Release

Published by Amogh-Bharadwaj 5 months ago

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.13.2...v0.13.3

peerdb - v0.13.2

Published by Amogh-Bharadwaj 5 months ago

New Contributors

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.13.1...v0.13.2

peerdb - v0.13.1

Published by Amogh-Bharadwaj 6 months ago

What's Changed

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.13.0...v0.13.1

peerdb - v0.13.0

Published by Amogh-Bharadwaj 6 months ago

What's Changed

This release of PeerDB stars the introduction of a brand new ElasticSearch connector for query replication. Secondly, it ships an in-house min.io bucket integration for the Clickhouse peer, boosting its usability. Also included is a major effort towards perfection of Postgres to Postgres CDC and improved performance of replication to queues. Improvements to UI, tests and telemetry are featured as well.

  • βš™οΈ Elasticsearch connector for QRep by @heavycrystal in #1618
  • πŸ“¦ In-house S3 stage for Clickhouse by @Amogh-Bharadwaj in #1634
  • πŸ”„ PgValue variant bypassing QValue for more efficient & correct PG<>PG mirroring by @serprex in #1565
  • 🌐 Lua: peerdb.LuaRowTable by @serprex in #1612
  • ⚑ Parallelized Lua script execution with serialized output by @serprex in #1613
  • πŸ–₯️ UI Create Peer: replace dropdown with columns of buttons by @serprex in #1611
  • ❄️ Snowflake: Addition of dynamic numeric columns by @Amogh-Bharadwaj in #1590
  • πŸ“Š Queues: communicate flushes back to source in order to keep slot size down by @serprex in #1626
  • πŸ’» UI for opting into PG type system by @Amogh-Bharadwaj in #1630
  • 🎯 Eventhubs E2E foundation by @Amogh-Bharadwaj in #1631
  • πŸ› οΈ AWS: add credentials loading mechanism using SDK config by @iamKunalGupta in #1507
  • πŸ§ͺ Framework for ES tests + basic QRep test by @heavycrystal in #1644
  • πŸ–ΌοΈ UI: Better Create QRep by @Amogh-Bharadwaj in #1643
  • πŸ—‘οΈ Soft delete false for Snowflake initial load by @Amogh-Bharadwaj in #1647
  • πŸ”” Alerting: enhance telemetry logging by @iamKunalGupta in #1642

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.12.2...v0.13.0

peerdb - v0.12.2

Published by Amogh-Bharadwaj 6 months ago

What's Changed

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.12.1...v0.12.2

peerdb -

Published by Amogh-Bharadwaj 6 months ago

peerdb - v0.12.0

Published by Amogh-Bharadwaj 7 months ago

What's Changed

Version 12 of PeerDB features a new wing to the product - queues. This release introduces support for mirrors to Kafka, PubSub and Azure Eventhubs with transformations using Lua scripts. Documentation for this feature is ongoing at: https://docs.peerdb.io/lua/reference
This version also provides new UI indicators for the mirror page, improved data type mapping and performance/robustness boosts for initial load and CDC.

New Contributors

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.11.4...v0.12.0

peerdb - v0.11.4

Published by Amogh-Bharadwaj 7 months ago

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.11.3...v0.11.4

peerdb - v0.11.3

Published by Amogh-Bharadwaj 7 months ago

What's Changed

This release includes some data type improvements for Snowflake, along with some more improvements and test additions to boost robustness

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.11.2...v0.11.3

peerdb - v0.11.2

Published by Amogh-Bharadwaj 7 months ago

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.11.1...v0.11.2

peerdb - v0.11.1

Published by Amogh-Bharadwaj 7 months ago

What's Changed

This release includes some important bug fixes in our CDC code, some related to how Postgres records LSNs and TID scanning in PostgreSQL 12.

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.11.0...v0.11.1

peerdb - v0.11.0

Published by Amogh-Bharadwaj 7 months ago

What's Changed

In this version of PeerDB, we restructured our flow architecture resulting in improved reliability (ex: drop mirror) and performance. We added support for a brand new connector in Clickhouse! On the UI side, we now have upgraded validation checks across the board, along with cutting-edge features such as pausing and editing mirrors.
This release sees a major investment in the alerting and monitoring systems, with support for email and slack alerts. We also added support for SSH connections to PostgreSQL peers. When it comes to the core ETL logic, we've solidified our data type handling, memory and storage management, logging and testing.

  • πŸ”„ Split cdc_flow into cdc_flow / sync_flow by @serprex in #1365
  • πŸ› οΈ Clickhouse cdc by @pankaj-peerdb in #1096
  • πŸ“§ feat(alerting): add email sender by @iamKunalGupta in #1433
  • ⏸️ Pause and resume mirror buttons, along with state reflection by @heavycrystal in #1133
  • βš™οΈ Add Tables Feature: Added dynamic table addition to existing props signal by @heavycrystal in #1106
  • πŸš€ feat: add telemetry/alerts via sns by @iamKunalGupta in #1411
  • πŸ“¦ Setup partitioning and clustering for raw table by @iskakaushik in #915
  • πŸ—οΈ SSH Support: Use SSHWrappedPool when querying the Postgres Peer by @iskakaushik in #1148
  • βš™οΈ Custom threshold support for slack alerts by @heavycrystal in #1277
  • πŸ”„ Normalize concurrently with sync flows by @serprex in #893
  • πŸ’Ύ Spill to disk based on flow-worker memory usage by @heavycrystal in #1231
  • πŸ”„ Update max batch size on signal by @iskakaushik in #910
  • πŸ”„ drop_flow: retry until both source/destination succeed by @serprex in #1201
  • βš™οΈ Adds idletimeout to flow config, ui and temporal signal by @Amogh-Bharadwaj in #952
  • βš™οΈ Added capability for BQ CDC across datasets by @heavycrystal in #904
  • πŸ’Ύ Support dynamic numeric with defaults by @Amogh-Bharadwaj in #1194
  • πŸ–₯️ UI: Edit Page and Refactor Edit Mirror Route by @Amogh-Bharadwaj in #1156
  • πŸ”„ Retry when WAL segment has not been found by @iskakaushik in #930
  • πŸ—ΊοΈ HStore and Geospatial for Postgres by @Amogh-Bharadwaj in #1091
  • βš™οΈ Dynamically add new tables to CDC mirrors by @heavycrystal in #1084
  • πŸ”„ cdc_flow: listen for shutdown request while sync flow in progress by @serprex in #1103
  • πŸ”„ Go 1.22 by @serprex in #1219
  • πŸ–₯️ Clickhouse UI by @pankaj-peerdb in #1022
  • πŸ› οΈ Validate Mirror: PostgreSQL Checks by @Amogh-Bharadwaj in #1110
  • βš™οΈ Support specifying host key for ssh config by @serprex in #1125
  • πŸ—‘οΈ drop_flow: drop destination/source concurrently by @serprex in #1101
  • πŸ“¦ BQ Peer: Support project dot dataset by @Amogh-Bharadwaj in #1073
  • πŸ› οΈ Validate peer: permission check for snowflake by @Amogh-Bharadwaj in #1126
  • πŸ—‘οΈ Removing deprecated params from QRep by @heavycrystal in #1154
  • πŸ’“ HeartbeatRoutine: use explicit ticker by @serprex in #1157
  • πŸ› οΈ Validate peer: check bigquery permissions by @Amogh-Bharadwaj in #1119
  • πŸ–₯️ UI: Show peer configuration on clicking peer by @Amogh-Bharadwaj in #1168
  • πŸ”„ Always set application_name when connecting to postgres by @serprex in #1169
  • πŸ–₯️ UI: Resync button by @Amogh-Bharadwaj in #1178
  • πŸ–₯️ UI: Line chart for slot growth by @Amogh-Bharadwaj in #1184
  • πŸ—‘οΈ Remove configurable postgres metadata database by @serprex in #1189
  • πŸ› οΈ Snowflake: stop storing metadata on warehouse; store in catalog by @serprex in #1179
  • πŸ› οΈ BigQuery: stop storing metadata on warehouse; store in catalog by @serprex in #1191
  • πŸ”„ Clickhouse cdc data types by @pankaj-peerdb in #1210
  • πŸ“ Better CDC error logging by @iskakaushik in #1275
  • πŸ› οΈ Connectors: build on a single GetConnector using generic GetConnectorAs function by @serprex in #1281

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.10.2...v0.11.0

peerdb - v0.10.2

Published by Amogh-Bharadwaj 10 months ago

peerdb - v0.10.1

Published by Amogh-Bharadwaj 10 months ago

peerdb - v0.10.0

Published by Amogh-Bharadwaj 10 months ago

What's Changed

In this version of PeerDB, we improved the way in which pull rows from Postgres, and we also track the WAL extensively. This release includes support for Temporal Cloud. Of course, it comes with exciting new features for PeerDB mirror such as Mirror Resync, major UI upgrades, and heavy focus on development experience.

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.9.2...v0.10.0

peerdb - v0.9.2

Published by iskakaushik 12 months ago

What's Changed

In this version of PeerDB the performance of streaming has improved all around by ~30%. This is because of push-while-pull rather than pull-then-push architecture we had before. We also spent a good chunk of time on the UI.

  • πŸš€ Either go through gRPC gateway or use prisma by @iskakaushik in PR#519
  • πŸ”„ Removed ENABLE_STATS option, checking for catalog connectivity by @heavycrystal in PR#517
  • πŸ› Fix docker builds for ui by @iskakaushik in PR#520
  • 🌍 Geospatial support for Snowflake by @Amogh-Bharadwaj in PR#516
  • πŸ› οΈ Make qrep status more useful by @iskakaushik in PR#522
  • πŸ”§ Add the ability to push to eventhubs in an asynchronous way by @iskakaushik in PR#523
  • πŸ“¦ [ui] Minor bugfixes and improvements by @iskakaushik in PR#524
  • πŸ“ Better COPY command quoting by @Amogh-Bharadwaj in PR#526
  • πŸ“Š Add tabs for cdc mirror status page by @iskakaushik in PR#525
  • πŸ”‘ Composite primary key support for SF, PG, and BQ by @heavycrystal in PR#499
  • πŸ“ Optionally create watermark table on destination for qrep mirrors by @heavycrystal in PR#528
  • πŸ“ˆ Some refinements to the status pages by @iskakaushik in PR#529
  • πŸ”— Improve connection params for postgres connector by @iskakaushik in PR#530
  • πŸ’‘ Optimize Avro Streaming with zstd Compression for Snowflake by @iskakaushik in PR#527
  • πŸ–₯️ UI for Create QRep Mirror by @Amogh-Bharadwaj in PR#532

Full Changelog: https://github.com/PeerDB-io/peerdb/compare/v0.8.1...v0.9.0

peerdb - v0.8.1

Published by iskakaushik about 1 year ago

πŸš€ Release Notes v0.8.1

🌟 Highlights:

  • Major improvements and features added to EventHub GA.
  • Introduction of GCS as a new destination.
  • Important schema changes and several new features to boost your workflow.
  • Miscellaneous changes for overall system improvements.
  • Warm welcome to our new contributors!

πŸŽ‰ EventHub GA:

  • πŸ“ Parameters for EH CDC by @Amogh-Bharadwaj in #375
  • πŸ“Š Event Hub CDC Logs by @Amogh-Bharadwaj in #374
  • ⏲ Metrics, timing for Eventhub CDC by @Amogh-Bharadwaj in #390
  • πŸ—‘οΈ Adding DROP MIRROR support for EventHub by @heavycrystal in #402
  • ❀️ Update heartbeat for eventhub by @iskakaushik in #409
    ... [and more]

🌐 GCS as a destination:

  • πŸ“¦ CDC to S3/GCS by @Amogh-Bharadwaj in #507
  • πŸ”„ Support GCS via S3 API for query replication by @Amogh-Bharadwaj in #502

πŸ”§ Basic Schema Changes:

  • πŸ“ Basic ADD COLUMN replay support for PG, BQ, and SF by @heavycrystal in #368

✨ New Features:

  • πŸ›‘ Support DROP MIRROR for Query Replication by @iskakaushik in #481
  • πŸ•‘ XMIN for Query Replication by @Amogh-Bharadwaj in #403
  • βœ‚οΈ Added QRep overwrite mode, to truncate destination table by @heavycrystal in #385
    ... [and more]

πŸ› Bug Fixes:

  • ⚠️ [Important] Return only at commit message in Postgres CDC by @iskakaushik in #503
  • βŒ› Wait for the workflow to actually close before proceeding by @iskakaushik in #417
  • βœ… Replica Identity Check by @Amogh-Bharadwaj in #392
    ... [and more]

πŸš€ Performance:

  • ⬆️ Pass QValue by pointer to significantly reduce gc pressure by @iskakaushik in #381
  • πŸ“€ [postgres] Copy to destination not staging by @iskakaushik in #498

πŸ”§ Miscellaneous:

  • πŸ”„ Split Connector interface to better represent capabilities by @heavycrystal in #376
  • πŸ“Š Optimizing flow tests by separating and parallelizing test suites by @heavycrystal in #356
    ... [and more]

πŸ†• New Contributors:

  • πŸŽ‰ @serprex made their first contribution in #451
  • πŸŽ‰ @iamKunalGupta made their first contribution in #508

Full Changelog: v0.7.1...v0.8.1

Package Rankings
Top 6.47% on Proxy.golang.org
Badges
Extracted from project README
Workflow Status ElV2 License Slack Community
Related Projects
pgcat

PostgreSQL pooler with sharding, load balancing and failover support.

29 Jan 2022 2,916
materialize

The data warehouse for operational workloads.

22 Feb 2019 5,667
pgwire

PostgreSQL wire protocol implemented as a rust library.

11 Jun 2022 509
pg-extend-rs

Postgres extension library for Rust

22 Dec 2018 276