clickhouse-sink-connector

Kafka version with support for passing JDBC parameters.

Added option to write UUID/BLOB(Debezium.BYTES) datatype as raw bytes instead of encoding as Base64 or HEX strings.

Config.yml needs to be updated
From offset.storage.offset.storage.jdbc.offset.table.name to offset.storage.jdbc.offset.table.name
From offset.storage.offset.storage.jdbc.offset.table.ddl to offset.storage.jdbc.offset.table.ddl

• Fix for gtid overflow integer error.

clickhouse-debezium-embedded-1.1-SNAPSHOT.jar

What's Changed

• Added logic to ignore deletes by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/282
• Fix kakfa metadata in Kafka sink connector by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/286
• Docusarus docs by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/283
• Added support for postgresql data types(including Array). by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/297
• 296 postgresql data types add test cases and support by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/298

Full Changelog: https://github.com/Altinity/clickhouse-sink-connector/compare/release_0.7.0...release_0.9.0

What's Changed

• Added logic to set lsn through REST API. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/275

Full Changelog: https://github.com/Altinity/clickhouse-sink-connector/compare/release_0.6.0...release_0.7.0

What's Changed

• Changes to support yaml file configuration in Integration tests. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/255
• Added logic to use yml file for Integration tests. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/269

Full Changelog: https://github.com/Altinity/clickhouse-sink-connector/compare/release_0.5.0...release_0.6.0

Docker: registry.gitlab.com/altinity-public/container-images/clickhouse_debezium_embedded:2023-07-26

What's Changed

• Updated README.md and postgres configuration files. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/256
• Updated standalone docker-compose files. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/264
• 259 - Fix bug where the version column might be retrieved from the wr… by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/261
• Fixed bug with retrieving metadata without database name by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/265
• Postgres fixes to support Debezium 2.3(Changes to configuration varia… by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/266
• Sink connector cli by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/260

Full Changelog: https://github.com/Altinity/clickhouse-sink-connector/compare/release_0.4.0...release_0.5.0

What's Changed

• Disable snapshot ddl by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/227
• Logic added to use replacingmergetree with the is_deleted column. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/234
• Debezium jdbc storage by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/240

Full Changelog: https://github.com/Altinity/clickhouse-sink-connector/compare/release_v5...release_0.4.0

What's Changed

• Added support for CREATE TABLE like. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/217
• 216 add support for create table like by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/222
• 216 add support for create table like by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/225

Full Changelog: https://github.com/Altinity/clickhouse-sink-connector/compare/release_v4...release_v5

What's Changed

• Remove references to clickhousetablename not used by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/72
• Useful MySQL settings by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/83
• Adding scripts to test classic databases by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/82
• python scripts to run checksums by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/80
• 78 the virtual columns should be _version and _sign by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/88
• Changed camel case functions to snake case. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/86
• 77 add sakila employees database by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/84
• Revert "77 add sakila employees database" by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/92
• Changes to sakila db. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/93
• Added documentation for Confluent Schema Registry REST API. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/91
• clickhouse_loader initial release by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/96
• Better support for dates, schema generation fixes for fractional dates and timestamps by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/97
• Sakila db fixes by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/98
• Integrate sakila employees database by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/103
• Support most data types by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/112
• Support all MySQL data types + bug fix + basic partitioning by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/117
• TestFlows Sink tests v.1 integration to develop by @AlmostIvanSidorov in https://github.com/Altinity/clickhouse-sink-connector/pull/115
• Add meaningful digits to data_types.sql by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/121
• Refactor insert batch by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/110
• 122 remove trailing zeros in timestamp and time columns by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/126
• Updating README.md by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/127
• Map Float64(Debezium) to Float32(CH) by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/131
• GitHub action push docker image by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/136
• Fixed trim of trailing '.' in mysql checksum calculation. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/138
• Added delay to configure connectors, added script to start docker con… by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/135
• Added lag metrics in Grafana dashboard. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/142
• 140 unique identifier to support mysql failover by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/143
• Added JVM metrics, updated Grafana dashboard by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/145
• Fixes for clickhouse_loader when the sql files are empty. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/153
• Grafana improvements by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/146
• Updated debezium to 2.1 Final by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/154
• Documented Sink connector configuration in README by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/156
• Integration test using testcontainer, added logic to auto create kafka offset table. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/159
• Fix version column parsing in ReplicatedReplacingMergeTree by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/165
• Added option in clickhouse checksum utility to connect to secure server by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/168
• Created Helm chart for sink connector. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/147
• Added test case for float values. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/171
• Removed out commented out statements in docker-compose.yaml by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/181
• Changes to support library by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/188
• 189 add mapping for postgresql UUID type by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/192
• testflows & github action tests by @AlmostIvanSidorov in https://github.com/Altinity/clickhouse-sink-connector/pull/124
• 194 Fixed auto creation of table when after struct is empty by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/195
• Updated instructions for mysql shell, dumping tables. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/198
• Add Mongo override docker and seed data. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/173
• 194 error auto creating table when after struct is not available by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/199
• Lightweight version merge by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/200
• Modified pom.xml in lightweight to use sink-connector as additional s… by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/201
• widget to include parts insertion by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/204
• Updated README.md and postgresql/mongo instructions. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/211
• Updated README.md and postgresql/mongo instructions. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/212

New Contributors

• @AlmostIvanSidorov made their first contribution in https://github.com/Altinity/clickhouse-sink-connector/pull/115

Full Changelog: https://github.com/Altinity/clickhouse-sink-connector/compare/release_0.2.0...release_v4

What's Changed

• Remove references to clickhousetablename not used by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/72
• Useful MySQL settings by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/83
• Adding scripts to test classic databases by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/82
• python scripts to run checksums by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/80
• 78 the virtual columns should be _version and _sign by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/88
• Changed camel case functions to snake case. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/86
• 77 add sakila employees database by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/84
• Revert "77 add sakila employees database" by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/92
• Changes to sakila db. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/93
• Added documentation for Confluent Schema Registry REST API. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/91
• clickhouse_loader initial release by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/96
• Better support for dates, schema generation fixes for fractional dates and timestamps by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/97
• Sakila db fixes by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/98
• Integrate sakila employees database by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/103
• Support most data types by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/112
• Support all MySQL data types + bug fix + basic partitioning by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/117
• TestFlows Sink tests v.1 integration to develop by @AlmostIvanSidorov in https://github.com/Altinity/clickhouse-sink-connector/pull/115
• Add meaningful digits to data_types.sql by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/121
• Refactor insert batch by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/110
• 122 remove trailing zeros in timestamp and time columns by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/126
• Updating README.md by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/127
• Map Float64(Debezium) to Float32(CH) by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/131
• Release 0_3_0 by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/128

New Contributors

• @AlmostIvanSidorov made their first contribution in https://github.com/Altinity/clickhouse-sink-connector/pull/115

Full Changelog: https://github.com/Altinity/clickhouse-sink-connector/compare/release_0.2.0...release_0.3.0

What's Changed

• Update README.md by @raphaelauv in https://github.com/Altinity/clickhouse-sink-connector/pull/32
• Removed links and replace with depends_on. Fix SINK_VERSION error. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/37
• Fix empty primary key by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/42
• 34 error loading debezium mysql connector dashboard in grafana by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/43
• Docker compose cleanup by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/45
• Document steps to connect to external MySQL server and instructions to load ontime dataset. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/49
• Auto create table: When PK is empty, set ORDER BY to ORDER BY tuple() by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/50
• Kubernetes updates - Added redpanda 3 node cluster configuration and schema registry confl… by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/52
• Fixed exception when metrics is disabled by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/57
• More changes to k8s manifest files. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/58
• Manually set binlog position before starting streaming. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/60
• 69 fix GitHub workflow to run unit tests by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/70
• Release 0.2.0 by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/71

New Contributors

• @raphaelauv made their first contribution in https://github.com/Altinity/clickhouse-sink-connector/pull/32

Full Changelog: https://github.com/Altinity/clickhouse-sink-connector/compare/release_0.1.0...release_0.2.0

Docker: https://hub.docker.com/layers/altinity/clickhouse-sink-connector/latest/images/sha256-be5a0d5b2187b5872c03e1489d1bd24d97525f9453f1bb0d629fb1bb7828a600?context=repo

• Confluent schema registry
• Auto create tables (Outstanding issues)
• JSON data type support
• Performance improvements.
• Monitoring(Grafana dashboard)

https://hub.docker.com/layers/clickhouse-sink-connector/altinity/clickhouse-sink-connector/latest/images/sha256-51d24adeeb3be6d6678f06fd9b545c8cd9ecfaf2788b50fcc5bdd4a02438b6e2?context=explore

What's Changed

• Added initial commit of files from the original repo by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/1
• Added logic to use Clickhouse JDBC connector, by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/2
• Added logic to create a new thread to handle flushing of records to C… by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/3
• Fix unix permissions by @aadant in https://github.com/Altinity/clickhouse-sink-connector/pull/10
• Refactor logic to buffer records and flush. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/14
• Create sysbench script combine tests by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/16
• Integration tests check count by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/17
• Added steps to create grafana docker image for debezium connector and… by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/19
• Map INT32 Kafka connect to INT256 CH type by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/23
• Confluent schema registry by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/24
• Revert 23 15 mysql bigint to ch int64 doesnt work correct with autocreatetables true setting by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/25
• Fixed grafana dashboard auto loading. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/26
• Support for JSON data type. by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/28
• Fix auto create table retry by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/29
• Add max.batch.size and max.queue.size to debezium configuration to im… by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/31
• Release 1.0 by @subkanthi in https://github.com/Altinity/clickhouse-sink-connector/pull/18

New Contributors

• @subkanthi made their first contribution in https://github.com/Altinity/clickhouse-sink-connector/pull/1
• @aadant made their first contribution in https://github.com/Altinity/clickhouse-sink-connector/pull/10

Full Changelog: https://github.com/Altinity/clickhouse-sink-connector/commits/release_0.1.0