A collection of open source Apache 2.0 Kafka Connector maintained by Lenses.io.
APACHE-2.0 License
Bot releases are visible (Hide)
Published by stheppi about 1 year ago
Fix: S3 sink when using envelope did not handle messages with null Value correctly
Published by github-actions[bot] about 1 year ago
Published by github-actions[bot] about 1 year ago
The source and sink has been the focus of this release.
PROPERTIES('store.envelope'=true)
STOREAS BYTES
PROPERTIES
to drive new settings required to drive the connectors' behaviour. The KCQL looks INSERT INTO ... SELECT ... FROM ... PROPERTIES(property=key, ...)
PARTITIONBY a, `field1.field2`
for enhanced partitioning control.For installations that have been using the preview version of the S3 connector and are upgrading to the release, there
are a few important considerations:
Previously, default padding was enabled for both "offset" and "partition" values starting in June.
However, in version 5.0, the decision to apply default padding to the "offset" value only, leaving the "
partition" value without padding. This change was made to enhance compatibility with querying in Athena.
If you have been using a build from the master branch since June, your connectors might have been configured with a
different default padding setting.
To maintain consistency and ensure your existing connector configuration remains valid, you will need to use KCQL
configuration properties to customize the padding fields accordingly.
INSERT INTO $bucket[:$prefix]
SELECT *
FROM $topic
...
PROPERTIES(
'padding.length.offset'=12,
'padding.length.partition'=12
)
Starting with version 5.0.0, the following configuration keys have been replaced.
Field | Old Property | New Property |
---|---|---|
AWS Secret Key | aws.secret.key | connect.s3.aws.secret.key |
Access Key | aws.access.key | connect.s3.aws.access.key |
Auth Mode | aws.auth.mode | connect.s3.aws.auth.mode |
Custom Endpoint | aws.custom.endpoint | connect.s3.custom.endpoint |
VHost Bucket | aws.vhost.bucket | connect.s3.vhost.bucket |
In version 4.1, padding options were available but were not enabled by default. At that time, the default padding
length, if not specified, was set to 8 characters.
However, starting from version 5.0, padding is now enabled by default, and the default padding length has been increased
to 12 characters.
Enabling padding has a notable advantage: it ensures that the files written are fully compatible with the Lenses Stream
Reactor S3 Source, enhancing interoperability and data integration.
Sinks created with 4.2.0 and 4.2.1 should retain the padding behaviour, and, therefore should disable padding:
INSERT INTO $bucket[:$prefix]
SELECT *
FROM $topic
...
PROPERTIES (
'padding.type'=NoOp
)
If padding was enabled in 4.1, then the padding length should be specified in the KCQL statement:
INSERT INTO $bucket[:$prefix]
SELECT *
FROM $topic
...
PROPERTIES (
'padding.length.offset'=12,
'padding.length.partition'=12
)
STOREAS Bytes_***
is usedThe Bytes_*** storage format has been removed. If you are using this storage format, you will need to install the
5.0.0-deprecated connector and upgrade the connector instances by changing the class name:
Source Before:
class.name=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector
...
Source After:
class.name=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnectorDeprecated
...
Sink Before:
class.name=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector
...
Sink After:
class.name=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnectorDeprecated
connect.s3.padding.strategy=NoOp
...
The deprecated connector won't be developed any further and will be removed in a future release.
If you want to talk to us about a migration plan, please get in touch with us at [email protected].
To migrate to the new configuration, please follow the following steps:
Published by github-actions[bot] over 1 year ago
Published by github-actions[bot] over 1 year ago
Published by github-actions[bot] about 2 years ago
Published by lanbot almost 3 years ago
3.0.1
Published by lanbot almost 3 years ago
3.0.0
Published by lanbot almost 4 years ago
2.1.3
Published by lanbot almost 4 years ago
2.1.2
Published by lanbot about 4 years ago
2.1.1
Published by lanbot about 4 years ago
2.1.0
Published by lanbot over 4 years ago
2.0.0
Published by lanbot over 4 years ago
1.2.7
Published by lanbot over 4 years ago
1.2.6
Published by lanbot over 4 years ago
1.2.5
Published by lanbot almost 5 years ago
Bug fixes
JMS Source
Ack the JMS messages was not always possible. Also there was an issue with producing the messages to Kafka out of order from the JMS queue.
Changes:
Published by lanbot about 5 years ago
Published by lanbot over 5 years ago
GEOADD
supportPUB/SUB
supportPublished by lanbot over 5 years ago
1.2.1