What's Changed

CI/CD Improvements

• Adding CodeQL Workflow for Java by @mukundnigam in https://github.com/Azure/azure-functions-kafka-extension/pull/480
• Weekly Sunday build for the CI by @vivekjilla in https://github.com/Azure/azure-functions-kafka-extension/pull/483
• changing cron to run at weekly schedule by @mukundnigam in https://github.com/Azure/azure-functions-kafka-extension/pull/485
• Adding code mirror pipeline yaml by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/488
• Adding Public and Official build pipelines by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/500
• Installing .NET8 sdk for Official Build Pipeline by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/512

Extension Improvements

• Upgrading package versions for E2E tests to .net8 by @vivekjilla in https://github.com/Azure/azure-functions-kafka-extension/pull/499
• Upgrading confluent package to 2.4.0 by @vivekjilla in https://github.com/Azure/azure-functions-kafka-extension/pull/504
• Fixed inconsistency between consumer and producer config by @gvdwiele in https://github.com/Azure/azure-functions-kafka-extension/pull/464
• Added OAuthBearer authentication mode by @nakah in https://github.com/Azure/azure-functions-kafka-extension/pull/437
• Updating extension version to 4.0.0 by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/510
• Resolving Oauthbearer Authentication Attributes from Settings for Output Binding by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/511

Bug Fixes

• Fix scale monitor descriptor function id by @pierre-sk in https://github.com/Azure/azure-functions-kafka-extension/pull/451
• Fixing Msg commit for Cancellation Scenarios by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/505
• Allowing schema registry creation for out of proc data types by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/506
• Schema registry Url and Config Count Fix by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/507

New Contributors

• @pierre-sk made their first contribution in https://github.com/Azure/azure-functions-kafka-extension/pull/451
• @gvdwiele made their first contribution in https://github.com/Azure/azure-functions-kafka-extension/pull/464
• @nakah made their first contribution in https://github.com/Azure/azure-functions-kafka-extension/pull/437

This pre-release includes Header support by the contribution of @gliljas Thank you for the great contribution.

• Add Header Support https://github.com/Azure/azure-functions-kafka-extension/pull/170
• Add Language E2E Testing https://github.com/Azure/azure-functions-kafka-extension/pull/199

Pre-Release for testing the new Confluent.Kafka 1.6.0-PRE4 https://github.com/Azure/azure-functions-kafka-extension/issues/193

The version looks solve the Timeout issue that is reported on https://github.com/edenhill/librdkafka/issues/3109

Update the Confluent.Kafka to the v1.5.2. It includes librdkafka v1.5.2. On this release, we can avoiding limitation for Azure about the long-lived TCP connection https://github.com/Azure/azure-functions-kafka-extension/issues/187

• Upgrade Confluent.Kafka Library to 1.5.2 PR https://github.com/Azure/azure-functions-kafka-extension/pull/186 Fix https://github.com/Azure/azure-functions-kafka-extension/issues/185 https://github.com/Azure/azure-functions-kafka-extension/issues/187
• Enable librdkafka message route to the Azure Functions Logger PR https://github.com/Azure/azure-functions-kafka-extension/pull/186
• Bump Junit From 4.1.2 to 4.13.1 https://github.com/Azure/azure-functions-kafka-extension/pull/184 https://github.com/Azure/azure-functions-kafka-extension/pull/182
• Remove Python/Java library. It is available on azure functions library now. https://github.com/Azure/azure-functions-kafka-extension/pull/180 https://github.com/Azure/azure-functions-kafka-extension/pull/174
• Update C# extension name Thank you @JoeRobich https://github.com/Azure/azure-functions-kafka-extension/pull/178

New Configuration

You can configure these additional configuration on your host.json

Breaking Change

We introduce the breaking change on configuration. It is to avoid nasty and hard-to-debug issue on Azure platform. For more details, refer to this issue. However, we recommend this new default value since it is also support the default configuration of EventHubs. For more details, refer to librdkafka configuration properties.

Do you want to avoid breaking change?

You can add this configuration on your host.json. For the original default value of librdkafka, refer to this document.

{
  "version": "2.0",
  "extensions": {
    "kafka": {
         "MetadataMaxAgeMs": 900000,
         "SocketKeepaliveEnable": false
    }
  }
}

This version is not available.

Notes

Update the Confluent.Kafka to the v1.4.3. It includes v1.1.0 Notable improvement for Windows SSL users: You no longer need to specify a CA certificate file/directory (SslCaLocation) - librdkafka will load the CA certs by default from the Windows Root Certificate Store.

• Support for EventHub Kafka API with Documentation (triggers, input & output bindings) PR: https://github.com/Azure/azure-functions-kafka-extension/pull/165
• Upgrade Confluent.Kafka Library to 1.4.3 https://github.com/Azure/azure-functions-kafka-extension/pull/151 Fix: https://github.com/Azure/azure-functions-kafka-extension/issues/134 https://github.com/Azure/azure-functions-kafka-extension/issues/159 https://github.com/Azure/azure-functions-kafka-extension/issues/157
• Fix KafkaOutput binding 'topic' attribute doesn't support environment variables PR: https://github.com/Azure/azure-functions-kafka-extension/pull/166
• Support read "ConsumerGroup" from settings PR: https://github.com/Azure/azure-functions-kafka-extension/pull/153
• Remove the PackageLicenseUrl for deprecation
• Bump lodash from 4.17.15 to 4.17.19 in /samples/javascript

Reference

• confluent-kafka-dotnet Release v1.4.3

Contribution

Thank you @ThomasManson for the great contribution!

• Fixed issue 157 where attribute binding would not load files correctly

NOTE

CA Certificate issue happens when we use it with intermediate certificate store. For more details refer to https://github.com/Azure/azure-functions-kafka-extension/issues/173

NOTES

The first GA release. No Update for the extension of .NET. The library version will be updated at 3.1.0. We are going to provide 3.1.0-rc for testing the new library version for the next release. This release provides:

• GA
• Update Samples with upgrading .NET Core 3.0 with supporting DevContainer (https://github.com/Azure/azure-functions-kafka-extension/pull/138 ) (https://github.com/Azure/azure-functions-kafka-extension/pull/143) (https://github.com/Azure/azure-functions-kafka-extension/pull/144)
• Update Document for each languages (https://github.com/Azure/azure-functions-kafka-extension/pull/140) (https://github.com/Azure/azure-functions-kafka-extension/pull/145)
• Update Python Li brary (https://github.com/Azure/azure-functions-kafka-extension/pull/139)
• Update Java and Python library to support Cardinality (https://github.com/Azure/azure-functions-kafka-extension/pull/137)

Related PR for the other Repo

• Include the Java and Python library into the official library (https://github.com/Azure/azure-functions-java-library/pull/116) (https://github.com/microsoft/azure-maven-plugins/pull/1092) (https://github.com/Azure/azure-functions-python-library/pull/62)

NEW FEATURE

• PowerShell Sample. Thank you @EwertonJordao ( https://github.com/Azure/azure-functions-kafka-extension/pull/135 )

• Supporting Runtime Scale Monitoring that will leverage auto scaling for premium plans.
• Enhancing consumer lag metrics by committing single dispatch after each function call.
• Improved support to Confluent Cloud in Azure.
• Suppressed Libradkafka load error log message in development
• Added JavaScript sample.

• Fixed issue preventing using extension in Azure (librdkafka.dll could not be loaded)
• Fixed issue preventing using EventHubs with Kafka header (cacert.pem file was missing)

This is a prerelease version of Microsoft.Azure.WebJobs.Extensions.Kafka.