What's Changed

CI/CD Improvements

• Adding CodeQL Workflow for Java by @mukundnigam in https://github.com/Azure/azure-functions-kafka-extension/pull/480
• Weekly Sunday build for the CI by @vivekjilla in https://github.com/Azure/azure-functions-kafka-extension/pull/483
• changing cron to run at weekly schedule by @mukundnigam in https://github.com/Azure/azure-functions-kafka-extension/pull/485
• Adding code mirror pipeline yaml by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/488
• Adding Public and Official build pipelines by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/500
• Installing .NET8 sdk for Official Build Pipeline by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/512

Extension Improvements

• Upgrading package versions for E2E tests to .net8 by @vivekjilla in https://github.com/Azure/azure-functions-kafka-extension/pull/499
• Upgrading confluent package to 2.4.0 by @vivekjilla in https://github.com/Azure/azure-functions-kafka-extension/pull/504
• Fixed inconsistency between consumer and producer config by @gvdwiele in https://github.com/Azure/azure-functions-kafka-extension/pull/464
• Added OAuthBearer authentication mode by @nakah in https://github.com/Azure/azure-functions-kafka-extension/pull/437
• Updating extension version to 4.0.0 by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/510
• Resolving Oauthbearer Authentication Attributes from Settings for Output Binding by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/511

Bug Fixes

• Fix scale monitor descriptor function id by @pierre-sk in https://github.com/Azure/azure-functions-kafka-extension/pull/451
• Fixing Msg commit for Cancellation Scenarios by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/505
• Allowing schema registry creation for out of proc data types by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/506
• Schema registry Url and Config Count Fix by @jainharsh98 in https://github.com/Azure/azure-functions-kafka-extension/pull/507

New Contributors

• @pierre-sk made their first contribution in https://github.com/Azure/azure-functions-kafka-extension/pull/451
• @gvdwiele made their first contribution in https://github.com/Azure/azure-functions-kafka-extension/pull/464
• @nakah made their first contribution in https://github.com/Azure/azure-functions-kafka-extension/pull/437

What's Changed

• Update Version from 3.8.0 to 3.9.0 (#450) @jainharsh98
• FabricBot: Onboarding to GitOps.ResourceManagement because of FabricBot decommissioning (#445) @microsoft-github-policy-service
• Adding Unit Tests for SSL Attributes from Settings Scenario (#448) @jainharsh98
• Session timeout default value change. (#443) @aloiva
• #358 Resolve KafkaAttribute SSL values from settings (#430) @Cameron-Goss
• Re-Enabling Confluent Tests and Adding Flags to disable Lang E2E tests from Pipeline (#449) @jainharsh98
• Support for Target based scaling (#438) @aloiva
• Improving Retry Test for Webjobs Version Upgrade (#446) @jainharsh98
• Add support for external Avro Schema Registry (#411) @florianluediger
• Fixing pipeline error (#444) @krishna-kariya

☕Java

• Bump json from 20211205 to 20230227 in /test/Microsoft.Azure.WebJobs.Extensions.Kafka.LangEndToEndTests/FunctionApps/java/Confluent (#432) @dependabot
• Bump json from 20211205 to 20230227 in /test/Microsoft.Azure.WebJobs.Extensions.Kafka.LangEndToEndTests/FunctionApps/java/EventHub (#433) @dependabot

🔗Dependency Updates

• Bump minimatch from 3.0.4 to 3.1.2 in /samples/typescript (#412) @dependabot
• Bump minimatch from 3.0.4 to 3.1.2 in /samples/javascript (#413) @dependabot
• Bump json from 20211205 to 20230227 in /test/Microsoft.Azure.WebJobs.Extensions.Kafka.LangEndToEndTests/FunctionApps/java/Confluent (#432) @dependabot
• Bump Azure.Storage.Queues from 12.9.0 to 12.11.0 in /test/Microsoft.Azure.WebJobs.Extensions.Kafka.LangEndToEndTests (#392) @dependabot
• Bump json from 20211205 to 20230227 in /test/Microsoft.Azure.WebJobs.Extensions.Kafka.LangEndToEndTests/FunctionApps/java/EventHub (#433) @dependabot

What's Changed

• Update to version 3.8.0 (#415) @github-actions
• Adding Dotnet Samples for Distributed Tracing (#414) @krishna-kariya
• Distributed Tracing for trigger (#396) @krishna-kariya

What's Changed

• Update to verison 3.7.0 (#410) @github-actions
• Adding support for Linger.ms (#407) @shrohilla
• Boosting the producer perf for output binding (#403) @shrohilla
• BUG: Remove nullability from attribute properties (#362) @gliljas

What's Changed

• Update to verison 3.6.0 (#399) @github-actions
• Fix for E2E pipeline (#398) @krishna-kariya
• Upgrade Confluent package to version 1.9.0 (#357) @krishna-kariya
• Python and Java Tests with E2E testing Framework (#354) @jainharsh98
• Bug Fix for the file location error (#391) @krishna-kariya
• Bug Fix for fail to load certificate location for KafkaTrigger (#386) @krishna-kariya
• adding the sample for accessing key in java function with avro (#381) @shrohilla
• adding the Error & Retries in doc (#378) @shrohilla

☕Java

• Bump gson from 2.2.4 to 2.8.9 in /samples/WalletProcessing_KafkademoSample (#352) @dependabot

🔗Dependency Updates

• Bump gson from 2.2.4 to 2.8.9 in /samples/WalletProcessing_KafkademoSample (#352) @dependabot
• Bump shell-quote from 1.7.2 to 1.7.3 in /samples/typescript (#373) @dependabot

What's Changed

• Update to verison 3.5.0 (#379) @github-actions
• Adding the support for key for trigger & output binding (#328) @shrohilla
• Fixing issues related to Application Insights Logging (#377) @krishna-kariya
• Migrate FabricBot Tasks to Config-as-Code (#359) @msftbot
• Updates in the samples (#376) @shrohilla
• Adding the samples support for retry (#375) @shrohilla
• Addition of retry support in Kafka extension (#363) @shrohilla
• Adding Microsoft SECURITY.MD (#350) @microsoft-github-policy-service
• Main (#374) @shrohilla
• Update to verison 3.4.1-PRE (#372) @github-actions
• Mykafkabranch (#368) @raorugan
• Revert "fixed yaml values" (#367) @shrohilla
• Exposing AutoOffsetReset property in KafkaOptions (#364) @jainharsh98
• fixed yaml values (#355) @raorugan
• Modified read me file for blog sample (#353) @raorugan
• adding the blog in samples (#349) @shrohilla
• Added Header samples for dotnet isolated runtime (#345) @krishna-kariya
• Avrosamples for Generic Records Typecript (#347) @shrohilla
• Shrohilla/avrosamples (#346) @shrohilla
• Correcting Public docs to indicate 3.x function support (#344) @jainharsh98
• Correcting the Samples (#343) @krishna-kariya

What's Changed

• Update to verison 3.4 (#342) @github-actions
• Add Strong name signing (https://github.com/Azure/azure-functions-kafka-extension/pull/339) @shrohilla
• Shrohilla/codesignin (https://github.com/Azure/azure-functions-kafka-extension/pull/330) @shrohilla
• Linked Trigger documentation in output binding (https://github.com/Azure/azure-functions-kafka-extension/pull/334) @jainharsh98
• Shrohilla/public doc (https://github.com/Azure/azure-functions-kafka-extension/pull/333) @shrohilla
• Adding Trigger documentation and Modifying Output documentation (https://github.com/Azure/azure-functions-kafka-extension/pull/325) @jainharsh98
• Adding headers example for documentation (https://github.com/Azure/azure-functions-kafka-extension/pull/331) @krishna-kariya
• Added Examples Section in Trigger and Output (https://github.com/Azure/azure-functions-kafka-extension/pull/329) @krishna-kariya
• Adding samples for documentation (https://github.com/Azure/azure-functions-kafka-extension/pull/318) @krishna-kariya
• Adding the support for Kafka headers output & input @shrohilla
• Adding the support for Avro generic for all language workers @shrohilla
• fixing the Little Endian issue (https://github.com/Azure/azure-functions-kafka-extension/pull/315) @shrohilla
• Shrohilla/kafka header bindings (https://github.com/Azure/azure-functions-kafka-extension/pull/308) @shrohilla
• Modifying function.json in samples to use local.settings.json (#276) @jainharsh98
• Shrohilla/kafka header bindings (#281) @shrohilla
• Updating Broker variable in Readme (#279) @krishna-kariya
• Removing Confluent Cloud Credentials from README.md (#278) @krishna-kariya
• adding the binding for supporting headers (#277) @shrohilla
• adding sbom stage in pipeline (#270) @shrohilla
• Change the log level since it could happens when we use extension bundles (#248) @TsuyoshiUshio

⚡Automation

• Upgrade GitHubActions (#235) @TsuyoshiUshio

📔Documentation

• Update node sample (#240) @TsuyoshiUshio

🔗Dependency Updates

What's Changed

• Update to verison 3.3.3.1 (#341) @github-actions
• Action/release.3.3.3 test (#340) @shrohilla
• Add Strong name signing (#339) @shrohilla
• Update to verison 3.3.3 (#338) @github-actions
• changing the release version (#337) @shrohilla
• Shrohilla/codesignin (#330) @shrohilla
• Linked Trigger documentation in output binding (#334) @jainharsh98
• Shrohilla/public doc (#333) @shrohilla
• Adding Trigger documentation and Modifying Output documentation (#325) @jainharsh98
• Adding headers example for documentation (#331) @krishna-kariya
• Added Examples Section in Trigger and Output (#329) @krishna-kariya
• Adding samples for documentation (#318) @krishna-kariya
• Adding the support for Kafka headers output & input
• Adding the support for Avro generic for all language workers

🔗Dependency Updates

• Bump minimist from 1.2.5 to 1.2.6 in /samples/javascript (#306) @dependabot
• Bump ansi-regex from 4.1.0 to 4.1.1 in /samples/javascript (#327) @dependabot
• Bump ajv from 6.12.0 to 6.12.6 in /samples/javascript (#287) @dependabot

What's Changed

• Update to verison 3.3.3 (#316) @github-actions
• fixing the Little Endian issue (#315) @shrohilla
• Shrohilla/kafka header bindings (#308) @shrohilla
• Addition to the support for 3.3.3-PRE

What's Changed

• Update to verison 3.3.3-PRE (#289) @github-actions
• Modifying function.json in samples to use local.settings.json (#276) @jainharsh98
• Revert "Update to verison 3.3.3-PRE" (#285) @shrohilla
• Update to verison 3.3.3-PRE (#284) @github-actions
• Shrohilla/kafka header bindings (#281) @shrohilla
• Updating Broker variable in Readme (#279) @krishna-kariya
• Removing Confluent Cloud Credentials from README.md (#278) @krishna-kariya
• adding the binding for supporting headers (#277) @shrohilla
• Use 1ES Hosted pool for builds (#271) @JatinSanghvi
• adding sbom stage in pipeline (#270) @shrohilla
• Change the log level since it could happens when we use extension bundles (#248) @TsuyoshiUshio

⚡Automation

• Upgrade GitHubActions (#235) @TsuyoshiUshio

📔Documentation

• Update node sample (#240) @TsuyoshiUshio

🔗Dependency Updates

• Bump path-parse from 1.0.6 to 1.0.7 in /samples/typescript/kafka-trigger (#253) @dependabot
• Bump glob-parent from 5.1.1 to 5.1.2 in /samples/javascript (#246) @dependabot
• Bump hosted-git-info from 2.8.8 to 2.8.9 in /samples/typescript/kafka-trigger (#242) @dependabot
• Bump lodash from 4.17.19 to 4.17.21 in /samples/javascript (#241) @dependabot

What's Changed

This release for fixing the issue of NuGet publishing. The App is the same as 3.3.1.

• Update to verison 3.3.2 (#234) @github-actions
• Update copyright for avoiding bom issue (#233) @TsuyoshiUshio
• fix encoding issue (#231) @TsuyoshiUshio

What's Changed

Thank you for your contribution @lilleman12 and @tomkerkhove !
This release also include Upgrade to Confluent.Kafka 1.6.3

• Update to verison 3.3.1 (#229) @github-actions
• Remove version prefix for release pipeline(GitHub Action) (#230) @TsuyoshiUshio
• Adding version up script (#228) @TsuyoshiUshio
• Enable setting LagThreshold via the kafkaAttribute (#198) @lilleman12
• Fix typo in .NET sample (#205) @tomkerkhove

This pre-release includes Header support by the contribution of @gliljas Thank you for the great contribution.

• Add Header Support https://github.com/Azure/azure-functions-kafka-extension/pull/170
• Add Language E2E Testing https://github.com/Azure/azure-functions-kafka-extension/pull/199

Pre-Release for testing the new Confluent.Kafka 1.6.0-PRE4 https://github.com/Azure/azure-functions-kafka-extension/issues/193

The version looks solve the Timeout issue that is reported on https://github.com/edenhill/librdkafka/issues/3109

Update the Confluent.Kafka to the v1.5.2. It includes librdkafka v1.5.2. On this release, we can avoiding limitation for Azure about the long-lived TCP connection https://github.com/Azure/azure-functions-kafka-extension/issues/187

• Upgrade Confluent.Kafka Library to 1.5.2 PR https://github.com/Azure/azure-functions-kafka-extension/pull/186 Fix https://github.com/Azure/azure-functions-kafka-extension/issues/185 https://github.com/Azure/azure-functions-kafka-extension/issues/187
• Enable librdkafka message route to the Azure Functions Logger PR https://github.com/Azure/azure-functions-kafka-extension/pull/186
• Bump Junit From 4.1.2 to 4.13.1 https://github.com/Azure/azure-functions-kafka-extension/pull/184 https://github.com/Azure/azure-functions-kafka-extension/pull/182
• Remove Python/Java library. It is available on azure functions library now. https://github.com/Azure/azure-functions-kafka-extension/pull/180 https://github.com/Azure/azure-functions-kafka-extension/pull/174
• Update C# extension name Thank you @JoeRobich https://github.com/Azure/azure-functions-kafka-extension/pull/178

New Configuration

You can configure these additional configuration on your host.json

Breaking Change

We introduce the breaking change on configuration. It is to avoid nasty and hard-to-debug issue on Azure platform. For more details, refer to this issue. However, we recommend this new default value since it is also support the default configuration of EventHubs. For more details, refer to librdkafka configuration properties.

Do you want to avoid breaking change?

You can add this configuration on your host.json. For the original default value of librdkafka, refer to this document.

{
  "version": "2.0",
  "extensions": {
    "kafka": {
         "MetadataMaxAgeMs": 900000,
         "SocketKeepaliveEnable": false
    }
  }
}

This version is not available.

Notes

Update the Confluent.Kafka to the v1.4.3. It includes v1.1.0 Notable improvement for Windows SSL users: You no longer need to specify a CA certificate file/directory (SslCaLocation) - librdkafka will load the CA certs by default from the Windows Root Certificate Store.

• Support for EventHub Kafka API with Documentation (triggers, input & output bindings) PR: https://github.com/Azure/azure-functions-kafka-extension/pull/165
• Upgrade Confluent.Kafka Library to 1.4.3 https://github.com/Azure/azure-functions-kafka-extension/pull/151 Fix: https://github.com/Azure/azure-functions-kafka-extension/issues/134 https://github.com/Azure/azure-functions-kafka-extension/issues/159 https://github.com/Azure/azure-functions-kafka-extension/issues/157
• Fix KafkaOutput binding 'topic' attribute doesn't support environment variables PR: https://github.com/Azure/azure-functions-kafka-extension/pull/166
• Support read "ConsumerGroup" from settings PR: https://github.com/Azure/azure-functions-kafka-extension/pull/153
• Remove the PackageLicenseUrl for deprecation
• Bump lodash from 4.17.15 to 4.17.19 in /samples/javascript

Reference

• confluent-kafka-dotnet Release v1.4.3

Contribution

Thank you @ThomasManson for the great contribution!

• Fixed issue 157 where attribute binding would not load files correctly

NOTE

CA Certificate issue happens when we use it with intermediate certificate store. For more details refer to https://github.com/Azure/azure-functions-kafka-extension/issues/173

NOTES

The first GA release. No Update for the extension of .NET. The library version will be updated at 3.1.0. We are going to provide 3.1.0-rc for testing the new library version for the next release. This release provides:

• GA
• Update Samples with upgrading .NET Core 3.0 with supporting DevContainer (https://github.com/Azure/azure-functions-kafka-extension/pull/138 ) (https://github.com/Azure/azure-functions-kafka-extension/pull/143) (https://github.com/Azure/azure-functions-kafka-extension/pull/144)
• Update Document for each languages (https://github.com/Azure/azure-functions-kafka-extension/pull/140) (https://github.com/Azure/azure-functions-kafka-extension/pull/145)
• Update Python Li brary (https://github.com/Azure/azure-functions-kafka-extension/pull/139)
• Update Java and Python library to support Cardinality (https://github.com/Azure/azure-functions-kafka-extension/pull/137)

Related PR for the other Repo

• Include the Java and Python library into the official library (https://github.com/Azure/azure-functions-java-library/pull/116) (https://github.com/microsoft/azure-maven-plugins/pull/1092) (https://github.com/Azure/azure-functions-python-library/pull/62)

NEW FEATURE

• PowerShell Sample. Thank you @EwertonJordao ( https://github.com/Azure/azure-functions-kafka-extension/pull/135 )

• Supporting Runtime Scale Monitoring that will leverage auto scaling for premium plans.
• Enhancing consumer lag metrics by committing single dispatch after each function call.
• Improved support to Confluent Cloud in Azure.
• Suppressed Libradkafka load error log message in development
• Added JavaScript sample.

• Fixed issue preventing using extension in Azure (librdkafka.dll could not be loaded)
• Fixed issue preventing using EventHubs with Kafka header (cacert.pem file was missing)