avrora

In version 0.28 two new configuration options were added: registry_ssl_cacerts and registry_ssl_cacert_path both used for controlling SSL verification of the Schema Registry connection.

If both options are set, the registry_ssl_cacerts takes precedence over registry_ssl_cacert_path, also registry_ssl_cacerts is a DER-encoded certificate, when registry_ssl_cacert_path is a PEM-encoded certificate file.

As always feel free to share your feedback or issues and happy coding 🤗

To reduce noise about deprecation of Logger.warn/1 and refresh the CI setup together with local dependencies we have new this release.

Enjoy! 🎉

In this release, @emilianobovetti fixed an issue with custom configurations for the registration mix task. If you want to use your special config, let's say config/runtime.exs, now you can specify it via --appconfig option, like that

mix avrora.reg.schema --name io.confluent.Payment --appconfig runtime
schema `io.confluent.Payment' will be registered

it will be loaded in addition to the default one (if it's present)

Enjoy! Thanks to @emilianobovetti 💙

Starting this release each request to the Schema Registry will have an additional header User-Agent. It's done in response to issue #101 when the AWS setup rejects the request.

The default value will contain the current version of the library, its name, and the language, for example: Avrora/0.25.0 Elixir

If you want the previous behaviour, i.e no User-Agent header or your custom header value, then you can adjust it via the registry_user_agent configuration option by setting it to nil or your String respectively.

defmodule MyClient
  use Avrora.Client, registry_user_agent: "Hello world" # you can set it to whatever you like
end

defmodule MyClientWithoutUserAgent
  use Avrora.Client, registry_user_agent: nil # or you can disable User-Agent header
end

Thanks to @azeemchauhan for bringing it up 💟

It turns out that Avrora.Config.registry_schemas_autoreg/0 configuration option was not returning false values if the library is configured and a private client is not used. And the only way to set it to false was to create your client and configure it like

defmodule MyClient do
  use Avrora.Client, registry_schemas_autoreg: false
end

Thanks to @ankhers it's fixed now! Enjoy folks! 🚢

In this release, we explicitly set the SSL option verify to be verify_none to reflect the changed default in OTP25. To avoid the false sense of security OTP25 has changed the default value for SSL verify to be verify_peer which requires you to source it certs.

We are going to add secure option in the next release, but for now let's keep it as it was 🤷🏼

Thanks to @goozzik for pointing this out!

Initially, the decoder hook was done as a non-configurable no-op function due to clarity – how much it will be used and will be at all?. Seems the time has come, thanks to the @LostKobrakai 💜

The very first use-case is tagging unions, check this out:

defmodule MyClient do
  use Avrora.Client,
    decoder_hook: &__MODULE__.hook/4

  def hook(type, sub_name_or_index, data, decode_fun) do
    tag_unions_hook = :avro_decoder_hooks.tag_unions()
    tag_unions_hook.(type, sub_name_or_index, data, decode_fun)
  end
end

Now all the complex unions will be tagged and result will be passed to the "private" Avrora decoder hook. But there is no limit on what you can do with it, just don't forget to call at the end decode_fun.(data) 😉

Have fun 🐟 🪝

In this release, all the private client configuration options become dynamic and may be resolved at runtime or at compile-time. It's defined by the presence of the otp_app option.

If you set the otp_app configuration the value lookup will follow this logic

OTP env value → Client value → Default value

Let's take a look at this example with private client MyClient and :my_app OTP application

# config/config.exs
config :my_app, MyClient, registry_url: "http://my-app.io"

# lib/my_client.ex
defmodule MyClient do
  use Avrora.Client,
    otp_app: :my_app,
    registry_url: "http://never-used.com",
    names_cache_ttl: 42
end

As the result, configuration values will look like this

MyClient.Config.registry_url() == "http://my-app.io" # (comes from runtime environment)
MyClient.Config.names_cache_ttl() == 42 # (comes from compile-time options)
MyClient.Config.registry_schemas_autoreg() == true # (comes from compile-time option defaults)

🚤 Bonus

Most of the generated Avrora.Client code for Config module gets rid of strings interpolation on runtime and moves it to the compile-rime and some hot paths were moved to compile-time.

Thanks a lot for this release to @juanperi 💙

In this release, few changes were done to the Avrora.Config.

The main change is about releases and schemas_path configuration. Especially if you have multiple apps in Umbrella and would like to have per-app schemas_path, now you can pass an otp_app option which should point to your OPT application which will be used to compute the root path for the schemas_path in a runtime.

# Private client
defmodule MyClient do
  use Avrora.Client,
    otp_app: :my_application,
    schemas_path: "./some/path/to/schemas"
end

# Shared client
config :avrora,
  otp_app: :my_application,
  schemas_path: "./some/path/to/schemas"

And as a side-effect, now all the schemas_path which is without otp_app set, will be resolved with Path.expand

Many thanks to @LostKobrakai for support 💚

Unfortunately, mistakes happen. A copy-paste bug sneaks into Avrora.Client module and breaks authentication. Thanks to @raphaklaus who spots and fixes the issue 💙

Happy coding 😸

This release getting Avrora closer to the AVRO specification. Now it will be possible to encode and decode all 3 named types Enum, Fixed and already working one –Record.

It's simple as it was and requires no hassle. Here is a low-level example with some internal changes 😉

json = ~s({"namespace":"io.confluent","name":"CardType","type":"enum","symbols":["MASTERCARD","VISA","AMERICANEXPRESS"]})
{:ok, schema} = Avrora.Schema.Encoder.from_json(json)
{:ok, encoded} = Avrora.Codec.Plain.encode("VISA", schema: schema)

encoded # => <<2>>

In this release, all warnings of the dialyzer about return types or dead branches of the code should vanish.

Dead code branches

It turns out that the private Avrora.Config modules were too optimized which caused the dialyzer to report dead branches inside the code which was using if/else conditions with that config. The solution was to make it a bit more dynamic, but with a constant access time.

Return type mismatch

Another problem was the private Avrora.Schema return type, which was mismatching for every private client. So now, all private clients will use the only one schema type Avrora.Schema, all the functionality was removed from this module and it became just a struct.

Kudos to @LostKobrakai for raising the issue 💙

Starting this version a new command-line argument --module MODULE added to the schema registration mix task. It allows you to use your private Avrora client modules to be used for schema registration.

Example

$ mix avrora.reg.schema --name io.confluent.Payment --as MyCustomName --module MyClient

Enjoy ✌️

Unfortunately, in release 0.18.0 two new methods were forgotten for auto-generated private clients. It's Avrora.encode_plain/2 and Avrora.decode_plain/2.

Luckily @LostKobrakai spot the issue ❤️ and fixed it.

Happy encoding, everyone 🥚

In this release, a potential collision between 2 formats was removed. In several cases (see #70) messages encoded with format: :plain could match a format: :registry magic byte sequence. It will lead to an error in decoding with format: :guess.

From now on it is recommended to use special crafted Avrora.encode_plain/2 and Avrora.decode_plain/2 while working with plain encoded Avro messages, thanks for the effort to @mw23 💙

{:ok, pid} = Avrora.start_link()

message = <<8, 116, 120, 45, 49, 123, 20, 174, 71, 225, 250, 47, 64>>
{:ok, decoded} = Avrora.decode_plain(message, schema_name: "io.confluent.Payment")
%{"id" => "tx-1", "amount" => 15.99}

message = %{"id" => "tx-1", "amount" => 15.99}
{:ok, encoded} = Avrora.encode_plain(message, schema_name: "io.confluent.Payment")
<<8, 116, 120, 45, 49, 123, 20, 174, 71, 225, 250, 47, 64>>

In addition, the project readme was clean up and adopted to be transformed as close as possible by ExDoc for hexdocs.pm. And as a bonus compilation should not throw any warnings about docs or specs, thanks for bringing it up to @fxn 💚

In this release, Avrora finally gets multi-client support. From now on you can use a shared client Avrora or make as many you need private clients which will 100% equivalent to the Avrora one.

Private clients are totally isolated. They don't share cache nor configuration. To have one you can use a special Avrora.Client macros to generate it.

defmodule MyClient do
  use Avrora.Client,
    registry_url: "http://localhost:8081",
    registry_auth: {:basic, ["username", "password"]}
    schemas_path: Path.expand("./priv/schemas"),
    registry_schemas_autoreg: false,
    convert_null_values: false,
    convert_map_to_proplist: false
    names_cache_ttl: :timer.minutes(5)
end

and then you can add it to your supervision tree or start the process manually

children = [
  MyClient
]

Supervisor.start_link(children, strategy: :one_for_one)

# or

{:ok, pid} = MyClient.start_link()

Enjoy!

Special thanks 👏🏼 for the feature request to @arusahni 💚

In this release, a new module responsible for schema registration arises. Welcome Avrora.Utils.Registrar. It's designed to be used in the client code because it's using Avrora.Storage.Memory to prevent performance issues.

It allows you to control the name your AVRO schemas will be registered at (i.esubject in Schema Registry).

defmodule Sample do
  alias Avrora.Utils.Registrar
  
  def loop do
    Enum.reduce_while(1..100, 0, fn x, acc ->
      if x < 100, do: {:cont, register("io.confluent.Payment")}, else: {:halt, acc}
    end)
  end
  
  defp register(schema_name), do: Registrar.register_schema_by_name(schema_name)
end

In addition, a mix task for schema registration was refactored (#65) to use this new functionality and now supports a new argument --as which in combination with --name option allows you to change the subject.

$ mix avrora.reg.schema --name io.confluent.Payment --as MyCustomName
schema `io.confluent.Payment' will be registered as `MyCustomName'

Thanks a lot to @arttsu #62 💚

In this release with help of @MichalDolata Avrora.Encoder start properly decoding AVRO map fields and instead of Erlang proplists they become Elixir Map

If you depend on the old behavior it can be easily restored with new configuration option:

config :avrora,
  # ...
  convert_map_to_proplist: true # optional: if you want to restore the old behavior for decoding map-type

Kudos to @soundmonster who spot that Avrora.Encoder has the wrong typespecs. We a few precautions and the new PR checks it should be possible to catch such mistakes early.

Starting this version Avrora gets two new amazing features.

Confluent® Schema References

Yes, Avrora already had support for Inter-schema references, but now it gains a native Confluent Schema Registry references support.

Starting Confluent Platform 5.5 you can have a true reference to another schema in the registry. This info is delivered via metadata of the schema response and will be handled by Avrora seamlessly.

⚠️ The only thing to remember – please disable automatic schema registration registry_schemas_autoreg: false to avoid the creation of a schema with de-referenced nested schemas. This is an official recommendation from the Confluent themself.

Many thanks to @FrancescoPessina for making it possible 🎉

The old new NULL handling

In this release, the handling of the decoded null values was unified for all codecs. Now the null can appear either as :null atom from Erlang or as nil from Elixir. It can be controlled via the new configuration option

config :avrora,
  convert_null_values: false, # optional: if you want to keep decoded `:null` values as is

💢 This is almost a backward-compatible change, except that ObjectContainerFIle behavior got changed. Now instead of the "null" string, you will get a :null atom. The string null value was an untested behavior before, so if you rely on string nulls in your code, please change them to nil or :null at your preference.

Kudos to @matreyes for bringing it up 👏🏼

Happy coding everyone! ❤️