A convenient Elixir library to work with Avro messages, schemas and Confluentยฎ Schema Registry
MIT License
In version 0.28 two new configuration options were added: registry_ssl_cacerts
and registry_ssl_cacert_path
both used for controlling SSL verification of the Schema Registry connection.
If both options are set, the registry_ssl_cacerts
takes precedence over registry_ssl_cacert_path
, also registry_ssl_cacerts
is a DER-encoded certificate, when registry_ssl_cacert_path
is a PEM-encoded certificate file.
As always feel free to share your feedback or issues and happy coding ๐ค
Published by Strech over 1 year ago
To reduce noise about deprecation of Logger.warn/1
and refresh the CI setup together with local dependencies we have new this release.
Enjoy! ๐
Published by Strech almost 2 years ago
In this release, @emilianobovetti fixed an issue with custom configurations for the registration mix task. If you want to use your special config, let's say config/runtime.exs
, now you can specify it via --appconfig
option, like that
mix avrora.reg.schema --name io.confluent.Payment --appconfig runtime
schema `io.confluent.Payment' will be registered
it will be loaded in addition to the default one (if it's present)
Enjoy! Thanks to @emilianobovetti ๐
Published by Strech almost 2 years ago
Starting this release each request to the Schema Registry will have an additional header User-Agent
. It's done in response to issue #101 when the AWS setup rejects the request.
The default value will contain the current version of the library, its name, and the language, for example: Avrora/0.25.0 Elixir
If you want the previous behaviour, i.e no User-Agent
header or your custom header value, then you can adjust it via the registry_user_agent
configuration option by setting it to nil
or your String respectively.
defmodule MyClient
use Avrora.Client, registry_user_agent: "Hello world" # you can set it to whatever you like
end
defmodule MyClientWithoutUserAgent
use Avrora.Client, registry_user_agent: nil # or you can disable User-Agent header
end
Thanks to @azeemchauhan for bringing it up ๐
Published by Strech about 2 years ago
It turns out that Avrora.Config.registry_schemas_autoreg/0
configuration option was not returning false
values if the library is configured and a private client is not used. And the only way to set it to false
was to create your client and configure it like
defmodule MyClient do
use Avrora.Client, registry_schemas_autoreg: false
end
Thanks to @ankhers it's fixed now! Enjoy folks! ๐ข
Published by Strech about 2 years ago
In this release, we explicitly set the SSL option verify
to be verify_none
to reflect the changed default in OTP25. To avoid the false sense of security OTP25 has changed the default value for SSL verify
to be verify_peer
which requires you to source it certs.
We are going to add secure option in the next release, but for now let's keep it as it was ๐คท๐ผ
Thanks to @goozzik for pointing this out!
Published by Strech over 2 years ago
Initially, the decoder hook was done as a non-configurable no-op function due to clarity โ how much it will be used and will be at all?. Seems the time has come, thanks to the @LostKobrakai ๐
The very first use-case is tagging unions, check this out:
defmodule MyClient do
use Avrora.Client,
decoder_hook: &__MODULE__.hook/4
def hook(type, sub_name_or_index, data, decode_fun) do
tag_unions_hook = :avro_decoder_hooks.tag_unions()
tag_unions_hook.(type, sub_name_or_index, data, decode_fun)
end
end
Now all the complex unions will be tagged and result will be passed to the "private" Avrora decoder hook. But there is no limit on what you can do with it, just don't forget to call at the end decode_fun.(data)
๐
Have fun ๐ ๐ช
Published by Strech over 3 years ago
In this release, all the private client configuration options become dynamic and may be resolved at runtime or at compile-time. It's defined by the presence of the otp_app
option.
If you set the otp_app
configuration the value lookup will follow this logic
OTP env value โ Client value โ Default value
Let's take a look at this example with private client MyClient
and :my_app
OTP application
# config/config.exs
config :my_app, MyClient, registry_url: "http://my-app.io"
# lib/my_client.ex
defmodule MyClient do
use Avrora.Client,
otp_app: :my_app,
registry_url: "http://never-used.com",
names_cache_ttl: 42
end
As the result, configuration values will look like this
MyClient.Config.registry_url() == "http://my-app.io" # (comes from runtime environment)
MyClient.Config.names_cache_ttl() == 42 # (comes from compile-time options)
MyClient.Config.registry_schemas_autoreg() == true # (comes from compile-time option defaults)
Most of the generated Avrora.Client
code for Config
module gets rid of strings interpolation on runtime and moves it to the compile-rime and some hot paths were moved to compile-time.
Thanks a lot for this release to @juanperi ๐
Published by Strech over 3 years ago
In this release, few changes were done to the Avrora.Config
.
The main change is about releases and schemas_path
configuration. Especially if you have multiple apps in Umbrella and would like to have per-app schemas_path
, now you can pass an otp_app
option which should point to your OPT application which will be used to compute the root path for the schemas_path
in a runtime.
# Private client
defmodule MyClient do
use Avrora.Client,
otp_app: :my_application,
schemas_path: "./some/path/to/schemas"
end
# Shared client
config :avrora,
otp_app: :my_application,
schemas_path: "./some/path/to/schemas"
And as a side-effect, now all the schemas_path
which is without otp_app
set, will be resolved with Path.expand
Many thanks to @LostKobrakai for support ๐
Published by Strech over 3 years ago
Unfortunately, mistakes happen. A copy-paste bug sneaks into Avrora.Client
module and breaks authentication. Thanks to @raphaklaus who spots and fixes the issue ๐
Happy coding ๐ธ
Published by Strech over 3 years ago
This release getting Avrora closer to the AVRO specification. Now it will be possible to encode and decode all 3 named types Enum
, Fixed
and already working one โRecord
.
It's simple as it was and requires no hassle. Here is a low-level example with some internal changes ๐
json = ~s({"namespace":"io.confluent","name":"CardType","type":"enum","symbols":["MASTERCARD","VISA","AMERICANEXPRESS"]})
{:ok, schema} = Avrora.Schema.Encoder.from_json(json)
{:ok, encoded} = Avrora.Codec.Plain.encode("VISA", schema: schema)
encoded # => <<2>>
Published by Strech over 3 years ago
In this release, all warnings of the dialyzer about return types or dead branches of the code should vanish.
It turns out that the private Avrora.Config
modules were too optimized which caused the dialyzer to report dead branches inside the code which was using if/else conditions with that config. The solution was to make it a bit more dynamic, but with a constant access time.
Another problem was the private Avrora.Schema
return type, which was mismatching for every private client. So now, all private clients will use the only one schema type Avrora.Schema
, all the functionality was removed from this module and it became just a struct.
Kudos to @LostKobrakai for raising the issue ๐
Published by Strech over 3 years ago
Starting this version a new command-line argument --module MODULE
added to the schema registration mix task. It allows you to use your private Avrora client modules to be used for schema registration.
Example
$ mix avrora.reg.schema --name io.confluent.Payment --as MyCustomName --module MyClient
Enjoy โ๏ธ
Published by Strech over 3 years ago
Unfortunately, in release 0.18.0
two new methods were forgotten for auto-generated private clients. It's Avrora.encode_plain/2
and Avrora.decode_plain/2
.
Luckily @LostKobrakai spot the issue โค๏ธ and fixed it.
Happy encoding, everyone ๐ฅ
Published by Strech over 3 years ago
In this release, a potential collision between 2 formats was removed. In several cases (see #70) messages encoded with format: :plain
could match a format: :registry
magic byte sequence. It will lead to an error in decoding with format: :guess
.
From now on it is recommended to use special crafted Avrora.encode_plain/2
and Avrora.decode_plain/2
while working with plain encoded Avro messages, thanks for the effort to @mw23 ๐
{:ok, pid} = Avrora.start_link()
message = <<8, 116, 120, 45, 49, 123, 20, 174, 71, 225, 250, 47, 64>>
{:ok, decoded} = Avrora.decode_plain(message, schema_name: "io.confluent.Payment")
%{"id" => "tx-1", "amount" => 15.99}
message = %{"id" => "tx-1", "amount" => 15.99}
{:ok, encoded} = Avrora.encode_plain(message, schema_name: "io.confluent.Payment")
<<8, 116, 120, 45, 49, 123, 20, 174, 71, 225, 250, 47, 64>>
In addition, the project readme was clean up and adopted to be transformed as close as possible by ExDoc for hexdocs.pm. And as a bonus compilation should not throw any warnings about docs or specs, thanks for bringing it up to @fxn ๐
Published by Strech over 3 years ago
In this release, Avrora finally gets multi-client support. From now on you can use a shared client Avrora
or make as many you need private clients which will 100% equivalent to the Avrora
one.
Private clients are totally isolated. They don't share cache nor configuration. To have one you can use a special Avrora.Client
macros to generate it.
defmodule MyClient do
use Avrora.Client,
registry_url: "http://localhost:8081",
registry_auth: {:basic, ["username", "password"]}
schemas_path: Path.expand("./priv/schemas"),
registry_schemas_autoreg: false,
convert_null_values: false,
convert_map_to_proplist: false
names_cache_ttl: :timer.minutes(5)
end
and then you can add it to your supervision tree or start the process manually
children = [
MyClient
]
Supervisor.start_link(children, strategy: :one_for_one)
# or
{:ok, pid} = MyClient.start_link()
Enjoy!
Special thanks ๐๐ผ for the feature request to @arusahni ๐
Published by Strech almost 4 years ago
In this release, a new module responsible for schema registration arises. Welcome Avrora.Utils.Registrar
. It's designed to be used in the client code because it's using Avrora.Storage.Memory
to prevent performance issues.
It allows you to control the name your AVRO schemas will be registered at (i.esubject
in Schema Registry).
defmodule Sample do
alias Avrora.Utils.Registrar
def loop do
Enum.reduce_while(1..100, 0, fn x, acc ->
if x < 100, do: {:cont, register("io.confluent.Payment")}, else: {:halt, acc}
end)
end
defp register(schema_name), do: Registrar.register_schema_by_name(schema_name)
end
In addition, a mix
task for schema registration was refactored (#65) to use this new functionality and now supports a new argument --as
which in combination with --name
option allows you to change the subject
.
$ mix avrora.reg.schema --name io.confluent.Payment --as MyCustomName
schema `io.confluent.Payment' will be registered as `MyCustomName'
Thanks a lot to @arttsu #62 ๐
Published by Strech almost 4 years ago
In this release with help of @MichalDolata Avrora.Encoder
start properly decoding AVRO map fields and instead of Erlang proplists they become Elixir Map
If you depend on the old behavior it can be easily restored with new configuration option:
config :avrora,
# ...
convert_map_to_proplist: true # optional: if you want to restore the old behavior for decoding map-type
Published by Strech almost 4 years ago
Kudos to @soundmonster who spot that Avrora.Encoder
has the wrong typespecs. We a few precautions and the new PR checks it should be possible to catch such mistakes early.
Published by Strech almost 4 years ago
Starting this version Avrora gets two new amazing features.
Yes, Avrora already had support for Inter-schema references, but now it gains a native Confluent Schema Registry references support.
Starting Confluent Platform 5.5 you can have a true reference to another schema in the registry. This info is delivered via metadata of the schema response and will be handled by Avrora seamlessly.
โ ๏ธ The only thing to remember โ please disable automatic schema registration registry_schemas_autoreg: false
to avoid the creation of a schema with de-referenced nested schemas. This is an official recommendation from the Confluent themself.
Many thanks to @FrancescoPessina for making it possible ๐
In this release, the handling of the decoded null
values was unified for all codecs. Now the null can appear either as :null
atom from Erlang or as nil
from Elixir. It can be controlled via the new configuration option
config :avrora,
convert_null_values: false, # optional: if you want to keep decoded `:null` values as is
๐ข This is almost a backward-compatible change, except that ObjectContainerFIle behavior got changed. Now instead of the "null"
string, you will get a :null
atom. The string null
value was an untested behavior before, so if you rely on string nulls in your code, please change them to nil
or :null
at your preference.
Kudos to @matreyes for bringing it up ๐๐ผ
Happy coding everyone! โค๏ธ