parquet-dotnet

This release contains completely rewritten class serializer using .NET expression trees. All the class serialisation documentation has been updated to reflect this. Existing ParquetConvert serializer is left untouched and marked obsolete, hence there is no increase in the major version number.
If you are using ParquetConvert, consider switching to ParquetSerializer soon, because no new features or fixes will be added to ParquetConvert.

New class serializer supports all primitive types and nested types (structs, lists, maps and their combinations). It also fully conforms to Dremel specification, which was a massive headache to implement properly.

Other improvements:

• Documentation updated for low-level and serializer API on how to use nested data types.
• Corrected how definition and repetition levels are calculated, making it more conformant to Parquet specification.
• Schema path calculation logic improved and does not rely on string splitting/joining, which allows you to use any characters anywhere in column names (#278)

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.5.4...4.6.0

What's Changed

• bitpacking range check bug on destination buffer (#267)

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.5.3...4.5.4

What's Changed

• Class serialization includes inherited properties by default by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/271
• Ability to use JsonIgnore and JsonPropertyName in serialisation

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.5.2...4.5.3

What's Changed

• support null entries in structures by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/264
• SchemaReflector and PropertyHelpers.GetDeclaredPropertyFromClassType … by @clarkfinger in https://github.com/aloneguid/parquet-dotnet/pull/269

New Contributors

• @clarkfinger made their first contribution in https://github.com/aloneguid/parquet-dotnet/pull/269

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.5.1...4.5.2

What's Changed

• Spark fails to read files if "created_by" metadata field is not in the format is expects. by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/262

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.5.0...4.5.1

What's Changed

• Performance and encodings by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/249
• Massive performance improvement in RLE encoder/decoder (tests to follow)
• Support DELTA_BINARY_PACKED encoding.
• Support DELTA_LENGTH_BYTE_ARRAY encoding.
• Support DELTA_BYTE_ARRAY encoding.
• Reading back the schema with adjusted encoding of DateTime, is not re… by @dxdjgl in https://github.com/aloneguid/parquet-dotnet/pull/248
• Testing the 'Unspecified Kind' behaviour (and Local behaviour) by @xclayl in https://github.com/aloneguid/parquet-dotnet/pull/253
• Fix issue destroying existing file with append in ParquetConvert.Seri… by @oyhovd in https://github.com/aloneguid/parquet-dotnet/pull/259

New Contributors

• @oyhovd made their first contribution in https://github.com/aloneguid/parquet-dotnet/pull/259

Performance

In general, this library outperforms native code in raw write performance which was the aim of this release (see chart). Unfortunately, writing strings is still slightly slower, which will improve in the next versions.

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.4.7...4.5.0

What's Changed

• Changes to test code to run on my machine by @xclayl in https://github.com/aloneguid/parquet-dotnet/pull/250
• Remove new() constraint for Serialize methods by @francoishill in https://github.com/aloneguid/parquet-dotnet/pull/254

New Contributors

• @xclayl made their first contribution in https://github.com/aloneguid/parquet-dotnet/pull/250
• @francoishill made their first contribution in https://github.com/aloneguid/parquet-dotnet/pull/254

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.4.6...4.4.7

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.4.5...4.4.6

What's Changed

• Fix signedbyte typo by @mukunku in https://github.com/aloneguid/parquet-dotnet/pull/246
• RLE encoding bug for different bit width (#243) by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/247

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.4.4...4.4.5

What's Changed

• Add option to decide compression level of for instance gzip compressi… by @dxdjgl in https://github.com/aloneguid/parquet-dotnet/pull/242

New Contributors

• @dxdjgl made their first contribution in https://github.com/aloneguid/parquet-dotnet/pull/242

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.4.3...4.4.4

Massive performance boost for RLE encoder.

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.4.2...4.4.3

What's Changed

• perform range checks when data page has more data than there is (#214) by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/241

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.4.1...4.4.2

What's Changed

• adding nullable annotations across the library by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/238
• byte[] was wrongly resolved to FIXED_LENGTH_BYTE_ARRAY instead of BYTE_ARRAY by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/239
• ReadAsTableAsync method supports progress callbacks

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.4.0...4.4.1

What's Changed

• support reading delta byte array by @dmishne in https://github.com/aloneguid/parquet-dotnet/pull/231
• 232 fix ReadDataPageV2Async memory handling by @dmishne in https://github.com/aloneguid/parquet-dotnet/pull/233
• dictionary encoding needs to write RLE with bit-width byte (#234) by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/236

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.3.4...4.4.0

Experimental managed Snappy implementation.

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.3.3...4.3.4

Improvements

• Slight performance improvements in string write.
• Reading file schema is now using only async methods.

Unrelated

Check out Parquet Online project, help wanted!

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.3.2...4.3.3

Performance Improvements

Write performance comparing to ParquetSharp:

Read performance comparing to ParquetSharp:

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.3.1...4.3.2

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.3.0...4.3.1

Breaking Changes

(I promise to stop these!)

• Due to massive ambiguity issues between DateTimeOffset and DateTime, the first one gets retired because DateTime is more often used by developers. If you are using DateTimeOffset (with Parquet) please stop now!
• DataField constructor now accepts nullable parameters for isNullable and isArray which allows to override reflected type metadata.

New Features

• Dictionary encoding is now supported. It is enabled by default for string columns with uniqueness factor of 0.8. You can tune it or turn off completely in ParquetOptions class (#223, #151).

Deprecations

• DataType enumeration is now obsolete. It was introduced in early versions when number of supported types was small, but now almost every basic CLR type is supported so there is no need for it.

Bugs Fixed

• Due to refactoring, data boundary check was missed out causing exception on deserialising columns with all nulls (#224).
• Creator metadata wasn't set after migrating to GitHub Actions (#104).

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.2.3...4.3.0

What's Changed

• Support decimals expressed as fixed-size byte array by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/220

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.2.2...4.2.3