parquet-dotnet

New

• Markdown documentation fully migrated to GitHub Pages. It was becoming slightly unmanageable and also recent GitHub updates made markdown files look awful. Also I kind of wanted to try Writerside by JetBrains, and publish docs with pride ;) @aloneguid
• Class deserializer will now skip class properties and not throw an exception if they are missing in the source parquet file. Thanks to @greenlynx in #361.
• Column statistics can be read with zero cost without reading the data. Thanks to @mirosuav in #252, #368.
• Support for DELTA_BINARY_PACKED encoding on write. This encoding is now default when writing INT32 and INT64 columns. Most of the work done by @ee-naveen in #382.

Improvements

• IronCompress was updated to v1.5.1 by @aloneguid.

Fixes

• Fix precision issues writing DateTime as milliseconds by @spanglerco in #312.
• In DataColumnWriter, RecycableMemoryStream wasn't used in a particular case, and instead MemoryStream was initialized directly. Thanks to @itayfisz in #373.
• Bitpacked Hybrid decoder was failing on columns containing exactly one value.

Bugs Fixed

• strings must be null by default (#360). Thanks @waf!

New Stuff

• You can force optionality of a schema field using the [ParquetRequired] attribute.
• ParquetSerializer validates class schema against actual file schema on deserialization and throws a helpful exception, like System.IO.InvalidDataException : property 'Id' is declared as Id (System.String?) but source data has it as Id (System.String) (you can spot the difference in nullability here).

• Added support for reading legacy array primitives collection serialized via legacy ParquetConvert class or some other legacy system, thanks to @PablitoCBR. This work was effectively taken from his PR and integrated more natively into this library. Thank you very much!

• Fixed deserializing parquet generated by Azure Data Explorer non-native writer by @mcbos in #357.

• re-worked build pipeline to separate build and release stage.

• use handcrafted release notes file and cut out last version notes with grep/head/tail on release. This is in order to improve release notes experience as autogenerated ones are often of sub-par quality.

What's Changed

• Add support for deserializing required strings by @mcbos in https://github.com/aloneguid/parquet-dotnet/pull/341
• Add support to .NET 6 TimeOnly type by @ramon-garcia in https://github.com/aloneguid/parquet-dotnet/pull/352
• Support for reading and writing column chunk metadata by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/354

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.12.0...4.13.0

What's Changed

• Support of DateOnly type in UntypedArrayExtensions by @ramon-garcia in https://github.com/aloneguid/parquet-dotnet/pull/338
• RLE/Hybrid encoder is up to 18 times faster by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/339

New Contributors

• @ramon-garcia made their first contribution in https://github.com/aloneguid/parquet-dotnet/pull/338

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.11.3...4.12.0

What's Changed

• Added missing flush, which makes my code hang, the flush was present … by @dxdjgl in https://github.com/aloneguid/parquet-dotnet/pull/333

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.11.2...4.11.3

What's Changed

• Added deserialization by rowGroupIndex by @mcbos in https://github.com/aloneguid/parquet-dotnet/pull/330
• support for serialisation of nullable guids by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/332

New Contributors

• @mcbos made their first contribution in https://github.com/aloneguid/parquet-dotnet/pull/330

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.11.1...4.11.2

What's Changed

• Managed Snappy performance improvement and managed Zstd support by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/327

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.11.0...4.11.1

What's Changed

• End of life for legacy class serializer and legacy DataType enumeration by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/324
• bug: decimal encoding didn't stamp ConvertedType (#323) by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/325
• Support UUID logical type (C# Guid). by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/326

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.10.1...4.11.0

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.10.0...4.10.1

This release has a low-level breaking change but you will only notice it if you are working with low-level thrift metadata. See the detailed reason and description for this change. In some cases you should see 10x performance increase due to a new handcrafted thrift compiler.

No functional changes are introduced in this release, performance only.

What's Changed

• Thrift Compiler by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/313

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.9.2...4.10.0

What's Changed

• bug: empty logical type container is created if there is no matching logical type by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/314

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.9.1...4.9.2

What's Changed

• minimize number of streaming calls when reading and writing thrift metadata by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/310

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.9.0...4.9.1

What's Changed

• Support [JsonPropertyOrder] in class serializer by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/307
• Library-Wide Schema improvements by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/308
• low-level: full logical types support in Parquet Thrift schema
• class serializer supports custom attributes to customize serialisation of dates, times, and decimals with [ParquetTimestamp], [ParquetMicroSecondsTime] and [ParquetDecimal] attributes. See the relevant docs.
• added support for .NET >= 6 DateOnly type.

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.8.1...4.9.0

What's Changed

• DataFrame write support by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/304

This release fully supports integration with Microsoft.Data.Analysis package and the docs are available.

There's also a sample Jupyter notebook in C# demonstrating it in action.

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.8.0...4.8.1

What's Changed

• support decoding legacy lists with no inner element (#286) by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/298
• Basic DataFrame support by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/299
• Empty Lists by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/303

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.7.1...4.8.0

• support for custom row group size

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.7.0...4.7.1

What's Changed

• Data Column now exposes definition and repetition levels #288
• ParquetSerializer correctly interprets nulls at various levels of nested structures #282

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.6.2...4.7.0

What's Changed

• Class Serialisation Performance Maintenance by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/284

This release has massive performance improvements for ParquetSerializer comparing to ParquetConvert (marked as "Legacy" in tests):

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.6.1...4.6.2

What's Changed

• ParquetSerializer support for append operations by @aloneguid in https://github.com/aloneguid/parquet-dotnet/pull/280

Full Changelog: https://github.com/aloneguid/parquet-dotnet/compare/4.6.0...4.6.1