parquet2

A new release is out there!

Thanks everyone for the fixes and improvements resulting in this stabler, easier to use and faster version of parquet2!

Breaking changes:

• Improved hybrid rle decoding performance ~-40% #203 (ritchie46)
• Improved API to read column chunks #195 (jorgecarleitao)
• Removed EncodedPage #191 (jorgecarleitao)

New features:

• Added serde support for RowGroupMetaData. #202 (youngsofun)

Fixed bugs:

• Removed un-necessary conversion #197 (jorgecarleitao)
• Avoid OOM on page streams #194 (jorgecarleitao)
• Fixed error requiring stats #193 (jorgecarleitao)

Enhancements:

• elide bound check in RLE decoder #201 (ritchie46)
• Replaced panics by errors on invalid pages #188 (evanrichter)

Yet another release of parquet2, mostly focused on avoiding panics and oom. No impact on performance, but improves reliability.

v0.16.1 (2022-08-17)

Full Changelog

Fixed bugs:

• Fixed error in FilteredHybridBitmapIter's trait bounds #187 (jorgecarleitao)

v0.16.0 (2022-08-17)

Full Changelog

Breaking changes:

• Improved Error #181 (jorgecarleitao)
• Made decoding fallible #178 (jorgecarleitao)
• Improved bitpacking #176 (jorgecarleitao)

New features:

• Added DELTA_BYTE_ARRAY encoder #183 (jorgecarleitao)

Fixed bugs:

• FixedLenByteArray max_precision integer overflow #184 (evanrichter)

Documentation updates:

• enable doc_cfg feature #186 (ritchie46)
• Improved decoding documentation #180 (jorgecarleitao)

We have a new release of parquet2 available!

Breaking changes:

• Add max_size to get_page_stream and get_page_iterator #173
• Optional async #174 (jorgecarleitao)
• Privatized CompressionLevel #170 (jorgecarleitao)
• Delay deserialization of dictionary pages #160 (jorgecarleitao)

New features:

• Added feature flag to use zlib-ng backend for gzip #165 (ritchie46)

Fixed bugs:

• Fixed OOM on malicious/malformed thrift #172 (jorgecarleitao)

Enhancements:

• Made compute_page_row_intervals public #171 (jorgecarleitao)
• Simplified interal code #168 (jorgecarleitao)
• cargo fmt #166 (ritchie46)

Testing updates:

• Improved coverage report #175 (jorgecarleitao)

A couple of bug fixes, by @jhorstmann and @v0y4g3r

Full Changelog

Fixed bugs:

• Fixed FileStreamer's end method to flush Parquet magic #163 (v0y4g3r)
• Fix compilation of parquet-tools #161 (jhorstmann)

Enhancements:

• Added Compressor::into_inner #158 (jorgecarleitao)

A small but important release to support legacy lz4, by @dantengsky 🚀

Full Changelog

New features:

• Added support for legacy lz4 decompression #151 (dantengsky)

Enhancements:

• Improved performance of reading #157 (jorgecarleitao)

A new release is here and in crates.io! 🎉🎉🎉

Full Changelog

Breaking changes:

• split_buffer should return Result #156

Fixed bugs:

• Removed panics on read #150 (jorgecarleitao)

Enhancements:

• Reduced reallocations #153 (jorgecarleitao)
• Removed AsyncSeek requirement from page stream #149 (medwards)

Another release of parquet2 is here!

We can now control the compression level of both GZIP and BROTLI compression thanks to @TurnOfACard 🙇

Thank you to everyone that contributed to this release!

Breaking changes:

• Removed unused cargo feature #145 (jorgecarleitao)
• Fix potential misuse of FileWriter API's (sync + async) #138 (TurnOfACard)

New features:

• Added new_with_page_meta to PageReader #136 (ygf11)
• Added compression options/levels for GZIP and BROTLI codecs. #132 (TurnOfACard)

Fixed bugs:

• Async FileStreamer does not write statistics #139
• Fixed error in compressing lz4raw with large offsets #140 (jorgecarleitao)

Enhancements:

• Improved read of metadata #143 (jorgecarleitao)
• Simplified async metadata read #137 (jorgecarleitao)

Testing updates:

• Lifted duplicated code to a function #141 (jorgecarleitao)
• Improved Integration test documentation and expanded tests #133 (TurnOfACard)

Full Changelog

Fixed bugs:

• Fixed error in compressing lz4raw with large offsets #140 (jorgecarleitao)

Full Changelog

Breaking changes:

• Add CompressionOptions, which allows for zstd compression levels. #128 (TurnOfACard)

Enhancements:

• Improved performance of RLE decoding (-18%) #130 (jorgecarleitao)
• Improved perf of bitpacking decoding (3.5x) #129 (jorgecarleitao)

Here we are for a new release of parquet2. This release has 3 main features:

• added optional support LZ4 compression and decompression in WASM builds (via LZ4-flex by @PSeitz)
• added support to read bloom filters
• added support to read and write page indexes

A summary of the Full Changelog is available below.

Thank you for everyone that contributed to this release! (credits to individual PRs below)

Breaking changes:

• Renamed ParquetError to Error #109
• Made .end not consume the parquet FileWriter #127 (jorgecarleitao)
• Removed compression from WriteOptions #125 (kornholi)
• Simplified API and converted some panics on read to errors #112 (jorgecarleitao)
• Improved typing to reduce clones and use of unwraps #106 (jorgecarleitao)
• Simplified PageIterator #103 (jorgecarleitao)

New features:

• Added support for page-level filter pushdown (indexes) #102
• Added support for bloom filters #98
• Added optional support for LZ4 via LZ4-flex crate (thus enabling wasm) #124 (jorgecarleitao)
• Added support for page-level filter pushdown (column and offset indexes) #107 (jorgecarleitao)
• Added support to read column and page indexes #100 (jorgecarleitao)

Fixed bugs:

• Fixed minimum version for LZ4 #122 (kornholi)
• Fixed Lz4Raw compression error (if input is tiny) #118 (dantengsky)
• Fixed LZ4 #95 (jorgecarleitao)

Enhancements:

• Made offsets be always written #123 (jorgecarleitao)
• Added specialized deserialization of one-level filtered pages #120 (jorgecarleitao)
• Added support to read and use bloom filters #99 (jorgecarleitao)
• Added ordinal and total_compressed_size to column meta #96 (jorgecarleitao)
• Added non-consuming function to get values of delta-decoder #94 (jorgecarleitao)
• Disabled bitpacking default-features and upgraded to edition 2021 #93 (light4)

Documentation updates:

• Fix deployment of guide #115 (jorgecarleitao)

Testing updates:

• Added tests for reducing statistics #116 (jorgecarleitao)
• Simplified tests #104 (jorgecarleitao)

Full Changelog

Enhancements:

• Update zstd dependency #82 (jhorstmann)
• Added file offset #81 (barrotsteindev)

v0.10.0 (2022-02-02)

A new release is now available in crates.io.

Breaking changes:

• Simplified API to write files #78 (jorgecarleitao)

Fixed bugs:

• Fixed panic in reading empty values in hybrid-RLE #80 (jorgecarleitao)

Fixed bugs:

• Fixed panic in reading empty values in hybrid-RLE #80 (jorgecarleitao)

A new release is here, with a performance improvement to bitpacking by @danburkert , an important fix to write data, and one backward incompatible change to allow serializing and writing to happen on separate thread pools (e.g. via tokio_rayon).

Breaking changes:

• Changed stream of groups to stream of futures of groups #71 (jorgecarleitao)

Enhancements:

• bitpacking: use stack-allocated temporary buffer #76 (danburkert)
• Added constructor to RowGroupMetaData and ColumnChunkMetaData #74 (yjshen)
• Improved performance of reading multiple pages #73 (jorgecarleitao)

Fixed bugs:

• Fixed error in declaring size of compressed dict page. #72 (jorgecarleitao)

Full Changelog

Full Changelog

Fixed bugs:

• Fixed error in declaring size of compressed dict page. #72 (jorgecarleitao)

Full Changelog

Breaking changes:

• Improved error message when a feature is not active #69 (jorgecarleitao)

Fixed bugs:

• Fixed error in finishing iterator. #68 (jorgecarleitao)

Full Changelog

Breaking changes:

• Use i64s for delta-bitpacked's interface #67 (kornholi)

New features:

• Added basic support to read nested types #64 (jorgecarleitao)

Fixed bugs:

• Fix off-by-one error in delta-bitpacked decoder #66 (kornholi)

Another release is here. It contains 3 breaking changes and a couple of bug fixes (backported to 0.5).

Full Changelog

Breaking changes:

• Improved performance of codec initialization #63 (jorgecarleitao)
• Made PageFilter Send #62 (dantengsky)
• Allowed reusing compression buffer #60 (jorgecarleitao)

Fixed bugs:

• Fixed delta-bitpacked mini-block decoding #56 (kornholi)
• Add descriptor to FixedLenStatistics #54 (potter420)
• Fixed error in reading zero-width bit from hybrid RLE. #53 (jorgecarleitao)

Enhancements:

• Added writing reduced statistics for FixedLenByteArray #55 (potter420)

Full Changelog

Fixed bugs:

• Fixed delta-bitpacked mini-block decoding #56 (kornholi)
• Add descriptor to FixedLenStatistics #54 (potter420)

Enhancements:

• Added writing reduced statistics for FixedLenByteArray #55 (potter420)

Full Changelog

Fixed bugs:

• Fixed error in reading zero-width bit from hybrid RLE. #53 (jorgecarleitao)