delta-rs

New features

• feat: allow merge_execute to release the GIL by @emcake in https://github.com/delta-io/delta-rs/pull/2091
• feat: arrow backed log replay and table state by @roeap in https://github.com/delta-io/delta-rs/pull/2037
• feat: update table config to contain new config keys by @roeap in https://github.com/delta-io/delta-rs/pull/2127
• feat: expose stats schema on Snapshot by @roeap in https://github.com/delta-io/delta-rs/pull/2128
• feat: implementation for replaceWhere by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1996
• feat: implement clone for DeltaTable struct by @mightyshazam in https://github.com/delta-io/delta-rs/pull/2160
• feat: introduce schema evolution on RecordBatchWriter by @rtyler in https://github.com/delta-io/delta-rs/pull/2024

Bug Fixes

• fix: properly deserialize percent-encoded file paths of Remove actions, to make sure tombstone and file paths match by @sigorbor in https://github.com/delta-io/delta-rs/pull/2035
• fix: reinstate copy-if-not-exists passthrough by @emcake in https://github.com/delta-io/delta-rs/pull/2083
• refactor: add deltalake-gcp crate by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2061
• fix: schema issue within writebuilder by @universalmind303 in https://github.com/delta-io/delta-rs/pull/2106
• fix: temporarily skip s3 roundtrip test by @roeap in https://github.com/delta-io/delta-rs/pull/2124
• fix: set partition values for added files when building compaction plan by @alexwilcoxson-rel in https://github.com/delta-io/delta-rs/pull/2119
• fix: clean-up paths created during tests by @roeap in https://github.com/delta-io/delta-rs/pull/2126
• fix: add missing pandas import by @Tim-Haarman in https://github.com/delta-io/delta-rs/pull/2116
• fix: order logical schema to match physical schema by @Blajda in https://github.com/delta-io/delta-rs/pull/2129
• fix: do not write empty parquet file/add on writer close; accurately … by @alexwilcoxson-rel in https://github.com/delta-io/delta-rs/pull/2123
• fix: prevent empty stats struct during parquet write by @alexwilcoxson-rel in https://github.com/delta-io/delta-rs/pull/2125
• fix(#2143): keep specific error type when writing fails by @abaerptc in https://github.com/delta-io/delta-rs/pull/2144
• fix(s3): restore working test for DynamoDb log store repair log on read by @dispanser in https://github.com/delta-io/delta-rs/pull/2120
• fix: made generalize_filter less permissive, also added more cases by @emcake in https://github.com/delta-io/delta-rs/pull/2149
• fix: allow loading of tables with identity columns by @rtyler in https://github.com/delta-io/delta-rs/pull/2155
• fix: replace BTreeMap with IndexMap to preserve insertion order by @roeap in https://github.com/delta-io/delta-rs/pull/2150

Other Changes

• chore(deps): update serial_test requirement from 2 to 3 by @dependabot in https://github.com/delta-io/delta-rs/pull/2052
• chore: update documentation for S3 / DynamoDb log store configuration by @dispanser in https://github.com/delta-io/delta-rs/pull/2041
• docs: make an overview tab visible in docs by @r3stl355 in https://github.com/delta-io/delta-rs/pull/2080
• docs: update docs for rust print statement by @skariyania in https://github.com/delta-io/delta-rs/pull/2077
• docs: add usage guide for check constraints by @hntd187 in https://github.com/delta-io/delta-rs/pull/2079
• docs: add page on why to use delta lake by @MrPowers in https://github.com/delta-io/delta-rs/pull/2076
• docs: how delta lake transactions work by @MrPowers in https://github.com/delta-io/delta-rs/pull/2089
• docs: move dynamo docs into new docs page by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2093
• docs: delta lake file skipping by @MrPowers in https://github.com/delta-io/delta-rs/pull/2096
• chore: removed unnecessary print statement from update method by @LilMonk in https://github.com/delta-io/delta-rs/pull/2111
• chore: temporarily ignore the repair on update test by @rtyler in https://github.com/delta-io/delta-rs/pull/2114
• chore: bump python by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2092
• docs: add dask page to integration docs by @avriiil in https://github.com/delta-io/delta-rs/pull/2122
• docs: fix arg indent by @wchatx in https://github.com/delta-io/delta-rs/pull/2103
• docs: delta lake is great for small data by @MrPowers in https://github.com/delta-io/delta-rs/pull/2113
• docs: use transparent logo in README by @roeap in https://github.com/delta-io/delta-rs/pull/2132
• chore: shorten up the crate folder names in the tree by @rtyler in https://github.com/delta-io/delta-rs/pull/2145
• refactor(python): drop custom filesystem in write_deltalake by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2137
• chore: upgrade to DataFusion 35.0 by @philippemnoel in https://github.com/delta-io/delta-rs/pull/2121
• chore: cleanup minor clippies and other warns by @rtyler in https://github.com/delta-io/delta-rs/pull/2161
• fix: allow checkpoints to contain metadata actions without a createdTime value by @rtyler in https://github.com/delta-io/delta-rs/pull/2059

New Contributors

• @skariyania made their first contribution in https://github.com/delta-io/delta-rs/pull/2077
• @LilMonk made their first contribution in https://github.com/delta-io/delta-rs/pull/2111
• @alexwilcoxson-rel made their first contribution in https://github.com/delta-io/delta-rs/pull/2119
• @Tim-Haarman made their first contribution in https://github.com/delta-io/delta-rs/pull/2116
• @avriiil made their first contribution in https://github.com/delta-io/delta-rs/pull/2122
• @wchatx made their first contribution in https://github.com/delta-io/delta-rs/pull/2103
• @abaerptc made their first contribution in https://github.com/delta-io/delta-rs/pull/2144
• @philippemnoel made their first contribution in https://github.com/delta-io/delta-rs/pull/2121
• @mightyshazam made their first contribution in https://github.com/delta-io/delta-rs/pull/2160

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.15.1...python-v0.15.2

New features

• feat(python, rust): expose custom_metadata for all operations by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2032
• feat: refactor WriterProperties class by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2030
• refactor: increase metadata action usage by @roeap in https://github.com/delta-io/delta-rs/pull/2027
• feat(rust): add more commit info to most operations by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2009
• feat(python): add schema conversion of FixedSizeBinaryArray and FixedSizeList by @balbok0 in https://github.com/delta-io/delta-rs/pull/2005
• feat: retry with exponential backoff for DynamoDb interaction by @dispanser in https://github.com/delta-io/delta-rs/pull/1975

Bug Fixes

• fix: ensure metadata cleanup do not corrupt tables without checkpoints by @Blajda in https://github.com/delta-io/delta-rs/pull/2044
• fix: remove casts of structs to record batch by @Blajda in https://github.com/delta-io/delta-rs/pull/2033
• fix: use temporary table names during the constraint checks by @r3stl355 in https://github.com/delta-io/delta-rs/pull/2017

Other Changes

• chore: refactoring AWS code out of the core crate by @rtyler in https://github.com/delta-io/delta-rs/pull/1995
• refactor: move azure integration to dedicated crate by @roeap in https://github.com/delta-io/delta-rs/pull/2023
• docs: update docs for merge by @Blajda in https://github.com/delta-io/delta-rs/pull/2042
• chore: update datafusion by @roeap in https://github.com/delta-io/delta-rs/pull/2029
• fix: github actions for releasing docs by @r3stl355 in https://github.com/delta-io/delta-rs/pull/2026
• docs: add alterer by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2014
• docs: add writer properties to docs by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2002

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.15.0...python-v0.15.1

New features

• feat: merge using partition filters by @emcake in https://github.com/delta-io/delta-rs/pull/1958
• feat: omit unmodified files during merge write by @Blajda in https://github.com/delta-io/delta-rs/pull/1969
• feat: check constraints by @hntd187 in https://github.com/delta-io/delta-rs/pull/1915
• feat(python): expose add constraint operation by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1973
• feat: add kernel ExpressionEvaluator by @roeap in https://github.com/delta-io/delta-rs/pull/1829
• feat: implement S3 log store with transactions backed by DynamoDb by @dispanser in https://github.com/delta-io/delta-rs/pull/1904
• feat(python): add writer_properties to all operations by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1980
• feat(python): combine load_version/load_with_datetime into load_as_version by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1968
• feat: update to include pyarrow-hotfix by @dennyglee in https://github.com/delta-io/delta-rs/pull/1930
• feat: cast list items to default before write with different item names by @JonasDev1 in https://github.com/delta-io/delta-rs/pull/1959
• feat(python): expose large_dtype param in merge @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2003
• feat(python): expose custom metadata to writers @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1994

Bug Fixes

• fix: respect case sensitivity on operations by @Blajda in https://github.com/delta-io/delta-rs/pull/1954
• fix: case sensitivity for z-order by @Blajda in https://github.com/delta-io/delta-rs/pull/1982
• fix: implement consistent formatting for constraint expressions by @Blajda in https://github.com/delta-io/delta-rs/pull/1985
• fix: remove the get_data_catalog() function by @rtyler in https://github.com/delta-io/delta-rs/pull/1941
• fix: handle empty table response in unity api by @JonasDev1 in https://github.com/delta-io/delta-rs/pull/1963
• fix: flakey gcs test by @roeap in https://github.com/delta-io/delta-rs/pull/1987
• fix: enable S3 integration tests to be configured via environment vars by @dispanser in https://github.com/delta-io/delta-rs/pull/1966
• fix: properly decode percent-encoded file paths coming from parquet checkpoints by @sigorbor in https://github.com/delta-io/delta-rs/pull/1970

Breaking Changes

To control the writer properties in .update you need to pass the deltalake.WriterProperties class instead of a dicationary.

Other Changes

• chore: relocate cast_record_batch into its own module to shed the datafusion dependency by @rtyler in https://github.com/delta-io/delta-rs/pull/1955
• chore: break Glue support into its own crate without rusoto by @rtyler in https://github.com/delta-io/delta-rs/pull/1825
• docs: add polars integration by @MrPowers in https://github.com/delta-io/delta-rs/pull/1949
• docs: add better installation instructions by @MrPowers in https://github.com/delta-io/delta-rs/pull/1951
• docs: datafusion integration by @MrPowers in https://github.com/delta-io/delta-rs/pull/1993
• fix: add arrow page back by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1944
• refactor: trigger metadata retrieval only during DeltaTable.metadata by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1979
• docs: add auto-release when docs are merged to main by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1962
• chore: fix CI breaking lint issues by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1999

New Contributors

• @JonasDev1 made their first contribution in https://github.com/delta-io/delta-rs/pull/1963

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.14.0...python-v0.15.0

New features

• feat: adopt kernel schemas and improve protocol support by @roeap in https://github.com/delta-io/delta-rs/pull/1756
• feat: drop python 3.7 and adopt 3.12 by @roeap in https://github.com/delta-io/delta-rs/pull/1859
• feat: expose cleanup_metadata in Python by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1826
• feat: handle protocol compatibility by @roeap in https://github.com/delta-io/delta-rs/pull/1807
• feat(python): expose convert_to_deltalake by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1842
• feat(python): add pyarrow to delta compatible schema conversion in writer/merge by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1820
• feat(python): expose create to DeltaTable class by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1932
• feat(python): expose rust writer as additional engine v2 by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1891
• feat: extend write_deltalake to accept Deltalake schema by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1922

Bug Fixes

• fix: prevent writing checkpoints with a version that does not exist in table state by @rtyler in https://github.com/delta-io/delta-rs/pull/1863
• fix: checkpoint error with Azure Synapse by @PierreDubrulle in https://github.com/delta-io/delta-rs/pull/1848
• fix: improve catalog failure error message, add missing Glue native-tls feature dependency by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1883
• fix: use physical name for column name lookup in partitions by @aersam in https://github.com/delta-io/delta-rs/pull/1836
• fix(rust/python): optimize.compact not working with tables with mixed large/normal arrow by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1926
• fix: support os.PathLike for table references by @bolkedebruin in https://github.com/delta-io/delta-rs/pull/1809
• fix: add buffer flushing to filesystem writes by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1911
• fix: fail fast for opening non-existent path by @dimonchik-suvorov in https://github.com/delta-io/delta-rs/pull/1917
• fix: compare timestamp partition values as timestamps instead of strings by @sigorbor in https://github.com/delta-io/delta-rs/pull/1895
• fix: add high-level checking for append-only tables by @junjunjd in https://github.com/delta-io/delta-rs/pull/1887
• fix: prune each merge bin with only 1 file by @haruband in https://github.com/delta-io/delta-rs/pull/1902
• fix: get rid of panic in during table by @dimonchik-suvorov in https://github.com/delta-io/delta-rs/pull/1928

Other Changes

• docs: add release action by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1801
• chore: upgrade to the latest dynamodb-lock crate by @rtyler in https://github.com/delta-io/delta-rs/pull/1816
• refactor: default logstore implementation by @dispanser in https://github.com/delta-io/delta-rs/pull/1742
• ci: adopt ruff format for formatting by @roeap in https://github.com/delta-io/delta-rs/pull/1841
• chore: upgrade datafusion 33 by @Blajda in https://github.com/delta-io/delta-rs/pull/1775
• docs: add docs on small file compaction with optimize by @MrPowers in https://github.com/delta-io/delta-rs/pull/1850
• refactor: express log schema in delta types by @roeap in https://github.com/delta-io/delta-rs/pull/1876
• refactor: merge to use logical plans by @Blajda in https://github.com/delta-io/delta-rs/pull/1720
• chore: create benchmarks for merge by @Blajda in https://github.com/delta-io/delta-rs/pull/1857
• docs: on append, overwrite, delete and z-ordering by @MrPowers in https://github.com/delta-io/delta-rs/pull/1897
• docs: update python docs link in readme.md by @thomasfrederikhoeck in https://github.com/delta-io/delta-rs/pull/1899
• docs: update docs home page and add pandas integration by @MrPowers in https://github.com/delta-io/delta-rs/pull/1905
• ci: run doctest in CI for Python API examples by @marijncv in https://github.com/delta-io/delta-rs/pull/1840
• docs: delta lake arrow integration page by @MrPowers in https://github.com/delta-io/delta-rs/pull/1914
• docs: fix all examples and change overall structure by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1931
• refactor: simplify DeltaTableState by @roeap in https://github.com/delta-io/delta-rs/pull/1877
• docs: add logo, dark mode, boost search by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1936
• refactor: prefer usage of metadata and protocol fields by @roeap in https://github.com/delta-io/delta-rs/pull/1935

New Contributors

• @PierreDubrulle made their first contribution in https://github.com/delta-io/delta-rs/pull/1848
• @thomasfrederikhoeck made their first contribution in https://github.com/delta-io/delta-rs/pull/1899
• @dimonchik-suvorov made their first contribution in https://github.com/delta-io/delta-rs/pull/1917
• @sigorbor made their first contribution in https://github.com/delta-io/delta-rs/pull/1895
• @bolkedebruin made their first contribution in https://github.com/delta-io/delta-rs/pull/1809

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.13.0...python-v0.14.0

⚠️ If you are upgrading from any release other than 0.16.4, please also read these release notes ⚠️

This release includes a number of minor bug fixes including one for users of create_checkpoint_for() which previously allowed the caller to specify a version which did not match the loaded table state, leading to incorrect _last_checkjpoint files and a broke Delta table.

The v0.16.4 version of the deltalake crate contains one notable and important fix: an upgrade to the dynamodb_lock crate to v0.6.1.

That release changes the expected of the format for leaseDuration in DynamoDb from String to Number, which is a long-overlooked bug in the lock code which prevented stale locks from being reaped automatically using DynamoDb's TTL attribute

⚠️ CAUTION: Users of DynamoDb-based locking should use caution when upgrading their applications. ⚠️

Pre-existing locks should be properly respected by this newer version of dynamodb_lock however the consequences of a lock not being respected can result in data corruption of Delta tables. It is therefore recommended that when upgrading:

• All writers using a given DynamoDb table for locking are stopped
• DynamoDb is inspected and stale locks are cleared.
• TTL is enabled on the table on the leaseDuration attribute (adjust if the application uses a different attribute name for lease duration).
• Writers are restarted.

New features

• feat(python): expose FSCK (repair) operation by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1730
• feat: add VACUUM operation as commit in transaction log by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1728
• fix(python): add support for pyarrow 13+ by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1804
• feat(python): allow python objects to be passed as new values in .update() by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1749
• feat(python): allow for multiple when calls in MERGE operation by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1750

Bug fixes

• fix: ignore binary columns for stats generation by @emcake in https://github.com/delta-io/delta-rs/pull/1766
• fix(python): add write support explicitly for pyarrow dataset by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1780
• fix(python): ignore infinity in stats by @wjones127 in https://github.com/delta-io/delta-rs/pull/1784
• fix: delta scan partition ordering bug by @Blajda in https://github.com/delta-io/delta-rs/pull/1789

Other changes

• fix: relax pyarrow pin by @dhirschfeld in https://github.com/delta-io/delta-rs/pull/1743
• fix: remove pandas pin by @dhirschfeld in https://github.com/delta-io/delta-rs/pull/1746
• refactor!: update operations to use delta scan by @Blajda in https://github.com/delta-io/delta-rs/pull/1639
• chore: update datafusion by @roeap in https://github.com/delta-io/delta-rs/pull/1741
• docs: convert docs to use mkdocs by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1731
• docs: dynamodb lock configuration by @brayanjuls in https://github.com/delta-io/delta-rs/pull/1752
• refactor: perform bulk deletes during metadata cleanup by @cmackenzie1 in https://github.com/delta-io/delta-rs/pull/1763
• docs: enhance docs to enable multi-lingual examples by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1781
• chore: refactor into the deltalake meta crate and deltalake-core crates by @rtyler in https://github.com/delta-io/delta-rs/pull/1774
• feat: add deltalake sql crate by @roeap in https://github.com/delta-io/delta-rs/pull/1757
• feat: initial table features implementation by @hntd187 in https://github.com/delta-io/delta-rs/pull/1796
• docs: add CI for docs by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1798

New Contributors

• @dhirschfeld made their first contribution in https://github.com/delta-io/delta-rs/pull/1743
• @brayanjuls made their first contribution in https://github.com/delta-io/delta-rs/pull/1752
• @emcake made their first contribution in https://github.com/delta-io/delta-rs/pull/1766
• @hntd187 made their first contribution in https://github.com/delta-io/delta-rs/pull/1796

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.12.0...python-v0.13.0

What's Changed

New features

• feat: allow to set large dtypes for the schema check in write_deltalake by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1668
• feat(python): expose delete operation by @guilhem-dvr in https://github.com/delta-io/delta-rs/pull/1687
• feat(python): expose UPDATE operation by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1694
• feat(python): expose MERGE operation by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1685
• feat: add version number in .history() and display in reversed chronological order by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1710

Bug fixes

• fix: exception string in writer.py by @sebdiem in https://github.com/delta-io/delta-rs/pull/1665
• fix: change partitioning schema from large to normal string for pyarrow<12 by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1671
• fix: use epoch instead of ce for date stats by @universalmind303 in https://github.com/delta-io/delta-rs/pull/1672
• fix: unify environment variables referenced by Databricks docs by @rtyler in https://github.com/delta-io/delta-rs/pull/1673
• fix!: ensure predicates are parsable by @Blajda in https://github.com/delta-io/delta-rs/pull/1690
• fix: merge operation with string predicates by @Blajda in https://github.com/delta-io/delta-rs/pull/1705
• fix: reorder encode_partition_value() checks and add tests by @ldacey in https://github.com/delta-io/delta-rs/pull/1733

Other contributions

• perf: improve read performance by 7x with prebuffer by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1709
• docs: small consistency update in guide and readme by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1666
• docs: fix typo in readme by @JosiahParry in https://github.com/delta-io/delta-rs/pull/1696
• docs: add Python API reference to mkdocs by @wjones127 in https://github.com/delta-io/delta-rs/pull/1563
• docs(python): document the delete operation by @guilhem-dvr in https://github.com/delta-io/delta-rs/pull/1704
• docs: add a write example to delta.rs by @r3stl355 in https://github.com/delta-io/delta-rs/pull/1711
• chore: remove deprecated functions by @wjones127 in https://github.com/delta-io/delta-rs/pull/1735

Breaking changes

The DeltaTable.history() method now returns transactions in reverse chronological order. This matches the Spark implementation.

DeltaTable.files_by_partitions() has been removed. It has been deprecated since 0.7.0. Use DeltaTable.file_uris() instead.

DeltaTable.pyarrow_schema() has been removed. it has been deprecated since 0.7.0. Use DeltaTable.schema().to_pyarrow() instead.

New Contributors

• @sebdiem made their first contribution in https://github.com/delta-io/delta-rs/pull/1665
• @universalmind303 made their first contribution in https://github.com/delta-io/delta-rs/pull/1672
• @JosiahParry made their first contribution in https://github.com/delta-io/delta-rs/pull/1696
• @r3stl355 made their first contribution in https://github.com/delta-io/delta-rs/pull/1711
• @ldacey made their first contribution in https://github.com/delta-io/delta-rs/pull/1733

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.11.0...python-v0.12.0

Full Changelog

Implemented enhancements:

• Expose Optimize option min_commit_interval in Python #1640
• Expose create_checkpoint_for #1513
• integration tests regularly fail for HDFS #1428
• Add Support for Microsoft OneLake #1418
• add support for atomic rename in R2 #1356

Fixed bugs:

• Writing with large arrow types (e.g. large_utf8), writes wrong partition encoding #1669
• [python] Different stringification of partition values in reader and writer #1653
• Unable to interface with data written from Spark Databricks #1651
• get_last_checkpoint does some unnecessary listing #1643
• PartitionWriter's buffer_len doesn't include incomplete row groups #1637
• Slack community invite link has expired #1636
• delta-rs does not appear to support tables with liquid clustering #1626
• Internal Parquet panic when using a Map type. #1619
• partition_by with "$" on local filesystem #1591
• ProtocolChanged error when perfoming append write #1585
• Unable to cargo update using git tag or rev on Rust 1.70 #1580
• NoMetadata error when reading detlatable #1562
• Cannot read delta table: Delta protocol violation #1557
• Update the CODEOWNERS to capture the current reviewers and contributors #1553
• [Python] Incorrect file URIs when partition values contain escape character #1533
• add documentation how to Query Delta natively from datafusion #1485
• Python: write_deltalake to ADLS Gen2 issue #1456
• Partition values that have been url encoded cannot be read when using deltalake #1446
• Error optimizing large table #1419
• Cannot read partitions with special characters (including space) with pyarrow >= 11 #1393
• ImportError: deltalake/_internal.abi3.so: cannot allocate memory in static TLS block #1380
• Invalid JSON in log record missing field schemaString for DLT tables #1302
• Special characters in partition path not handled locally #1299

Merged pull requests:

• chore: bump rust crate version #1675 (rtyler)
• fix: change partitioning schema from large to normal string for pyarrow<12 #1671 (ion-elgreco)
• feat: allow to set large dtypes for the schema check in write_deltalake #1668 (ion-elgreco)
• docs: small consistency update in guide and readme #1666 (ion-elgreco)
• fix: exception string in writer.py #1665 (sebdiem)
• chore: increment python library version #1664 (wjones127)
• docs: fix some typos #1662 (ion-elgreco)
• fix: more consistent handling of partition values and file paths #1661 (roeap)
• docs: add docstring to protocol method #1660 (MrPowers)
• docs: make docs.rs build docs with all features enabled #1658 (simonvandel)
• fix: enable offset listing for s3 #1654 (eeroel)
• chore: fix the incorrect Slack link in our readme #1649 (rtyler)
• fix: compensate for invalid log files created by Delta Live Tables #1647 (rtyler)
• chore: proposed updated CODEOWNERS to allow better review notifications #1646 (rtyler)
• feat: expose min_commit_interval to optimize.compact and optimize.z_order #1645 (ion-elgreco)
• fix: avoid excess listing of log files #1644 (eeroel)
• fix: introduce support for Microsoft OneLake #1642 (rtyler)
• fix: explicitly require chrono 0.4.31 or greater #1641 (rtyler)
• fix: include in-progress row group when calculating in-memory buffer length #1638 (BnMcG)
• chore: relax chrono pin to 0.4 #1635 (houqp)
• chore: update datafusion to 31, arrow to 46 and object_store to 0.7 #1634 (houqp)
• docs: update Readme #1633 (dennyglee)
• chore: pin the chrono dependency #1631 (rtyler)
• feat: pass known file sizes to filesystem in Python #1630 (eeroel)
• feat: implement parsing for the new domainMetadata actions in the commit log #1629 (rtyler)
• ci: fix python release #1624 (wjones127)
• ci: extend azure timeout #1622 (wjones127)
• feat: allow multiple incremental commits in optimize #1621 (kvap)
• fix: change map nullable value to false #1620 (cmackenzie1)
• Introduce the changelog for the last couple releases #1617 (rtyler)
• chore: bump python version to 0.10.2 #1616 (wjones127)
• perf: avoid holding GIL in DeltaFileSystemHandler #1615 (wjones127)
• fix: don't re-encode paths #1613 (wjones127)
• feat: use url parsing from object store #1592 (roeap)
• feat: buffered reading of transaction logs #1549 (eeroel)
• feat: merge operation #1522 (Blajda)
• feat: expose create_checkpoint_for to the public #1514 (haruband)
• docs: update Readme #1440 (roeap)
• refactor: re-organize top level modules #1434 (roeap)
• feat: integrate unity catalog with datafusion #1338 (roeap)

What's Changed

New Features

• feat: expose min_commit_interval to optimize.compact and optimize.z_order by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1645
• feat: allow multiple incremental commits in optimize by @kvap in https://github.com/delta-io/delta-rs/pull/1621
• feat: introduce support for Microsoft OneLake by @rtyler in https://github.com/delta-io/delta-rs/pull/1642

Performance Improvements

• feat: pass known file sizes to filesystem in Python by @eeroel in https://github.com/delta-io/delta-rs/pull/1630
• fix: avoid excess listing of log files by @eeroel in https://github.com/delta-io/delta-rs/pull/1644
• fix: enable offset listing for s3 by @eeroel in https://github.com/delta-io/delta-rs/pull/1654

Other

• chore: update datafusion to 31, arrow to 46 and object_store to 0.7 by @houqp in https://github.com/delta-io/delta-rs/pull/1634
• feat: implement parsing for the new domainMetadata actions in the commit log by @rtyler in https://github.com/delta-io/delta-rs/pull/1629
• feat: integrate unity catalog with datafusion by @roeap in https://github.com/delta-io/delta-rs/pull/1338
• fix: compensate for invalid log files created by Delta Live Tables by @rtyler in https://github.com/delta-io/delta-rs/pull/1647
• docs: add docstring to protocol method by @MrPowers in https://github.com/delta-io/delta-rs/pull/1660
• docs: fix some typos by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1662
• feat: use url parsing from object store by @roeap in https://github.com/delta-io/delta-rs/pull/1592
• chore: proposed updated CODEOWNERS to allow better review notifications by @rtyler in https://github.com/delta-io/delta-rs/pull/1646
• fix: more consistent handling of partition values and file paths by @roeap in https://github.com/delta-io/delta-rs/pull/1661

New Contributors

• @kvap made their first contribution in https://github.com/delta-io/delta-rs/pull/1621

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.10.2...python-v0.11.0

What's Changed

New features

• feat: add restore command in python binding by @loleek in https://github.com/delta-io/delta-rs/pull/1529
• feat: buffered reading of transaction logs by @eeroel in https://github.com/delta-io/delta-rs/pull/1549

Bug fixes

• fix: correct whitespace in delta protocol reader minimum version error message by @polynomialherder in https://github.com/delta-io/delta-rs/pull/1576
• fix: just make pyarrow 12 the max by @wjones127 in https://github.com/delta-io/delta-rs/pull/1603
• fix: support partial statistics in JSON by @CurtHagenlocher in https://github.com/delta-io/delta-rs/pull/1599
• perf: avoid holding GIL in DeltaFileSystemHandler by @wjones127 in https://github.com/delta-io/delta-rs/pull/1615
• fix: change map nullable value to false by @cmackenzie1 in https://github.com/delta-io/delta-rs/pull/1620
• fix: don't re-encode paths by @wjones127 in https://github.com/delta-io/delta-rs/pull/1613

Other

• ci: don't run benchmark in debug mode by @wjones127 in https://github.com/delta-io/delta-rs/pull/1566
• chore: update datafusion to 28 and arrow to 43 by @cmackenzie1 in https://github.com/delta-io/delta-rs/pull/1571
• chore: move deps to [workspace.dependencies] by @cmackenzie1 in https://github.com/delta-io/delta-rs/pull/1575
• fix: remove alpha classifier by @marcelotrevisani in https://github.com/delta-io/delta-rs/pull/1578
• refactor: use pa.table.cast in delta_arrow_schema_from_pandas by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/1573
• feat: add metadata for operations::write::WriteBuilder by @abhimanyusinghgaur in https://github.com/delta-io/delta-rs/pull/1584
• feat: add metadata for deletion vectors by @aersam in https://github.com/delta-io/delta-rs/pull/1583
• refactor: clean up arrow schema defs by @polynomialherder in https://github.com/delta-io/delta-rs/pull/1590
• fix: update python test by @wjones127 in https://github.com/delta-io/delta-rs/pull/1608
• chore: update datafusion to 30, arrow to 45 by @scsmithr in https://github.com/delta-io/delta-rs/pull/1606
• chore: bump python version to 0.10.2 by @wjones127 in https://github.com/delta-io/delta-rs/pull/1616
• ci: extend azure timeout by @wjones127 in https://github.com/delta-io/delta-rs/pull/1622
• ci: fix python release by @wjones127 in https://github.com/delta-io/delta-rs/pull/1624

New Contributors

• @polynomialherder made their first contribution in https://github.com/delta-io/delta-rs/pull/1576
• @marcelotrevisani made their first contribution in https://github.com/delta-io/delta-rs/pull/1578
• @ion-elgreco made their first contribution in https://github.com/delta-io/delta-rs/pull/1573
• @aersam made their first contribution in https://github.com/delta-io/delta-rs/pull/1583
• @CurtHagenlocher made their first contribution in https://github.com/delta-io/delta-rs/pull/1599
• @scsmithr made their first contribution in https://github.com/delta-io/delta-rs/pull/1606
• @eeroel made their first contribution in https://github.com/delta-io/delta-rs/pull/1549

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.10.1...python-v0.10.2

What's Changed

• fix: revert premature merge of an attempted fix for binary column statistics by @rtyler in https://github.com/delta-io/delta-rs/pull/1544
• chore: disable incremental builds in CI for saving space by @rtyler in https://github.com/delta-io/delta-rs/pull/1545
• chore: address some integration test bloat of disk usage for development by @rtyler in https://github.com/delta-io/delta-rs/pull/1552
• chore: increment python version by @wjones127 in https://github.com/delta-io/delta-rs/pull/1542
• docs: port docs to mkdocs by @MrPowers in https://github.com/delta-io/delta-rs/pull/1548
• ci: install newer rust for macos python release by @wjones127 in https://github.com/delta-io/delta-rs/pull/1565
• feat!: bulk delete for vacuum by @Blajda in https://github.com/delta-io/delta-rs/pull/1556
• feat: make find_files public by @yjshen in https://github.com/delta-io/delta-rs/pull/1560
• ci: don't run benchmark in debug mode by @wjones127 in https://github.com/delta-io/delta-rs/pull/1566
• chore: update datafusion to 28 and arrow to 43 by @cmackenzie1 in https://github.com/delta-io/delta-rs/pull/1571
• chore: move deps to [workspace.dependencies] by @cmackenzie1 in https://github.com/delta-io/delta-rs/pull/1575
• fix: correct whitespace in delta protocol reader minimum version error message by @polynomialherder in https://github.com/delta-io/delta-rs/pull/1576
• feat: add restore command in python binding by @loleek in https://github.com/delta-io/delta-rs/pull/1529

New Contributors

• @yjshen made their first contribution in https://github.com/delta-io/delta-rs/pull/1560
• @polynomialherder made their first contribution in https://github.com/delta-io/delta-rs/pull/1576

Full Changelog: https://github.com/delta-io/delta-rs/compare/rust-v0.13.0...rust-v0.14.0

What's Changed

New features

• feat: handle larger z-order jobs with streaming output and spilling by @wjones127 in https://github.com/delta-io/delta-rs/pull/1461
• feat: implement restore operation by @loleek in https://github.com/delta-io/delta-rs/pull/1502
• feat!: bulk delete for vacuum by @Blajda in https://github.com/delta-io/delta-rs/pull/1556

Fixes

• fix(python): match Field signatures by @guilhem-dvr in https://github.com/delta-io/delta-rs/pull/1463
• fix: add sizeInBytes to _last_checkpoint and change size to # of actions by @cmackenzie1 in https://github.com/delta-io/delta-rs/pull/1477
• fix: tiny typo in AggregatedStats by @haruband in https://github.com/delta-io/delta-rs/pull/1516
• fix: handle nulls in file-level stats by @wjones127 in https://github.com/delta-io/delta-rs/pull/1520

Other

• docs: show data catalog options in Python API reference by @omkar-foss in https://github.com/delta-io/delta-rs/pull/1532
• chore: fix mypy failure by @wjones127 in https://github.com/delta-io/delta-rs/pull/1500
• chore: increment python version by @wjones127 in https://github.com/delta-io/delta-rs/pull/1542
• ci: install newer rust for macos python release by @wjones127 in https://github.com/delta-io/delta-rs/pull/1565

New Contributors

• @guilhem-dvr made their first contribution in https://github.com/delta-io/delta-rs/pull/1463
• @haruband made their first contribution in https://github.com/delta-io/delta-rs/pull/1516
• @omkar-foss made their first contribution in https://github.com/delta-io/delta-rs/pull/1532

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.10.0...python-v0.10.1

Full Changelog

Implemented enhancements:

• Add nested struct supports #1518
• Support FixedLenByteArray UUID statistics as a logical scalar #1483
• Exposing create_add in the API #1458
• Update features table on README #1404
• docs(python): show data catalog options in Python API reference #1347
• Add optimization to only list log files starting at a certain name #1252
• Support configuring parquet compression #1235
• parallel processing in Optimize command #1171

Fixed bugs:

• get_add_actions() MAX is not showing complete value #1534
• Can't get stats's minValues in add actions #1515
• Pyarrow is_null filter not working as expected after loading using deltalake #1496
• Can't write to table that uses generated columns #1495
• Json error: Binary is not supported by JSON when writing checkpoint files #1493
• _last_checkpoint size field is incorrect #1468
• Error when Z Ordering a larger dataset #1459
• Timestamp parsing issue #1455
• File options are ignored when writing delta #1444
• Slack Invite Link No Longer Valid #1425
• cleanup_metadata doesn't remove .checkpoint.parquet files #1420
• The test of reading the data from the blob storage located in Azurite container failed #1415
• The test of reading the data from the bucket located in Minio container failed #1408
• Datafusion: unreachable code reached when parsing statistics with missing columns #1374
• vacuum is very slow on Cloudflare R2 #1366

Closed issues:

• Expose Compression Options or WriterProperties for writing to Delta #1469
• Support out-of-core Z-order using DataFusion #1460
• Expose Z-order in Python #1442

Merged pull requests:

• chore: fix the latest clippy warnings with the newer rustc's #1536 (rtyler)
• docs: show data catalog options in Python API reference #1532 (omkar-foss)
• fix: handle nulls in file-level stats #1520 (wjones127)
• feat: add nested struct supports #1519 (haruband)
• fix: tiny typo in AggregatedStats #1516 (haruband)
• refactor: unify with_predicate for delete ops #1512 (Blajda)
• chore: remove deprecated table functions #1511 (roeap)
• chore: update datafusion and related crates #1504 (roeap)
• feat: implement restore operation #1502 (loleek)
• chore: fix mypy failure #1500 (wjones127)
• fix: avoid writing statistics for binary columns to fix JSON error #1498 (ChewingGlass)
• feat(rust): expose WriterProperties method on RecordBatchWriter and DeltaWriter #1497 (theelderbeever)
• feat: add UUID statistics handling #1484 (atefsaw)
• feat: expose create_add to the public #1482 (atefsaw)
• fix: add sizeInBytes to _last_checkpoint and change size to # of actions #1477 (cmackenzie1)
• fix(python): match Field signatures #1463 (guilhem-dvr)
• feat: handle larger z-order jobs with streaming output and spilling #1461 (wjones127)
• chore: increment python version #1449 (wjones127)
• chore: upgrade to arrow 40 and datafusion 26 #1448 (rtyler)
• feat(python): expose z-order in Python #1443 (wjones127)
• ci: prune CI/CD pipelines #1433 (roeap)
• refactor: remove LoadCheckpointError and ApplyLogError #1432 (roeap)
• feat: update writers to include compression method in file name #1431 (Blajda)
• refactor: move checkpoint and errors into separate module #1430 (roeap)
• feat: add z-order optimize #1429 (wjones127)
• fix: casting when data to be written does not match table schema #1427 (Blajda)
• docs: update README.adoc to fix expired Slack link #1426 (dennyglee)
• chore: remove no-longer-necessary build.rs for Rust bindings #1424 (rtyler)
• chore: remove the delta-checkpoint lambda which I have moved to a new repo #1423 (rtyler)
• refactor: rewrite redundant_async_block #1422 (cmackenzie1)
• fix: update cleanup regex to include checkpoint.parquet files #1421 (cmackenzie1)
• docs: update features table in README #1414 (ognis1205)
• fix: get_prune_stats returns homogenous ArrayRef #1413 (cmackenzie1)
• feat: explicit python exceptions #1409 (roeap)
• feat: implement update operation #1390 (Blajda)
• feat: allow concurrent file compaction #1383 (wjones127)

What's Changed

• feat(python): expose z-order in Python by @wjones127 in https://github.com/delta-io/delta-rs/pull/1443
• feat: add z-order optimize by @wjones127 in https://github.com/delta-io/delta-rs/pull/1429
• feat(python): add filters argument to DeltaTable.to_pandas() for filter pushdown by @ognis1205 in https://github.com/delta-io/delta-rs/pull/1349
• feat: add datafusion storage catalog by @roeap in https://github.com/delta-io/delta-rs/pull/1381
• feat: allow concurrent file compaction by @wjones127 in https://github.com/delta-io/delta-rs/pull/1383
• feat: vacuum with concurrent requests by @wjones127 in https://github.com/delta-io/delta-rs/pull/1382
• feat: more efficient parquet writer and more statistics by @wjones127 in https://github.com/delta-io/delta-rs/pull/1397
• feat: explicit python exceptions by @roeap in https://github.com/delta-io/delta-rs/pull/1409
• feat: update writers to include compression method in file name by @Blajda in https://github.com/delta-io/delta-rs/pull/1431
• fix: include stats for all columns (#1223) by @mrjoe7 in https://github.com/delta-io/delta-rs/pull/1342
• fix: add py.typed marker by @SchutteJan in https://github.com/delta-io/delta-rs/pull/1350
• fix: allow user defined config keys by @roeap in https://github.com/delta-io/delta-rs/pull/1365
• fix: add conversion for string for Field::TimestampMicros (#1372) by @cmackenzie1 in https://github.com/delta-io/delta-rs/pull/1373
• perf: improve record batch partitioning by @roeap in https://github.com/delta-io/delta-rs/pull/1396
• chore: type-check friendlier exports by @roeap in https://github.com/delta-io/delta-rs/pull/1407

New Contributors

• @SchutteJan made their first contribution in https://github.com/delta-io/delta-rs/pull/1350
• @cmackenzie1 made their first contribution in https://github.com/delta-io/delta-rs/pull/1373
• @rahulj51 made their first contribution in https://github.com/delta-io/delta-rs/pull/1377

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.9.0...python-v0.10.0

Full Changelog

Implemented enhancements:

• Implement simple delete case #832

Merged pull requests:

• chore: update Rust package version #1346 (rtyler)
• fix: replace deprecated arrow::json::reader::Decoder #1226 (rtyler)
• feat: delete operation #1176 (Blajda)
• feat: add wasbs to known schemes #1345 (iajoiner)
• test: add some missing unit and doc tests for DeltaTablePartition #1341 (rtyler)
• feat: write command improvements #1267 (roeap)
• feat: added support for Databricks Unity Catalog #1331 (nohajc)
• fix: double url encode of partition key #1324 (mrjoe7)

What's Changed

New features

• added support for Databricks Unity Catalog by @nohajc in https://github.com/delta-io/delta-rs/pull/1331
• add optimize command in python binding by @loleek in https://github.com/delta-io/delta-rs/pull/1313
• optimistic transaction protocol by @roeap in https://github.com/delta-io/delta-rs/pull/632
• use new conflict checker in Python by @wjones127 in https://github.com/delta-io/delta-rs/pull/1275
• Add Max Partitions Arg to Write by @ColeMurray in https://github.com/delta-io/delta-rs/pull/1242
• Write support for additional Arrow datatypes by @chitralverma in https://github.com/delta-io/delta-rs/pull/1044
• add package version by @wjones127 in https://github.com/delta-io/delta-rs/pull/1243
• improve err msg on use of non-partitioned column by @marijncv in https://github.com/delta-io/delta-rs/pull/1221
• update incremental after operations by @wjones127 in https://github.com/delta-io/delta-rs/pull/1337

Fixes

• fix: double url encode of partition key by @mrjoe7 in https://github.com/delta-io/delta-rs/pull/1324
• fix: documentation typo fix by @benrutter in https://github.com/delta-io/delta-rs/pull/1332
• fix: allow special characters in storage prefix by @wjones127 in https://github.com/delta-io/delta-rs/pull/1311
• fix: Fixed Documentation for get_add_actions function by @JHibbard in https://github.com/delta-io/delta-rs/pull/1253
• fix: use native-tls for python deltalake releases by @wjones127 in https://github.com/delta-io/delta-rs/pull/1244
• refactor: Simplify the Store Backend Configuration code by @mrjoe7 in https://github.com/delta-io/delta-rs/pull/1265

New Contributors

• @ColeMurray made their first contribution in https://github.com/delta-io/delta-rs/pull/1242
• @JHibbard made their first contribution in https://github.com/delta-io/delta-rs/pull/1253
• @chitralverma made their first contribution in https://github.com/delta-io/delta-rs/pull/1044
• @mrjoe7 made their first contribution in https://github.com/delta-io/delta-rs/pull/1265
• @loleek made their first contribution in https://github.com/delta-io/delta-rs/pull/1313
• @nohajc made their first contribution in https://github.com/delta-io/delta-rs/pull/1331

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.8.1...python-v0.9.0

Full Changelog

Implemented enhancements:

• Support Optimize on non-append-only tables #1125

Fixed bugs:

• DataFusion integration incorrectly handles partition columns defined "first" in schema #1168
• Datafusion: SQL projection returns wrong column for partitioned data #1292
• Unable to query partitioned tables #1291

Merged pull requests:

• chore: add deprecation notices for commit logic on DeltaTable #1323 (roeap)
• fix: handle local paths on windows #1322 (roeap)
• fix: scan partitioned tables with datafusion #1303 (roeap)
• fix: allow special characters in storage prefix #1311 (wjones127)
• feat: upgrade to Arrow 37 and Datafusion 23 #1314 (rtyler)
• Hide the parquet/json feature behind our own JSON feature #1307 (rtyler)
• Enable the json feature for the parquet crate #1300 (rtyler)

Full Changelog

Implemented enhancements:

• hdfs support #300
• Add decimal primitive type to document #1280
• Improve error message when filtering on non-existant partition columns #1218

Fixed bugs:

• Datafusion table provider: issues with timestamp types #441
• Not matching column names when creating a RecordBatch from MapArray #1257
• All stores created using DeltaObjectStore::new have an identical object_store_url #1188

Merged pull requests:

• Upgrade datafusion to 22 which brings arrow upgrades with it #1249 (rtyler)
• chore: df / arrow changes after update #1288 (roeap)
• feat: read schema from parquet files in datafusion scans #1266 (roeap)
• HDFS storage support via datafusion-objectstore-hdfs #1279 (iajoiner)
• Add description of decimal primitive to SchemaDataType #1281 (ognis1205)
• Fix names and nullability when creating RecordBatch from MapArray #1258 (balbok0)
• Simplify the Store Backend Configuration code #1265 (mrjoe7)
• feat: optimistic transaction protocol #632 (roeap)
• Write support for additional Arrow datatypes #1044(chitralverma)
• Unique delta object store url #1212 (gruuya)
• improve err msg on use of non-partitioned column #1221 (marijncv)