Bot releases are hidden (Show)

delta-rs - python-v0.19.2: objectstore conditional put Latest Release

Published by ion-elgreco about 1 month ago

New features

perf: conditional put for default log store (e.g. azure, gcs, minio, cloudflare) by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2813
feat: make Add::get_json_stats public by @gruuya in https://github.com/delta-io/delta-rs/pull/2822
refactor(python): add pymergebuilder by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2823
feat: public method to get partitions for DeltaTable (#2671) by @omkar-foss in https://github.com/delta-io/delta-rs/pull/2816
feat(rust): add operationMetrics to WRITE by @gavinmead in https://github.com/delta-io/delta-rs/pull/2838

Bug Fixes

fix: enable feature flags to which deltalake-core build tokio with enable_io by @rtyler in https://github.com/delta-io/delta-rs/pull/2803
fix(rust): set token provider explicitly by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2817
fix(python, rust): allow in pushdowns in early_filter by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2807
fix: use table config target file size, expose target_file_size in python by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2811

Other Changes

chore(python): raise not implemented in from_data_catalog by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2799
docs: added WriterProperties documentation by @sherlockbeard in https://github.com/delta-io/delta-rs/pull/2804
chore(python): remove deprecated or duplicated functions by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2801
test(python): fix optimize call in benchmark by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2812
docs: fix documentation about max_spill_size by @junhl in https://github.com/delta-io/delta-rs/pull/2835
refactor(python): post_commit_hook_properties derive by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2824
docs: fix docstring of set_table_properties by @astrojuanlu in https://github.com/delta-io/delta-rs/pull/2820
chore: enable codecov reporting by @rtyler in https://github.com/delta-io/delta-rs/pull/2836
chore(aws): use backon to replace backoff by @Xuanwo in https://github.com/delta-io/delta-rs/pull/2840
chore: update python by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2845
docs: concurrent writes permission missing by @poguez in https://github.com/delta-io/delta-rs/pull/2846

New Contributors

@junhl made their first contribution in https://github.com/delta-io/delta-rs/pull/2835
@Xuanwo made their first contribution in https://github.com/delta-io/delta-rs/pull/2840
@gavinmead made their first contribution in https://github.com/delta-io/delta-rs/pull/2838
@poguez made their first contribution in https://github.com/delta-io/delta-rs/pull/2846

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.19.1...python-v0.19.2

delta-rs - python-v0.19.1: separate IO runtime

Published by ion-elgreco 2 months ago

New features

feat: configurable IO runtime by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2789
feat(python, rust): added statistics_truncate_length in WriterProperties by @sherlockbeard in https://github.com/delta-io/delta-rs/pull/2784
feat(python, rust): add ColumnProperties And rework in python WriterProperties by @sherlockbeard in https://github.com/delta-io/delta-rs/pull/2793

Bug Fixes

fix: pin maturin verison by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2778
fix(rust): max_spill_size default value by @mrjsj in https://github.com/delta-io/delta-rs/pull/2795
fix: trim trailing slash in url storage options (#2656) by @omkar-foss in https://github.com/delta-io/delta-rs/pull/2775

Other Changes

chore: update the changelog with the 0.19.0 release by @rtyler in https://github.com/delta-io/delta-rs/pull/2774
style: more consistent imports by @roeap in https://github.com/delta-io/delta-rs/pull/2786
chore: remove some file_actions call sites by @roeap in https://github.com/delta-io/delta-rs/pull/2787
chore(deps): update sqlparser requirement from 0.49 to 0.50 by @dependabot in https://github.com/delta-io/delta-rs/pull/2792

New Contributors

@mrjsj made their first contribution in https://github.com/delta-io/delta-rs/pull/2795

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.19.0...python-v0.19.1

delta-rs - python-v0.19.0: complete CDF support, add column operation, faster MERGE

Published by ion-elgreco 2 months ago

Breaking changes!

Default writer engine has changed to rust. Replace your partition_filters with a predicate (sql) instead. PyArrow engine is deprecated now, and will be removed in v1.0.

Highlights

CDF support in write_deltalake, delete, and merge operation
Expired logs cleanup during post-commit. Can be disabled with delta.enableExpiredLogCleanup = false
Improved MERGE performance by using predicate non-partition columns min/max for prefiltering
ADD column operation
Speed up log parsing

Performance improvements

perf: apply projection when reading checkpoint parquet by @alexwilcoxson-rel in https://github.com/delta-io/delta-rs/pull/2717
perf: grab file size in rust by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2734
feat: improve merge performance by using predicate non-partition columns min/max for prefiltering by @JonasDev1 in https://github.com/delta-io/delta-rs/pull/2513
perf: early stop if all values in arr are null by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2764

New features

feat(python, rust): cdc write-support for delete operation by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2721
feat(python, rust): cdc write-support for overwrite and replacewhere writes by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2722
feat: introduce CDC generation for merge operations by @rtyler in https://github.com/delta-io/delta-rs/pull/2747
feat: use logical plan in delete, delta planner refactoring by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2725
feat: use logical plan in update, refactor/simplify CDCTracker by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2727
feat(python, rust): arrow large/view types passthrough, rust default engine by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2738
feat(python, rust): cleanup expired logs post-commit hook by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2459
feat(python, rust): add column operation by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2562
feat(python): handle PyCapsule interface objects in write_deltalake by @kylebarron in https://github.com/delta-io/delta-rs/pull/2534
feat(rust): fix size_in_bytes in last_checkpoint_ to i64 by @sherlockbeard in https://github.com/delta-io/delta-rs/pull/2649
feat(rust,python): cast each parquet file to delta schema by @HawaiianSpork in https://github.com/delta-io/delta-rs/pull/2615
feat: support userMetadata in CommitInfo by @jkylling in https://github.com/delta-io/delta-rs/pull/2670
feat(python, rust): add projection in CDF reads by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2704
feat(python): add DeltaTable.is_deltatable static method (#2662) by @omkar-foss in https://github.com/delta-io/delta-rs/pull/2715
feat: improved test fixtures by @roeap in https://github.com/delta-io/delta-rs/pull/2749
feat: fail fast on forked process by @Tom-Newton in https://github.com/delta-io/delta-rs/pull/2765
feat: restore the TryFrom for DeltaTablePartition by @rtyler in https://github.com/delta-io/delta-rs/pull/2767
feat: more economic data skipping with datafusion by @roeap in https://github.com/delta-io/delta-rs/pull/2772

Bug Fixes

fix(rust): inconsistent order of partitioning columns (#2494) by @aditanase in https://github.com/delta-io/delta-rs/pull/2614
fix(rust,python): checkpoint with column nullable false by @sherlockbeard in https://github.com/delta-io/delta-rs/pull/2680
fix: update delta kernel version by @jeppe742 in https://github.com/delta-io/delta-rs/pull/2685
fix(python): empty dataset fix for "pyarrow" engine by @sherlockbeard in https://github.com/delta-io/delta-rs/pull/2689
fix: ensure DataFusion SessionState Parquet options are applied to DeltaScan by @alexwilcoxson-rel in https://github.com/delta-io/delta-rs/pull/2702
fix(python, rust): use url encoder when encoding partition values by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2705
fix(python, rust): use input schema to get correct schema in cdf reads by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2723
fix: change arrow map root name to follow with parquet root name by @sclmn in https://github.com/delta-io/delta-rs/pull/2538
fix: schema adapter doesn't map partial batches correctly by @alexwilcoxson-rel in https://github.com/delta-io/delta-rs/pull/2735
fix: optimize Spark written tables by @rtyler in https://github.com/delta-io/delta-rs/pull/1650
fix(python, rust): cdc in writer not creating inserts by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2751
fix(python, rust): don't flatten fields during cdf read by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2763
fix: column parsing to include nested columns and enclosing char by @gtrawinski in https://github.com/delta-io/delta-rs/pull/2737

Other Changes

chore: missed one macos runner reference in actions by @rtyler in https://github.com/delta-io/delta-rs/pull/2645
chore: add a reproduction case for merge failures with struct by @rtyler in https://github.com/delta-io/delta-rs/pull/2644
ci: update CODEOWNERS by @hntd187 in https://github.com/delta-io/delta-rs/pull/2650
chore: increase subcrate versions by @rtyler in https://github.com/delta-io/delta-rs/pull/2648
docs: fix bullets on hdfs docs by @Kimahriman in https://github.com/delta-io/delta-rs/pull/2653
docs: improve navigation fixes by @avriiil in https://github.com/delta-io/delta-rs/pull/2660
docs: add integration docs for s3 backend by @avriiil in https://github.com/delta-io/delta-rs/pull/2658
chore: bump ruff to 0.5.2 by @fpgmaas in https://github.com/delta-io/delta-rs/pull/2673
chore: enable RUF ruleset for ruff by @fpgmaas in https://github.com/delta-io/delta-rs/pull/2677
chore: pin ruff and mypy versions in the lint stage in the CI pipeline by @fpgmaas in https://github.com/delta-io/delta-rs/pull/2679
chore: update README.md by @veronewra in https://github.com/delta-io/delta-rs/pull/2684
chore: create separate action to setup python and rust in the cicd pipeline by @fpgmaas in https://github.com/delta-io/delta-rs/pull/2687
chore: add test coverage command to Makefile by @fpgmaas in https://github.com/delta-io/delta-rs/pull/2688
chore: improve contributing.md by @fpgmaas in https://github.com/delta-io/delta-rs/pull/2672
chore: remove stale code for conditional import of Literal by @fpgmaas in https://github.com/delta-io/delta-rs/pull/2676
chore: remove references to black from the project by @fpgmaas in https://github.com/delta-io/delta-rs/pull/2674
chore: refactor write_deltalake in writer.py by @fpgmaas in https://github.com/delta-io/delta-rs/pull/2695
chore: upgrade to datafusion 40 by @rtyler in https://github.com/delta-io/delta-rs/pull/2661
chore: prepare python release 0.18.3 by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2707
chore: enabling actions for merge groups by @rtyler in https://github.com/delta-io/delta-rs/pull/2718
chore(deps): update sqlparser requirement from 0.47 to 0.49 by @dependabot in https://github.com/delta-io/delta-rs/pull/2714
chore: try an alternative docke compose invocation syntax by @rtyler in https://github.com/delta-io/delta-rs/pull/2724
chore(deps): update which requirement from 4 to 6 by @dependabot in https://github.com/delta-io/delta-rs/pull/2730
chore: update changelog and versions for next release by @rtyler in https://github.com/delta-io/delta-rs/pull/2740
chore: add to code_owner crates by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2741
chore: update delta_kernel to 0.3.0 by @alexwilcoxson-rel in https://github.com/delta-io/delta-rs/pull/2742
docs: fix broken link in docs by @astrojuanlu in https://github.com/delta-io/delta-rs/pull/2746
chore: upgrade to datafusion 41 by @rtyler in https://github.com/delta-io/delta-rs/pull/2761
chore: prepare the next notable release of 0.19.0 by @rtyler in https://github.com/delta-io/delta-rs/pull/2768
chore: fix a bunch of clippy lints and re-enable tests by @rtyler in https://github.com/delta-io/delta-rs/pull/2773

New Contributors

@aditanase made their first contribution in https://github.com/delta-io/delta-rs/pull/2614
@fpgmaas made their first contribution in https://github.com/delta-io/delta-rs/pull/2673
@kylebarron made their first contribution in https://github.com/delta-io/delta-rs/pull/2534
@veronewra made their first contribution in https://github.com/delta-io/delta-rs/pull/2684
@jeppe742 made their first contribution in https://github.com/delta-io/delta-rs/pull/2685
@sclmn made their first contribution in https://github.com/delta-io/delta-rs/pull/2538
@astrojuanlu made their first contribution in https://github.com/delta-io/delta-rs/pull/2746
@gtrawinski made their first contribution in https://github.com/delta-io/delta-rs/pull/2737

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.18.2...python-v0.19.0

delta-rs - python-v0.18.2: HDFS support

Published by ion-elgreco 4 months ago

New features

feat(#2597): allow pyarrow.dataset.Expression in filters kwarg by @giacomorebecchi in https://github.com/delta-io/delta-rs/pull/2600
feat(rust, python): add HDFS support via hdfs-native package by @Kimahriman in https://github.com/delta-io/delta-rs/pull/2612
feat: report DataFusion metrics for DeltaScan by @alexwilcoxson-rel in https://github.com/delta-io/delta-rs/pull/2617

Bug Fixes

fix: enable parquet pushdown for DeltaScan via TableProvider impl for DeltaTable (rebase) by @rtyler in https://github.com/delta-io/delta-rs/pull/2637
fix(rust, python): fix writing empty structs when creating checkpoint by @sherlockbeard in https://github.com/delta-io/delta-rs/pull/2627
fix(python): fixed large_dtype to schema convert by @sherlockbeard in https://github.com/delta-io/delta-rs/pull/2635
fix(rust, python): fix merge schema with overwrite by @sherlockbeard in https://github.com/delta-io/delta-rs/pull/2623
fix(python): constrain multipart upload size to fixed length by @abhiaagarwal in https://github.com/delta-io/delta-rs/pull/2606
fix: update changelog by @rtyler in https://github.com/delta-io/delta-rs/pull/2599

Other Changes

chore: migrate to pyo3 Bounds API by @abhiaagarwal in https://github.com/delta-io/delta-rs/pull/2596
chore(deps): update dashmap requirement from 5 to 6 by @dependabot in https://github.com/delta-io/delta-rs/pull/2641
chore: remove macos builders from pull request flow by @rtyler in https://github.com/delta-io/delta-rs/pull/2638
docs: add Daft writer by @avriiil in https://github.com/delta-io/delta-rs/pull/2594
chore: fix documentation generation with a pin of griffe by @rtyler in https://github.com/delta-io/delta-rs/pull/2636
chore: bump python 0.18.2 by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2621
chore: implement regression test for push down panic by @rtyler in https://github.com/delta-io/delta-rs/pull/2604
docs: fix typo by @avriiil in https://github.com/delta-io/delta-rs/pull/2603
test: reintroduce azurite SAS integration tests by @giacomorebecchi in https://github.com/delta-io/delta-rs/pull/2598

New Contributors

@giacomorebecchi made their first contribution in https://github.com/delta-io/delta-rs/pull/2598
@Kimahriman made their first contribution in https://github.com/delta-io/delta-rs/pull/2612
@sherlockbeard made their first contribution in https://github.com/delta-io/delta-rs/pull/2623

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.18.1...python-v0.18.2

delta-rs - python-v0.18.1

Published by ion-elgreco 4 months ago

New features

feat: add custom dynamodb endpoint configuration by @hnaoto in https://github.com/delta-io/delta-rs/pull/2575
chore: bump to datafusion 39, arrow 52, pyo3 0.21 by @abhiaagarwal in https://github.com/delta-io/delta-rs/pull/2581

Bug Fixes

chore: bump macOS runners, maybe resolve import error by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2588

Other Changes

docs: improve S3 access docs by @avriiil in https://github.com/delta-io/delta-rs/pull/2589
chore: expose files_by_partition to public api by @edmondop in https://github.com/delta-io/delta-rs/pull/2533

New Contributors

@abhiaagarwal made their first contribution in https://github.com/delta-io/delta-rs/pull/2581
@edmondop made their first contribution in https://github.com/delta-io/delta-rs/pull/2533

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.18.0...python-v0.18.1

delta-rs - python-v0.18.0: CDC for update operation, added `set table properties` operation

Published by ion-elgreco 4 months ago

New features

feat: adopt kernel schema types by @roeap in https://github.com/delta-io/delta-rs/pull/2495
feat: add stats to convert-to-delta operation by @gruuya in https://github.com/delta-io/delta-rs/pull/2491
feat(python, rust): add set table properties operation by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2264
feat: implement transaction identifiers - continued by @roeap in https://github.com/delta-io/delta-rs/pull/2539
feat: introduce CDC write-side support for the Update operations by @rtyler in https://github.com/delta-io/delta-rs/pull/2486

Bug Fixes

fix(rust, python): fixed differences in storage options between log and object stores by @mightyshazam in https://github.com/delta-io/delta-rs/pull/2500
fix: enable field_with_name to support nested fields with '.' delimiter by @alexwilcoxson-rel in https://github.com/delta-io/delta-rs/pull/2519
fix(python): release GIL on most operations by @adriangb in https://github.com/delta-io/delta-rs/pull/2512
fix: clippy warnings by @imor in https://github.com/delta-io/delta-rs/pull/2548
fix: remove deprecated overwrite_schema configuration which has incorrect behavior by @rtyler in https://github.com/delta-io/delta-rs/pull/2554
fix: update deltalake crate examples for crate layout and TimestampNtz by @jhoekx in https://github.com/delta-io/delta-rs/pull/2559
fix: consistently use raise_if_key_not_exists in CreateBuilder by @vegarsti in https://github.com/delta-io/delta-rs/pull/2569
fix: cast support fields nested in lists and maps by @HawaiianSpork in https://github.com/delta-io/delta-rs/pull/2541

Other Changes

docs: fix typo by @avriiil in https://github.com/delta-io/delta-rs/pull/2508
chore: tidying up builds without datafusion feature and clippy by @rtyler in https://github.com/delta-io/delta-rs/pull/2516
chore: fixing some clips by @rtyler in https://github.com/delta-io/delta-rs/pull/2521
fix: msrv in workspace by @roeap in https://github.com/delta-io/delta-rs/pull/2524
feat(rust): make PartitionWriter public by @adriangb in https://github.com/delta-io/delta-rs/pull/2525
docs: improve daft integration docs by @avriiil in https://github.com/delta-io/delta-rs/pull/2496
chore: bump python 0.17.5 by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2531
chore(deps): update itertools requirement from 0.12 to 0.13 by @dependabot in https://github.com/delta-io/delta-rs/pull/2526
docs: dask write syntax fix by @avriiil in https://github.com/delta-io/delta-rs/pull/2543
docs: pull delta from conda not pip by @avriiil in https://github.com/delta-io/delta-rs/pull/2535
docs: clarify locking mechanism requirement for S3 by @inigohidalgo in https://github.com/delta-io/delta-rs/pull/2558
chore(deps): update sqlparser requirement from 0.46 to 0.47 by @dependabot in https://github.com/delta-io/delta-rs/pull/2563
docs: dt.delete add context + api docs link by @avriiil in https://github.com/delta-io/delta-rs/pull/2560

New Contributors

@imor made their first contribution in https://github.com/delta-io/delta-rs/pull/2548
@inigohidalgo made their first contribution in https://github.com/delta-io/delta-rs/pull/2558
@vegarsti made their first contribution in https://github.com/delta-io/delta-rs/pull/2565
@HawaiianSpork made their first contribution in https://github.com/delta-io/delta-rs/pull/2541

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.17.4...python-v0.18.0

delta-rs - python-v0.17.4: stats collection according config

Published by ion-elgreco 5 months ago

New features

feat(python): add parameter to DeltaTable.to_pyarrow_dataset() by @adriangb in https://github.com/delta-io/delta-rs/pull/2465
feat(python, rust): respect column stats collection configurations by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2428

Bug Fixes

fix(rust): implement abort commit for S3DynamoDBLogStore by @PeterKeDer in https://github.com/delta-io/delta-rs/pull/2452
fix(python, rust): use new schema for stats parsing instead of old by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2480
fix: check to see if the file exists before attempting to rename by @rtyler in https://github.com/delta-io/delta-rs/pull/2482
fix(rust): unable to read delta table when table contains both null and non-null add stats by @yjshen in https://github.com/delta-io/delta-rs/pull/2476
fix(python, rust): region lookup wasn't working correctly for dynamo by @mightyshazam in https://github.com/delta-io/delta-rs/pull/2488
fix: return unsupported error for merging schemas in the presence of partition columns by @emcake in https://github.com/delta-io/delta-rs/pull/2469
fix(python): reuse state in to_pyarrow_dataset by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2485

Other Changes

chore(deps): update sqlparser requirement from 0.44 to 0.46 by @dependabot in https://github.com/delta-io/delta-rs/pull/2483
test: add test for concurrent checkpoint during table load by @alexwilcoxson-rel in https://github.com/delta-io/delta-rs/pull/2151

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.17.3...python-v0.17.4

delta-rs - python-v0.17.3: CDF read support

Published by ion-elgreco 6 months ago

New features

feat(rust): advance state in post commit by @ion-elgreco in #2396
feat: cdf reader for delta tables by @hntd187 in #2048
feat(python, rust): add OBJECT_STORE_CONCURRENCY_LIMIT setting for ObjectStoreFactory by @zZKato in #2458

Bug Fixes

fix(rust): handle 429 from GCS by @adriangb in #2454

Other changes

chore(rust): bump arrow v51 and datafusion v37.1 by @lasantosr in #2395

New Contributors

@adriangb made their first contribution in #2454
@zZKato made their first contribution in #2458

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.17.2...python-v0.17.3

delta-rs - rust-v0.17.3

Published by rtyler 6 months ago

rust-v0.17.3 (2024-05-01)

Full Changelog

Implemented enhancements:

Limit concurrent ObjectStore access to avoid resource limitations in constrained environments #2457
How to get a DataFrame in Rust? #2404
Allow checkpoint creation when partion column is "timestampNtz " #2381
is there a way to make writing timestamp_ntz optional #2339
Update arrow dependency #2328
Release GIL in deltalake.write_deltalake #2234
Unable to retrieve custom metadata from tables in rust #2153
Refactor commit interface to be a Builder #2131

Fixed bugs:

Handle rate limiting during write contention #2451
regression : delta.logRetentionDuration don't seems to be respected #2447
Issue writing to mounted storage in AKS using delta-rs library #2445
TableMerger - when_matched_delete() fails when Column names contain special characters #2438
Generic DeltaTable error: External error: Arrow error: Invalid argument error: arguments need to have the same data type - while merge data in to delta table #2423
Merge on predicate throw error on date colum: Unable to convert expression to string #2420
Writing Tables with Append mode errors if the schema metadata is different #2419
Logstore issues on AWS Lambda #2410
Datafusion timestamp type doesn't respect delta lake schema #2408
Compacting produces smaller row groups than expected #2386
ValueError: Partition value cannot be parsed from string. #2380
Very slow s3 connection after 0.16.1 #2377
Merge update+insert truncates a delta table if the table is big enough #2362
Do not add readerFeatures or writerFeatures keys under checkpoint files if minReaderVersion or minWriterVersion do not satisfy the requirements #2360
Create empty table failed on rust engine #2354
Getting error message when running in lambda: message: "Too many open files" #2353
Temporary files filling up _delta_log folder - increasing table load time #2351
compact fails with merged schemas #2347
Cannot merge into table partitioned by date type column on 0.16.3 #2344
Merge breaks using logical datatype decimal128 #2343
Decimal types are not checked against max precision/scale at table creation #2331
Merge update+insert truncates a delta table #2320
Extract add.stats_parsed with wrong type #2312
Process fails without error message when executing merge #2310
delta_rs don't seems to respect the row group size #2309
Auth error when running inside VS Code #2306
Unable to read deltatables with binary columns: Binary is not supported by JSON #2302
Schema evolution not coercing with Large arrow types #2298
Panic in deltalake_core::kernel::snapshot::log_segment::list_log_files_with_checkpoint::{{closure}} #2290
Checkpoint does not preserve reader and writer features for the table protocol. #2288
Z-Order with larger dataset resulting in memory error #2284
Successful writes return error when using concurrent writers #2279
Rust writer should raise when decimal types are incompatible (currently writers and puts table in invalid state) #2275
Generic DeltaTable error: Version mismatch with new schema merge functionality in AWS S3 #2262
DeltaTable is not resilient to corrupted checkpoint state #2258
Inconsistent units of time #2256
Partition column comparison is an assertion rather than if block with raise exception #2242
Unable to merge column names starting from numbers #2230
Merging to a table with multiple distinct partitions in parallel fails #2227
cleanup_metadata not respecting custom logRetentionDuration #2180
Merge predicate fails with a field with a space #2167
When_matched_update causes records to be lost with explicit predicate #2158
Merge execution time grows exponetially with the number of column #2107
_internal.DeltaError when merging #2084

delta-rs - python-v0.17.2

Published by ion-elgreco 6 months ago

What's Changed

chore: introduce the Operation trait to enforce consistency between operations by @rtyler in https://github.com/delta-io/delta-rs/pull/2435
fix(python): reuse table state in write engine by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2453

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.17.1...python-v0.17.2

delta-rs - python-v0.17.1

Published by ion-elgreco 6 months ago

Bug Fixes

fix(python, rust): use from_name during column projection creation by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2441
fix(python, rust): check timestamp_ntz in nested fields, add check_can_write in pyarrow writer by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2443
fix(python, rust): remove imds calls from profile auth and region by @mightyshazam in https://github.com/delta-io/delta-rs/pull/2442

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.17.0...python-v0.17.1

delta-rs - python-v0.17.0: checkpoint hook

Published by ion-elgreco 6 months ago

New features

feat(rust): post commit hook (v2), create checkpoint hook by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2391
feat: added configuration variables to handle EC2 metadata service by @mightyshazam in https://github.com/delta-io/delta-rs/pull/2385
feat: lazy static runtime in python by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2424
feat: implement repartitioned for DeltaScan by @jkylling in https://github.com/delta-io/delta-rs/pull/2421

Bug Fixes

fix(python, rust): expr parsing date/timestamp by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2357
fix(rust): remove flush after writing every batch by @PeterKeDer in https://github.com/delta-io/delta-rs/pull/2387
fix: return error when checkpoints and metadata get out of sync by @esarili in https://github.com/delta-io/delta-rs/pull/2406
fix: time travel when checkpointed and logs removed by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2389
fix(rust): timestamp deserialization format, missing type by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2383
fix(rust): stats_parsed has different number of records with stats by @yjshen in https://github.com/delta-io/delta-rs/pull/2405
fix(python): load_as_version with datetime object with no timezone specified by @t1g0rz in https://github.com/delta-io/delta-rs/pull/2429
fix(python,rust): missing remove actions during create_or_replace specified by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2437

Other Changes

chore: bump chrono by @universalmind303 in https://github.com/delta-io/delta-rs/pull/2372
docs: document required aws permissions by @ale-rinaldi in https://github.com/delta-io/delta-rs/pull/2393
docs: add Daft integration by @avriiil in https://github.com/delta-io/delta-rs/pull/2402

New Contributors

@PeterKeDer made their first contribution in https://github.com/delta-io/delta-rs/pull/2387
@ale-rinaldi made their first contribution in https://github.com/delta-io/delta-rs/pull/2393
@esarili made their first contribution in https://github.com/delta-io/delta-rs/pull/2406
@jkylling made their first contribution in https://github.com/delta-io/delta-rs/pull/2421
@t1g0rz made their first contribution in https://github.com/delta-io/delta-rs/pull/2429

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.16.4...python-v0.17.0

delta-rs - python-v0.16.4

Published by ion-elgreco 7 months ago

Bug Fixes

fix(python): wrong batch size by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2314
fix(rust): raise schema mismatch when decimal is not subset by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2330
fix: make struct fields nullable in stats schema by @qinix in https://github.com/delta-io/delta-rs/pull/2346
fix: remove tmp files in cleanup_metadata by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2356
fix(python,rust): optimize compact on schema evolved table by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2358
fix: add config for parquet pushdown on delta scan by @Blajda in https://github.com/delta-io/delta-rs/pull/2364
feat(rust): derive Copy on some public enums by @lasantosr in https://github.com/delta-io/delta-rs/pull/2329
fix: add snappy compression on checkpoint files by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2365

Other Changes

chore(rust): bump datafusion to 36 by @universalmind303 in https://github.com/delta-io/delta-rs/pull/2249
chore: bump python 0.16.4 by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2371

New Contributors

@lasantosr made their first contribution in https://github.com/delta-io/delta-rs/pull/2329

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.16.3...python-v0.16.4

delta-rs - python-v0.16.3

Published by ion-elgreco 7 months ago

New features

feat: logical Node for find files by @hntd187 in https://github.com/delta-io/delta-rs/pull/2194

Bug Fixes

fix: try to fix timeouts by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2318
fix: handle conflict checking in optimize correctly by @emcake in https://github.com/delta-io/delta-rs/pull/2208
fix: merge concurrency control by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2324
fix: merge pushdown handling by @Blajda in https://github.com/delta-io/delta-rs/pull/2326
fix(rust): serialize MetricDetails from compaction runs to a string by @liamphmurphy in https://github.com/delta-io/delta-rs/pull/2317
fix(rust): adhere to protocol for Decimal by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2332

Other Changes

chore: object store 0.9.1 by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2311
docs: add example in to_pyarrow_dataset by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2315
Revert 2291 merge predicate fix by @Blajda in https://github.com/delta-io/delta-rs/pull/2323

New Contributors

@liamphmurphy made their first contribution in https://github.com/delta-io/delta-rs/pull/2317

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.16.2...python-v0.16.3

delta-rs - python-v0.16.2

Published by ion-elgreco 7 months ago

Bug Fixes

fix: schema evolution not coercing with large arrow types by @aersam in https://github.com/delta-io/delta-rs/pull/2305
fix: clean up some non-datafusion builds by @rtyler in https://github.com/delta-io/delta-rs/pull/2303
fix: checkpoint features format below v3,7 by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2307
fix: merge predicate for concurrent writes by @JonasDev1 in https://github.com/delta-io/delta-rs/pull/2291
fix(rust): add missing chrono-tz feature by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2295

Other Changes

docs: fix typo by @LauH1987 in https://github.com/delta-io/delta-rs/pull/2300
docs: make replaceWhere example compile by @LauH1987 in https://github.com/delta-io/delta-rs/pull/2299

New Contributors

@LauH1987 made their first contribution in https://github.com/delta-io/delta-rs/pull/2300

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.16.1...python-v0.16.2

delta-rs - python-v0.16.2

Published by ion-elgreco 7 months ago

Bug Fixes

fix: clean up some non-datafusion builds by @rtyler in https://github.com/delta-io/delta-rs/pull/2303
fix: checkpoint features format below v3,7 by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2307
fix: merge predicate for concurrent writes by @JonasDev1 in https://github.com/delta-io/delta-rs/pull/2291
fix(rust): add missing chrono-tz feature by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2295

Other Changes

docs: fix typo by @LauH1987 in https://github.com/delta-io/delta-rs/pull/2300
docs: make replaceWhere example compile by @LauH1987 in https://github.com/delta-io/delta-rs/pull/2299

New Contributors

@LauH1987 made their first contribution in https://github.com/delta-io/delta-rs/pull/2300

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.16.1...python-v0.16.2

delta-rs - python-v0.16.1

Published by ion-elgreco 7 months ago

New features

feat(python): release GIL in the write_deltalake function by @franz101 in https://github.com/delta-io/delta-rs/pull/2257

Bug Fixes

fix(rust): typo deletionvectors by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2251
fix: fixes panic on empty write by @aersam in https://github.com/delta-io/delta-rs/pull/2254
fix(rust): make interval parsing compatible with plural form by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2250
fix(#2256): use consistent units of time by @cmackenzie1 in https://github.com/delta-io/delta-rs/pull/2260
fix(python): always encapsulate column names in backticks in _all functions by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2271
fix(rust): read only checkpoints that match _last_checkpoint version by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2270
fix: replace assert and AssertionError with appropriate exceptions by @joe-sharman in https://github.com/delta-io/delta-rs/pull/2286
fix(python, rust): prevent table scan returning large arrow dtypes by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2274
fix: compatible to write to local file systems that do not support hard link by @RobinLin666 in https://github.com/delta-io/delta-rs/pull/1868
fix(rust): features not maintained in protocol after checkpoint @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2293

Other Changes

refactor!: use builder for commit interface by @Blajda in https://github.com/delta-io/delta-rs/pull/2154
docs: update comment about r2 requiring locks by @cmackenzie1 in https://github.com/delta-io/delta-rs/pull/2261
docs: create Dagster integration page by @avriiil in https://github.com/delta-io/delta-rs/pull/2159
chore!: replace rusoto with AWS SDK by @mightyshazam in https://github.com/delta-io/delta-rs/pull/2243
docs: use dagster deltalake polars library by @avriiil in https://github.com/delta-io/delta-rs/pull/2263
fix: add .venv to .gitignore by @gacharya in https://github.com/delta-io/delta-rs/pull/2268
docs: fix typo in delta-lake-polars.md by @vladdoster in https://github.com/delta-io/delta-rs/pull/2285
chore: update the changelog for rust-v0.17.1 by @rtyler in https://github.com/delta-io/delta-rs/pull/2259

New Contributors

@franz101 made their first contribution in https://github.com/delta-io/delta-rs/pull/2257
@vladdoster made their first contribution in https://github.com/delta-io/delta-rs/pull/2285
@joe-sharman made their first contribution in https://github.com/delta-io/delta-rs/pull/2286
@RobinLin666 made their first contribution in https://github.com/delta-io/delta-rs/pull/1868

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.16.0...python-v0.16.1

delta-rs - python-v0.16.0: schema evolution, timestampNtz, faster MERGE, drop constraints

Published by ion-elgreco 8 months ago

Performance improvements

perf: directly create projection instead of using DataFrame::with_column by @emcake in https://github.com/delta-io/delta-rs/pull/2222

Breaking changes

This version introduces timestampNtz datatype, this means if your writer before wrote timestamp with no timezones to a timestamp column, this will now fail. The new behavior is that you can only write timestamps with UTC time zone to timestamp primitive type.

New features

feat: implement string representation for PartitionFilter by @sonhmai in https://github.com/delta-io/delta-rs/pull/2183
feat(rust, python): add drop constraint operation by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2071
feat: merge schema support for the write operation and Python by @aersam @rtyler in https://github.com/delta-io/delta-rs/pull/2246
feat(python, rust): timestampNtz support by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2236

Bug Fixes

feat: add comment to explain why assert has failed and show state by @braaannigan in https://github.com/delta-io/delta-rs/pull/2179
fix: removed panic in method by @mightyshazam in https://github.com/delta-io/delta-rs/pull/2185
fix: correct map field names by @emcake in https://github.com/delta-io/delta-rs/pull/2182
fix: add data_type and nullable to StructField hash (#2045) by @sonhmai in https://github.com/delta-io/delta-rs/pull/2190
fix: is_commit_file should only catch commit jsons by @emcake in https://github.com/delta-io/delta-rs/pull/2213
fix(python): sort before schema comparison by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2209
fix: canonicalize config keys by @emcake in https://github.com/delta-io/delta-rs/pull/2206
fix(writer): retry storage.put on temporary network errors by @qinix in https://github.com/delta-io/delta-rs/pull/2207
fix: fix ruff and mypy version and do formatting by @aersam in https://github.com/delta-io/delta-rs/pull/2240
fix: object_store 0.9.0 since 0.9.1 causes CI failure by @aersam in https://github.com/delta-io/delta-rs/pull/2245

Other Changes

chore: 0.17.0 publish changes by @rtyler in https://github.com/delta-io/delta-rs/pull/2171
docs: include the 0.17.0 changelog by @rtyler in https://github.com/delta-io/delta-rs/pull/2173
docs: add delta lake best practices by @MrPowers in https://github.com/delta-io/delta-rs/pull/2147
docs: dask integration fix formatting typo by @avriiil in https://github.com/delta-io/delta-rs/pull/2196
docs: update README code samples for newer versions by @jhoekx in https://github.com/delta-io/delta-rs/pull/2202
chore: remove caches from github actions by @rtyler in https://github.com/delta-io/delta-rs/pull/2215
docs: fixing example in CONTRIBUTING.md by @gacharya in https://github.com/delta-io/delta-rs/pull/2224
chore: fix the Cargo.tomls to publish information properly on docs.rs by @rtyler in https://github.com/delta-io/delta-rs/pull/2211
chore: bump to 0.16 by @ion-elgreco in https://github.com/delta-io/delta-rs/pull/2248
chore: clean up some compilation failures and un-ignore some tests by @rtyler in https://github.com/delta-io/delta-rs/pull/2231

New Contributors

@braaannigan made their first contribution in https://github.com/delta-io/delta-rs/pull/2179
@sonhmai made their first contribution in https://github.com/delta-io/delta-rs/pull/2183
@jhoekx made their first contribution in https://github.com/delta-io/delta-rs/pull/2202
@gacharya made their first contribution in https://github.com/delta-io/delta-rs/pull/2224
@qinix made their first contribution in https://github.com/delta-io/delta-rs/pull/2207

Full Changelog: https://github.com/delta-io/delta-rs/compare/python-v0.15.3...python-v0.16.0

delta-rs - rust-v0.17.0

Published by rtyler 8 months ago

⚠️ The release of 0.17.0 removes the legacy dynamodb lock functionality, AWS users must read these release notes! ⚠️

File handlers

The 0.17.0 release moves storage implementations into their own crates, such as
deltalake-aws. A consequence of that refactoring is that custom storage and
file scheme handlers must be registered/initialized at runtime. Storage
subcrates conventionally define a register_handlers function which performs
that task. Users may see errors such as:

thread 'main' panicked at /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deltalake-core-0.17.0/src/table/builder.rs:189:48:
The specified table_uri is not valid: InvalidTableLocation("Unknown scheme: s3")

Users of the meta-crate (deltalake) can call the storage crate via: deltalake::aws::register_handlers(None); at the entrypoint for their code.
Users who adopt core and storage crates independently (e.g. deltalake-aws) can register via deltalake_aws::register_handlers(None);.

The AWS, Azure, and GCP crates must all have their custom file schemes registered in this fashion.

dynamodblock to S3DynamoDbLogStore

The locking mechanism is fundamentally different between deltalake v0.16.x and v0.17.0, starting with this release the deltalake and deltalake-aws crates this library now relies on the same protocol for concurrent writes on AWS as the Delta Lake/Spark implementation.

Fundamentally the DynamoDB table structure changes, which is documented here. The configuration of a Rust process should continue to use the AWS_S3_LOCKING_PROVIDER environment value of dynamodb. The new table must be specified with the DELTA_DYNAMO_TABLE_NAME environment or configuration variable, and that should name the new S3DynamoDbLogStore compatible DynamoDB table.

Because locking is required to ensure safe cconsistent writes, there is no iterative migration, 0.16 and 0.17 writers cannot safely coexist. The following steps should be taken when upgrading:

Stop all 0.16.x writers
Ensure writes are completed, and lock table is empty.
Deploy 0.17.0 writers

Full Changelog

Implemented enhancements:

Expose the ability to compile DataFusion with SIMD #2118
Updating Table log retention configuration with write_deltalake silently changes nothing #2108
ALTER table, ALTER Column, Add/Modify Comment, Add/remove/rename partitions, Set Tags, Set location, Set TBLProperties #2088
Docs: Update docs for check constraints #2063
Don't ensure_table_uri when creating a table with_log_store #2036
Exposing custom_metadata in merge operation #2031
Support custom table properties via TableAlterer and write/merge #2022
Remove parquet2 crate support #2004
Merge operation that only touches necessary partitions #1991
store userMetadata on write operations #1990
Create Dask integration page #1956
Merge: Filtering on partitions #1918
Rethink the load_version and load_with_datetime interfaces #1910
docs: Delta Lake + Arrow Integration #1908
docs: Delta Lake + Polars integration #1906
Rethink decision to expose the public interface in namespaces #1900
Add documentation on how to build and run documentation locally #1893
Add API to create an empty Delta Lake table #1892
Implementing CHECK constraints #1881
Check Invariants are respecting table features for write paths #1880
Organize docs with single lefthand sidebar #1873
Make sure invariants are handled properly throughout the codebase #1870
Unable to use deltalake Schema in write_deltalake #1862
Add a Rust-backed engine for write_deltalake #1861
Run doctest in CI for Python API examples #1783
[RFC] Use arrow for checkpoint reading and state handling #1776
Expose Python exceptions in public module #1771
Expose cleanup_metadata or create_checkpoint_from_table_uri_and_cleanup to the Python API #1768
Expose convert_to_delta to Python API #1767
Add high-level checking for append-only tables #1759

Fixed bugs:

Row order no longer preserved after merge operation #2165
Error when reading delta table with IDENTITY column #2152
Merge on IS NULL condition doesn't work for empty table #2148
JsonWriter converts structured parsing error into plain string #2143
Pandas import error when merging tables #2112
test_repair_on_update broken in main #2109
WriteBuilder::with_input_execution_plan does not apply the schema to the log's metadata fields #2105
MERGE logical plan vs execution plan schema mismatch #2104
Partitions not pushed down #2090
Cant create empty table with write_deltalake #2086
Unexpected high costs on Google Cloud Storage #2085
Unable to read s3 table: Unknown scheme: s3 #2065
write_deltalake not respecting writer_properties #2064
Unable to read/write tables with the "gs" schema in the table_uri in 0.15.1 #2060
LockClient requiered error for S3 backend in 0.15.1 python #2057
Error while writing Pandas DataFrame to Delta Lake (S3) #2051
Error with dynamo locking provider on 0.15 #2034
Conda version 0.15.0 is missing files #2021
Rust panicking through Python library when a delete predicate uses a nullable field #2019
No snapshot or version 0 found, perhaps /Users/watsy0007/resources/test_table/ is an empty dir? #2016
Generic DeltaTable error: type_coercion in Struct column in merge operation #1998
Constraint expr not formatted during commit action #1971
.load_with_datetime() is incorrectly rounding to nearest second #1967
vacuuming log files #1965
Unable to merge uppercase column names #1960
Schema error: Invalid data type for Delta Lake: Null #1946
Python v0.14 wheel files not up to date #1945
python Release 0.14 is missing Windows wheels #1942
CI integration test fails randomly: test_restore_by_datetime #1925
Merge data freezes indefenetely #1920
Load DeltaTable from non-existing folder causing empty folder creation #1916
Reoptimizes merge bins with only 1 file, even though they have no effect. #1901
The Python Docs link in README.MD points to old docs #1898
optimize.compact() fails with bad schema after updating to pyarrow 8.0 #1889
Python build is broken on main #1856
Checkpoint error with Azure Synapse #1847
merge very slow compared to delete + append on larger dataset #1846
get_add_actions fails with deltalake 0.13 #1835
Handle PyArrow CVE-2023-47248 #1834
Delta-rs writer hangs with to many file handles open (Azure) #1832
Encountering NotATable("No snapshot or version 0 found, perhaps xxx is an empty dir?") #1831
write_deltalake is not creating checkpoints #1815
Problem writing tables in directory named with char ~ #1806
DeltaTable Merge throws in merging if there are uppercase in Schema. #1797
rust merge error - datafusion panics #1790
expose use_dictionary=False when writing Delta Table and running optimize #1772