lance

What's Changed

• fix benchmarks by @changhiskhan in https://github.com/eto-ai/lance/pull/862
• [Python] Expose Fragment data files in Python by @eddyxu in https://github.com/eto-ai/lance/pull/864

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.10...v0.4.11

Welcome our newest contributor @LiWeiJie!

This release introduces several performance improvements, including index caching and customized prefetching.

What's Changed

• remove unnecessary simd alignment check by @changhiskhan in https://github.com/eto-ai/lance/pull/852
• Fix projection for null struct fields by @LiWeiJie in https://github.com/eto-ai/lance/pull/857
• [Rust] Caching index in LRU by @eddyxu in https://github.com/eto-ai/lance/pull/829
• Expose batch_readahead via Python scanner API by @eddyxu in https://github.com/eto-ai/lance/pull/859

New Contributors

• @LiWeiJie made their first contribution in https://github.com/eto-ai/lance/pull/857

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.9...v0.4.10

What's Changed

• [Doc] improvement on read related functionalities. by @eddyxu in https://github.com/eto-ai/lance/pull/848
• Expose schema in Fragment object by @eddyxu in https://github.com/eto-ai/lance/pull/850
• Pickle Fragment Metadata by @eddyxu in https://github.com/eto-ai/lance/pull/851

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.8...v0.4.9

Previously predicates on nested (and deeply nested) fields were not properly supported. This release adds support for filtering on struct sub-fields or deeply nested structs.

We also add support for more filter predicates and fixed a regression in NULL handling for string columns.

What's Changed

• Fix nested schema merge by @changhiskhan in https://github.com/eto-ai/lance/pull/836
• Fix nested field filtering by @changhiskhan in https://github.com/eto-ai/lance/pull/837
• Fix Projection for struct fields by @changhiskhan in https://github.com/eto-ai/lance/pull/844
• [Bug] Calculating the nulls from position slices in chunks. by @eddyxu in https://github.com/eto-ai/lance/pull/846
• Add tests for is_NULL, is_not_null, and invert in filter by @eddyxu in https://github.com/eto-ai/lance/pull/847

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.7...v0.4.8

In this version, we improve the random access over cloud storage by allowing a higher number of parallel I/Os.

What's Changed

• [Rust] increase parallelism, reduce array build overhead by @eddyxu in https://github.com/eto-ai/lance/pull/830
• Recursively merge record batches by @changhiskhan in https://github.com/eto-ai/lance/pull/833
• Allow user to customize BlockSize on ObjectStore. by @eddyxu in https://github.com/eto-ai/lance/pull/835

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.6...v0.4.7

Allows the creation of a distributed lance dataset from scratch

What's Changed

• Allow create fragment on non-existed dataset. by @eddyxu in https://github.com/eto-ai/lance/pull/825

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.5...v0.4.6

Welcome @Mause as our newest contributor! Also, a big thank you for your work on the duckdb extension framework.

In this release we added a preview of the feature to do distributed column additions. This makes it possible to distribute Lance Fragments across nodes, add a new column to each Fragment, and then write a new Lance dataset version manifest with the updated schema and files.

What's Changed

• add support for aws profile by @Renkai in https://github.com/eto-ai/lance/pull/807
• Upgrade Arrow to 37 by @changhiskhan in https://github.com/eto-ai/lance/pull/810
• Schema intersection by @eddyxu in https://github.com/eto-ai/lance/pull/814
• Add a check to make sure field names don't contain periods by @changhiskhan in https://github.com/eto-ai/lance/pull/816
• fix(docs): correct link to docs.rs by @Mause in https://github.com/eto-ai/lance/pull/819
• update arrow version in duckdb extension by @changhiskhan in https://github.com/eto-ai/lance/pull/817
• Do not use lifetime on FileWriter by @eddyxu in https://github.com/eto-ai/lance/pull/820
• Setting field ID after merging the fields. by @eddyxu in https://github.com/eto-ai/lance/pull/821
• [Rust] Project schema by schema by @eddyxu in https://github.com/eto-ai/lance/pull/822
• Merge batches from multiple datafiles in the same Fragment by @eddyxu in https://github.com/eto-ai/lance/pull/815
• Update README.md by @jaichopra in https://github.com/eto-ai/lance/pull/809
• [Python] Provide a private / distributed add column api in Python by @eddyxu in https://github.com/eto-ai/lance/pull/823

New Contributors

• @Mause made their first contribution in https://github.com/eto-ai/lance/pull/819

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.4...v0.4.5

#805 fixed an integer overflow bug in the plain decoder that resulted in high latency for Take (and consequently high latency for the vector search). We'll be adding continuous performance benchmarks soon to prevent issues like this from being released in the future.

We also fixed a gap in cosine similarity where the vectors does not line up perfectly with SIMD strides on the platform.

DiskANN progress is continuing. First milestone will be an in-memory version to support smaller datasets. A compressed, disk-based version will follow soon after that.

What's Changed

• Fix L2 simd benchmark by @eddyxu in https://github.com/eto-ai/lance/pull/793
• bugfix for dataset overwrite method by @gsilvestrin in https://github.com/eto-ai/lance/pull/794
• [Rust] Minor SIMD benchmark fix set minimal CPU target for AVX2 by @eddyxu in https://github.com/eto-ai/lance/pull/795
• Persist simple diskann index by @eddyxu in https://github.com/eto-ai/lance/pull/787
• Fix offset overflow in plain decoder by @eddyxu in https://github.com/eto-ai/lance/pull/805
• Fix cosine similarity when missing simd alignment by @changhiskhan in https://github.com/eto-ai/lance/pull/808

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.3...v0.4.4

What's Changed

• [Rust] L2 distance on not aligned data by @eddyxu in https://github.com/eto-ai/lance/pull/779
• [Rust] Move L2 to linalg module by @eddyxu in https://github.com/eto-ai/lance/pull/781
• [Rust] Build DiskANN index by @eddyxu in https://github.com/eto-ai/lance/pull/763
• Refactor cosine distance into linalg module by @eddyxu in https://github.com/eto-ai/lance/pull/786
• google cloud storage fixes by @gsilvestrin in https://github.com/eto-ai/lance/pull/782
• Fix unaligned normalization bug on arm64 by @eddyxu in https://github.com/eto-ai/lance/pull/789
• Speed up vector index tests by reducing dataset size by @changhiskhan in https://github.com/eto-ai/lance/pull/790

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.2...v0.4.3

A warm welcome to @hzhang86 as Lance's newest contributor. Thanks for adding TPCH benchmarks for Lance to establish a baseline. This is really helpful for us to focus performance optimization roadmap.

This release is packed with valuable features:

1. Direct polars scan without needing to pull everything into memory is added.
2. We expose FileFragment's to allow distributed processing engines like Spark to access parts of a Lance dataset easily
3. Last but not least, we've added support for reading Lance data directly from GS buckets

What's Changed

• [Rust] FileReader read range API by @eddyxu in https://github.com/eto-ai/lance/pull/752
• Support direct polars scan by @changhiskhan in https://github.com/eto-ai/lance/pull/755
• [Rust] Persist graph using lance file format. by @eddyxu in https://github.com/eto-ai/lance/pull/756
• Refactor PQ and OPQ training function to make it usable widely by @eddyxu in https://github.com/eto-ai/lance/pull/758
• Matrix::centroids method by @eddyxu in https://github.com/eto-ai/lance/pull/759
• [Python] Set minimal version of Polars for python tests by @eddyxu in https://github.com/eto-ai/lance/pull/765
• [Rust] Refactor RecordBatchStream trait by @eddyxu in https://github.com/eto-ai/lance/pull/766
• [Rust] Expose DataFragment as pubilc dataset api. by @eddyxu in https://github.com/eto-ai/lance/pull/769
• Revert "[Python] Set minimal version of Polars for python tests (#765)" by @gsilvestrin in https://github.com/eto-ai/lance/pull/770
• add python script to compare lance performance vs parquet TPCH by @hzhang86 in https://github.com/eto-ai/lance/pull/749
• Expose index metadata by @changhiskhan in https://github.com/eto-ai/lance/pull/768
• Google Cloud Storage support. by @gsilvestrin in https://github.com/eto-ai/lance/pull/773
• [Python] Expose DataFragment via dataset by @eddyxu in https://github.com/eto-ai/lance/pull/774
• Get S3 credentials from_env by @changhiskhan in https://github.com/eto-ai/lance/pull/775
• Fix duckdb build by @eddyxu in https://github.com/eto-ai/lance/pull/776
• [Rust] A arrow kernel to compute hash value of the array. by @eddyxu in https://github.com/eto-ai/lance/pull/777

New Contributors

• @hzhang86 made their first contribution in https://github.com/eto-ai/lance/pull/749

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.1...v0.4.2

The vector search in Lance now supports live updates. Previously, when you added new vectors to the dataset, you would be required to rebuild the index. Now, the index is "inherited" and the vector search results are the combination of ANN search on the indexed data and KNN on the new Appended data. So there's a small latency increase and the recall should be the same or better.

This provides a smooth performance curve until you have inserted enough new data that re-indexing is warranted.

What's Changed

• Adding secret to publish task by @gsilvestrin in https://github.com/eto-ai/lance/pull/742
• [Rust] make distance function to take slice instead of Float32Array by @eddyxu in https://github.com/eto-ai/lance/pull/748
• Vector search should support appending new rows by @changhiskhan in https://github.com/eto-ai/lance/pull/593
• windows lapack support by @gsilvestrin in https://github.com/eto-ai/lance/pull/743
• Fix LanceDataset.to_batches by @changhiskhan in https://github.com/eto-ai/lance/pull/751

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.0...v0.4.1

A warm welcome to @gsajko ! Thanks for making our tutorial notebook easier to use and understand!

Note: OPQ is disabled in windows for the vector index. This will be addressed once LAPACK support is added.

What's Changed

• small fixes by @gsajko in https://github.com/eto-ai/lance/pull/725
• Windows support by @gsilvestrin in https://github.com/eto-ai/lance/pull/724

New Contributors

• @gsajko made their first contribution in https://github.com/eto-ai/lance/pull/725

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.19...v0.4.0

Also fix publishing to crates.io

What's Changed

• Make contract clear for KNN nodes by @eddyxu in https://github.com/eto-ai/lance/pull/729
• Refactor Scan I/O plan by @eddyxu in https://github.com/eto-ai/lance/pull/731
• [Rust] Use folked sqlparser to unblock rust crate release by @eddyxu in https://github.com/eto-ai/lance/pull/732
• [Rust] Fix filter on large UTF8 columns by @eddyxu in https://github.com/eto-ai/lance/pull/733

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.18...v0.3.19

Fix for incorrect offset for string/variable list columns as reported in https://github.com/eto-ai/lance/issues/720#issuecomment-1479716990

Thanks @lucazanna for the feedback!

What's Changed

• Train OPQ and write rotation matrix to index file by @eddyxu in https://github.com/eto-ai/lance/pull/713
• removing warnings by @gsilvestrin in https://github.com/eto-ai/lance/pull/721
• [Bug] Fix IVF merge sort when refine factor is presented. by @eddyxu in https://github.com/eto-ai/lance/pull/722
• Add input / output schema contract to Global Take by @eddyxu in https://github.com/eto-ai/lance/pull/728
• Fix offsets for Binary/Lists/LargeLists by @gsilvestrin in https://github.com/eto-ai/lance/pull/727

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.17...v0.3.18

A warm welcome to @haoxins , a new contributor who has helped improve Lance documentation.

This release adds support for list-of-dict columns (thanks @lucazanna for reporting the bug in #715).

Also included in this release are various vector index improvements for scalability and more progress towards OPQ implementation.

What's Changed

• docs: fix the links by @haoxins in https://github.com/eto-ai/lance/pull/701
• repair macos build for duckdb extension by @changhiskhan in https://github.com/eto-ai/lance/pull/705
• filter evaluation with flat search by @changhiskhan in https://github.com/eto-ai/lance/pull/704
• fix flaky test by @changhiskhan in https://github.com/eto-ai/lance/pull/706
• [Bug] Fix transpose in MatrixView.data() by @eddyxu in https://github.com/eto-ai/lance/pull/711
• Refactored variable length encoders by @gsilvestrin in https://github.com/eto-ai/lance/pull/710
• add notebook for q&a bot by @changhiskhan in https://github.com/eto-ai/lance/pull/707
• Allow iteratively train PQ by @eddyxu in https://github.com/eto-ai/lance/pull/712
• Use relative eq and fix a compiling warning by @eddyxu in https://github.com/eto-ai/lance/pull/714
• docs: fix the mod path by @haoxins in https://github.com/eto-ai/lance/pull/718
• Composable vector search pipeline by @eddyxu in https://github.com/eto-ai/lance/pull/716
• Fix CI failure by increasing epsilon for test_train_pq_iteratively by @eddyxu in https://github.com/eto-ai/lance/pull/719
• Implement support for list of Dictionaries by @gsilvestrin in https://github.com/eto-ai/lance/pull/664

New Contributors

• @haoxins made their first contribution in https://github.com/eto-ai/lance/pull/701

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.16...v0.3.17

Welcome @wangfenjin to lance contributors. Thanks for submitting a bug fix for the Lance DuckDB extensions 🔥

This release contains 2 workarounds for arrow limitations:

1. Lance datasets now support <field> LIKE '%' and <field> IN (<values>) filters to be passed in as string. Generic SQL syntax supported by datafusion is now accepted. This is a break from standard pyarrow Dataset behavior which only accepts arrow compute Expression, which is not present in rust and also does not support introspection in python for developers to build custom adapter.

2. When concatenating arrow dictionary arrays, the dict values are duplicated. There is currently no concrete plans to change this behavior in Arrow. Instead, we fix that at write time in Lance.

What's Changed

• Changed encoders to handle multiple Arrays by @gsilvestrin in https://github.com/eto-ai/lance/pull/681
• Train kmeans iteratively by @eddyxu in https://github.com/eto-ai/lance/pull/688
• Changed writers to handle multiple Arrays by @gsilvestrin in https://github.com/eto-ai/lance/pull/691
• Streaming PQ by @eddyxu in https://github.com/eto-ai/lance/pull/689
• [Bug] PQ training generates empty centroids by @eddyxu in https://github.com/eto-ai/lance/pull/693
• Allow append mode even if dataset doesn't already exist by @ananis25 in https://github.com/eto-ai/lance/pull/690
• Support "LIKE" and "IN" in filters by @eddyxu in https://github.com/eto-ai/lance/pull/696
• fix typo by @wangfenjin in https://github.com/eto-ai/lance/pull/697
• Improve indexing performance by @eddyxu in https://github.com/eto-ai/lance/pull/699
• Compute PQ distortion. by @eddyxu in https://github.com/eto-ai/lance/pull/695
• Bugfix for BinaryEncoder positions by @gsilvestrin in https://github.com/eto-ai/lance/pull/698

New Contributors

• @wangfenjin made their first contribution in https://github.com/eto-ai/lance/pull/697

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.15...v0.3.16

Thanks to @cemoody for the bug report!

What's Changed

• Missing column when both nearest and filter are applied by @changhiskhan in https://github.com/eto-ai/lance/pull/686

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.14...v0.3.15

This is a patch release that adds support for Arrow Timestamp type. Thanks @kesavkolla for the bug report!

Thanks to @Renkai we also an optimized Take for Boolean arrays.

What's Changed

• OPQ rotation matrix training by @eddyxu in https://github.com/eto-ai/lance/pull/669
• Optimize boolean by @Renkai in https://github.com/eto-ai/lance/pull/676
• Support timestamp type by @eddyxu in https://github.com/eto-ai/lance/pull/684

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.13...v0.3.14

What's Changed

• update arrow-rs version in duckdb-ext for lance as well by @changhiskhan in https://github.com/eto-ai/lance/pull/670
• Support take operation on List by @eddyxu in https://github.com/eto-ai/lance/pull/671

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.12...v0.3.13

• Upgraded arrow-rs dependency to 33.0 (Waiting on datafusion for 34.0 upgrade).
• Nested Dictionary fields are now parsed and written correctly.
• More progress towards OPQ implementation.

What's Changed

• Matrix mul and transpose by @eddyxu in https://github.com/eto-ai/lance/pull/661
• Recursively set dictionaries in struct fields by @gsilvestrin in https://github.com/eto-ai/lance/pull/662
• Upgrading arrow version to 33.0 by @gsilvestrin in https://github.com/eto-ai/lance/pull/665
• [Rust] sampling over matrix. by @eddyxu in https://github.com/eto-ai/lance/pull/666
• Sorting dataset versions by @gsilvestrin in https://github.com/eto-ai/lance/pull/668

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.11...v0.3.12