lance

What's Changed

Breaking Changes 🛠

• feat!: index_statistics returns the concrete index type by @BubbleCal in https://github.com/lancedb/lance/pull/2716
• feat: stop writing _latest.manifest by @wjones127 in https://github.com/lancedb/lance/pull/2776
• refactor: build & search vector index with new indexer & IVF impl by @BubbleCal in https://github.com/lancedb/lance/pull/2552

New Features 🎉

• feat: add max_bytes_per_file and batch_size to CompactionOptions by @westonpace in https://github.com/lancedb/lance/pull/2728
• feat: add fixed size binary encoding to lance by @raunaks13 in https://github.com/lancedb/lance/pull/2707
• feat: make the I/O buffer size configurable by @westonpace in https://github.com/lancedb/lance/pull/2736
• feat: pushdown limit & offset into file reader if there is no filter by @westonpace in https://github.com/lancedb/lance/pull/2747
• feat: support phrase query for full text search by @BubbleCal in https://github.com/lancedb/lance/pull/2751
• feat: make the # of CPU threads configurable and document cpu/memory patterns by @westonpace in https://github.com/lancedb/lance/pull/2773
• feat: add logging to compaction by @westonpace in https://github.com/lancedb/lance/pull/2791
• feat(python): account for both multi-process and distributed torch workers by @tonyf in https://github.com/lancedb/lance/pull/2761
• feat: allow duckdb / polars pushdown to operate with non-substrait types in schema but not filter by @westonpace in https://github.com/lancedb/lance/pull/2796
• feat: add an environment variable LANCE_INITIAL_UPLOAD_SIZE by @westonpace in https://github.com/lancedb/lance/pull/2806
• feat: extend python bindings for the v2 reader/writer by @ankitvij-db in https://github.com/lancedb/lance/pull/2800
• feat: provide param to control whether to build FTS index with positions by @BubbleCal in https://github.com/lancedb/lance/pull/2795
• feat: allow creating fragments from v2 files, expose rewrite operation to python by @westonpace in https://github.com/lancedb/lance/pull/2811
• feat: use a considerably higher default for frag readahead in v2 by @westonpace in https://github.com/lancedb/lance/pull/2797
• feat: support update tags by @broccoliSpicy in https://github.com/lancedb/lance/pull/2813
• feat(java): add create vector search index, list index names, and vector search scan by @LuQQiu in https://github.com/lancedb/lance/pull/2782
• feat: make default I/O buffer size configurable via env var by @westonpace in https://github.com/lancedb/lance/pull/2826
• feat: constant-time manifest lookup on object stores by @wjones127 in https://github.com/lancedb/lance/pull/2798

Bug Fixes 🐛

• fix: drop indices if all fragment ids are missing by @wjones127 in https://github.com/lancedb/lance/pull/2720
• fix: ignore pages with zero rows by @westonpace in https://github.com/lancedb/lance/pull/2724
• fix: various changes to fix backpressure by @westonpace in https://github.com/lancedb/lance/pull/2721
• fix: raise exception if user passes unsorted indices to take_rows() of lance file api by @raunaks13 in https://github.com/lancedb/lance/pull/2729
• fix: flaky test caused by PQ distortion by @BubbleCal in https://github.com/lancedb/lance/pull/2732
• fix: don't panic when v2 scans end early (reader is dropped) by @westonpace in https://github.com/lancedb/lance/pull/2690
• fix: fix a crash that could sometimes happen reading largish string/binary data in a list by @westonpace in https://github.com/lancedb/lance/pull/2731
• fix: fix two cases in the v2 decoder where the decode order didn't match the scheduling priority by @westonpace in https://github.com/lancedb/lance/pull/2754
• fix: fix several situations where we were incorrectly inferring the storage version by @westonpace in https://github.com/lancedb/lance/pull/2756
• fix: rework priority handling for lists. We now properly schedule list item pages in priority order by @westonpace in https://github.com/lancedb/lance/pull/2769
• fix: try and fix an invalid data storage version automatically by @westonpace in https://github.com/lancedb/lance/pull/2759
• fix: add not-linux version of benches to avoid bench compiler error on mac by @westonpace in https://github.com/lancedb/lance/pull/2794
• fix: fix v2 error that can happen when writing list<struct<...>> with many empty lists by @westonpace in https://github.com/lancedb/lance/pull/2762
• fix: torch to_tensor for FixedShapeTensorType data by @jacketsj in https://github.com/lancedb/lance/pull/2824
• fix: cleanup external staging manifests by @wjones127 in https://github.com/lancedb/lance/pull/2792
• fix: the DataFile version doesn't respect the writer's version by @BubbleCal in https://github.com/lancedb/lance/pull/2825
• fix: merge-insert and update can't work on v2 format by @BubbleCal in https://github.com/lancedb/lance/pull/2833

Documentation 📚

• docs: clean up quickstart notebook and add tags example usage by @dsgibbons in https://github.com/lancedb/lance/pull/2735
• docs: fix version table so it renders by @wjones127 in https://github.com/lancedb/lance/pull/2745
• docs: document newly added compaction options in optimize.py by @westonpace in https://github.com/lancedb/lance/pull/2746
• docs: fix transaction conflict table by @wjones127 in https://github.com/lancedb/lance/pull/2744
• docs: fix #2748, "Merge Insert" doc error by @broccoliSpicy in https://github.com/lancedb/lance/pull/2788

Performance Improvements 🚀

• perf: coalesce ids before executing take by @westonpace in https://github.com/lancedb/lance/pull/2680
• perf: cache default session contexts by @dsgibbons in https://github.com/lancedb/lance/pull/2709
• perf: various fixes to improve shuffling performance at high scales by @westonpace in https://github.com/lancedb/lance/pull/2710
• perf: calculate the max scores for posting lists then have a tighter upper bound by @BubbleCal in https://github.com/lancedb/lance/pull/2763
• perf: stable row id prefilter by @wjones127 in https://github.com/lancedb/lance/pull/2706
• perf: add prefetching to hnsw greedy_search by @jacketsj in https://github.com/lancedb/lance/pull/2783
• perf: parallelize FTS indexing by @BubbleCal in https://github.com/lancedb/lance/pull/2807
• perf: split MaterializeIndex stream into batches by @wjones127 in https://github.com/lancedb/lance/pull/2770

Other Changes

• refactor: convert DataBlock to an enum, add conversion from arrow, normalize dictionaries by @westonpace in https://github.com/lancedb/lance/pull/2789

New Contributors

• @dentiny made their first contribution in https://github.com/lancedb/lance/pull/2740
• @jacketsj made their first contribution in https://github.com/lancedb/lance/pull/2783
• @tonyf made their first contribution in https://github.com/lancedb/lance/pull/2761
• @ankitvij-db made their first contribution in https://github.com/lancedb/lance/pull/2800

Full Changelog: https://github.com/lancedb/lance/compare/v0.16.1...v0.17.0

What's Changed

New Features 🎉

• feat: make default I/O buffer size configurable via env var by @westonpace in https://github.com/lancedb/lance/pull/2826

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.12...v0.17.0-beta.13

What's Changed

New Features 🎉

• feat: use a considerably higher default for frag readahead in v2 by @westonpace in https://github.com/lancedb/lance/pull/2797
• feat: support update tags by @broccoliSpicy in https://github.com/lancedb/lance/pull/2813
• feat(java): add create vector search index, list index names, and vector search scan by @LuQQiu in https://github.com/lancedb/lance/pull/2782

Bug Fixes 🐛

• fix: fix v2 error that can happen when writing list<struct<...>> with many empty lists by @westonpace in https://github.com/lancedb/lance/pull/2762

Performance Improvements 🚀

• perf: parallelize FTS indexing by @BubbleCal in https://github.com/lancedb/lance/pull/2807

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.11...v0.17.0-beta.12

What's Changed

New Features 🎉

• feat: extend python bindings for the v2 reader/writer by @ankitvij-db in https://github.com/lancedb/lance/pull/2800
• feat: provide param to control whether to build FTS index with positions by @BubbleCal in https://github.com/lancedb/lance/pull/2795
• feat: allow creating fragments from v2 files, expose rewrite operation to python by @westonpace in https://github.com/lancedb/lance/pull/2811

New Contributors

• @ankitvij-db made their first contribution in https://github.com/lancedb/lance/pull/2800

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.10...v0.17.0-beta.11

What's Changed

New Features 🎉

• feat: allow duckdb / polars pushdown to operate with non-substrait types in schema but not filter by @westonpace in https://github.com/lancedb/lance/pull/2796
• feat: add an environment variable LANCE_INITIAL_UPLOAD_SIZE by @westonpace in https://github.com/lancedb/lance/pull/2806

Other Changes

• refactor: convert DataBlock to an enum, add conversion from arrow, normalize dictionaries by @westonpace in https://github.com/lancedb/lance/pull/2789

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.9...v0.17.0-beta.10

What's Changed

Breaking Changes 🛠

• feat: stop writing _latest.manifest by @wjones127 in https://github.com/lancedb/lance/pull/2776
• refactor: build & search vector index with new indexer & IVF impl by @BubbleCal in https://github.com/lancedb/lance/pull/2552

New Features 🎉

• feat: support phrase query for full text search by @BubbleCal in https://github.com/lancedb/lance/pull/2751
• feat: make the # of CPU threads configurable and document cpu/memory patterns by @westonpace in https://github.com/lancedb/lance/pull/2773
• feat: add logging to compaction by @westonpace in https://github.com/lancedb/lance/pull/2791
• feat(python): account for both multi-process and distributed torch workers by @tonyf in https://github.com/lancedb/lance/pull/2761

Bug Fixes 🐛

• fix: add not-linux version of benches to avoid bench compiler error on mac by @westonpace in https://github.com/lancedb/lance/pull/2794

Documentation 📚

• docs: fix #2748, "Merge Insert" doc error by @broccoliSpicy in https://github.com/lancedb/lance/pull/2788

Performance Improvements 🚀

• perf: calculate the max scores for posting lists then have a tighter upper bound by @BubbleCal in https://github.com/lancedb/lance/pull/2763
• perf: stable row id prefilter by @wjones127 in https://github.com/lancedb/lance/pull/2706
• perf: add prefetching to hnsw greedy_search by @jacketsj in https://github.com/lancedb/lance/pull/2783

New Contributors

• @jacketsj made their first contribution in https://github.com/lancedb/lance/pull/2783
• @tonyf made their first contribution in https://github.com/lancedb/lance/pull/2761

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.8...v0.17.0-beta.9

What's Changed

Bug Fixes 🐛

• fix: rework priority handling for lists. We now properly schedule list item pages in priority order by @westonpace in https://github.com/lancedb/lance/pull/2769
• fix: try and fix an invalid data storage version automatically by @westonpace in https://github.com/lancedb/lance/pull/2759

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.7...v0.17.0-beta.8

What's Changed

New Features 🎉

• feat: pushdown limit & offset into file reader if there is no filter by @westonpace in https://github.com/lancedb/lance/pull/2747

Bug Fixes 🐛

• fix: fix two cases in the v2 decoder where the decode order didn't match the scheduling priority by @westonpace in https://github.com/lancedb/lance/pull/2754
• fix: fix several situations where we were incorrectly inferring the storage version by @westonpace in https://github.com/lancedb/lance/pull/2756

Performance Improvements 🚀

• perf: various fixes to improve shuffling performance at high scales by @westonpace in https://github.com/lancedb/lance/pull/2710

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.6...v0.17.0-beta.7

What's Changed

New Features 🎉

• feat: make the I/O buffer size configurable by @westonpace in https://github.com/lancedb/lance/pull/2736

Documentation 📚

• docs: fix version table so it renders by @wjones127 in https://github.com/lancedb/lance/pull/2745
• docs: document newly added compaction options in optimize.py by @westonpace in https://github.com/lancedb/lance/pull/2746
• docs: fix transaction conflict table by @wjones127 in https://github.com/lancedb/lance/pull/2744

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.5...v0.17.0-beta.6

What's Changed

New Features 🎉

• feat: add fixed size binary encoding to lance by @raunaks13 in https://github.com/lancedb/lance/pull/2707

Bug Fixes 🐛

• fix: raise exception if user passes unsorted indices to take_rows() of lance file api by @raunaks13 in https://github.com/lancedb/lance/pull/2729
• fix: flaky test caused by PQ distortion by @BubbleCal in https://github.com/lancedb/lance/pull/2732
• fix: don't panic when v2 scans end early (reader is dropped) by @westonpace in https://github.com/lancedb/lance/pull/2690
• fix: fix a crash that could sometimes happen reading largish string/binary data in a list by @westonpace in https://github.com/lancedb/lance/pull/2731

Documentation 📚

• docs: clean up quickstart notebook and add tags example usage by @dsgibbons in https://github.com/lancedb/lance/pull/2735

Performance Improvements 🚀

• perf: cache default session contexts by @dsgibbons in https://github.com/lancedb/lance/pull/2709

New Contributors

• @dentiny made their first contribution in https://github.com/lancedb/lance/pull/2740

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.4...v0.17.0-beta.5

What's Changed

New Features 🎉

• feat: add max_bytes_per_file and batch_size to CompactionOptions by @westonpace in https://github.com/lancedb/lance/pull/2728

Performance Improvements 🚀

• perf: coalesce ids before executing take by @westonpace in https://github.com/lancedb/lance/pull/2680

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.2...v0.17.0-beta.4

What's Changed

Bug Fixes 🐛

• fix: drop indices if all fragment ids are missing by @wjones127 in https://github.com/lancedb/lance/pull/2720
• fix: ignore pages with zero rows by @westonpace in https://github.com/lancedb/lance/pull/2724
• fix: various changes to fix backpressure by @westonpace in https://github.com/lancedb/lance/pull/2721

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.1...v0.17.0-beta.3

What's Changed

Bug Fixes 🐛

• fix: drop indices if all fragment ids are missing by @wjones127 in https://github.com/lancedb/lance/pull/2720
• fix: ignore pages with zero rows by @westonpace in https://github.com/lancedb/lance/pull/2724
• fix: various changes to fix backpressure by @westonpace in https://github.com/lancedb/lance/pull/2721

Full Changelog: https://github.com/lancedb/lance/compare/v0.17.0-beta.1...v0.17.0-beta.2

What's Changed

Breaking Changes 🛠

• feat!: index_statistics returns the concrete index type by @BubbleCal in https://github.com/lancedb/lance/pull/2716

Full Changelog: https://github.com/lancedb/lance/compare/v0.16.1...v0.17.0-beta.1

What's Changed

New Features 🎉

• feat: make scalar index training configurable by @dsgibbons in https://github.com/lancedb/lance/pull/2686
• feat: migrate Bitmap & LabelList index file to v2 format by @BubbleCal in https://github.com/lancedb/lance/pull/2677
• feat: make default batch size configurable via environment variable by @westonpace in https://github.com/lancedb/lance/pull/2693
• feat: move-stable row ids in compaction by @wjones127 in https://github.com/lancedb/lance/pull/2544
• feat: load multiple ivf-pq sorted files and commit index into dataset by @raunaks13 in https://github.com/lancedb/lance/pull/2681

Bug Fixes 🐛

• fix: out of bounds error in bitpacking by @albertlockett in https://github.com/lancedb/lance/pull/2692
• fix: add fix for slicing single array into many pages but don't have enough rows to do so by @westonpace in https://github.com/lancedb/lance/pull/2702
• fix: optimize cosine IVF_PQ index by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2698
• fix: search results are not correct after appending index by @BubbleCal in https://github.com/lancedb/lance/pull/2701
• fix: v2 writer used the wrong value to calculate how many rows were available in array by @westonpace in https://github.com/lancedb/lance/pull/2704
• perf: bitpacking exclude sign bit if not needed by @albertlockett in https://github.com/lancedb/lance/pull/2696
• fix: correct index type of inverted index by @BubbleCal in https://github.com/lancedb/lance/pull/2715

Performance Improvements 🚀

• perf: various fixes to improve vector transform performance at high scales by @westonpace in https://github.com/lancedb/lance/pull/2703

Full Changelog: https://github.com/lancedb/lance/compare/v0.16.0...v0.16.1

What's Changed

New Features 🎉

• feat: load multiple ivf-pq sorted files and commit index into dataset by @raunaks13 in https://github.com/lancedb/lance/pull/2681

Bug Fixes 🐛

• perf: bitpacking exclude sign bit if not needed by @albertlockett in https://github.com/lancedb/lance/pull/2696
• fix: correct index type of inverted index by @BubbleCal in https://github.com/lancedb/lance/pull/2715

Performance Improvements 🚀

• perf: various fixes to improve vector transform performance at high scales by @westonpace in https://github.com/lancedb/lance/pull/2703

Full Changelog: https://github.com/lancedb/lance/compare/v0.16.1-beta.3...v0.16.1-beta.4

What's Changed

New Features 🎉

• feat: move-stable row ids in compaction by @wjones127 in https://github.com/lancedb/lance/pull/2544

Bug Fixes 🐛

• fix: optimize cosine IVF_PQ index by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2698
• fix: search results are not correct after appending index by @BubbleCal in https://github.com/lancedb/lance/pull/2701
• fix: v2 writer used the wrong value to calculate how many rows were available in array by @westonpace in https://github.com/lancedb/lance/pull/2704

Full Changelog: https://github.com/lancedb/lance/compare/v0.16.1-beta.2...v0.16.1-beta.3

What's Changed

Bug Fixes 🐛

• fix: out of bounds error in bitpacking by @albertlockett in https://github.com/lancedb/lance/pull/2692
• fix: add fix for slicing single array into many pages but don't have enough rows to do so by @westonpace in https://github.com/lancedb/lance/pull/2702

Full Changelog: https://github.com/lancedb/lance/compare/v0.16.1-beta.1...v0.16.1-beta.2

What's Changed

New Features 🎉

• feat: make scalar index training configurable by @dsgibbons in https://github.com/lancedb/lance/pull/2686
• feat: migrate Bitmap & LabelList index file to v2 format by @BubbleCal in https://github.com/lancedb/lance/pull/2677
• feat: make default batch size configurable via environment variable by @westonpace in https://github.com/lancedb/lance/pull/2693

Full Changelog: https://github.com/lancedb/lance/compare/v0.16.0...v0.16.1-beta.1

What's Changed

Breaking Changes 🛠

• feat!: simplify take row api by @eddyxu in https://github.com/lancedb/lance/pull/2664
• feat!: distinguishable scalar index types by @eddyxu in https://github.com/lancedb/lance/pull/2671

New Features 🎉

• feat: standalone vector transform stage by @westonpace in https://github.com/lancedb/lance/pull/2566
• feat: coalesce scheduling of reads to speed up random access by @raunaks13 in https://github.com/lancedb/lance/pull/2636
• feat: add version tags by @dsgibbons in https://github.com/lancedb/lance/pull/2482
• feat: object store registry for custom object store providers by @maxburke in https://github.com/lancedb/lance/pull/2513
• feat: merge_insert update subcolumns by @wjones127 in https://github.com/lancedb/lance/pull/2639
• feat: implement disk-based inverted index by @BubbleCal in https://github.com/lancedb/lance/pull/2643
• feat: allow round-tripping of dictionary data through the v2 format by @westonpace in https://github.com/lancedb/lance/pull/2656
• feat: support transforming selected fragments in vector transform stage for ivf_pq index by @raunaks13 in https://github.com/lancedb/lance/pull/2657
• feat: expand tags api by @dsgibbons in https://github.com/lancedb/lance/pull/2679
• feat: add standalone shuffle for transformed ivf-pq vectors file by @raunaks13 in https://github.com/lancedb/lance/pull/2670
• feat: support bitpacking for signed types by @albertlockett in https://github.com/lancedb/lance/pull/2662
• feat: support loading huggingface image dataset and convert image to PIL by @eddyxu in https://github.com/lancedb/lance/pull/2684
• feat: return BM25 scores for FTS by @BubbleCal in https://github.com/lancedb/lance/pull/2687
• feat: add data file format / version information to manifest by @westonpace in https://github.com/lancedb/lance/pull/2673
• feat: add support for the null data type to v2 by @westonpace in https://github.com/lancedb/lance/pull/2685
• feat: add backpressure to v2 I/O scheduler by @westonpace in https://github.com/lancedb/lance/pull/2683

Bug Fixes 🐛

• fix: correctly encode a list type when all items are empty by @westonpace in https://github.com/lancedb/lance/pull/2653
• fix: improve error message when can't train PQ on too small dataset by @albertlockett in https://github.com/lancedb/lance/pull/2644
• fix: slight cleanups to path handling so that the indices builder tool properly supports Windows by @westonpace in https://github.com/lancedb/lance/pull/2689

Documentation 📚

• docs: schema evolution by @wjones127 in https://github.com/lancedb/lance/pull/1911
• docs: reorg the scalar index python docstring to make the index type clear by @eddyxu in https://github.com/lancedb/lance/pull/2678

Performance Improvements 🚀

• perf: add v2 fragment file metadata to the FileMetadataCache by @jiachengdb in https://github.com/lancedb/lance/pull/2647
• perf: add random take benchmark by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2654
• perf: benchmark lance vs parquet read time, write time, and compressed size by @raunaks13 in https://github.com/lancedb/lance/pull/2383

Other Changes

• refactor: new buffer abstractions in decoders by @westonpace in https://github.com/lancedb/lance/pull/2648

New Contributors

• @dsgibbons made their first contribution in https://github.com/lancedb/lance/pull/2482
• @maxburke made their first contribution in https://github.com/lancedb/lance/pull/2513

Full Changelog: https://github.com/lancedb/lance/compare/v0.15.0...v0.16.0