lance

What's Changed

New Features 🎉

• feat: standalone vector transform stage by @westonpace in https://github.com/lancedb/lance/pull/2566

Bug Fixes 🐛

• fix: correctly encode a list type when all items are empty by @westonpace in https://github.com/lancedb/lance/pull/2653

Performance Improvements 🚀

• perf: add v2 fragment file metadata to the FileMetadataCache by @jiachengdb in https://github.com/lancedb/lance/pull/2647
• perf: add random take benchmark by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2654

Other Changes

• refactor: new buffer abstractions in decoders by @westonpace in https://github.com/lancedb/lance/pull/2648

Full Changelog: https://github.com/lancedb/lance/compare/v0.15.0...v0.15.1-beta.1

What's Changed

Breaking Changes 🛠

• feat: upgrade DataFusion, Arrow, PyO3, ObjectStore by @wjones127 in https://github.com/lancedb/lance/pull/2594

New Features 🎉

• feat: add fast search to to_table method by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2591
• feat: loosen pyarrow pin by @wjones127 in https://github.com/lancedb/lance/pull/2597
• feat: add FSST string compression by @broccoliSpicy in https://github.com/lancedb/lance/pull/2470
• feat: integrate inverted index into lance index APIs by @BubbleCal in https://github.com/lancedb/lance/pull/2577
  • You can now create full text search indices and pass text search queries into the full_text_query argument
• feat: add projection to take and take_rows by @westonpace in https://github.com/lancedb/lance/pull/2623
• feat: add storage_options to _BaseLanceDatasink, LanceDatasink, LanceCommitter by @BitPhinix in https://github.com/lancedb/lance/pull/2619
• feat: add bitpack encoding for LanceV2 by @albertlockett in https://github.com/lancedb/lance/pull/2333
• feat: add a packed struct encoding to lance by @raunaks13 in https://github.com/lancedb/lance/pull/2593
• feat: support deletion with stable row id and scalar indices by @wjones127 in https://github.com/lancedb/lance/pull/2609

Bug Fixes 🐛

• fix: avoid making empty range requests as these will err on S3 by @westonpace in https://github.com/lancedb/lance/pull/2592
• fix: upgrade AWS SDK by @wjones127 in https://github.com/lancedb/lance/pull/2616
• fix: specify behavior version when creating dynamodb config by @westonpace in https://github.com/lancedb/lance/pull/2624
• fix: pin numpy version to avoid numpy >= 2.0 until pyarrow supports it by @westonpace in https://github.com/lancedb/lance/pull/2637

Performance Improvements 🚀

• perf: improve inverted index performance by @BubbleCal in https://github.com/lancedb/lance/pull/2574
• perf: improve v2 scan performance by @westonpace in https://github.com/lancedb/lance/pull/2604
• perf: allow fragment scan for nearest query if there is a prefilter by @jiachengdb in https://github.com/lancedb/lance/pull/2631
• perf: speed up fsst decompression by @broccoliSpicy in https://github.com/lancedb/lance/pull/2626
• perf: skip documents with WAND by @BubbleCal in https://github.com/lancedb/lance/pull/2632

Other Changes

• refactor: move planner.rs, logical_expr.rs, and sql.rs from lance to lance_datafusion by @westonpace in https://github.com/lancedb/lance/pull/2622
• refactor: remove custom ProjectionExec by @wjones127 in https://github.com/lancedb/lance/pull/2608

New Contributors

• @BitPhinix made their first contribution in https://github.com/lancedb/lance/pull/2619

Full Changelog: https://github.com/lancedb/lance/compare/v0.14.1...v0.15.0

What's Changed

New Features 🎉

• feat: add FSST string compression by @broccoliSpicy in https://github.com/lancedb/lance/pull/2470
• feat: integrate inverted index into lance index APIs by @BubbleCal in https://github.com/lancedb/lance/pull/2577
• feat: add projection to take and take_rows by @westonpace in https://github.com/lancedb/lance/pull/2623

Bug Fixes 🐛

• fix: upgrade AWS SDK by @wjones127 in https://github.com/lancedb/lance/pull/2616
• fix: specify behavior version when creating dynamodb config by @westonpace in https://github.com/lancedb/lance/pull/2624

Other Changes

• refactor: move planner.rs, logical_expr.rs, and sql.rs from lance to lance_datafusion by @westonpace in https://github.com/lancedb/lance/pull/2622

Full Changelog: https://github.com/lancedb/lance/compare/v0.15.0-beta.1...v0.15.0-beta.2

What's Changed

Breaking Changes 🛠

• feat: upgrade DataFusion, Arrow, PyO3, ObjectStore by @wjones127 in https://github.com/lancedb/lance/pull/2594

New Features 🎉

• feat: loosen pyarrow pin by @wjones127 in https://github.com/lancedb/lance/pull/2597

Performance Improvements 🚀

• perf: improve inverted index performance by @BubbleCal in https://github.com/lancedb/lance/pull/2574
• perf: improve v2 scan performance by @westonpace in https://github.com/lancedb/lance/pull/2604

Full Changelog: https://github.com/lancedb/lance/compare/v0.14.2-beta.1...v0.15.0-beta.1

What's Changed

New Features 🎉

• feat: add fast search to to_table method by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2591

Bug Fixes 🐛

• fix: avoid making empty range requests as these will err on S3 by @westonpace in https://github.com/lancedb/lance/pull/2592

Full Changelog: https://github.com/lancedb/lance/compare/v0.14.1...v0.14.2-beta.1

What's Changed

New Features 🎉

• feat: add inverted index by @BubbleCal in https://github.com/lancedb/lance/pull/2526
• feat(python): expose fast search flag in python by @eddyxu in https://github.com/lancedb/lance/pull/2573
• feat: use hyperloglog for cardinality estimation for dictionary encoding by @niyue in https://github.com/lancedb/lance/pull/2555
• feat: add scalar bitmap index to lance by @raunaks13 in https://github.com/lancedb/lance/pull/2560
• feat: add a standalone method for calculating the PQ model by @westonpace in https://github.com/lancedb/lance/pull/2562
• feat: integrate bitmap index into top-level lance APIs by @westonpace in https://github.com/lancedb/lance/pull/2575
• feat: support to set ef for search from python by @BubbleCal in https://github.com/lancedb/lance/pull/2569
• feat: dataset update accepts binary value by @eddyxu in https://github.com/lancedb/lance/pull/2579
• feat: do flat search if too many rows are filtered out by @BubbleCal in https://github.com/lancedb/lance/pull/2583
• feat: add label list index by @westonpace in https://github.com/lancedb/lance/pull/2581
• feat: allow passing in precomputed centroids to lance.util.KMeans by @jiachengdb in https://github.com/lancedb/lance/pull/2586
• feat: add standalone partition assignment operation by @westonpace in https://github.com/lancedb/lance/pull/2556

Bug Fixes 🐛

• fix: add support for list fields with names other than 'item' by @westonpace in https://github.com/lancedb/lance/pull/2580
• fix: propagate storage_options to v2 writer by @jiachengdb in https://github.com/lancedb/lance/pull/2578
• fix: be compatible with v1 index format by @BubbleCal in https://github.com/lancedb/lance/pull/2582
• fix: search panic with prefilter and HNSW by @BubbleCal in https://github.com/lancedb/lance/pull/2584

Performance Improvements 🚀

• perf: change scalar index to return RowIdTreeMap instead of u64 array by @westonpace in https://github.com/lancedb/lance/pull/2587

Full Changelog: https://github.com/lancedb/lance/compare/v0.14.0...v0.14.1

What's Changed

New Features 🎉

• feat: support to set ef for search from python by @BubbleCal in https://github.com/lancedb/lance/pull/2569

Full Changelog: https://github.com/lancedb/lance/compare/v0.14.1-beta.1...v0.14.1-beta.2

What's Changed

New Features 🎉

• feat: add inverted index by @BubbleCal in https://github.com/lancedb/lance/pull/2526
• feat(python): expose fast search flag in python by @eddyxu in https://github.com/lancedb/lance/pull/2573
• feat: use hyperloglog for cardinality estimation for dictionary encoding by @niyue in https://github.com/lancedb/lance/pull/2555
• feat: add scalar bitmap index to lance by @raunaks13 in https://github.com/lancedb/lance/pull/2560
• feat: add a standalone method for calculating the PQ model by @westonpace in https://github.com/lancedb/lance/pull/2562
• feat: integrate bitmap index into top-level lance APIs by @westonpace in https://github.com/lancedb/lance/pull/2575

Full Changelog: https://github.com/lancedb/lance/compare/v0.14.0...v0.14.1-beta.1

What's Changed

Breaking Changes 🛠

• fix: return correct sub index statistics by @BubbleCal in https://github.com/lancedb/lance/pull/2532
• feat: change to snapshot isolation by @wjones127 in https://github.com/lancedb/lance/pull/2537

New Features 🎉

• feat: enhance binary array encoding, make it the default by @westonpace in https://github.com/lancedb/lance/pull/2521
• feat: stable row id support in queries by @wjones127 in https://github.com/lancedb/lance/pull/2452
• feat: make it possible to create v2 fragments using the fragment API by @westonpace in https://github.com/lancedb/lance/pull/2534
• feat: add dictionary encoding to lance by @raunaks13 in https://github.com/lancedb/lance/pull/2409
• feat: provide a fast ANN search plan by @eddyxu in https://github.com/lancedb/lance/pull/2554
• feat: create standalone IVF training API in python by @westonpace in https://github.com/lancedb/lance/pull/2553
• feat: support u8 for existing distance types by @BubbleCal in https://github.com/lancedb/lance/pull/2501
• feat: reserve cores for IO by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2496
• feat: allow users to create a file writer without a schema by @westonpace in https://github.com/lancedb/lance/pull/2558

Bug Fixes 🐛

• fix: pytorch dot inference pass unwanted parameters by @eddyxu in https://github.com/lancedb/lance/pull/2539
• fix: show underlying cause of DynamoDB service error by @wjones127 in https://github.com/lancedb/lance/pull/2541
• fix: build HNSW with wrong distance type by @BubbleCal in https://github.com/lancedb/lance/pull/2533
• fix: propagate storage_options to LanceFragment.create and LanceDataset.commit by @jiachengdb in https://github.com/lancedb/lance/pull/2547
• fix: propagate shuffle_partition_concurrency from Python SDK by @jiachengdb in https://github.com/lancedb/lance/pull/2564
• fix: avoid multiple threads loading same index partition by @albertlockett in https://github.com/lancedb/lance/pull/2559
• fix: failed to return index stats if the first partition is empty by @BubbleCal in https://github.com/lancedb/lance/pull/2565
• fix: propagate storage_options to create_index for reading precomputed partition file by @jiachengdb in https://github.com/lancedb/lance/pull/2568

Documentation 📚

• docs: artefact_management_doc by @tanaymeh in https://github.com/lancedb/lance/pull/2504

Performance Improvements 🚀

• perf: avoid allocating bitvec for each insertion while indexing HNSW by @BubbleCal in https://github.com/lancedb/lance/pull/2527

Other Changes

• refactor: move IVF_HNSW_SQ & IVF_FLAT to new buliding & search path by @BubbleCal in https://github.com/lancedb/lance/pull/2469
• refactor: flat search to use datafusion top k by @eddyxu in https://github.com/lancedb/lance/pull/2535
• refactor: remove generic PQ impl and trait by @BubbleCal in https://github.com/lancedb/lance/pull/2529

New Contributors

• @jiachengdb made their first contribution in https://github.com/lancedb/lance/pull/2547
• @walterddr made their first contribution in https://github.com/lancedb/lance/pull/2382

Full Changelog: https://github.com/lancedb/lance/compare/v0.13.0...v0.14.0

What's Changed

New Features 🎉

• feat: add dictionary encoding to lance by @raunaks13 in https://github.com/lancedb/lance/pull/2409
• feat: provide a fast ANN search plan by @eddyxu in https://github.com/lancedb/lance/pull/2554
• feat: create standalone IVF training API in python by @westonpace in https://github.com/lancedb/lance/pull/2553
• feat: support u8 for existing distance types by @BubbleCal in https://github.com/lancedb/lance/pull/2501
• feat: reserve cores for IO by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2496
• feat: allow users to create a file writer without a schema by @westonpace in https://github.com/lancedb/lance/pull/2558

Bug Fixes 🐛

• fix: propagate shuffle_partition_concurrency from Python SDK by @jiachengdb in https://github.com/lancedb/lance/pull/2564
• fix: avoid multiple threads loading same index partition by @albertlockett in https://github.com/lancedb/lance/pull/2559

Other Changes

• refactor: remove generic PQ impl and trait by @BubbleCal in https://github.com/lancedb/lance/pull/2529

New Contributors

• @walterddr made their first contribution in https://github.com/lancedb/lance/pull/2382

Full Changelog: https://github.com/lancedb/lance/compare/v0.14.0-beta.1...v0.14.0-beta.2

What's Changed

Breaking Changes 🛠

• fix: return correct sub index statistics by @BubbleCal in https://github.com/lancedb/lance/pull/2532
• feat: change to snapshot isolation by @wjones127 in https://github.com/lancedb/lance/pull/2537

New Features 🎉

• feat: stable row id support in queries by @wjones127 in https://github.com/lancedb/lance/pull/2452
• feat: make it possible to create v2 fragments using the fragment API by @westonpace in https://github.com/lancedb/lance/pull/2534

Bug Fixes 🐛

• fix: pytorch dot inference pass unwanted parameters by @eddyxu in https://github.com/lancedb/lance/pull/2539
• fix: show underlying cause of DynamoDB service error by @wjones127 in https://github.com/lancedb/lance/pull/2541
• fix: build HNSW with wrong distance type by @BubbleCal in https://github.com/lancedb/lance/pull/2533
• fix: propagate storage_options to LanceFragment.create and LanceDataset.commit by @jiachengdb in https://github.com/lancedb/lance/pull/2547

Performance Improvements 🚀

• perf: avoid allocating bitvec for each insertion while indexing HNSW by @BubbleCal in https://github.com/lancedb/lance/pull/2527

Other Changes

• refactor: flat search to use datafusion top k by @eddyxu in https://github.com/lancedb/lance/pull/2535

New Contributors

• @jiachengdb made their first contribution in https://github.com/lancedb/lance/pull/2547

Full Changelog: https://github.com/lancedb/lance/compare/v0.13.1-beta.1...v0.14.0-beta.1

What's Changed

New Features 🎉

• feat: enhance binary array encoding, make it the default by @westonpace in https://github.com/lancedb/lance/pull/2521

Documentation 📚

• docs: artefact_management_doc by @tanaymeh in https://github.com/lancedb/lance/pull/2504

Other Changes

• refactor: move IVF_HNSW_SQ & IVF_FLAT to new buliding & search path by @BubbleCal in https://github.com/lancedb/lance/pull/2469

Full Changelog: https://github.com/lancedb/lance/compare/v0.13.0...v0.13.1-beta.1

What's Changed

Breaking Changes 🛠

• fix: dot distance so kmeans can converge by @eddyxu in https://github.com/lancedb/lance/pull/2509

New Features 🎉

• feat: convert binary logical encoding/decoding to physical array encoding/page decoding by @raunaks13 in https://github.com/lancedb/lance/pull/2426
• feat: support stable row ids in Dataset::take_rows() by @wjones127 in https://github.com/lancedb/lance/pull/2447
• feat: enable aarch64-pc-windows builds by @NickDarvey in https://github.com/lancedb/lance/pull/2512
• feat: expose DatasetPreFilter, PreFilter, and FilterLoader to the public API by @westonpace in https://github.com/lancedb/lance/pull/2522

Bug Fixes 🐛

• fix: low recall if many partitions with only 1 row by @BubbleCal in https://github.com/lancedb/lance/pull/2487
• fix: allow row id only in v2 by @westonpace in https://github.com/lancedb/lance/pull/2481

Documentation 📚

• docs: flickr8k_dataset_creation_example by @tanaymeh in https://github.com/lancedb/lance/pull/2489
• docs(python): note multiprocessing incompatibility by @wjones127 in https://github.com/lancedb/lance/pull/2506

Performance Improvements 🚀

• perf: don't write precomputed partitions to trace file by @westonpace in https://github.com/lancedb/lance/pull/2493
• perf: use faster kmean find partition routing for pq assignment by @eddyxu in https://github.com/lancedb/lance/pull/2515

Other Changes

• refactor: combined capacity updates and decoding for all physical decoders by @raunaks13 in https://github.com/lancedb/lance/pull/2508

New Contributors

• @NickDarvey made their first contribution in https://github.com/lancedb/lance/pull/2512

Full Changelog: https://github.com/lancedb/lance/compare/v0.12.3...v0.13.0

What's Changed

New Features 🎉

• feat: enable aarch64-pc-windows builds by @NickDarvey in https://github.com/lancedb/lance/pull/2512
• feat: expose DatasetPreFilter, PreFilter, and FilterLoader to the public API by @westonpace in https://github.com/lancedb/lance/pull/2522

Bug Fixes 🐛

• fix: allow row id only in v2 by @westonpace in https://github.com/lancedb/lance/pull/2481

Other Changes

• refactor: combined capacity updates and decoding for all physical decoders by @raunaks13 in https://github.com/lancedb/lance/pull/2508

New Contributors

• @NickDarvey made their first contribution in https://github.com/lancedb/lance/pull/2512

Full Changelog: https://github.com/lancedb/lance/compare/v0.13.0-beta.1...v0.13.0-beta.2

What's Changed

Breaking Changes 🛠

• fix: dot distance so kmeans can converge by @eddyxu in https://github.com/lancedb/lance/pull/2509

New Features 🎉

• feat: convert binary logical encoding/decoding to physical array encoding/page decoding by @raunaks13 in https://github.com/lancedb/lance/pull/2426
• feat: support stable row ids in Dataset::take_rows() by @wjones127 in https://github.com/lancedb/lance/pull/2447

Bug Fixes 🐛

• fix: low recall if many partitions with only 1 row by @BubbleCal in https://github.com/lancedb/lance/pull/2487

Documentation 📚

• docs: flickr8k_dataset_creation_example by @tanaymeh in https://github.com/lancedb/lance/pull/2489
• docs(python): note multiprocessing incompatibility by @wjones127 in https://github.com/lancedb/lance/pull/2506

Performance Improvements 🚀

• perf: don't write precomputed partitions to trace file by @westonpace in https://github.com/lancedb/lance/pull/2493
• perf: use faster kmean find partition routing for pq assignment by @eddyxu in https://github.com/lancedb/lance/pull/2515

Full Changelog: https://github.com/lancedb/lance/compare/v0.12.3...v0.13.0-beta.1

What's Changed

New Features 🎉

• feat: add arguments to allow resuming gpu indexing by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2473
• feat: make IVF_HNSW_SQ work with dot distance by @eddyxu in https://github.com/lancedb/lance/pull/2472
• feat(rust): add support for loongarch by @heiher in https://github.com/lancedb/lance/pull/2406
• feat: added some additional logging to the indexing process by @westonpace in https://github.com/lancedb/lance/pull/2488

Bug Fixes 🐛

• fix: correct with_new_children for knn flat by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2471

Documentation 📚

• docs: clip training example by @tanaymeh in https://github.com/lancedb/lance/pull/2477

Full Changelog: https://github.com/lancedb/lance/compare/v0.12.2...v0.12.3

What's Changed

New Features 🎉

• feat: add arguments to allow resuming gpu indexing by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2473

Bug Fixes 🐛

• fix: correct with_new_children for knn flat by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2471

Full Changelog: https://github.com/lancedb/lance/compare/v0.12.2...v0.12.3-beta.1

What's Changed

New Features 🎉

• feat: support IVF_FLAT index by @BubbleCal in https://github.com/lancedb/lance/pull/2418
• feat: add support for arbitrary global buffers to the v2 reader/writer by @westonpace in https://github.com/lancedb/lance/pull/2445
• feat: add pushdown to the read path by @westonpace in https://github.com/lancedb/lance/pull/2444
• feat: scan with stable row id by @wjones127 in https://github.com/lancedb/lance/pull/2441
• feat(java): add lance spark connector for read by @LuQQiu in https://github.com/lancedb/lance/pull/2429
• feat: pickle the manifest when pickling a dataset by @westonpace in https://github.com/lancedb/lance/pull/2459
• feat: store IVF in global buffer by @BubbleCal in https://github.com/lancedb/lance/pull/2449

Bug Fixes 🐛

• fix: lance.torch.LanceDataset spelt readahead wrong by @wjones127 in https://github.com/lancedb/lance/pull/2453
• fix: respect storage options on create_index by @joshua-auchincloss in https://github.com/lancedb/lance/pull/2460
• fix: incorrect logic when decoding manifest from last block by @westonpace in https://github.com/lancedb/lance/pull/2464

Documentation 📚

• docs: add spark-lance-connector readme by @LuQQiu in https://github.com/lancedb/lance/pull/2456

Performance Improvements 🚀

• perf: speed up hnsw build by @eddyxu in https://github.com/lancedb/lance/pull/2448
• perf: better L2 code generation by @eddyxu in https://github.com/lancedb/lance/pull/2450

Other Changes

• refactor: pull row_id and deletion vector out of v1 file reader by @wjones127 in https://github.com/lancedb/lance/pull/2436
• refactor: move Prefilter to a trait and move VectorIndex to lance-vector by @eddyxu in https://github.com/lancedb/lance/pull/2442
• refactor: move hnsw from lance crate to lance-index crate by @eddyxu in https://github.com/lancedb/lance/pull/2446
• refactor: impl the sub index trait for HNSW and clean code by @BubbleCal in https://github.com/lancedb/lance/pull/2462

New Contributors

• @joshua-auchincloss made their first contribution in https://github.com/lancedb/lance/pull/2460

Full Changelog: https://github.com/lancedb/lance/compare/v0.12.1...v0.12.2

What's Changed

New Features 🎉

• feat: store IVF in global buffer by @BubbleCal in https://github.com/lancedb/lance/pull/2449

Bug Fixes 🐛

• fix: respect storage options on create_index by @joshua-auchincloss in https://github.com/lancedb/lance/pull/2460
• fix: incorrect logic when decoding manifest from last block by @westonpace in https://github.com/lancedb/lance/pull/2464

Documentation 📚

• docs: add spark-lance-connector readme by @LuQQiu in https://github.com/lancedb/lance/pull/2456

New Contributors

• @joshua-auchincloss made their first contribution in https://github.com/lancedb/lance/pull/2460

Full Changelog: https://github.com/lancedb/lance/compare/v0.12.2-beta.1...v0.12.2-beta.2

What's Changed

New Features 🎉

• feat: support IVF_FLAT index by @BubbleCal in https://github.com/lancedb/lance/pull/2418
• feat: add support for arbitrary global buffers to the v2 reader/writer by @westonpace in https://github.com/lancedb/lance/pull/2445
• feat: add pushdown to the read path by @westonpace in https://github.com/lancedb/lance/pull/2444
• feat: scan with stable row id by @wjones127 in https://github.com/lancedb/lance/pull/2441
• feat(java): add lance spark connector for read by @LuQQiu in https://github.com/lancedb/lance/pull/2429
• feat: pickle the manifest when pickling a dataset by @westonpace in https://github.com/lancedb/lance/pull/2459

Bug Fixes 🐛

• fix: lance.torch.LanceDataset spelt readahead wrong by @wjones127 in https://github.com/lancedb/lance/pull/2453

Performance Improvements 🚀

• perf: speed up hnsw build by @eddyxu in https://github.com/lancedb/lance/pull/2448
• perf: better L2 code generation by @eddyxu in https://github.com/lancedb/lance/pull/2450

Other Changes

• refactor: pull row_id and deletion vector out of v1 file reader by @wjones127 in https://github.com/lancedb/lance/pull/2436
• refactor: move Prefilter to a trait and move VectorIndex to lance-vector by @eddyxu in https://github.com/lancedb/lance/pull/2442
• refactor: move hnsw from lance crate to lance-index crate by @eddyxu in https://github.com/lancedb/lance/pull/2446

Full Changelog: https://github.com/lancedb/lance/compare/v0.12.1...v0.12.2-beta.1