lance

What's Changed

• docs: update to_batches batch_size documentation by @alexkohler in https://github.com/lancedb/lance/pull/1779
• chore: minor improvement over kmeans algorithm by @eddyxu in https://github.com/lancedb/lance/pull/1777
• perf: reduce torch memory allocation and avoid calculating y^2 multiple times in one iteration by @eddyxu in https://github.com/lancedb/lance/pull/1783
• chore: fix cuda-based indexing with f16 by @eddyxu in https://github.com/lancedb/lance/pull/1786
• fix: nan handling during training and dist accumulation by @eddyxu in https://github.com/lancedb/lance/pull/1787
• chore: revert async_dataset usage in cuda kmeans training by @eddyxu in https://github.com/lancedb/lance/pull/1788

New Contributors

• @alexkohler made their first contribution in https://github.com/lancedb/lance/pull/1779

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.3...v0.9.4

What's Changed

• chore: create f16 and bf16 test dataset from sift and gist by @eddyxu in https://github.com/lancedb/lance/pull/1761
• fix: fix query over f16 knn node and sift benchmark by @eddyxu in https://github.com/lancedb/lance/pull/1762
• fix: fix shuffler k-list merge and save GPU trained centroids to file by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1763
• fix: pick up metric type from the existing index for the refine step by @eddyxu in https://github.com/lancedb/lance/pull/1764
• fix: switch back to stable sampling, and tune memory usage during IVF shuffle by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1765
• fix: handle nan and zeros gracefully in kmean training by @eddyxu in https://github.com/lancedb/lance/pull/1766
• fix: fix latent sample size bug in torch dataset by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1772
• chore: handle nans in GPU training by @eddyxu in https://github.com/lancedb/lance/pull/1769
• chore(python): prevent segfault using LanceFragment by @changhiskhan in https://github.com/lancedb/lance/pull/1768
• chore: skip flaky test by @changhiskhan in https://github.com/lancedb/lance/pull/1774
• feat(python): add with_row_id to fragment scanner by @changhiskhan in https://github.com/lancedb/lance/pull/1770
• chore: fix f16 distance computation on CPU by @eddyxu in https://github.com/lancedb/lance/pull/1776

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.2...v0.9.3

What's Changed

• chore: add test for f16 embeddings by @eddyxu in https://github.com/lancedb/lance/pull/1744
• chore(rust): make clippy happy for the pyo3 code by @changhiskhan in https://github.com/lancedb/lance/pull/1748
• chore: clean up pyarrow 12 check and pandas version by @eddyxu in https://github.com/lancedb/lance/pull/1749
• feat: improve ergonomic of bf16 array by @eddyxu in https://github.com/lancedb/lance/pull/1752
• feat: cast between fsl of floats and fsl of bf16 by @eddyxu in https://github.com/lancedb/lance/pull/1753
• chore: add utility to materialize in-sample ground truth by @eddyxu in https://github.com/lancedb/lance/pull/1755
• fix: proper mixed-percision training by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1757
• chore: make lance-arrow compilable for wasm target by @eddyxu in https://github.com/lancedb/lance/pull/1758
• fix: fix gpu cosine and add early stop for async dataset by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1759

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.1...v0.9.2

New features

• feat: support drop column api in Dataset (Rust only) by @rok in https://github.com/lancedb/lance/pull/1695
• feat: allow fixed size tensors to be used as query vectors by @westonpace in https://github.com/lancedb/lance/pull/1736

Performance improvements

• perf: implement external shuffle when building IVF by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1735

Bug fixes

• fix: take fails on non-unique indexes by @rok in https://github.com/lancedb/lance/pull/1724
• fix: raise the manylinux version to 2_24 on aarch64 linux builds to avoid cross compilation issues by @westonpace in https://github.com/lancedb/lance/pull/1737
• fix: the routing to find ground truth in sift benchmark by @eddyxu in https://github.com/lancedb/lance/pull/1740

Other changes

• chore(python): update sift benchmarks to include GIST1M as well by @changhiskhan in https://github.com/lancedb/lance/pull/1732
• chore(benchmarks): update the full benchmark results in csv format by @changhiskhan in https://github.com/lancedb/lance/pull/1734
• chore: upgrade to DataFusion 34 by @wjones127 in https://github.com/lancedb/lance/pull/1728

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.0...v0.9.1

Summary

• Stats-based predicate pushdown
• Scalar index
• Tensorflow and PyTorch data loader
• Pre / post filter combined with vector search
• Performance improvement across the stack

Breaking changes:

• Change IVF_PQ algorithm for cosine distance. Requires rebuilding index with cosine distance.
• Bump pyarrow version to 12.0+

What's Changed

• feat: add a cache for dynamodb schema validation by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1308
• chore: speed up kmean training for cosine by @eddyxu in https://github.com/lancedb/lance/pull/1334
• feat: add take() method to RecordBatchExt by @eddyxu in https://github.com/lancedb/lance/pull/1337
• feat(python): expose row id in python API by @eddyxu in https://github.com/lancedb/lance/pull/1339
• feat: data generation of dbpedia dataset by @eddyxu in https://github.com/lancedb/lance/pull/1340
• feat: build ivf partition using disk based shuffler by @eddyxu in https://github.com/lancedb/lance/pull/1312
• feat: friendlier error messages in nearest api by @rok in https://github.com/lancedb/lance/pull/1336
• chore: index / recall benchmark over dbpedia by @eddyxu in https://github.com/lancedb/lance/pull/1348
• feat: support storing page-level stats by @wjones127 in https://github.com/lancedb/lance/pull/1316
• fix: pq cosine fast lookup table by @eddyxu in https://github.com/lancedb/lance/pull/1354
• chore: compute distance using pytorch and GPU/MPS by @eddyxu in https://github.com/lancedb/lance/pull/1351
• feat: train kmean using pytorch by @eddyxu in https://github.com/lancedb/lance/pull/1358
• build: use larger runner for doc build by @eddyxu in https://github.com/lancedb/lance/pull/1364
• feat(python): gpu based ivf partition training by @eddyxu in https://github.com/lancedb/lance/pull/1361
• fix: stop reading latest manifest by @wjones127 in https://github.com/lancedb/lance/pull/1365
• chore: improve kmean training performance on CUDA by @eddyxu in https://github.com/lancedb/lance/pull/1368
• chore: improve kmean performance on MPS by @eddyxu in https://github.com/lancedb/lance/pull/1370
• chore: use torch.index_add to compute new centroids, to improve training performance on MPS by @eddyxu in https://github.com/lancedb/lance/pull/1371
• feat: schema::field_by_id() by @eddyxu in https://github.com/lancedb/lance/pull/1375
• feat(python): design an image extension type by @rok in https://github.com/lancedb/lance/pull/1272
• perf: improve KNN and ANN performance by @wjones127 in https://github.com/lancedb/lance/pull/1367
• feat: collect int/float/boolean/date page-level statistics on write by @rok in https://github.com/lancedb/lance/pull/1346
• chore: part 1/N of refactoring vector index into separate crate by @eddyxu in https://github.com/lancedb/lance/pull/1388
• fix: handle larger arrays in take by @wjones127 in https://github.com/lancedb/lance/pull/1383
• chore: cleanup tests to avoid errors when optional components are not present by @westonpace in https://github.com/lancedb/lance/pull/1374
• chore: object write trait by @eddyxu in https://github.com/lancedb/lance/pull/1389
• refactor: fast path to find fragments for flat scan by @eddyxu in https://github.com/lancedb/lance/pull/1394
• refactor: move reader trait to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1393
• refactor: move pq to lance-index by @eddyxu in https://github.com/lancedb/lance/pull/1400
• refactor: make pq a batch transformer by @eddyxu in https://github.com/lancedb/lance/pull/1401
• feat: run pq portion of ivf_pq in parallel by @westonpace in https://github.com/lancedb/lance/pull/1386
• feat: generic shuffler over RecordBatchStream by @eddyxu in https://github.com/lancedb/lance/pull/1402
• feat: remap indices on compaction by @westonpace in https://github.com/lancedb/lance/pull/1403
• chore: remove unused crate by @eddyxu in https://github.com/lancedb/lance/pull/1405
• docs: make overwrite row green by @wjones127 in https://github.com/lancedb/lance/pull/1409
• feat: add removed_indices to CreateIndex transaction operation by @eddyxu in https://github.com/lancedb/lance/pull/1408
• feat(rust): incremental index update by @eddyxu in https://github.com/lancedb/lance/pull/1406
• feat(python): expose index optimization via python by @eddyxu in https://github.com/lancedb/lance/pull/1412
• test: add test case to ensure optimize returns to flat KNN by @westonpace in https://github.com/lancedb/lance/pull/1416
• fix: fix bug in index remapping when plan contained multiple rewrite groups by @westonpace in https://github.com/lancedb/lance/pull/1415
• chore: upgrade to datafusion 32 by @wjones127 in https://github.com/lancedb/lance/pull/1391
• ci: cross compile arm wheels by @wjones127 in https://github.com/lancedb/lance/pull/1407
• test: add new ann scenarios to the python benchmarks by @westonpace in https://github.com/lancedb/lance/pull/1411
• chore: instrument various steps in the ann search by @westonpace in https://github.com/lancedb/lance/pull/1404
• refactor: refactor flat search to lance-index by @eddyxu in https://github.com/lancedb/lance/pull/1419
• refactor: move encodings to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1425
• feat: expose latest version id api by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1426
• refactor: use function pointers instead of trait objects by @wjones127 in https://github.com/lancedb/lance/pull/1424
• feat: automatically convert image to tensors in TF data pipeline by @rok in https://github.com/lancedb/lance/pull/1420
• refactor: migrate schema and data types to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1429
• perf: better parallelism in delete vector prefiltering by @westonpace in https://github.com/lancedb/lance/pull/1428
• test: fix flaky tests involving tokio::fs::File by @westonpace in https://github.com/lancedb/lance/pull/1430
• perf: use selection vector strategy to improve exact knn performance with deletions by @wjones127 in https://github.com/lancedb/lance/pull/1418
• chore: use arrow 47 function by @eddyxu in https://github.com/lancedb/lance/pull/1439
• refactor: move format definitions to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1440
• refactor: migrate object reader and object writer by @eddyxu in https://github.com/lancedb/lance/pull/1442
• fix: fix an issue where the GPU index trainer was taking too much data into memory by @westonpace in https://github.com/lancedb/lance/pull/1447
• feat: store a separate tensor blob for IVF centroids by @eddyxu in https://github.com/lancedb/lance/pull/1446
• refactor: move all Python operations to the same runtime by @wjones127 in https://github.com/lancedb/lance/pull/1445
• chore: bump prost version to latest by @eddyxu in https://github.com/lancedb/lance/pull/1449
• chore: update half to 2.3.1 by @jacobBaumbach in https://github.com/lancedb/lance/pull/1450
• feat: allow prefiltering to be used with an index by @westonpace in https://github.com/lancedb/lance/pull/1435
• feat: benchmark and improve L2 partition compute by @eddyxu in https://github.com/lancedb/lance/pull/1453
• chore: increase ivf assignment parallism during indexing by @eddyxu in https://github.com/lancedb/lance/pull/1451
• feat: support keyboard interrupt in Python by @wjones127 in https://github.com/lancedb/lance/pull/1438
• feat: add parameter to split by file size by @wjones127 in https://github.com/lancedb/lance/pull/1444
• ci: fix ARM build due to Ring dependency by @wjones127 in https://github.com/lancedb/lance/pull/1462
• refactor: move read and write manifest file to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1467
• feat: create_index take torch.device object by @eddyxu in https://github.com/lancedb/lance/pull/1465
• feat: added dataset stats api by @albertlockett in https://github.com/lancedb/lance/pull/1452
• refactor: move commit traits to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1469
• chore: use ruff format to replace isort and black by @eddyxu in https://github.com/lancedb/lance/pull/1472
• refactor: move ObjectStore, FileReader and FileWriter to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1473
• perf: support multi-threading shuffler by @eddyxu in https://github.com/lancedb/lance/pull/1474
• feat: poor man's SIMD lib by @eddyxu in https://github.com/lancedb/lance/pull/1478
• perf: use simd lib to implement dot by @eddyxu in https://github.com/lancedb/lance/pull/1480
• feat: expose progress on write_fragments and write_dataset by @wjones127 in https://github.com/lancedb/lance/pull/1464
• feat: split out datagen utilities, expand them, expose to python by @westonpace in https://github.com/lancedb/lance/pull/1315
• chore: remove outdated warnings about prefiltering with a vector index by @westonpace in https://github.com/lancedb/lance/pull/1484
• fix: fix L2 computation on GPU by @eddyxu in https://github.com/lancedb/lance/pull/1485
• perf: improve kmeans and make pq training multi-threaded by @eddyxu in https://github.com/lancedb/lance/pull/1479
• chore: mention GPU support in README by @eddyxu in https://github.com/lancedb/lance/pull/1489
• fix: fix PQ training metric type is not appropriately propogated by @eddyxu in https://github.com/lancedb/lance/pull/1493
• docs: clarify behaviour of refine_factor by @albertlockett in https://github.com/lancedb/lance/pull/1496
• ci: cancel in progress runs on new push by @albertlockett in https://github.com/lancedb/lance/pull/1497
• chore: remove unused value settings by @eddyxu in https://github.com/lancedb/lance/pull/1494
• feat: provide a f32x16 abstraction to make unrolling 256-bit code easier by @eddyxu in https://github.com/lancedb/lance/pull/1495
• fix: remove channel closed messages by @wjones127 in https://github.com/lancedb/lance/pull/1502
• perf: dimension-based kernel for L2 and Cosine by @eddyxu in https://github.com/lancedb/lance/pull/1503
• feat: add location for all error by @Weijun-H in https://github.com/lancedb/lance/pull/1475
• feat: add sorting to the scanner by @westonpace in https://github.com/lancedb/lance/pull/1498
• feat: add tf.data APIs for reading batches by @wjones127 in https://github.com/lancedb/lance/pull/1488
• feat: experimental avx512 features by @eddyxu in https://github.com/lancedb/lance/pull/1506
• feat: add read ahead for take scan by @wjones127 in https://github.com/lancedb/lance/pull/1501
• feat: use caller location in error conversion functions by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1510
• chore(rust): reduce debug message log level by @changhiskhan in https://github.com/lancedb/lance/pull/1512
• feat: collect page-level statistics on write by @rok in https://github.com/lancedb/lance/pull/1335
• feat(rust): simd ops of reduce min, min, find and gather by @eddyxu in https://github.com/lancedb/lance/pull/1514
• feat: add btree scalar index by @westonpace in https://github.com/lancedb/lance/pull/1476
• feat: support true in deletion logic by @Weijun-H in https://github.com/lancedb/lance/pull/1515
• fix: make sure we have physical rows by @wjones127 in https://github.com/lancedb/lance/pull/1511
• chore: benchmark of large IVF parrtitions by @eddyxu in https://github.com/lancedb/lance/pull/1524
• feat: make dot generic to support bf16/f16/f32 with one dot_distance interface. by @eddyxu in https://github.com/lancedb/lance/pull/1522
• chore: add same target-features to python pyo3 build by @eddyxu in https://github.com/lancedb/lance/pull/1527
• feat: expose index cache configure via open dataset API by @eddyxu in https://github.com/lancedb/lance/pull/1523
• fix: fix assertion of cosine values by @eddyxu in https://github.com/lancedb/lance/pull/1530
• feat: generic cosine code by @eddyxu in https://github.com/lancedb/lance/pull/1537
• perf: improve f16 performance for norm L2 on aarch64 by @eddyxu in https://github.com/lancedb/lance/pull/1539
• feat: make L2 generic to work with all float numbers by @eddyxu in https://github.com/lancedb/lance/pull/1532
• fix: pq index does not handle dot product metric correctly during search by @rok in https://github.com/lancedb/lance/pull/1536
• chore: move scalar_index benchmark to break circular dependency by @westonpace in https://github.com/lancedb/lance/pull/1540
• feat: safer API for physical_rows by @wjones127 in https://github.com/lancedb/lance/pull/1529
• feat: implement datafusion tableprovider trait for Dataset by @universalmind303 in https://github.com/lancedb/lance/pull/1526
• feat: expose Dataset.validate() in Python by @wjones127 in https://github.com/lancedb/lance/pull/1538
• fix: add versioning and bypass broken row counts by @wjones127 in https://github.com/lancedb/lance/pull/1534
• feat: generic kmeans that supports bf16 and f16 by @eddyxu in https://github.com/lancedb/lance/pull/1544
• chore: disable avx512 for now by @eddyxu in https://github.com/lancedb/lance/pull/1546
• chore: fix type inference errors in benchmarks by @westonpace in https://github.com/lancedb/lance/pull/1556
• chore: provide a trait to dynamically dispatch different pq based on different vector data type by @eddyxu in https://github.com/lancedb/lance/pull/1555
• chore: update the CI build to check/build all crates in the workspace and not just the lance crate by @westonpace in https://github.com/lancedb/lance/pull/1557
• feat: make it possible to create and load scalar indices for a dataset by @westonpace in https://github.com/lancedb/lance/pull/1516
• feat: generic Product Quantizatoin by @eddyxu in https://github.com/lancedb/lance/pull/1560
• test: add property-based testing for statistics by @wjones127 in https://github.com/lancedb/lance/pull/1554
• feat: ffi to accelerate norm_l2 for f16 if the instruction set is available by @eddyxu in https://github.com/lancedb/lance/pull/1562
• feat: extend FSL with sample by @eddyxu in https://github.com/lancedb/lance/pull/1572
• feat: allow for more advanced storage options in objectstore by @universalmind303 in https://github.com/lancedb/lance/pull/1547
• feat: implement as_slice for bfloat16 array by @eddyxu in https://github.com/lancedb/lance/pull/1574
• perf: add bf16 benchmarks by @eddyxu in https://github.com/lancedb/lance/pull/1575
• feat: f16 for L2 by @eddyxu in https://github.com/lancedb/lance/pull/1577
• chore: update cc dependency to 1.0.83 by @westonpace in https://github.com/lancedb/lance/pull/1578
• feat: make IVF model support f16 and bf16 by @eddyxu in https://github.com/lancedb/lance/pull/1573
• feat: allow the scanner to take advantage of scalar indices by @westonpace in https://github.com/lancedb/lance/pull/1543
• chore: dotprod should be on mac target, not haswell and better randomness for bf16 by @westonpace in https://github.com/lancedb/lance/pull/1579
• feat: make Dataset::nearest() accepts arbitrary query type by @eddyxu in https://github.com/lancedb/lance/pull/1582
• chore(rust): remove extraneous dbg message by @changhiskhan in https://github.com/lancedb/lance/pull/1598
• feat: torch cache-able dataset, with sampling support by @eddyxu in https://github.com/lancedb/lance/pull/1591
• fix: tell writer correct schema when writing index file by @wjones127 in https://github.com/lancedb/lance/pull/1518
• feat: add support for remapping scalar indices during compaction by @westonpace in https://github.com/lancedb/lance/pull/1571
• refactor: switch to using DataFusions physical expr by @wjones127 in https://github.com/lancedb/lance/pull/1581
• chore: various fixes for Python benchmarks by @wjones127 in https://github.com/lancedb/lance/pull/1513
• feat: adaptive cuda allocation for l2/cosine distance computation by @eddyxu in https://github.com/lancedb/lance/pull/1601
• fix: fix a memory leak where a dataset would not be fully deleted by @westonpace in https://github.com/lancedb/lance/pull/1606
• fix: google objectstore uses proper gs configuration by @universalmind303 in https://github.com/lancedb/lance/pull/1608
• perf: kmean fit uses cached torch dataset by @eddyxu in https://github.com/lancedb/lance/pull/1603
• fix: add migration for bad fragment bitmaps by @westonpace in https://github.com/lancedb/lance/pull/1611
• feat: allow scalar indices to be updated with new data by @westonpace in https://github.com/lancedb/lance/pull/1576
• feat: add python bindings for creating scalar indices by @westonpace in https://github.com/lancedb/lance/pull/1592
• fix: handle no max value for string by @wjones127 in https://github.com/lancedb/lance/pull/1600
• feat: expose index cache size by @rok in https://github.com/lancedb/lance/pull/1587
• feat: track index cache hit rate by @rok in https://github.com/lancedb/lance/pull/1586
• feat: serialize arbitrary float type of PQ to protobuf by @eddyxu in https://github.com/lancedb/lance/pull/1624
• ci: use M1 runner for now for release by @wjones127 in https://github.com/lancedb/lance/pull/1623
• feat: coerce float array for nearest query by @eddyxu in https://github.com/lancedb/lance/pull/1618
• chore: expose avx512fp16 feature via main lance crate by @eddyxu in https://github.com/lancedb/lance/pull/1626
• feat: make partition calculation parallel by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1625
• feat(rust): simplify object store option API by @wjones127 in https://github.com/lancedb/lance/pull/1627
• fix: fix chunk size issue by @wjones127 in https://github.com/lancedb/lance/pull/1630
• perf: more efficient treemap implementation for row ids by @wjones127 in https://github.com/lancedb/lance/pull/1632
• feat(python): add index_cache_hit_rate to index_stats() by @rok in https://github.com/lancedb/lance/pull/1631
• chore: make lance-linalg benchmark ready to test bf16 data by @eddyxu in https://github.com/lancedb/lance/pull/1634
• perf: fast L2 distance table build by @eddyxu in https://github.com/lancedb/lance/pull/1639
• fix: correctly avg centroids in update logic in GPU IVF training by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1646
• perf: add a fast path for converting bytes into array when the bytes has the correct alignment by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1652
• fix: prevent OOM when IVF centroids are provided by @wjones127 in https://github.com/lancedb/lance/pull/1653
• test: fix for test by @wjones127 in https://github.com/lancedb/lance/pull/1644
• perf: minor change to cleanup allowing for size to be collected in parallel by @westonpace in https://github.com/lancedb/lance/pull/1649
• perf: add type coersion for in-list expressions by @westonpace in https://github.com/lancedb/lance/pull/1655
• chore: minor changes to tracing instrumentation by @westonpace in https://github.com/lancedb/lance/pull/1619
• fix: fix error message for invalid nprobes by @albertlockett in https://github.com/lancedb/lance/pull/1666
• feat: add support for update queries by @wjones127 in https://github.com/lancedb/lance/pull/1585
• fix: support no-op filters again by @wjones127 in https://github.com/lancedb/lance/pull/1669
• fix: row_id range fix for index training on gpu by @jerryyifei in https://github.com/lancedb/lance/pull/1663
• feat: better warnings when the PQ assignment over cosine distance is wrong by @eddyxu in https://github.com/lancedb/lance/pull/1672
• fix: add retries for failed response stream by @wjones127 in https://github.com/lancedb/lance/pull/1671
• chore: add utility to compute ground truth for benchmarks by @eddyxu in https://github.com/lancedb/lance/pull/1668
• fix: dont use scalar indices unless we are prefiltering by @westonpace in https://github.com/lancedb/lance/pull/1678
• fix: lance pytorch dataset parameter to load with row_id by @eddyxu in https://github.com/lancedb/lance/pull/1676
• feat: a tensor dataset that shared with the same behavior as Lance torch Dataset by @eddyxu in https://github.com/lancedb/lance/pull/1679
• chore: add new python benchmarks for testing scalar indices by @westonpace in https://github.com/lancedb/lance/pull/1658
• feat: add option to pass in precomputed row_id -> ivf partiton mapping and compute partiiton on GPU by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1680
• fix: make sure to prefilter the flat portion of a combined knn by @westonpace in https://github.com/lancedb/lance/pull/1583
• perf: use datafusion to shuffle index partition data by @wjones127 in https://github.com/lancedb/lance/pull/1645
• feat: add batch buffering and async loading to torch.LanceDataset by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1687
• feat: optimized pushdown scanner by @wjones127 in https://github.com/lancedb/lance/pull/1328
• fix: add shutdown to async loader by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1690
• fix: use eplison to handle all zero cosine values by @eddyxu in https://github.com/lancedb/lance/pull/1696
• fix: prevent stats meta from breaking old readers by @wjones127 in https://github.com/lancedb/lance/pull/1699
• fix: add _rowid when use_stats=False by @wjones127 in https://github.com/lancedb/lance/pull/1700
• perf: revert back to hashmap by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1692
• fix: remove default memory cap for index training by @wjones127 in https://github.com/lancedb/lance/pull/1702
• feat: do not use residual vector for cosine similarity by @eddyxu in https://github.com/lancedb/lance/pull/1708
• feat: add support for new and deleted data to scalar indices by @westonpace in https://github.com/lancedb/lance/pull/1689
• fix: update list_indices to report if an index is vector or scalar by @westonpace in https://github.com/lancedb/lance/pull/1710
• perf: allow take to process multiple fragments in parallel by @westonpace in https://github.com/lancedb/lance/pull/1713
• feat: turn on argument tracking in tracing by @wjones127 in https://github.com/lancedb/lance/pull/1706
• perf: make sure we use multiple threads when scanning by @wjones127 in https://github.com/lancedb/lance/pull/1705
• chore: kmeans fit takes pyarrow FixedSizeListArray by @eddyxu in https://github.com/lancedb/lance/pull/1714
• revert: use eplison to handle all zero cosine values (#1696) by @eddyxu in https://github.com/lancedb/lance/pull/1715
• chore: add ruff copyright check by @eddyxu in https://github.com/lancedb/lance/pull/1716
• chore: compute pairwise cosine using pytorch by @eddyxu in https://github.com/lancedb/lance/pull/1717
• chore: normalize vector kernel by @eddyxu in https://github.com/lancedb/lance/pull/1720
• fix: fix l2 normalize by @eddyxu in https://github.com/lancedb/lance/pull/1722
• perf: use an asynchronous open function even for local files by @westonpace in https://github.com/lancedb/lance/pull/1721
• perf: small performance fixes for scan by @wjones127 in https://github.com/lancedb/lance/pull/1719
• fix: cosine kmeans by @eddyxu in https://github.com/lancedb/lance/pull/1723
• fix: cosine kmeans on GPU by @eddyxu in https://github.com/lancedb/lance/pull/1726
• fix: pq code for cosine distance by @eddyxu in https://github.com/lancedb/lance/pull/1727
• chore: adjust cosine value from l2 distance by @eddyxu in https://github.com/lancedb/lance/pull/1730
• fix: various fixes to GPU kmeans by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1731
• feat: handroll ivf partition shuffle by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1729

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.0...v0.9.0

New features

• feat: add support for new and deleted data to scalar indices by @westonpace in https://github.com/lancedb/lance/pull/1689
• feat: turn on argument tracking in tracing by @wjones127 in https://github.com/lancedb/lance/pull/1706

Bug fixes

• fix: do not use residual vector for cosine similarity by @eddyxu in https://github.com/lancedb/lance/pull/1708
• fix: update list_indices to report if an index is vector or scalar by @westonpace in https://github.com/lancedb/lance/pull/1710
• revert: use eplison to handle all zero cosine values (#1696) by @eddyxu in https://github.com/lancedb/lance/pull/1715

Performance improvements

• perf: allow take to process multiple fragments in parallel by @westonpace in https://github.com/lancedb/lance/pull/1713
• perf: make sure we use multiple threads when scanning by @wjones127 in https://github.com/lancedb/lance/pull/1705

Other changes

• chore: kmeans fit takes pyarrow FixedSizeListArray by @eddyxu in https://github.com/lancedb/lance/pull/1714
• chore: add ruff copyright check by @eddyxu in https://github.com/lancedb/lance/pull/1716

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.20...v0.8.21

Bug Fixes

• fix: add shutdown to async loader by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1690
• fix: use eplison to handle all zero cosine values by @eddyxu in https://github.com/lancedb/lance/pull/1696
• fix: prevent stats meta from breaking old readers by @wjones127 in https://github.com/lancedb/lance/pull/1699
• fix: add _rowid when use_stats=False by @wjones127 in https://github.com/lancedb/lance/pull/1700
• perf: revert back to hashmap by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1692
• fix: remove default memory cap for index training by @wjones127 in https://github.com/lancedb/lance/pull/1702

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.19...v0.8.20

New Features

• feat: a tensor dataset that shared with the same behavior as Lance torch Dataset by @eddyxu in https://github.com/lancedb/lance/pull/1679
• feat: add option to pass in precomputed row_id -> ivf partiton mapping and compute partiiton on GPU by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1680
• feat: add batch buffering and async loading to torch.LanceDataset by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1687
• feat: optimized pushdown scanner by @wjones127 in https://github.com/lancedb/lance/pull/1328

Bug Fixes

• fix: dont use scalar indices unless we are prefiltering by @westonpace in https://github.com/lancedb/lance/pull/1678
• fix: lance pytorch dataset parameter to load with row_id by @eddyxu in https://github.com/lancedb/lance/pull/1676
• fix: make sure to prefilter the flat portion of a combined knn by @westonpace in https://github.com/lancedb/lance/pull/1583

Performance Improvements

• perf: use datafusion to shuffle index partition data by @wjones127 in https://github.com/lancedb/lance/pull/1645

Other Changes

• chore: add utility to compute ground truth for benchmarks by @eddyxu in https://github.com/lancedb/lance/pull/1668
• chore: add new python benchmarks for testing scalar indices by @westonpace in https://github.com/lancedb/lance/pull/1658

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.18...v0.8.19

New features

• feat: add support for update queries by @wjones127 in https://github.com/lancedb/lance/pull/1585
• feat: better warnings when the PQ assignment over cosine distance is wrong by @eddyxu in https://github.com/lancedb/lance/pull/1672

Bug fixes

• fix: correctly avg centroids in update logic in GPU IVF training by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1646
• fix: prevent OOM when IVF centroids are provided by @wjones127 in https://github.com/lancedb/lance/pull/1653
• fix: fix error message for invalid nprobes by @albertlockett in https://github.com/lancedb/lance/pull/1666
• fix: support no-op filters again by @wjones127 in https://github.com/lancedb/lance/pull/1669
• fix: row_id range fix for index training on gpu by @jerryyifei in https://github.com/lancedb/lance/pull/1663
• fix: add retries for failed response stream by @wjones127 in https://github.com/lancedb/lance/pull/1671

Performance Improvements

• perf: fast L2 distance table build by @eddyxu in https://github.com/lancedb/lance/pull/1639
• perf: add a fast path for converting bytes into array when the bytes has the correct alignment by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1652
• perf: minor change to cleanup allowing for size to be collected in parallel by @westonpace in https://github.com/lancedb/lance/pull/1649
• perf: add type coersion for in-list expressions by @westonpace in https://github.com/lancedb/lance/pull/1655

Other changes

• chore: make lance-linalg benchmark ready to test bf16 data by @eddyxu in https://github.com/lancedb/lance/pull/1634
• test: fix for test by @wjones127 in https://github.com/lancedb/lance/pull/1644
• chore: minor changes to tracing instrumentation by @westonpace in https://github.com/lancedb/lance/pull/1619

New Contributors

• @jerryyifei made their first contribution in https://github.com/lancedb/lance/pull/1663

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.17...v0.8.18

New features

• feat: allow scalar indices to be updated with new data by @westonpace in https://github.com/lancedb/lance/pull/1576
• feat: add python bindings for creating scalar indices by @westonpace in https://github.com/lancedb/lance/pull/1592
• feat: expose index cache size by @rok in https://github.com/lancedb/lance/pull/1587
• feat: track index cache hit rate by @rok in https://github.com/lancedb/lance/pull/1586
• feat: serialize arbitrary float type of PQ to protobuf by @eddyxu in https://github.com/lancedb/lance/pull/1624
• feat: coerce float array for nearest query by @eddyxu in https://github.com/lancedb/lance/pull/1618
• feat: make partition calculation parallel by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1625
• feat(rust): simplify object store option API by @wjones127 in https://github.com/lancedb/lance/pull/1627
• feat(python): add index_cache_hit_rate to index_stats() by @rok in https://github.com/lancedb/lance/pull/1631

Bug fixes

• fix: handle no max value for string by @wjones127 in https://github.com/lancedb/lance/pull/1600
• fix: fix chunk size issue by @wjones127 in https://github.com/lancedb/lance/pull/1630

Performance improvements

Previously, there was a performance bug where ANN search would be much slower if an entire fragment was deleted. This release fixes that so there is no drop in performance.

• perf: more efficient treemap implementation for row ids by @wjones127 in https://github.com/lancedb/lance/pull/1632

Other changes

• ci: use M1 runner for now for release by @wjones127 in https://github.com/lancedb/lance/pull/1623
• chore: expose avx512fp16 feature via main lance crate by @eddyxu in https://github.com/lancedb/lance/pull/1626

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.16...v0.8.17

What's Changed

• perf: kmean fit uses cached torch dataset by @eddyxu in https://github.com/lancedb/lance/pull/1603
• fix: add migration for bad fragment bitmaps by @westonpace in https://github.com/lancedb/lance/pull/1611

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.15...v0.8.16

What's Changed

• chore: fix type inference errors in benchmarks by @westonpace in https://github.com/lancedb/lance/pull/1556
• chore: provide a trait to dynamically dispatch different pq based on different vector data type by @eddyxu in https://github.com/lancedb/lance/pull/1555
• chore: update the CI build to check/build all crates in the workspace and not just the lance crate by @westonpace in https://github.com/lancedb/lance/pull/1557
• feat: make it possible to create and load scalar indices for a dataset by @westonpace in https://github.com/lancedb/lance/pull/1516
• feat: generic Product Quantizatoin by @eddyxu in https://github.com/lancedb/lance/pull/1560
• test: add property-based testing for statistics by @wjones127 in https://github.com/lancedb/lance/pull/1554
• feat: ffi to accelerate norm_l2 for f16 if the instruction set is available by @eddyxu in https://github.com/lancedb/lance/pull/1562
• feat: extend FSL with sample by @eddyxu in https://github.com/lancedb/lance/pull/1572
• feat: allow for more advanced storage options in objectstore by @universalmind303 in https://github.com/lancedb/lance/pull/1547
• feat: implement as_slice for bfloat16 array by @eddyxu in https://github.com/lancedb/lance/pull/1574
• perf: add bf16 benchmarks by @eddyxu in https://github.com/lancedb/lance/pull/1575
• feat: f16 for L2 by @eddyxu in https://github.com/lancedb/lance/pull/1577
• chore: update cc dependency to 1.0.83 by @westonpace in https://github.com/lancedb/lance/pull/1578
• feat: make IVF model support f16 and bf16 by @eddyxu in https://github.com/lancedb/lance/pull/1573
• feat: allow the scanner to take advantage of scalar indices by @westonpace in https://github.com/lancedb/lance/pull/1543
• chore: dotprod should be on mac target, not haswell and better randomness for bf16 by @westonpace in https://github.com/lancedb/lance/pull/1579
• feat: make Dataset::nearest() accepts arbitrary query type by @eddyxu in https://github.com/lancedb/lance/pull/1582
• chore(rust): remove extraneous dbg message by @changhiskhan in https://github.com/lancedb/lance/pull/1598
• feat: torch cache-able dataset, with sampling support by @eddyxu in https://github.com/lancedb/lance/pull/1591
• fix: tell writer correct schema when writing index file by @wjones127 in https://github.com/lancedb/lance/pull/1518
• feat: add support for remapping scalar indices during compaction by @westonpace in https://github.com/lancedb/lance/pull/1571
• refactor: switch to using DataFusions physical expr by @wjones127 in https://github.com/lancedb/lance/pull/1581
• chore: various fixes for Python benchmarks by @wjones127 in https://github.com/lancedb/lance/pull/1513
• feat: adaptive cuda allocation for l2/cosine distance computation by @eddyxu in https://github.com/lancedb/lance/pull/1601
• fix: fix a memory leak where a dataset would not be fully deleted by @westonpace in https://github.com/lancedb/lance/pull/1606
• fix: google objectstore uses proper gs configuration by @universalmind303 in https://github.com/lancedb/lance/pull/1608

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.14...v0.8.15

What's Changed

• feat: generic kmeans that supports bf16 and f16 by @eddyxu in https://github.com/lancedb/lance/pull/1544
• chore: disable avx512 for now by @eddyxu in https://github.com/lancedb/lance/pull/1546

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.13...v0.8.14

Critical fix: tables written prior to v0.8.0 may have corrupted stats

If a table was written with a Lance version prior to v0.8.0, and then later written by a version >=0.8.0<=0.8.13, it may have incorrect statistics. You can detect whether this affects your table using the LanceDataset.validate() method. If this affects your table, Lance versions prior to 0.8.13 may not be able to read the table correct. If you do not plan on using older versions of Lance going forward, no action is needed. To fix reads on older Lance versions, commit any write transaction to the table with Lance v0.8.13 or newer. A simple way to make a transaction without changing the data would be:

import lance

dataset = lance.dataset('...')
operation = lance.LanceOperation.Append([])
dataset = lance.LanceDataset.commit(
    dataset.uri,
    operation,
    read_version=dataset.version,
)

(This makes an empty Append commit)

New features

• feat: generic cosine code by @eddyxu in https://github.com/lancedb/lance/pull/1537
• feat: make L2 generic to work with all float numbers by @eddyxu in https://github.com/lancedb/lance/pull/1532
• feat: safer API for physical_rows by @wjones127 in https://github.com/lancedb/lance/pull/1529
• feat: implement datafusion tableprovider trait for Dataset by @universalmind303 in https://github.com/lancedb/lance/pull/1526
• feat: expose Dataset.validate() in Python by @wjones127 in https://github.com/lancedb/lance/pull/1538

Bug fixes

• fix: add versioning and bypass broken row counts by @wjones127 in https://github.com/lancedb/lance/pull/1534
• fix: fix assertion of cosine values by @eddyxu in https://github.com/lancedb/lance/pull/1530
• fix: pq index does not handle dot product metric correctly during search by @rok in https://github.com/lancedb/lance/pull/1536

Performance improvements

• perf: improve f16 performance for norm L2 on aarch64 by @eddyxu in https://github.com/lancedb/lance/pull/1539

Other changes

• chore: move scalar_index benchmark to break circular dependency by @westonpace in https://github.com/lancedb/lance/pull/1540

New Contributors

• @universalmind303 made their first contribution in https://github.com/lancedb/lance/pull/1526

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.12...v0.8.13

What's Changed

• feat: use caller location in error conversion functions by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1510
• chore(rust): reduce debug message log level by @changhiskhan in https://github.com/lancedb/lance/pull/1512
• feat: collect page-level statistics on write by @rok in https://github.com/lancedb/lance/pull/1335
• feat(rust): simd ops of reduce min, min, find and gather by @eddyxu in https://github.com/lancedb/lance/pull/1514
• feat: add btree scalar index by @westonpace in https://github.com/lancedb/lance/pull/1476
• feat: support true in deletion logic by @Weijun-H in https://github.com/lancedb/lance/pull/1515
• fix: make sure we have physical rows by @wjones127 in https://github.com/lancedb/lance/pull/1511
• chore: benchmark of large IVF parrtitions by @eddyxu in https://github.com/lancedb/lance/pull/1524
• feat: make dot generic to support bf16/f16/f32 with one dot_distance interface. by @eddyxu in https://github.com/lancedb/lance/pull/1522
• chore: add same target-features to python pyo3 build by @eddyxu in https://github.com/lancedb/lance/pull/1527
• feat: expose index cache configure via open dataset API by @eddyxu in https://github.com/lancedb/lance/pull/1523

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.11...v0.8.12

New features

• feat: add location for all error by @Weijun-H in https://github.com/lancedb/lance/pull/1475
• feat: add sorting to the scanner by @westonpace in https://github.com/lancedb/lance/pull/1498
• feat: add tf.data APIs for reading batches by @wjones127 in https://github.com/lancedb/lance/pull/1488
• feat: experimental avx512 features by @eddyxu in https://github.com/lancedb/lance/pull/1506
• feat: add read ahead for take scan by @wjones127 in https://github.com/lancedb/lance/pull/1501

Bug fixes

• fix: remove channel closed messages by @wjones127 in https://github.com/lancedb/lance/pull/1502

Performance improvements

• perf: dimension-based kernel for L2 and Cosine by @eddyxu in https://github.com/lancedb/lance/pull/1503

New Contributors

• @Weijun-H made their first contribution in https://github.com/lancedb/lance/pull/1475

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.10...v0.8.11

What's Changed

• fix: fix PQ training metric type is not appropriately propogated by @eddyxu in https://github.com/lancedb/lance/pull/1493
• docs: clarify behaviour of refine_factor by @albertlockett in https://github.com/lancedb/lance/pull/1496
• ci: cancel in progress runs on new push by @albertlockett in https://github.com/lancedb/lance/pull/1497
• chore: remove unused value settings by @eddyxu in https://github.com/lancedb/lance/pull/1494
• feat: provide a f32x16 abstraction to make unrolling 256-bit code easier by @eddyxu in https://github.com/lancedb/lance/pull/1495

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.9...v0.8.10

New features

• feat: expose progress on write_fragments and write_dataset by @wjones127 in https://github.com/lancedb/lance/pull/1464
• feat: split out datagen utilities, expand them, expose to python by @westonpace in https://github.com/lancedb/lance/pull/1315

Bug fixes

• fix: fix L2 computation on GPU by @eddyxu in https://github.com/lancedb/lance/pull/1485

Performance Improvements

• perf: support multi-threading shuffler by @eddyxu in https://github.com/lancedb/lance/pull/1474
• perf: use simd lib to implement dot by @eddyxu in https://github.com/lancedb/lance/pull/1480
• perf: improve kmeans and make pq training multi-threaded by @eddyxu in https://github.com/lancedb/lance/pull/1479

Other changes

• refactor: move commit traits to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1469
• chore: use ruff format to replace isort and black by @eddyxu in https://github.com/lancedb/lance/pull/1472
• refactor: move ObjectStore, FileReader and FileWriter to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1473
• feat: poor man's SIMD lib by @eddyxu in https://github.com/lancedb/lance/pull/1478
• chore: remove outdated warnings about prefiltering with a vector index by @westonpace in https://github.com/lancedb/lance/pull/1484
• chore: mention GPU support in README by @eddyxu in https://github.com/lancedb/lance/pull/1489

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.8...v0.8.9

What's Changed

• ci: fix ARM build due to Ring dependency by @wjones127 in https://github.com/lancedb/lance/pull/1462
• refactor: move read and write manifest file to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1467
• feat: create_index take torch.device object by @eddyxu in https://github.com/lancedb/lance/pull/1465
• feat: added dataset stats api by @albertlockett in https://github.com/lancedb/lance/pull/1452

New Contributors

• @albertlockett made their first contribution in https://github.com/lancedb/lance/pull/1452

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.7...v0.8.8

What's Changed

• chore: use arrow 47 function by @eddyxu in https://github.com/lancedb/lance/pull/1439
• refactor: move format definitions to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1440
• refactor: migrate object reader and object writer by @eddyxu in https://github.com/lancedb/lance/pull/1442
• fix: fix an issue where the GPU index trainer was taking too much data into memory by @westonpace in https://github.com/lancedb/lance/pull/1447
• feat: store a separate tensor blob for IVF centroids by @eddyxu in https://github.com/lancedb/lance/pull/1446
• refactor: move all Python operations to the same runtime by @wjones127 in https://github.com/lancedb/lance/pull/1445
• chore: bump prost version to latest by @eddyxu in https://github.com/lancedb/lance/pull/1449
• chore: update half to 2.3.1 by @jacobBaumbach in https://github.com/lancedb/lance/pull/1450
• feat: allow prefiltering to be used with an index by @westonpace in https://github.com/lancedb/lance/pull/1435
• feat: benchmark and improve L2 partition compute by @eddyxu in https://github.com/lancedb/lance/pull/1453
• chore: increase ivf assignment parallism during indexing by @eddyxu in https://github.com/lancedb/lance/pull/1451
• feat: support keyboard interrupt in Python by @wjones127 in https://github.com/lancedb/lance/pull/1438
• feat: add parameter to split by file size by @wjones127 in https://github.com/lancedb/lance/pull/1444

New Contributors

• @jacobBaumbach made their first contribution in https://github.com/lancedb/lance/pull/1450

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.6...v0.8.7