lance

What's Changed

• fix: assume default rust toolchain as stable by @kerryeon in https://github.com/lancedb/lance/pull/2055
• chore: extend JNI to get strings by @eddyxu in https://github.com/lancedb/lance/pull/2047
• chore: bump datafusion version by @universalmind303 in https://github.com/lancedb/lance/pull/2035
• chore: skip empty batch while chunking batches by @BubbleCal in https://github.com/lancedb/lance/pull/2026
• chore: utility to convert RecordBatchStream to FFI_ArrowArrayStream by @eddyxu in https://github.com/lancedb/lance/pull/2065
• feat: use a scalar index, if available, during a merge insert operation by @westonpace in https://github.com/lancedb/lance/pull/1987
• chore: write HNSW partitions by @BubbleCal in https://github.com/lancedb/lance/pull/2056
• chore: add struct for parameters of HNSW by @BubbleCal in https://github.com/lancedb/lance/pull/2057
• docs: add llm dataset creation example by @tanaymeh in https://github.com/lancedb/lance/pull/2060
• fix: update merge_insert code to use latest df version by @westonpace in https://github.com/lancedb/lance/pull/2071
• feat: support filter while searching in HNSW by @BubbleCal in https://github.com/lancedb/lance/pull/2058
• feat(java): fragment reader by @eddyxu in https://github.com/lancedb/lance/pull/2072
• fix: force fragments to be stored in the manifest in id-order by @westonpace in https://github.com/lancedb/lance/pull/2075
• chore: building IVF_HNSW index by @BubbleCal in https://github.com/lancedb/lance/pull/2066
• fix: fix bug in indexed merge insert where new data could cause merge insert to panic by @westonpace in https://github.com/lancedb/lance/pull/2076
• feat: emit warnings if f16 kernels not built by @westonpace in https://github.com/lancedb/lance/pull/2077

New Contributors

• @kerryeon made their first contribution in https://github.com/lancedb/lance/pull/2055
• @tanaymeh made their first contribution in https://github.com/lancedb/lance/pull/2060

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.3...v0.10.4

What's Changed

• ci: fix compilers for release by @wjones127 in https://github.com/lancedb/lance/pull/2032
• fix: use None not zero for limit default by @wjones127 in https://github.com/lancedb/lance/pull/2033
• fix: stronger numeric guarantees for distance kernels by @wjones127 in https://github.com/lancedb/lance/pull/2013
• feat(java): dataset get fragments by @eddyxu in https://github.com/lancedb/lance/pull/2034
• feat: config plumbing for vector benchmark framework by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2036
• fix: pin away from pyarrow 15.0.1 as it is causing segmentation fault in tests by @westonpace in https://github.com/lancedb/lance/pull/2045
• chore: upgrade chrono and fix deprecation warnings by @eddyxu in https://github.com/lancedb/lance/pull/2048
• feat: add support for scalar indices on temporal columns by @westonpace in https://github.com/lancedb/lance/pull/1968
• feat: use out-of-core sort to train btree indices by @westonpace in https://github.com/lancedb/lance/pull/2043
• chore: store ivf in arrow schema by @eddyxu in https://github.com/lancedb/lance/pull/2053
• chore: on disk pq storage by @eddyxu in https://github.com/lancedb/lance/pull/2049

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.2...v0.10.3

What's Changed

• chore(java): provide conversion utilities between jni objects and rust objects by @eddyxu in https://github.com/lancedb/lance/pull/2009
• perf: avoid re-calculating the distances while building HNSW by @BubbleCal in https://github.com/lancedb/lance/pull/2010
• chore: select less but enough neighbors to establish edges by @BubbleCal in https://github.com/lancedb/lance/pull/2011
• fix: fp16 kernels computed in f32 by @wjones127 in https://github.com/lancedb/lance/pull/1990
• chore: add fixture test for IVF index by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2014
• perf: greedy search for finding entry point by @BubbleCal in https://github.com/lancedb/lance/pull/2012
• chore: normalize transform by @eddyxu in https://github.com/lancedb/lance/pull/2017
• refactor: refactor ivf into transformer by @eddyxu in https://github.com/lancedb/lance/pull/2023
• docs: fix typo by @krlmlr in https://github.com/lancedb/lance/pull/2025
• chore: fix cosine residual calculation by @eddyxu in https://github.com/lancedb/lance/pull/2015
• feat: add recall report notebook by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2018
• ci: fix nightly build on apple silicon GHA by @eddyxu in https://github.com/lancedb/lance/pull/2024
• chore: update HNSW example by @BubbleCal in https://github.com/lancedb/lance/pull/2016

New Contributors

• @krlmlr made their first contribution in https://github.com/lancedb/lance/pull/2025

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.1...v0.10.2

What's Changed

• feat: implement java bindings by @beinan in https://github.com/lancedb/lance/pull/1928
• chore: simplify output schema in scanner by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1999
• fix: escape column names correctly by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2007
• fix: crate publish by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2008

New Contributors

• @beinan made their first contribution in https://github.com/lancedb/lance/pull/1928

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.0...v0.10.1

Breaking Change

• Vector queries no longer return the vector column when columns= or project() does not include the vector column. (default behavior did not change, all columns are returned)

What's Changed

• chore: break the dependency cycle lance-index -> lance by @BubbleCal in https://github.com/lancedb/lance/pull/1998
• chore: improve hnsw build time by @eddyxu in https://github.com/lancedb/lance/pull/1996
• feat: allow projection expression by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1989
• feat: add _cast_list_f16 function for casting list of floats to f16 by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2000
• fix: use 1 for beam search candidate length for the levels above the assigned level by @eddyxu in https://github.com/lancedb/lance/pull/2001
• fix: search stops early than expected by @BubbleCal in https://github.com/lancedb/lance/pull/2005
• perf: remove the unnecessary selection while searching by @BubbleCal in https://github.com/lancedb/lance/pull/2004
• feat: expose dynamic projection to pylance by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2003

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.19...v0.10.0

What's Changed

• chore: make mac apple silicon build more reliable by @eddyxu in https://github.com/lancedb/lance/pull/1979
• fix: nightly compile error by @albertlockett in https://github.com/lancedb/lance/pull/1981
• feat: persist hnsw graph on disk by @eddyxu in https://github.com/lancedb/lance/pull/1976
• feat: hamming distance by @eddyxu in https://github.com/lancedb/lance/pull/1986
• chore: use residual in cosine inference by @eddyxu in https://github.com/lancedb/lance/pull/1984
• feat: allow casting in alter_columns() by @wjones127 in https://github.com/lancedb/lance/pull/1909
• feat: binary quantization by @eddyxu in https://github.com/lancedb/lance/pull/1988
• fix: deliver the last batch from the sharded sampler by @westonpace in https://github.com/lancedb/lance/pull/1993
• feat: implement heuristic select neighbor algorithm by @eddyxu in https://github.com/lancedb/lance/pull/1991
• chore: remove diskann implementation by @eddyxu in https://github.com/lancedb/lance/pull/1994
• fix: map the ErrorKind::NotFound to Error::NotFound by @BubbleCal in https://github.com/lancedb/lance/pull/1995
• chore: use v0.2.44 cargo-workspace for bumping version by @BubbleCal in https://github.com/lancedb/lance/pull/1997

New Contributors

• @BubbleCal made their first contribution in https://github.com/lancedb/lance/pull/1995

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.18...v0.9.19

What's Changed

• feat: expose object writer and schema from lance file writer by @eddyxu in https://github.com/lancedb/lance/pull/1956
• feat: create an I/O scheduler that will eventually be used by the new file reader by @westonpace in https://github.com/lancedb/lance/pull/1918
• chore: refactor distance table to separate function by @eddyxu in https://github.com/lancedb/lance/pull/1970
• fix: compile error when older versions of tokio were used by @westonpace in https://github.com/lancedb/lance/pull/1971
• test: fix faulty test_l2_distance by @eddyxu in https://github.com/lancedb/lance/pull/1972
• ci: pin ruff and ignore E302 in Github action by @eddyxu in https://github.com/lancedb/lance/pull/1974
• chore: remove blas dependency by @eddyxu in https://github.com/lancedb/lance/pull/1973
• chore: pq storage by @eddyxu in https://github.com/lancedb/lance/pull/1967
• ci: enable rust nightly in CI by @eddyxu in https://github.com/lancedb/lance/pull/1966

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.17...v0.9.18

What's Changed

• feat: add support for filtering to the ShardedBatchSampler by @westonpace in https://github.com/lancedb/lance/pull/1950

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.16...v0.9.17

What's Changed

• feat: simple in memory hnsw implementation by @eddyxu in https://github.com/lancedb/lance/pull/1933
• chore: take f16 codebook by @eddyxu in https://github.com/lancedb/lance/pull/1940
• perf: fast pq lookup by @eddyxu in https://github.com/lancedb/lance/pull/1941
• feat: support large types for image ext types by @wjones127 in https://github.com/lancedb/lance/pull/1944
• feat: make it possible to only load the row id column by @westonpace in https://github.com/lancedb/lance/pull/1949
• fix: allow image extension types to be pickled in PyArrow 12 by @wjones127 in https://github.com/lancedb/lance/pull/1955
• fix: do not panic when when_matched condition yields 0 matches by @westonpace in https://github.com/lancedb/lance/pull/1952

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.15...v0.9.16

New features

• feat(rust): make ReadParams Debug and Clone by @wjones127 in https://github.com/lancedb/lance/pull/1931

Bug Fixes

• fix: statically link lzma by @wjones127 in https://github.com/lancedb/lance/pull/1934
• fix: repair the filter keyword in the torch dataset loader by @westonpace in https://github.com/lancedb/lance/pull/1935

Other Changes

• ci: remove stdsimd reference by @wjones127 in https://github.com/lancedb/lance/pull/1925
• chore: need to specify version by @wjones127 in https://github.com/lancedb/lance/pull/1938

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.14...v0.9.15

New Features

• feat: can now specify a condition on merge insert's 'when matched' behavior by @westonpace in https://github.com/lancedb/lance/pull/1919

Bug Fixes

• fix: prevent timeouts while writing to object storage by @wjones127 in https://github.com/lancedb/lance/pull/1921
• fix: support large_string dictionaries by @wjones127 in https://github.com/lancedb/lance/pull/1920

What's Changed

• chore: update formatting to ruff 0.2 expectations by @westonpace in https://github.com/lancedb/lance/pull/1922
• chore: remove sqlparser as dependency by @eddyxu in https://github.com/lancedb/lance/pull/1923

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.13...v0.9.14

New features

• feat: add alter columns for names and nullability by @wjones127 in https://github.com/lancedb/lance/pull/1903
• feat: dataset add_columns method by @wjones127 in https://github.com/lancedb/lance/pull/1468
• feat(python): expose drop_columns() in Python by @wjones127 in https://github.com/lancedb/lance/pull/1904
• feat: allow opening FragmentReader with just row id by @wjones127 in https://github.com/lancedb/lance/pull/1902

Performance improvements

• perf: avoid a buffer copy and share the array to reduce RAM cost by @westonpace in https://github.com/lancedb/lance/pull/1907

Bug fixes

• fix: handle out-of-order columns by @wjones127 in https://github.com/lancedb/lance/pull/1901

Other changes

• docs: add torch integration docs by @eddyxu in https://github.com/lancedb/lance/pull/1906
• chore: support python 3.12 by @eddyxu in https://github.com/lancedb/lance/pull/1908
• docs: fix a path issue in benchmarks/sift/README.md by @frankliee in https://github.com/lancedb/lance/pull/1913
• ci: fix docs deployment by @wjones127 in https://github.com/lancedb/lance/pull/1915

New Contributors

• @frankliee made their first contribution in https://github.com/lancedb/lance/pull/1913

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.12...v0.9.13

What's Changed

• fix: allow merge() with dataset as input by @wjones127 in https://github.com/lancedb/lance/pull/1869
• fix: small bug in _updated_centroids() in kmeans.py, wrong initialization of zero_counts by @lillianye in https://github.com/lancedb/lance/pull/1885
• docs: document the merge insert operation in python by @westonpace in https://github.com/lancedb/lance/pull/1892
• fix: handle null return from predicate evaluation in pushdown by @westonpace in https://github.com/lancedb/lance/pull/1898
• perf: speed up materialization when there is no block list by @westonpace in https://github.com/lancedb/lance/pull/1897
• fix: merge insert operation fails if any of the payload columns are non-nullable by @westonpace in https://github.com/lancedb/lance/pull/1899
• feat: enable composable and customizable sampler in PyTorch data loader by @eddyxu in https://github.com/lancedb/lance/pull/1900

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.11...v0.9.12

What's Changed

• chore: do not run test on intel mac anymore, and reduce python target by @eddyxu in https://github.com/lancedb/lance/pull/1875
• fix: improve ability to control dynamo creds cache by @albertlockett in https://github.com/lancedb/lance/pull/1870
• feat: add support for accepting Substrait ExtendedExpression messages as filters by @westonpace in https://github.com/lancedb/lance/pull/1863
• fix: remove lru_cache from instance method and add lint to catch future issues by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1883
• feat(python): lance write huggingface dataset directly by @eddyxu in https://github.com/lancedb/lance/pull/1882
• feat: add "merge insert" operation based on merge operation in other databases by @westonpace in https://github.com/lancedb/lance/pull/1647
• ci: bump GHA to macos-14 to use free m1 GHA by @eddyxu in https://github.com/lancedb/lance/pull/1886
• fix: make avx512 fp16 a runtime check by @albertlockett in https://github.com/lancedb/lance/pull/1884
• fix: shard batch iterator can reads partial batches by @eddyxu in https://github.com/lancedb/lance/pull/1889
• feat: customize to tensor function in pytorch dataset by @eddyxu in https://github.com/lancedb/lance/pull/1890

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.10...v0.9.11

What's Changed

• chore(rust): allow index type to be copied by @eddyxu in https://github.com/lancedb/lance/pull/1864
• docs: object store configuration by @wjones127 in https://github.com/lancedb/lance/pull/1849
• fix: latent bug in index append by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1865
• refactor: remove unused dependencies by @westonpace in https://github.com/lancedb/lance/pull/1853
• feat: upgrade arrow to 50 and datafusion to 35 by @wjones127 in https://github.com/lancedb/lance/pull/1868
• fix: updating requirements.txt by @lillianye in https://github.com/lancedb/lance/pull/1867
• fix: do not panic when performing a pushdown scan on a multi-data-file fragment by @westonpace in https://github.com/lancedb/lance/pull/1873
• chore: bump github action version due to github node deprecation by @eddyxu in https://github.com/lancedb/lance/pull/1872

New Contributors

• @lillianye made their first contribution in https://github.com/lancedb/lance/pull/1867

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.9...v0.9.10

What's Changed

• refactor: split lance-core into lance-core, lance-io, lance-file, and lance-table by @westonpace in https://github.com/lancedb/lance/pull/1852
• fix: skip empty batches in shuffler by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1858
• test: make empty cluster message less noisy by @wjones127 in https://github.com/lancedb/lance/pull/1859
• chore: add instruction to sift dataset by @eddyxu in https://github.com/lancedb/lance/pull/1860
• fix: aws creds resolution panic by @albertlockett in https://github.com/lancedb/lance/pull/1861
• fix: import issues after refactor by @albertlockett in https://github.com/lancedb/lance/pull/1862

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.8...v0.9.9

What's Changed

• fix: log the actual dynamodb error by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1847
• fix: correctly split large cluster and use nicer splitting method by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1851
• ci: pin version of cargo publish action by @albertlockett in https://github.com/lancedb/lance/pull/1846
• feat: add a fast way to checkout a different version by @wjones127 in https://github.com/lancedb/lance/pull/1850
• refactor: move commit_handler out of ObjectStore and into Dataset by @westonpace in https://github.com/lancedb/lance/pull/1842
• refactor: split any reference to fragment or manifest out of the file reader by @westonpace in https://github.com/lancedb/lance/pull/1848

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.7...v0.9.8

What's Changed

• feat: use lazy import for pylance dependencies by @yliang412 in https://github.com/lancedb/lance/pull/1741
• chore: merge multiple ivf_pq indices by @eddyxu in https://github.com/lancedb/lance/pull/1815
• feat: improve error message when schema mismatch occurs by @westonpace in https://github.com/lancedb/lance/pull/1817
• refactor: do not leak precomputed partition map into IVF by @eddyxu in https://github.com/lancedb/lance/pull/1819
• chore(rust): increase minimum rust version to 1.75 to prevent build issues by @changhiskhan in https://github.com/lancedb/lance/pull/1821
• chore: bump object store version to 0.9 by @eddyxu in https://github.com/lancedb/lance/pull/1826
• chore: bump half deps for HW acceleration in python by @eddyxu in https://github.com/lancedb/lance/pull/1827
• refactor!: index stats to handle error and bunch of bugs by @eddyxu in https://github.com/lancedb/lance/pull/1828
• chore: remove empty trait by @eddyxu in https://github.com/lancedb/lance/pull/1830
• fix: spurious bug during compaction by @westonpace in https://github.com/lancedb/lance/pull/1829
• chore: add E2E tests for torch training with nans by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1825
• feat: roundtrip bfloat16 vectors to Lance format by @wjones127 in https://github.com/lancedb/lance/pull/1810
• perf: cache loaded index metadata by @eddyxu in https://github.com/lancedb/lance/pull/1831
• feat: expose custom PQ codebook in python APIs by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1838
• chore: expose externally computed shuffle buffer to python by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1805
• fix: function name by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1840
• chore: unindexed fragment can handle mulitple deltas by @eddyxu in https://github.com/lancedb/lance/pull/1839
• docs: code blocks in lance/read_and_write.html not showing by @martindurant in https://github.com/lancedb/lance/pull/1843
• feat: optimize indices by @eddyxu in https://github.com/lancedb/lance/pull/1841
• feat: support reading from delta indices by @eddyxu in https://github.com/lancedb/lance/pull/1845

New Contributors

• @yliang412 made their first contribution in https://github.com/lancedb/lance/pull/1741
• @martindurant made their first contribution in https://github.com/lancedb/lance/pull/1843

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.6...v0.9.7

New features

• feat: switch to using datafusion to parse functions by @westonpace in https://github.com/lancedb/lance/pull/1793
• feat: allow externally generated shuffle buffer and multiple of them by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1801
• feat: expose shuffler parameters for controlling memory usage by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1790

Bug fixes

• fix: ensure query plans are correct by @wjones127 in https://github.com/lancedb/lance/pull/1785
  • This fixes an issue where ANN queries with prefilters can have irrelevant results if there is a scalar index that is not up-to-date.

Other changes

• chore: init sharded iterator from pytorch distributed environment by @eddyxu in https://github.com/lancedb/lance/pull/1795
• chore: make create_plan a public API by @eddyxu in https://github.com/lancedb/lance/pull/1796
• chore: fix clippy lints on rust 1.75 by @eddyxu in https://github.com/lancedb/lance/pull/1767
• chore: migrate shuffle to lance-index by @eddyxu in https://github.com/lancedb/lance/pull/1803
• fix: fix 1.75 clippy by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1806
• chore: relax the footer-max-line-length rule (github takes care of this) by @westonpace in https://github.com/lancedb/lance/pull/1811
• chore: use local temp directory to store spilled precomputed partitions by @eddyxu in https://github.com/lancedb/lance/pull/1812
• ci: add set -e to linux script by @wjones127 in https://github.com/lancedb/lance/pull/1816

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.5...v0.9.6

What's Changed

• chore: binary search fragments from the row id range by @eddyxu in https://github.com/lancedb/lance/pull/1789
• chore: support sharded dataset in pytorch by @eddyxu in https://github.com/lancedb/lance/pull/1791

Full Changelog: https://github.com/lancedb/lance/compare/v0.9.4...v0.9.5