lance

What's Changed

• feat: pass through aws environment vars by @wjones127 in https://github.com/lancedb/lance/pull/1077

Full Changelog: https://github.com/lancedb/lance/compare/v0.5.8...v0.5.9

What's Changed

• [Python] Remove OPQ cargo feature by @gsilvestrin in https://github.com/lancedb/lance/pull/1069
• [Rust] Expose JsonSchema struct as public struct by @eddyxu in https://github.com/lancedb/lance/pull/1070
• [Rust] Provide schema conv for both schema and reference by @eddyxu in https://github.com/lancedb/lance/pull/1071
• [Rust] fix: make sure we can append with dictionary columns by @wjones127 in https://github.com/lancedb/lance/pull/1053
• [Docs]: reduce docs warnings by @wjones127 in https://github.com/lancedb/lance/pull/1068

Full Changelog: https://github.com/lancedb/lance/compare/v0.5.7...v0.5.8

Highlight

• Lance adds initial bfloat16 support for Machine Learning workloads!

https://lancedb.github.io/lance/arrays.html

What's Changed

• fix: make write_impl private by @wjones127 in https://github.com/lancedb/lance/pull/1061
• [Rust] Convert between json / arrow schema by @eddyxu in https://github.com/lancedb/lance/pull/1064
• [Python] Expose schema and json conversion via Python SDK by @eddyxu in https://github.com/lancedb/lance/pull/1067
• Add bfloat16 extension array by @wjones127 in https://github.com/lancedb/lance/pull/1002

Full Changelog: https://github.com/lancedb/lance/compare/v0.5.6...v0.5.7

What's Changed

• [Doc, Rust] Improve docstring about how to use Dataset write in Rust by @eddyxu in https://github.com/lancedb/lance/pull/1060
• [Rust] Fix new warnings on rust 1.71 by @eddyxu in https://github.com/lancedb/lance/pull/1063
• windows: make openblas optional and remove vcpkg by @gsilvestrin in https://github.com/lancedb/lance/pull/1062
• pin half by @wjones127 in https://github.com/lancedb/lance/pull/1065
• Removed duplicated dependency by @gsilvestrin in https://github.com/lancedb/lance/pull/1066

Full Changelog: https://github.com/lancedb/lance/compare/v0.5.5...v0.5.6

What's Changed

• allow aws cred injection in dataset write as well by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1059
• Arm quick fix by @unkn-wn in https://github.com/lancedb/lance/pull/1057

Full Changelog: https://github.com/lancedb/lance/compare/v0.5.4...v0.5.5

What's Changed

• feat: upgrade arrow, make write Send by @wjones127 in https://github.com/lancedb/lance/pull/1033
• Bring commented out clippy rule back after GHA rust upgrade by @eddyxu in https://github.com/lancedb/lance/pull/1039
• contributing guide by @changhiskhan in https://github.com/lancedb/lance/pull/1036
• use a large runner for linux tests by @changhiskhan in https://github.com/lancedb/lance/pull/1038
• bugfix: remove extra character from jq filter in GHA by @gsilvestrin in https://github.com/lancedb/lance/pull/941
• upgrade arrow version in duckdb extension by @changhiskhan in https://github.com/lancedb/lance/pull/1040
• Docs: Dataset write takes reader by owner now by @dnsco in https://github.com/lancedb/lance/pull/1042
• allow custom AWS credentials when constructing object store by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1049
• fix: don't compare metadata when checking schema in append by @wjones127 in https://github.com/lancedb/lance/pull/1051
• Add support for ARM architecture by @unkn-wn in https://github.com/lancedb/lance/pull/1043

New Contributors

• @unkn-wn made their first contribution in https://github.com/lancedb/lance/pull/1043

Full Changelog: https://github.com/lancedb/lance/compare/v0.5.3...v0.5.4

What's Changed

• [rust] Added support for creating empty dataset by @trueutkarsh in https://github.com/lancedb/lance/pull/1016
• [Rust] Use unaligned load in cosine on Intel CPUs by @eddyxu in https://github.com/lancedb/lance/pull/1035
• cache aws credentials until 30s before expiry by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1034
• fix record batch writing to dataset by @TevinWang in https://github.com/lancedb/lance/pull/1030
• docs: update format docs with feature flags and row indices by @wjones127 in https://github.com/lancedb/lance/pull/1037

Full Changelog: https://github.com/lancedb/lance/compare/v0.5.2...v0.5.3

What's Changed

• Support Azure blob storage as backend by @alikefia in https://github.com/lancedb/lance/pull/1017
• wire in DefaultCredentialsChain from aws_config as the credential provider for object_store by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/917
• split clippy to a separate workflow, disable incremental build for CI by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1018
• [rust][python] allow create fragment from existed datafile by @LiWeiJie in https://github.com/lancedb/lance/pull/981
• Support Dot product as metric type by @eddyxu in https://github.com/lancedb/lance/pull/1021
• use default value for null boolean by @wjones127 in https://github.com/lancedb/lance/pull/1028
• Throw error instead of panic if kmeans training does not have sufficient data. by @eddyxu in https://github.com/lancedb/lance/pull/1029
• Improve support for timestamp cmp and remove custom sql parser by @wjones127 in https://github.com/lancedb/lance/pull/1026

New Contributors

• @alikefia made their first contribution in https://github.com/lancedb/lance/pull/1017

Full Changelog: https://github.com/lancedb/lance/compare/v0.5.1...v0.5.2

What's Changed

• allow object store wrapper as part of dataset construction param by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1014

Full Changelog: https://github.com/lancedb/lance/compare/v0.5.0...v0.5.1

What's Changed

New features

• Delete rows in a dataset with Dataset.delete()
  • Add delete() method to Python API by @wjones127 in https://github.com/lancedb/lance/pull/953
  • feat: handle deletes in count_rows, updater, merge by @wjones127 in https://github.com/lancedb/lance/pull/995
  • feat: add feature flags to the dataset manifest by @wjones127 in https://github.com/lancedb/lance/pull/979
• You can now customize the log level to reduce verbose messages
  • Use log and allow users to customize log via LANCE_LOG env by @eddyxu in https://github.com/lancedb/lance/pull/987
• Negative numbers are now fully supported in SQL expressions
  • [rust] Added support for minus operator for numeric literals by @trueutkarsh in https://github.com/lancedb/lance/pull/983
• openblas is now an optional dependency
  • Add feature flag for opq by @TevinWang in https://github.com/lancedb/lance/pull/998
• Remove a directory (dataset) recursively. by @eddyxu in https://github.com/lancedb/lance/pull/1011

Bugfixes

• fix s3 get_range issues: 416, range not satisfiable by @LiWeiJie in https://github.com/lancedb/lance/pull/975
• fix: make sure we are always running with a single partition by @wjones127 in https://github.com/lancedb/lance/pull/977
• fix: handle NaN values in argmin by @wjones127 in https://github.com/lancedb/lance/pull/1000

Performance Improvments

• [Rust] Reduce arrow overhead during kmean training by @eddyxu in https://github.com/lancedb/lance/pull/990
• Zero copy during kmean membership computation by @eddyxu in https://github.com/lancedb/lance/pull/992
• [Rust] Improve kmean training performance by removing dynamic dispatch by @eddyxu in https://github.com/lancedb/lance/pull/996

Other

• upgrade to arrow 40 and datafusion 26 by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1005
• bump arrow versions on duckdb intergration as well by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1007
• Remove a directory (dataset) recursively. by @eddyxu in https://github.com/lancedb/lance/pull/1011

New Contributors

• @trueutkarsh made their first contribution in https://github.com/lancedb/lance/pull/983
• @TevinWang made their first contribution in https://github.com/lancedb/lance/pull/998

Full Changelog: https://github.com/lancedb/lance/compare/v0.4.21...v0.5.0

What's Changed

• [Rust] Use lance local object store by @eddyxu in https://github.com/lancedb/lance/pull/959
• [Rust] Allow to provide pre-existing IVF_PQ centroids during index creation by @eddyxu in https://github.com/lancedb/lance/pull/963
• remove deletion_vector method and handle Result from reading deletion file properly by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/965
• [Python] Pass IVF centroids via create_index API by @eddyxu in https://github.com/lancedb/lance/pull/970
• implement deletion vector handling in index scan by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/958

Full Changelog: https://github.com/lancedb/lance/compare/v0.4.20...v0.4.21

What's Changed

• Add support for cast in expressions by @wjones127 in https://github.com/lancedb/lance/pull/957

Full Changelog: https://github.com/lancedb/lance/compare/v0.4.19...v0.4.20

What's Changed

• Merge a pre-computed column / array. by @eddyxu in https://github.com/lancedb/lance/pull/824
• Use backtick to escape column names by @wjones127 in https://github.com/lancedb/lance/pull/935
• seed l2 benchmark data generation and unroll ARM Neon simd by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/937
• add a benchmark for cosine distance and unroll ARM Neon simd by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/942

Full Changelog: https://github.com/lancedb/lance/compare/v0.4.18...v0.4.19

What's Changed

• [Python] relax minimum pandas version to 1.4 by @changhiskhan in https://github.com/lancedb/lance/pull/921
• [Doc] Update README.md by @jaichopra in https://github.com/lancedb/lance/pull/924
• [Rust] argmin benchmark by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/927
• [Docs]: add branding to docs by @wjones127 in https://github.com/lancedb/lance/pull/928
• [Rust] Add deletion files to Lance format by @wjones127 in https://github.com/lancedb/lance/pull/920
• [Rust] Allow recreate index with the same name on the same column by @eddyxu in https://github.com/lancedb/lance/pull/929
• [Doc] make sure we can publish on docs.rs by @wjones127 in https://github.com/lancedb/lance/pull/925

Full Changelog: https://github.com/lancedb/lance/compare/v0.4.17...v0.4.18

What's Changed

• feat: typecheck arrow objects before sending to Rust by @wjones127 in https://github.com/lancedb/lance/pull/914
• Implementing more granular Errors using snafu. by @gsilvestrin in https://github.com/lancedb/lance/pull/913
• Check an object store is on local file system by @eddyxu in https://github.com/lancedb/lance/pull/918

Full Changelog: https://github.com/lancedb/lance/compare/v0.4.16...v0.4.17

What's Changed

• [CI] fix typo in make-release-commit workflow by @changhiskhan in https://github.com/lancedb/lance/pull/903
• [Rust] allow filter with regex expression on columns by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/899
• [CI] bump windows-build storage to 1gb by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/905
• [Doc] cloud storage support by @eddyxu in https://github.com/lancedb/lance/pull/907
• [CI] fix action yaml, storage config needs to be on the first step by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/906
• [Rust] ObjectStore::read_dir method by @eddyxu in https://github.com/lancedb/lance/pull/911

New Contributors

• @chebbyChefNEQ made their first contribution in https://github.com/lancedb/lance/pull/899

Full Changelog: https://github.com/lancedb/lance/compare/v0.4.15...v0.4.16

DuckDB was unnecessarily included as a required dependency, which increases the overall footprint by ~42mb. Removing it makes Lance easier to install in small containers with limited space or more stringent startup time needs.

What's Changed

• ci: fix path in cargo release ci by @wjones127 in https://github.com/lancedb/lance/pull/893
• Update docs page link by @gsilvestrin in https://github.com/lancedb/lance/pull/900
• update lancedb org references by @jaichopra in https://github.com/lancedb/lance/pull/897
• Remove duckdb as python package dependency by @eddyxu in https://github.com/lancedb/lance/pull/902

Full Changelog: https://github.com/lancedb/lance/compare/v0.4.14...v0.4.15

What's Changed

New features:

• feat: allow setting offset without limit by @wjones127 in https://github.com/eto-ai/lance/pull/892
• Customize index type in index creation benchmark by @eddyxu in https://github.com/eto-ai/lance/pull/886
• [Rust] Experimental: DiskANN search by @eddyxu in https://github.com/eto-ai/lance/pull/798

Bug fixes:

• [DuckDB] Fix linux build by @eddyxu in https://github.com/eto-ai/lance/pull/887

Internal changes:

• fix: update minimal futures version by @wjones127 in https://github.com/eto-ai/lance/pull/883
• ci: pass both mac os targets to macos release job by @wjones127 in https://github.com/eto-ai/lance/pull/882
• ci: use rust cache by @wjones127 in https://github.com/eto-ai/lance/pull/881
• Update GHA workflow to make a new commit by @changhiskhan in https://github.com/eto-ai/lance/pull/889
• Github workflow to run cargo publish on release by @changhiskhan in https://github.com/eto-ai/lance/pull/888
• resolve name conflict for workflows by @changhiskhan in https://github.com/eto-ai/lance/pull/890

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.13...v0.4.14

What's Changed

• fix: fetch remainder of metadata if it is large by @wjones127 in https://github.com/eto-ai/lance/pull/873
• wip: allow scanning data in out of order by @wjones127 in https://github.com/eto-ai/lance/pull/874
• feat: add fragments option to the scanner itself by @wjones127 in https://github.com/eto-ai/lance/pull/877

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.12...v0.4.13

What's Changed

• Add format string for the assertion by @haoxins in https://github.com/eto-ai/lance/pull/866
• GH-860: Create a scanner from fragments by @wjones127 in https://github.com/eto-ai/lance/pull/869

New Contributors

• @wjones127 made their first contribution in https://github.com/eto-ai/lance/pull/869

Full Changelog: https://github.com/eto-ai/lance/compare/v0.4.11...v0.4.12