lance

What's Changed

Bug Fixes 🐛

• fix: incorrect chunking was making lance datasets use too much RAM by @westonpace in https://github.com/lancedb/lance/pull/2438

Full Changelog: https://github.com/lancedb/lance/compare/v0.12.0...v0.12.1

What's Changed

Breaking Changes 🛠

• feat: change dataset uri to return full qualified url instead of object store path by @eddyxu in https://github.com/lancedb/lance/pull/2416

New Features 🎉

• feat: new shuffler by @BubbleCal in https://github.com/lancedb/lance/pull/2404
• feat: new index builder by @BubbleCal in https://github.com/lancedb/lance/pull/2401
• feat: stable row id manifest changes by @wjones127 in https://github.com/lancedb/lance/pull/2363
• feat: once a table has been created with v1 or v2 format then it should always use that format by @westonpace in https://github.com/lancedb/lance/pull/2435

Bug Fixes 🐛

• fix: fix file writer which was not writing page buffers in the correct order by @westonpace in https://github.com/lancedb/lance/pull/2413

Other Changes

• refactor: refactor logical decoders into "field decoders" by @westonpace in https://github.com/lancedb/lance/pull/2407
• refactor: rename use_experimental_writer to use_legacy_format by @westonpace in https://github.com/lancedb/lance/pull/2433
• refactor: minor refactor to allow I/O scheduler to be cloned in page schedulers by @westonpace in https://github.com/lancedb/lance/pull/2432

Full Changelog: https://github.com/lancedb/lance/compare/v0.11.1...v0.12.0

What's Changed

Breaking Changes 🛠

• feat: change dataset uri to return full qualified url instead of object store path by @eddyxu in https://github.com/lancedb/lance/pull/2416

New Features 🎉

• feat: new shuffler by @BubbleCal in https://github.com/lancedb/lance/pull/2404
• feat: new index builder by @BubbleCal in https://github.com/lancedb/lance/pull/2401
• feat: stable row id manifest changes by @wjones127 in https://github.com/lancedb/lance/pull/2363

Bug Fixes 🐛

• fix: fix file writer which was not writing page buffers in the correct order by @westonpace in https://github.com/lancedb/lance/pull/2413

Other Changes

• refactor: refactor logical decoders into "field decoders" by @westonpace in https://github.com/lancedb/lance/pull/2407
• refactor: rename use_experimental_writer to use_legacy_format by @westonpace in https://github.com/lancedb/lance/pull/2433

Full Changelog: https://github.com/lancedb/lance/compare/v0.11.1...v0.12.0-beta.1

What's Changed

New Features 🎉

• feat(java): support jdk8 by @LuQQiu in https://github.com/lancedb/lance/pull/2362
• feat: support kmode with hamming distance by @eddyxu in https://github.com/lancedb/lance/pull/2366
• feat: row id index structures (experimental) by @wjones127 in https://github.com/lancedb/lance/pull/2303
• feat: update merge_insert to add statistics for inserted, updated, deleted rows by @raunaks13 in https://github.com/lancedb/lance/pull/2357
• feat: define Flat index as a scan over VectorStorage by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2380
• feat: add some schema utility methods to the v2 reader/writer by @westonpace in https://github.com/lancedb/lance/pull/2389
• feat: general compression for value page buffer by @niyue in https://github.com/lancedb/lance/pull/2368
• feat: make the index cache size (in bytes) available by @westonpace in https://github.com/lancedb/lance/pull/2381
• feat: add special uri scheme to use CloudFileReader for local fs by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2402
• feat: add encoder utilities for pushdown by @westonpace in https://github.com/lancedb/lance/pull/2388

Bug Fixes 🐛

• fix: concat batches before writing to avoid small IO slow down by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2384
• fix: low recall if the num partitions is more than num rows by @BubbleCal in https://github.com/lancedb/lance/pull/2386
• fix: f32 reduce_min for x86 by @heiher in https://github.com/lancedb/lance/pull/2385
• fix: fix incorrect validation logic in updater by @westonpace in https://github.com/lancedb/lance/pull/2408

Performance Improvements 🚀

• perf: make VectorStorage and DistCalculator static to generate better code by @BubbleCal in https://github.com/lancedb/lance/pull/2355
• perf: optimize IO path for reading manifest by @wjones127 in https://github.com/lancedb/lance/pull/2396

Other Changes

• refactor: make proto conversion fallible and not copy by @wjones127 in https://github.com/lancedb/lance/pull/2371
• refactor: separate take and schema evolution impls to own files by @wjones127 in https://github.com/lancedb/lance/pull/2372
• Revert "fix: concat batches before writing to avoid small IO slow down (#2384)" by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2387
• refactor: shuffle around v2 metadata sections to allow read-on-demand statistics by @westonpace in https://github.com/lancedb/lance/pull/2400

New Contributors

• @niyue made their first contribution in https://github.com/lancedb/lance/pull/2368
• @heiher made their first contribution in https://github.com/lancedb/lance/pull/2385

Full Changelog: https://github.com/lancedb/lance/compare/v0.11.0...v0.11.1

What's Changed

Breaking Changes 🛠

• feat(rust)!: use BoxedError in Error::IO by @broccoliSpicy in https://github.com/lancedb/lance/pull/2329

New Features 🎉

• feat: add v2 support to fragment merge / update paths by @westonpace in https://github.com/lancedb/lance/pull/2311
• feat: add priority to I/O scheduler by @westonpace in https://github.com/lancedb/lance/pull/2315
• feat: add take_rows operation to the v2 file reader's python bindings by @westonpace in https://github.com/lancedb/lance/pull/2331
• feat: added example for reading and writing dataset in rust by @raunaks13 in https://github.com/lancedb/lance/pull/2349
• feat: new HNSW implementation by @BubbleCal in https://github.com/lancedb/lance/pull/2353
• feat: add fragment take / fixed-size-binary support to v2 format by @westonpace in https://github.com/lancedb/lance/pull/2354

Bug Fixes 🐛

• fix: recognize a simple expression like 'is_foo' as a scalar index query by @westonpace in https://github.com/lancedb/lance/pull/2356
• fix: rework list encoder to handle list-struct by @westonpace in https://github.com/lancedb/lance/pull/2344
• fix: minor bug fixes for v2 by @westonpace in https://github.com/lancedb/lance/pull/2361

Documentation 📚

• docs: clearify comments in table.proto -> message DataFragment -> physical_rows by @broccoliSpicy in https://github.com/lancedb/lance/pull/2346

Performance Improvements 🚀

• perf: use the file metadata cache in scalar indices by @westonpace in https://github.com/lancedb/lance/pull/2330

Other Changes

• chore: remove m_max and use_heuristic params from HNSW builder by @BubbleCal in https://github.com/lancedb/lance/pull/2336
• fix(java): fix JNI jar loader issue by @LuQQiu in https://github.com/lancedb/lance/pull/2340
• ci: fix labeler permissions by @wjones127 in https://github.com/lancedb/lance/pull/2348
• fix: rework decoding to fix bugs in nested struct decoding by @westonpace in https://github.com/lancedb/lance/pull/2337

New Contributors

• @broccoliSpicy made their first contribution in https://github.com/lancedb/lance/pull/2346
• @raunaks13 made their first contribution in https://github.com/lancedb/lance/pull/2349

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.18...v0.11.0

What's Changed

New Features 🎉

• feat: don't load all list items before returning a batch by @westonpace in https://github.com/lancedb/lance/pull/2262
• feat(java): support dataset/fragment scan with string filter, columns filter by @LuQQiu in https://github.com/lancedb/lance/pull/2266
• feat: support storage options in write fragments by @eddyxu in https://github.com/lancedb/lance/pull/2289
• feat: ray sink takes storage options parameter by @eddyxu in https://github.com/lancedb/lance/pull/2293
• feat(python): support auto conversion between fixed size int array to pytorch tensor by @eddyxu in https://github.com/lancedb/lance/pull/2294
• feat: add experimental limit variables for GCS by @wjones127 in https://github.com/lancedb/lance/pull/2295
• feat: add support for large string & large binary to the v2 encoder & decoder by @westonpace in https://github.com/lancedb/lance/pull/2297
• feat: hamming distance by @eddyxu in https://github.com/lancedb/lance/pull/2110
• feat: add projection support to the v2 format by @westonpace in https://github.com/lancedb/lance/pull/2296

Bug Fixes 🐛

• fix: fix some corner cases that can arise in v2 list encoding by @westonpace in https://github.com/lancedb/lance/pull/2291
• fix: flush after writing pages so that we don't yield to the user with in-progress writes by @westonpace in https://github.com/lancedb/lance/pull/2298
• fix: fix a panic that could happen when scanning only the row id from fragment with deleted rows by @westonpace in https://github.com/lancedb/lance/pull/2302
• fix: relax 'take out of bounds' check which could cause failure if flat searching deleted rows by @westonpace in https://github.com/lancedb/lance/pull/2314

Documentation 📚

• docs: update quickstart doc for supporting IVF_HNSW_* by @BubbleCal in https://github.com/lancedb/lance/pull/2301

Performance Improvements 🚀

• perf: re-enable late materialization on full scans by @westonpace in https://github.com/lancedb/lance/pull/2290

Other Changes

• perf: do HNSW search with threads of CPU runtime by @BubbleCal in https://github.com/lancedb/lance/pull/2251
• chore: add ossrh plugins to jni package by @eddyxu in https://github.com/lancedb/lance/pull/2265

New Contributors

• @kafonek made their first contribution in https://github.com/lancedb/lance/pull/2281

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.17...v0.10.18

What's Changed

New Features 🎉

• feat: don't load all list items before returning a batch by @westonpace in https://github.com/lancedb/lance/pull/2262
• feat(java): support dataset/fragment scan with string filter, columns filter by @LuQQiu in https://github.com/lancedb/lance/pull/2266
• feat: support storage options in write fragments by @eddyxu in https://github.com/lancedb/lance/pull/2289
• feat: ray sink takes storage options parameter by @eddyxu in https://github.com/lancedb/lance/pull/2293
• feat(python): support auto conversion between fixed size int array to pytorch tensor by @eddyxu in https://github.com/lancedb/lance/pull/2294
• feat: add experimental limit variables for GCS by @wjones127 in https://github.com/lancedb/lance/pull/2295
• feat: add support for large string & large binary to the v2 encoder & decoder by @westonpace in https://github.com/lancedb/lance/pull/2297
• feat: hamming distance by @eddyxu in https://github.com/lancedb/lance/pull/2110

Bug Fixes 🐛

• fix: fix some corner cases that can arise in v2 list encoding by @westonpace in https://github.com/lancedb/lance/pull/2291
• fix: flush after writing pages so that we don't yield to the user with in-progress writes by @westonpace in https://github.com/lancedb/lance/pull/2298
• fix: fix a panic that could happen when scanning only the row id from fragment with deleted rows by @westonpace in https://github.com/lancedb/lance/pull/2302

Performance Improvements 🚀

• perf: re-enable late materialization on full scans by @westonpace in https://github.com/lancedb/lance/pull/2290

Other Changes

• perf: do HNSW search with threads of CPU runtime by @BubbleCal in https://github.com/lancedb/lance/pull/2251

New Contributors

• @kafonek made their first contribution in https://github.com/lancedb/lance/pull/2281

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.17...v0.10.18-beta.1

What's Changed

New Features 🎉

• feat: add nullability and u64 support to v2 list codec by @westonpace in https://github.com/lancedb/lance/pull/2255

Bug Fixes 🐛

• fix: don't use pushdown scan if a custom batch size has been set by @westonpace in https://github.com/lancedb/lance/pull/2277

Documentation 📚

• docs: document LANCE_BYPASS_SPILLING environment variable by @alexkohler in https://github.com/lancedb/lance/pull/2276

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.16...v0.10.17

Bug fixes

• fix: use 1/ln(m) as the mL value for better performance by @BubbleCal in https://github.com/lancedb/lance/pull/2239
• fix: various fixes to ensure that the v2 writer is used by the ray sink if requested by @westonpace in https://github.com/lancedb/lance/pull/2248
• fix: bugs in list codec when dealing with a list of empty lists or multiple lists in a single page by @westonpace in https://github.com/lancedb/lance/pull/2222
• fix: creating IVF_HNSW index fails with cosine metric type by @BubbleCal in https://github.com/lancedb/lance/pull/2235
• fix: build & search causes panic with empty partition for IVF_HNSW by @BubbleCal in https://github.com/lancedb/lance/pull/2172
• fix: clippy on main by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2254
• fix: copy arrays when placing them in the v2 writer's accumulation queue by @westonpace in https://github.com/lancedb/lance/pull/2249
• fix: out of bound panics in take rows by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2259
• fix: allow for all-null pages in btree by @westonpace in https://github.com/lancedb/lance/pull/2245

New Features

• feat: add use_experimental_writer to write_fragments API by @westonpace in https://github.com/lancedb/lance/pull/2226
• feat: use v2 writer api in ray data sink by @eddyxu in https://github.com/lancedb/lance/pull/2231
• feat(java): add LanceOperation, commitAppend for batch write by @LuQQiu in https://github.com/lancedb/lance/pull/2207
• feat(python): expose FragmentMetadata attrs and compaction repr by @wjones127 in https://github.com/lancedb/lance/pull/2191
• feat: update arrow to 51, datafusion to 37 by @westonpace in https://github.com/lancedb/lance/pull/2240
• feat: improvement of Ray sink API by @eddyxu in https://github.com/lancedb/lance/pull/2237
• feat: forbid creating index if num_sub_vectors doesn't divide dim by @BubbleCal in https://github.com/lancedb/lance/pull/2234
• feat: add create_index to vector index extensions by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2250
• feat: support delta merge for IVF_HNSW_SQ by @BubbleCal in https://github.com/lancedb/lance/pull/2132
• feat(java): add version latestVersion api by @LuQQiu in https://github.com/lancedb/lance/pull/2238
• feat: raise GCS object size limit to 2.5TB by @wjones127 in https://github.com/lancedb/lance/pull/2261

Performance Improvements

• perf: concurrent building for HNSW graph by @BubbleCal in https://github.com/lancedb/lance/pull/2210
• perf: v2 fsl decode perf fix and some benchmarking utilities by @westonpace in https://github.com/lancedb/lance/pull/2214
• perf: avoid copying of creating memory dist calculator by @BubbleCal in https://github.com/lancedb/lance/pull/2219
• perf: keep less edges in non-base levels to improve greedy search performance by @BubbleCal in https://github.com/lancedb/lance/pull/2244

Other changes

• chore: remove --preview from ruff invocations by @westonpace in https://github.com/lancedb/lance/pull/2220
• chore: fix formatting in test_schema.py by @westonpace in https://github.com/lancedb/lance/pull/2227
• chore: bump to ruff 0.4.1 by @eddyxu in https://github.com/lancedb/lance/pull/2229
• chore: bump minimal python version to 39 by @eddyxu in https://github.com/lancedb/lance/pull/2230
• chore: use VectorStorage in HNSWBuilder by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2242
• test: test write methods against more varied layouts by @wjones127 in https://github.com/lancedb/lance/pull/2233

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.15...v0.10.16

What's Changed

• fix: pickle LanceSchema nested fields correctly by @wjones127 in https://github.com/lancedb/lance/pull/2223
• fix: do not create directory when open an non-existing dataset by @eddyxu in https://github.com/lancedb/lance/pull/2215
• perf: build HNSW and quantization storage in parallel by @BubbleCal in https://github.com/lancedb/lance/pull/2196
• chore: remove extend_candidates option by @BubbleCal in https://github.com/lancedb/lance/pull/2217
• perf: avoid copying every vectors multiple times during building by @BubbleCal in https://github.com/lancedb/lance/pull/2216
• chore: search for enough candidates to build connections by @BubbleCal in https://github.com/lancedb/lance/pull/2218
• feat: add support v2 fragments on the scan path by @westonpace in https://github.com/lancedb/lance/pull/2213
• feat: allow loading extension vector index by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2221

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.14...v0.10.15

What's Changed

• fix: llm pre-training docs by @tanaymeh in https://github.com/lancedb/lance/pull/2205
• refactor: rework fragment API in preparation for lance v2 by @westonpace in https://github.com/lancedb/lance/pull/2194
• feat: add v2 writer to the write dataset path by @westonpace in https://github.com/lancedb/lance/pull/2206
• refactor: migrate scan path from v1-only to generic reader by @westonpace in https://github.com/lancedb/lance/pull/2201
• feat: replace LanceFragment.add_columns with merge_columns by @wjones127 in https://github.com/lancedb/lance/pull/2208
• feat: support complex schemas in append by @wjones127 in https://github.com/lancedb/lance/pull/2209

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.13...v0.10.14

What's Changed

• feat: add FragmentWriter and Commiter for streaming ray write by @eddyxu in https://github.com/lancedb/lance/pull/2190
• perf: optimize construction & search for HNSW by @BubbleCal in https://github.com/lancedb/lance/pull/2193
• feat(java): add lance jni create empty dataset, create fragment, and get schema by @LuQQiu in https://github.com/lancedb/lance/pull/2175
• refactor: move existing utilities from file reader into fragment-level operations by @westonpace in https://github.com/lancedb/lance/pull/2197
• feat: more validation of fragments by @wjones127 in https://github.com/lancedb/lance/pull/2203
• feat: debug functions by @wjones127 in https://github.com/lancedb/lance/pull/2202

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.12...v0.10.13

What's Changed

• feat: add v2 nullability for primitive / fsl by @westonpace in https://github.com/lancedb/lance/pull/2169
• feat: add support for binary encoding by @westonpace in https://github.com/lancedb/lance/pull/2183
• feat: python binding for HNSW utils by @BubbleCal in https://github.com/lancedb/lance/pull/2182
• fix: add env variables to bypass/configure spill pool size by @westonpace in https://github.com/lancedb/lance/pull/2189

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.11...v0.10.12

New features

• feat: lance ray data sink by @eddyxu in https://github.com/lancedb/lance/pull/2180
• feat: cast ivf_centroids to appropriate type by @wjones127 in https://github.com/lancedb/lance/pull/2097
• feat: add python bindings for the v2 reader/writer by @westonpace in https://github.com/lancedb/lance/pull/2158
• feat: mvp for lance version 0.2 reader / writer by @westonpace in https://github.com/lancedb/lance/pull/1965
• feat: add string arrays to the set of v2 encodings by @westonpace in https://github.com/lancedb/lance/pull/2163

Bug fixes

• fix: handle varying field ids by @wjones127 in https://github.com/lancedb/lance/pull/2187

Other changes

• fix: pin h5py version to avoid missing arm wheels by @westonpace in https://github.com/lancedb/lance/pull/2178
• fix: select the wrong neighbors by @BubbleCal in https://github.com/lancedb/lance/pull/2181
• chore: refine the IVF_HNSW index building path by @BubbleCal in https://github.com/lancedb/lance/pull/2162
• ci: full python 3.12 ci by @eddyxu in https://github.com/lancedb/lance/pull/2170
• perf: impl heuristic pruning for HNSW by @BubbleCal in https://github.com/lancedb/lance/pull/2171
• chore: fix license headers by @westonpace in https://github.com/lancedb/lance/pull/2177
• docs: ray integration by @eddyxu in https://github.com/lancedb/lance/pull/2185
• fix: fix ray test broken on windows by @westonpace in https://github.com/lancedb/lance/pull/2188

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.10...v0.10.11

Features

• feat: easier and consistent S3 configuration by @wjones127 in https://github.com/lancedb/lance/pull/2147
  • When using AWS S3, you no longer have to specify the region.
  • S3 configuration is now more consistent in how it is picked up. It will now always read explicit storage_options first before looking at environment variables.
• feat: add encoders/decoders for basic types by @westonpace in https://github.com/lancedb/lance/pull/2142
• feat: initial reader/writer for the v2 format by @westonpace in https://github.com/lancedb/lance/pull/2153
• feat: add take to the v2 schedulers by @westonpace in https://github.com/lancedb/lance/pull/2156
• feat: add ef as query parameter by @BubbleCal in https://github.com/lancedb/lance/pull/2155
• feat: fast l2 for uint8 by @eddyxu in https://github.com/lancedb/lance/pull/2161
• perf: use fast u8 l2 route in hnsw beam search by @eddyxu in https://github.com/lancedb/lance/pull/2164
• feat: make substrait optional to allow avoiding libgit2 by @westonpace in https://github.com/lancedb/lance/pull/2168

Bug fixes

• fix: set prost to minimum 0.12.2 by @albertlockett in https://github.com/lancedb/lance/pull/2167

Other changes

• chore: apply clippy suggestions newly introduced in latest compiler by @westonpace in https://github.com/lancedb/lance/pull/2150

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.9...v0.10.10

What's Changed

• feat: support IVF_HNSW_SQ in Python by @BubbleCal in https://github.com/lancedb/lance/pull/2149
• feat: add outer retry loop to CloudObjectReader::size by @westonpace in https://github.com/lancedb/lance/pull/2151
• docs: fix typo in lance-arrow by @rgbkrk in https://github.com/lancedb/lance/pull/2152

New Contributors

• @rgbkrk made their first contribution in https://github.com/lancedb/lance/pull/2152

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.8...v0.10.9

What's Changed

• fix: may get lower recall from HNSW with quantization by @BubbleCal in https://github.com/lancedb/lance/pull/2145
• feat: add core traits for encoders & decoders in the v2 format by @westonpace in https://github.com/lancedb/lance/pull/2141
• feat: support filter in count rows by @eddyxu in https://github.com/lancedb/lance/pull/2146
• fix: variable file fragments by @wjones127 in https://github.com/lancedb/lance/pull/2148

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.7...v0.10.8

Bug fixes

❗ There was an bug with drop_columns(). If you've called this on your dataset, you should check if your dataset was affected by running dataset.validate(). If this raises an error, you can call dataset.delete("false") to force a repair operation on your dataset. Afterward it will work as expected.

• fix: remove data files with all dropped columns by @wjones127 in https://github.com/lancedb/lance/pull/2130

New Features

🚀 You can now configure object storage connection in the kwargs of lance.dataset() and lance.write_dataset() with storage_options. For example:

import lance
ds = lance.dataset(
    "s3://bucket/path",
    storage_options={
        "region": "us-east-1",
        "access_key_id": "my-access-key",
        "secret_access_key": "my-secret-key",
        "session_token": "my-session-token",
    }
)

Read more in https://lancedb.github.io/lance/read_and_write.html#object-store-configuration

• feat(python): expose storage options by @wjones127 in https://github.com/lancedb/lance/pull/2131
• feat: extend datagen to cover more types by @westonpace in https://github.com/lancedb/lance/pull/2138
• feat: add a protobuf file describing encodings by @westonpace in https://github.com/lancedb/lance/pull/2137
• feat: add a basic encodings crate by @westonpace in https://github.com/lancedb/lance/pull/2139
• feat: support IVF_HNSW_SQ by @BubbleCal in https://github.com/lancedb/lance/pull/2136

Other Changes

• chore: expose dynamic projection on fragment API by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2144

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.6...v0.10.7

What's Changed

• feat: expose migration check by @wjones127 in https://github.com/lancedb/lance/pull/2074
• chore: loading HNSW levels in parallel by @BubbleCal in https://github.com/lancedb/lance/pull/2093
• chore: construct the dist table only once while searching by @BubbleCal in https://github.com/lancedb/lance/pull/2094
• feat: enable fp16kernels in Mac and x86 Linux Python wheels by @wjones127 in https://github.com/lancedb/lance/pull/2098
• feat: support IVF_HNSW index by @BubbleCal in https://github.com/lancedb/lance/pull/2080
• perf: create ood dataset in bigann benchmark by @eddyxu in https://github.com/lancedb/lance/pull/2084
• chore: fix the time unit in logs by @BubbleCal in https://github.com/lancedb/lance/pull/2101
• chore: dynamically detect the schema while shuffling data by @BubbleCal in https://github.com/lancedb/lance/pull/2105
• chore: expose find_partition method by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2106
• fix: very low recall on IVF_HNSW by @BubbleCal in https://github.com/lancedb/lance/pull/2104
• fix: load_partition return Error when partition_id out of bounds by @LeoReeYang in https://github.com/lancedb/lance/pull/2107
• chore: expose query residulization by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2108
• chore: move java core api to sub java module by @LuQQiu in https://github.com/lancedb/lance/pull/2115
• docs: update image_to_tensor to to_tensor by @vipul-maheshwari in https://github.com/lancedb/lance/pull/2116
• perf: independent parallel building for IVF_HNSW partitions by @BubbleCal in https://github.com/lancedb/lance/pull/2109
• chore: add example of IVF_HNSW by @BubbleCal in https://github.com/lancedb/lance/pull/2112
• perf: fully building HNSW partitions in parallel by @BubbleCal in https://github.com/lancedb/lance/pull/2117
• perf: load HNSW levels in parallel by @BubbleCal in https://github.com/lancedb/lance/pull/2111
• chore: drop data after copied to reduce memory footprint by @BubbleCal in https://github.com/lancedb/lance/pull/2120
• fix: populate index cache at the end of loading by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2123
• fix: use the fair spill pool instead of the greedy spill pool by @westonpace in https://github.com/lancedb/lance/pull/2126
• feat: support create IVF_HNSW_PQ index in Python by @BubbleCal in https://github.com/lancedb/lance/pull/2127
• feat: add scalar quantizer by @BubbleCal in https://github.com/lancedb/lance/pull/2134
• fix: the HNSW index doesn't respect to refine factor by @BubbleCal in https://github.com/lancedb/lance/pull/2122
• feat: add sq storage and transformer by @BubbleCal in https://github.com/lancedb/lance/pull/2135

New Contributors

• @LeoReeYang made their first contribution in https://github.com/lancedb/lance/pull/2107
• @vipul-maheshwari made their first contribution in https://github.com/lancedb/lance/pull/2116

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.5...v0.10.6

Fixes a potential panic when reading a fragment that had multiple data files.

What's Changed

• chore: expose internal index APIs by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2082
• chore: expose prefilter traits by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2083
• feat: add java fragment create by @LuQQiu in https://github.com/lancedb/lance/pull/2081
• chore: enable codecov by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/2088
• perf: add a set of benchmark dataset by @eddyxu in https://github.com/lancedb/lance/pull/2090
• perf: text2image benchmark by @eddyxu in https://github.com/lancedb/lance/pull/2091
• docs: add llm training example by @tanaymeh in https://github.com/lancedb/lance/pull/2087
• fix: only read the data file's fields from the page table and not the whole dataset's fields by @westonpace in https://github.com/lancedb/lance/pull/2095

New Contributors

• @LuQQiu made their first contribution in https://github.com/lancedb/lance/pull/2081

Full Changelog: https://github.com/lancedb/lance/compare/v0.10.4...v0.10.5