lance

Also including a substantial refactor

What's Changed

• refactor: refactor flat search to lance-index by @eddyxu in https://github.com/lancedb/lance/pull/1419
• refactor: move encodings to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1425
• feat: expose latest version id api by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1426
• refactor: use function pointers instead of trait objects by @wjones127 in https://github.com/lancedb/lance/pull/1424
• feat: automatically convert image to tensors in TF data pipeline by @rok in https://github.com/lancedb/lance/pull/1420
• refactor: migrate schema and data types to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1429
• perf: better parallelism in delete vector prefiltering by @westonpace in https://github.com/lancedb/lance/pull/1428
• test: fix flaky tests involving tokio::fs::File by @westonpace in https://github.com/lancedb/lance/pull/1430
• perf: use selection vector strategy to improve exact knn performance with deletions by @wjones127 in https://github.com/lancedb/lance/pull/1418

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.5...v0.8.6

What's Changed

• chore: object write trait by @eddyxu in https://github.com/lancedb/lance/pull/1389
• refactor: fast path to find fragments for flat scan by @eddyxu in https://github.com/lancedb/lance/pull/1394
• refactor: move reader trait to lance-core by @eddyxu in https://github.com/lancedb/lance/pull/1393
• refactor: move pq to lance-index by @eddyxu in https://github.com/lancedb/lance/pull/1400
• refactor: make pq a batch transformer by @eddyxu in https://github.com/lancedb/lance/pull/1401
• feat: run pq portion of ivf_pq in parallel by @westonpace in https://github.com/lancedb/lance/pull/1386
• feat: generic shuffler over RecordBatchStream by @eddyxu in https://github.com/lancedb/lance/pull/1402
• feat: remap indices on compaction by @westonpace in https://github.com/lancedb/lance/pull/1403
• chore: remove unused crate by @eddyxu in https://github.com/lancedb/lance/pull/1405
• docs: make overwrite row green by @wjones127 in https://github.com/lancedb/lance/pull/1409
• feat: add removed_indices to CreateIndex transaction operation by @eddyxu in https://github.com/lancedb/lance/pull/1408
• feat(rust): incremental index update by @eddyxu in https://github.com/lancedb/lance/pull/1406
• feat(python): expose index optimization via python by @eddyxu in https://github.com/lancedb/lance/pull/1412
• test: add test case to ensure optimize returns to flat KNN by @westonpace in https://github.com/lancedb/lance/pull/1416
• fix: fix bug in index remapping when plan contained multiple rewrite groups by @westonpace in https://github.com/lancedb/lance/pull/1415
• chore: upgrade to datafusion 32 by @wjones127 in https://github.com/lancedb/lance/pull/1391
• ci: cross compile arm wheels by @wjones127 in https://github.com/lancedb/lance/pull/1407
• test: add new ann scenarios to the python benchmarks by @westonpace in https://github.com/lancedb/lance/pull/1411
• chore: instrument various steps in the ann search by @westonpace in https://github.com/lancedb/lance/pull/1404

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.4...v0.8.5

What's Changed

• chore: improve kmean training performance on CUDA by @eddyxu in https://github.com/lancedb/lance/pull/1368
• chore: improve kmean performance on MPS by @eddyxu in https://github.com/lancedb/lance/pull/1370
• chore: use torch.index_add to compute new centroids, to improve training performance on MPS by @eddyxu in https://github.com/lancedb/lance/pull/1371
• feat: schema::field_by_id() by @eddyxu in https://github.com/lancedb/lance/pull/1375
• feat(python): design an image extension type by @rok in https://github.com/lancedb/lance/pull/1272
• perf: improve KNN and ANN performance by @wjones127 in https://github.com/lancedb/lance/pull/1367
• feat: collect int/float/boolean/date page-level statistics on write by @rok in https://github.com/lancedb/lance/pull/1346
• chore: part 1/N of refactoring vector index into separate crate by @eddyxu in https://github.com/lancedb/lance/pull/1388
• fix: handle larger arrays in take by @wjones127 in https://github.com/lancedb/lance/pull/1383
• chore: cleanup tests to avoid errors when optional components are not present by @westonpace in https://github.com/lancedb/lance/pull/1374

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.3...v0.8.4

What's Changed

• chore: compute distance using pytorch and GPU/MPS by @eddyxu in https://github.com/lancedb/lance/pull/1351
• feat: train kmean using pytorch by @eddyxu in https://github.com/lancedb/lance/pull/1358
• build: use larger runner for doc build by @eddyxu in https://github.com/lancedb/lance/pull/1364
• feat(python): gpu based ivf partition training by @eddyxu in https://github.com/lancedb/lance/pull/1361
• fix: stop reading latest manifest by @wjones127 in https://github.com/lancedb/lance/pull/1365

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.2...v0.8.3

What's Changed

• chore: index / recall benchmark over dbpedia by @eddyxu in https://github.com/lancedb/lance/pull/1348
• feat: support storing page-level stats by @wjones127 in https://github.com/lancedb/lance/pull/1316
• fix: pq cosine fast lookup table by @eddyxu in https://github.com/lancedb/lance/pull/1354

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.1...v0.8.2

What's Changed

• feat: add a cache for dynamodb schema validation by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1308
• chore: speed up kmean training for cosine by @eddyxu in https://github.com/lancedb/lance/pull/1334
• feat: add take() method to RecordBatchExt by @eddyxu in https://github.com/lancedb/lance/pull/1337
• feat(python): expose row id in python API by @eddyxu in https://github.com/lancedb/lance/pull/1339
• feat: data generation of dbpedia dataset by @eddyxu in https://github.com/lancedb/lance/pull/1340
• feat: build ivf partition using disk based shuffler by @eddyxu in https://github.com/lancedb/lance/pull/1312
• feat: friendlier error messages in nearest api by @rok in https://github.com/lancedb/lance/pull/1336

Full Changelog: https://github.com/lancedb/lance/compare/v0.8.0...v0.8.1

What's Changed

• fix: bump version workflow by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1297
• fix: knn search works after compaction by @wjones127 in https://github.com/lancedb/lance/pull/1304
• feat: add naive prefiltering by @westonpace in https://github.com/lancedb/lance/pull/1276
• feat: streaming read selective batches in a single Lance data file by @eddyxu in https://github.com/lancedb/lance/pull/1306
• fix: take various types as fragments args and test it by @wjones127 in https://github.com/lancedb/lance/pull/1248
• refactor: switch to thin LTO for faster builds by @westonpace in https://github.com/lancedb/lance/pull/1299
• feat: disk-based shuffle for IVF partition assignment by @eddyxu in https://github.com/lancedb/lance/pull/1300
• fix: tell user column and q are required in nearest by @wjones127 in https://github.com/lancedb/lance/pull/1249
• fix: make CompactionPlan pickle-able and import-able by @wjones127 in https://github.com/lancedb/lance/pull/1307
• perf: improve scan with filter performance by @wjones127 in https://github.com/lancedb/lance/pull/1252
• feat: track fragment size and fragments in indices by @wjones127 in https://github.com/lancedb/lance/pull/1290
• feat: improve DX for matrix struct by @eddyxu in https://github.com/lancedb/lance/pull/1313
• feat(api!): make create_index mutate self by @wjones127 in https://github.com/lancedb/lance/pull/1118
• docs: document commit API as basis for distributed writes by @wjones127 in https://github.com/lancedb/lance/pull/1157
• chore: add pre-commit config file to make it easier for future devs to setup python commit hooks by @westonpace in https://github.com/lancedb/lance/pull/1319
• feat: add prefilter to the to_table method by @westonpace in https://github.com/lancedb/lance/pull/1320
• fix: allow limit as k by @wjones127 in https://github.com/lancedb/lance/pull/1323
• feat: allow customizing kmean sample rate for IVF and PQ by @eddyxu in https://github.com/lancedb/lance/pull/1322
• feat: add basic integration with rust tracing library by @westonpace in https://github.com/lancedb/lance/pull/1310
• feat: add python bindings for the cleanup_old_versions function by @westonpace in https://github.com/lancedb/lance/pull/1270
• feat: fused argmin and argmax to avoid unnecessary memory allocation by @eddyxu in https://github.com/lancedb/lance/pull/1325
• feat: implement wiki cohere benchmark by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1317
• chore: improve dx of wiki benchmark by @eddyxu in https://github.com/lancedb/lance/pull/1327
• chore(rust): improve kmean training time complexity by @eddyxu in https://github.com/lancedb/lance/pull/1330
• chore: convenience methods for reading and writing local files by @eddyxu in https://github.com/lancedb/lance/pull/1329
• fix: make PageTable handle multiple files by @wjones127 in https://github.com/lancedb/lance/pull/1332

Full Changelog: https://github.com/lancedb/lance/compare/v0.7.5...v0.8.0

What's Changed

• ci: fix clippy lints in Python by @wjones127 in https://github.com/lancedb/lance/pull/1281
• feat: support all float types in MatrixView by @eddyxu in https://github.com/lancedb/lance/pull/1282
• chore: simplify float traits by @eddyxu in https://github.com/lancedb/lance/pull/1285
• refactor: don't convert schema to protobuf just for pre-order traversal of fields by @westonpace in https://github.com/lancedb/lance/pull/1259
• feat: fix error message to be more explicit by @eddyxu in https://github.com/lancedb/lance/pull/1294
• fix: search should not panic on empty table by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1289
• chore: add location to all IO errors by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1295
• fix: search empty after delete test by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1296

Full Changelog: https://github.com/lancedb/lance/compare/v0.7.4...v0.7.5

What's Changed

• [Rust] Refactor lance rust crate into sub-modules by @eddyxu in https://github.com/lancedb/lance/pull/1250
• refactor: move main macro benchmarks to python by @wjones127 in https://github.com/lancedb/lance/pull/1235
• [Rust] Refactor to separate linalg, arrow and testing utils into separate modules by @eddyxu in https://github.com/lancedb/lance/pull/1251
• Fix duckdb build by @eddyxu in https://github.com/lancedb/lance/pull/1253
• [Rust] Bump workspace version by @eddyxu in https://github.com/lancedb/lance/pull/1254
• feat: update the commit function to work with transactions by @westonpace in https://github.com/lancedb/lance/pull/1193
• [Python] Support creating index over tensor type and allow Dataset::take() with projection by @eddyxu in https://github.com/lancedb/lance/pull/1258
• move python test entrypoint to makefile by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1260
• [Bug] Fix crashes when take extension types by @eddyxu in https://github.com/lancedb/lance/pull/1261
• [Python] Expose kmeans via python SDK by @eddyxu in https://github.com/lancedb/lance/pull/1263
• feat: add file compaction by @wjones127 in https://github.com/lancedb/lance/pull/1095
• feat: expose query plan as a debug string by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1267
• chore: add pr title check by @westonpace in https://github.com/lancedb/lance/pull/1266
• fix: can't filter on columns with uppercase names by @rok in https://github.com/lancedb/lance/pull/1232
• fix: add test for writing to a dataset with new external store by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1273
• feat: add target cpu to python dir by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1278
• refactor: move release-with-debug by @wjones127 in https://github.com/lancedb/lance/pull/1279
• fix: crate config for release by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1280
• fix: cargo toml to follow crates.io convenetion by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1283

Full Changelog: https://github.com/lancedb/lance/compare/v0.7.3...v0.7.4

What's Changed

• [Rust] fix API breakage in lancecb build by @eddyxu in https://github.com/lancedb/lance/pull/1247

Full Changelog: https://github.com/lancedb/lance/compare/v0.7.2...v0.7.3

What's Changed

• Revert "upgrade manylinux to 2_28 (#1234)" by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1244

Full Changelog: https://github.com/lancedb/lance/compare/v0.7.1...v0.7.2

What's Changed

• fix rust ci by @wjones127 in https://github.com/lancedb/lance/pull/1197
• feat: add count_rows to scanner by @westonpace in https://github.com/lancedb/lance/pull/1150
• move manifest path logic into commit handler by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1198
• implement ExternalManifestCommitHandler and trait ExternalManifestStore by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1190
• fix window test and sleep longer on asof test by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1204
• refactor: allow creating empty dataset fragments by @rok in https://github.com/lancedb/lance/pull/1229
• add documentation for external store commit procedure by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1207
• Implement path finalization for external manifest store by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1233
• implement dynamodb external manifest store by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1205
• upgrade manylinux to 2_28 by @eddyxu in https://github.com/lancedb/lance/pull/1234
• fix clippy on main by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1236
• perf: reduce copies and skip zeroing memory we read into by @wjones127 in https://github.com/lancedb/lance/pull/1238
• add parsing in object store for s3+ddb scheme by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1239
• add documentation for dynamodb uri by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1242
• fix images references in documents by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1243
• feat: add a fast path for pq index if there are no missing fragments/rows by @westonpace in https://github.com/lancedb/lance/pull/1240

Full Changelog: https://github.com/lancedb/lance/compare/v0.7.0...v0.7.1

What's Changed

• [Doc] Add deep dive presentation to README by @eddyxu in https://github.com/lancedb/lance/pull/1167
• run format and clippy on latest rust by @wjones127 in https://github.com/lancedb/lance/pull/1171
• feat: expose index stats for debugging by @rok in https://github.com/lancedb/lance/pull/1097
• feat: add cleanup_old_versions method which will remove unreferenced old files by @westonpace in https://github.com/lancedb/lance/pull/1140
• [bench] Fix up rust benchmarks by @changhiskhan in https://github.com/lancedb/lance/pull/1176
• feat: add a background executor for Python by @wjones127 in https://github.com/lancedb/lance/pull/1172
• [python] Validate schema metadata by @changhiskhan in https://github.com/lancedb/lance/pull/1186
• use commit handler to find the latest version of a dataset by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1187
• Remove request dependency as it's only used for its re-export of http by @dnsco in https://github.com/lancedb/lance/pull/1189
• feat: add support for field metdata by @westonpace in https://github.com/lancedb/lance/pull/1173
• feat: parse TFRecords as Arrow data by @wjones127 in https://github.com/lancedb/lance/pull/1166
• feat: add function to cleanup partial writes by @wjones127 in https://github.com/lancedb/lance/pull/1182
• feat: tf.data support for binary, struct, and FixedShapeTensor by @wjones127 in https://github.com/lancedb/lance/pull/1191
• fix: make field name consistent by @wjones127 in https://github.com/lancedb/lance/pull/1192
• refactor: pass params by reference in from_uri_and_params method by @westonpace in https://github.com/lancedb/lance/pull/1194

Full Changelog: https://github.com/lancedb/lance/compare/v0.6.5...v0.7.0

What's Changed

• [Rust] Compute dot distance as negative dot product by @eddyxu in https://github.com/lancedb/lance/pull/1160
• fix: Scanner schema fix by @gsilvestrin in https://github.com/lancedb/lance/pull/1161
• implement flat vector index benchmark by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1163
• clean cached wheels by @wjones127 in https://github.com/lancedb/lance/pull/1159

Full Changelog: https://github.com/lancedb/lance/compare/v0.6.4...v0.6.5

What's Changed

• feat: rebase restore feature on top of transactions feature by @westonpace in https://github.com/lancedb/lance/pull/1143
• feat: pass commit handler on dataset load by @wjones127 in https://github.com/lancedb/lance/pull/1154
• Track writing fragment progress by @eddyxu in https://github.com/lancedb/lance/pull/1146

Full Changelog: https://github.com/lancedb/lance/compare/v0.6.3...v0.6.4

What's Changed

• fix rust build error on Mac and remove warning messages from running benchmark by @QianZhu in https://github.com/lancedb/lance/pull/1137
• feat: transaction files and commit conflict resolution by @wjones127 in https://github.com/lancedb/lance/pull/1127
• [Rust] Benchmark for dot product by @eddyxu in https://github.com/lancedb/lance/pull/1142
• feat: quick restore command by @wjones127 in https://github.com/lancedb/lance/pull/1138
• Revert "feat: quick restore command (#1138)" by @eddyxu in https://github.com/lancedb/lance/pull/1148
• Fragment::write honors row group size by @eddyxu in https://github.com/lancedb/lance/pull/1145
• expose aws region configurability by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1153
• fix docs build by @changhiskhan in https://github.com/lancedb/lance/pull/1156

New Contributors

• @QianZhu made their first contribution in https://github.com/lancedb/lance/pull/1137

Full Changelog: https://github.com/lancedb/lance/compare/v0.6.2...v0.6.3

What's Changed

• propagate credential when opening dataset for write by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/1136

Full Changelog: https://github.com/lancedb/lance/compare/v0.6.1...v0.6.2

What's Changed

• Implement L2/norm/cosine/dot for bf16/f16/f64 by @eddyxu in https://github.com/lancedb/lance/pull/1121
• fix: cast when passed an iterator by @wjones127 in https://github.com/lancedb/lance/pull/1123
• feat: detect commit conflicts by @wjones127 in https://github.com/lancedb/lance/pull/1104
• feat: upgrade to datafusion 28.0.0 and arrow 43.0 by @westonpace in https://github.com/lancedb/lance/pull/1125
• feat: upgrade arrow to 43.0 in duckdb-lance by @westonpace in https://github.com/lancedb/lance/pull/1126
• feat: switch session caches to use moka cache instead of LruCache by @westonpace in https://github.com/lancedb/lance/pull/1131
• Added append method for dataset which takes self as arguemnt with tests by @trueutkarsh in https://github.com/lancedb/lance/pull/1122
• feat: Persist schema metadata by @gsilvestrin in https://github.com/lancedb/lance/pull/1133

New Contributors

• @westonpace made their first contribution in https://github.com/lancedb/lance/pull/1125

Full Changelog: https://github.com/lancedb/lance/compare/v0.6.0...v0.6.1

⚠️ Breaking Change: The score column has been renamed to _distance to more accurately describe the semantics (smaller means closer / better).

What's Changed

• Rename "score" to "_distance" by @trueutkarsh in https://github.com/lancedb/lance/pull/1055
• fix: take and search now work properly with deletions by @wjones127 in https://github.com/lancedb/lance/pull/1113
• Improve grammar of README by @michaelmior in https://github.com/lancedb/lance/pull/1101
• Serde Fragment to json by @eddyxu in https://github.com/lancedb/lance/pull/1105
• ci: fix the release commit script by @wjones127 in https://github.com/lancedb/lance/pull/1119

New Contributors

• @michaelmior made their first contribution in https://github.com/lancedb/lance/pull/1101

Full Changelog: https://github.com/lancedb/lance/compare/v0.5.10...v0.6.0

What's Changed

• feat: make pandas optional dependency by @rok in https://github.com/lancedb/lance/pull/1084
• fix bug in pandas check by @changhiskhan in https://github.com/lancedb/lance/pull/1085
• [Python] Set batch_size in Dataset::scanner() by @eddyxu in https://github.com/lancedb/lance/pull/1088
• [Python] Scanner dataset and projection schema by @eddyxu in https://github.com/lancedb/lance/pull/1089
• docs: add clarifications to format about fields by @wjones127 in https://github.com/lancedb/lance/pull/1052
• feat: add additional caching and expose batch size by @wjones127 in https://github.com/lancedb/lance/pull/1082
• feat: enforce max rows per group and file by @wjones127 in https://github.com/lancedb/lance/pull/1093
• [WIP] Tf.data pipeline by @eddyxu in https://github.com/lancedb/lance/pull/1087
• feat: track the max used fragment id by @wjones127 in https://github.com/lancedb/lance/pull/1092
• [Doc] Use Lance with tf.data by @eddyxu in https://github.com/lancedb/lance/pull/1096

New Contributors

• @rok made their first contribution in https://github.com/lancedb/lance/pull/1084

Full Changelog: https://github.com/lancedb/lance/compare/v0.5.9...v0.5.10