lancedb

What's Changed

• [Docs]Versioning docs by @AyushExel in https://github.com/lancedb/lancedb/pull/586
• [Docs] Embeddings API: Add multi-lingual semantic search example by @AyushExel in https://github.com/lancedb/lancedb/pull/582
• [Docs]: Minor Fixes by @AyushExel in https://github.com/lancedb/lancedb/pull/587
• feat: include manifest files in mirrow store by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/589
• feat: add checkout method to table to reuse existing store and connections by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/593
• fix(python): fix multiple embedding functions bug by @changhiskhan in https://github.com/lancedb/lancedb/pull/597
• update lance to 0.8.7 by @albertlockett in https://github.com/lancedb/lancedb/pull/598
• feat: expose optimize index api by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/602
• expose remap index api by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/603
• feat: added data stats apis by @albertlockett in https://github.com/lancedb/lancedb/pull/596
• feat: added dataset stats api to node by @albertlockett in https://github.com/lancedb/lancedb/pull/604

Lance updated from 0.8.6 -> 0.8.8

v0.8.8 Addition of dataset stats API
v0.8.7 Vector index performance improvement, customize file size during writing lance dataset, bug fixes

New Contributors

• @albertlockett made their first contribution in https://github.com/lancedb/lancedb/pull/598

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.3.3...v0.3.4

What's Changed

• [DOCS][PYTHON] Update embeddings API docs & Example by @AyushExel in https://github.com/lancedb/lancedb/pull/516
• chore: bump lance to 0.8.5 by @eddyxu in https://github.com/lancedb/lancedb/pull/561
• docs: switch python examples to be row based by @rok in https://github.com/lancedb/lancedb/pull/554
• feat(python,js): deletion operation on remote tables by @eddyxu in https://github.com/lancedb/lancedb/pull/568
• docs: show source of documented functions by @rok in https://github.com/lancedb/lancedb/pull/569
• implement remote api calls for table mutation by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/567
• doc: fix broken link and add README by @changhiskhan in https://github.com/lancedb/lancedb/pull/573
• Add pyarrow date and timestamp type conversion from pydantic by @prrao87 in https://github.com/lancedb/lancedb/pull/576
• list table pagination draft by @QianZhu in https://github.com/lancedb/lancedb/pull/574
• [Docs] Add posthog telemetry to docs by @AyushExel in https://github.com/lancedb/lancedb/pull/577
• [Python]Embeddings API refactor by @AyushExel in https://github.com/lancedb/lancedb/pull/580
• [Docs] Update embedding function docs by @AyushExel in https://github.com/lancedb/lancedb/pull/581
• chore: bump lance version in python/rust lancedb by @changhiskhan in https://github.com/lancedb/lancedb/pull/584
• [Docs]Versioning docs by @AyushExel in https://github.com/lancedb/lancedb/pull/586
• [Docs] Embeddings API: Add multi-lingual semantic search example by @AyushExel in https://github.com/lancedb/lancedb/pull/582
• [Docs]: Minor Fixes by @AyushExel in https://github.com/lancedb/lancedb/pull/587
• feat: include manifest files in mirrow store by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/589
• feat: add checkout method to table to reuse existing store and connections by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/593
• fix(python): fix multiple embedding functions bug by @changhiskhan in https://github.com/lancedb/lancedb/pull/597
• update lance to 0.8.7 by @albertlockett in https://github.com/lancedb/lancedb/pull/598

Lance updated from 0.8.4 -> 0.8.7

• v0.8.7 Vector index performance improvement, customize file size during writing lance dataset, bug fixes
• v0.8.6 Improve vector search performance when deletions are present
• v0.8.5: Incremental Index update, Index Compaction, and bug fixes
• v0.8.4: ANN training and querying performance improvements

New Contributors

• @rok made their first contribution in https://github.com/lancedb/lancedb/pull/554
• @albertlockett made their first contribution in https://github.com/lancedb/lancedb/pull/598

Full Changelog: https://github.com/lancedb/lancedb/compare/python-v0.3.1...python-v0.3.2

Thanks @prrao87 !

What's Changed

• doc: fix broken link and add README by @changhiskhan in https://github.com/lancedb/lancedb/pull/573
• Add pyarrow date and timestamp type conversion from pydantic by @prrao87 in https://github.com/lancedb/lancedb/pull/576
• list table pagination draft by @QianZhu in https://github.com/lancedb/lancedb/pull/574
• [Docs] Add posthog telemetry to docs by @AyushExel in https://github.com/lancedb/lancedb/pull/577
• [Python]Embeddings API refactor by @AyushExel in https://github.com/lancedb/lancedb/pull/580
• [Docs] Update embedding function docs by @AyushExel in https://github.com/lancedb/lancedb/pull/581
• chore: bump lance version in python/rust lancedb by @changhiskhan in https://github.com/lancedb/lancedb/pull/584

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.3.2...v0.3.3

What's Changed

• [DOCS][PYTHON] Update embeddings API docs & Example by @AyushExel in https://github.com/lancedb/lancedb/pull/516
• chore: bump lance to 0.8.5 by @eddyxu in https://github.com/lancedb/lancedb/pull/561
• docs: switch python examples to be row based by @rok in https://github.com/lancedb/lancedb/pull/554
• feat(python,js): deletion operation on remote tables by @eddyxu in https://github.com/lancedb/lancedb/pull/568
• docs: show source of documented functions by @rok in https://github.com/lancedb/lancedb/pull/569
• implement remote api calls for table mutation by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/567

New Contributors

• @rok made their first contribution in https://github.com/lancedb/lancedb/pull/554

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.3.1...v0.3.2

What's Changed

• implement mirroring object store by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/537
• fix stackoverflow by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/542
• Typo cleanup by @JoshWein in https://github.com/lancedb/lancedb/pull/539
• feat: use GPU for index creation. by @eddyxu in https://github.com/lancedb/lancedb/pull/540
• feat: telemetry, error tracking, CLI & config manager by @AyushExel in https://github.com/lancedb/lancedb/pull/538
• Use query.limit(..) in README by @ankrgyl in https://github.com/lancedb/lancedb/pull/543
• feat: add to_list and to_pandas api's by @changhiskhan in https://github.com/lancedb/lancedb/pull/556
• [Docs] Improve visibility of table ops by @AyushExel in https://github.com/lancedb/lancedb/pull/553
• feat: cleanup and compaction by @wjones127 in https://github.com/lancedb/lancedb/pull/518
• Add cohere embedding function by @AyushExel in https://github.com/lancedb/lancedb/pull/550

New Contributors

• @JoshWein made their first contribution in https://github.com/lancedb/lancedb/pull/539
• @ankrgyl made their first contribution in https://github.com/lancedb/lancedb/pull/543

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.3.0...v0.3.1

What's Changed

• implement mirroring object store by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/537
• fix stackoverflow by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/542
• Typo cleanup by @JoshWein in https://github.com/lancedb/lancedb/pull/539
• feat: use GPU for index creation. by @eddyxu in https://github.com/lancedb/lancedb/pull/540
• feat: telemetry, error tracking, CLI & config manager by @AyushExel in https://github.com/lancedb/lancedb/pull/538
• Use query.limit(..) in README by @ankrgyl in https://github.com/lancedb/lancedb/pull/543
• feat: add to_list and to_pandas api's by @changhiskhan in https://github.com/lancedb/lancedb/pull/556
• [Docs] Improve visibility of table ops by @AyushExel in https://github.com/lancedb/lancedb/pull/553
• feat: cleanup and compaction by @wjones127 in https://github.com/lancedb/lancedb/pull/518
• Add cohere embedding function by @AyushExel in https://github.com/lancedb/lancedb/pull/550

New Contributors

• @JoshWein made their first contribution in https://github.com/lancedb/lancedb/pull/539
• @ankrgyl made their first contribution in https://github.com/lancedb/lancedb/pull/543

Full Changelog: https://github.com/lancedb/lancedb/compare/python-v0.3.0...python-v0.3.1

What's Changed

• [doc] make the tensor width differnt from height by @litanlitudan in https://github.com/lancedb/lancedb/pull/533
• chore: upgrade lance to 0.8.1 by @eddyxu in https://github.com/lancedb/lancedb/pull/536

New Contributors

• @hynek made their first contribution in https://github.com/lancedb/lancedb/pull/508
• @prrao87 made their first contribution in https://github.com/lancedb/lancedb/pull/531
• @litanlitudan made their first contribution in https://github.com/lancedb/lancedb/pull/533

Lance core-format change

Bump lance format to 0.8.1 to have faster filtering, support pre-filtering, and fast vector index creation time

https://github.com/lancedb/lance/releases/tag/v0.8.0
https://github.com/lancedb/lance/releases/tag/v0.8.1

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.2.6...v0.3.0

What's Changed

Summar: including features in Lance 0.8.1

• [DOCS][PYTHON] Update docs for clarity by @prrao87 in https://github.com/lancedb/lancedb/pull/531
• [doc] make the tensor width differnt from height by @litanlitudan in https://github.com/lancedb/lancedb/pull/533
• chore: upgrade lance to 0.8.1 by @eddyxu in https://github.com/lancedb/lancedb/pull/536

New Contributors

• @prrao87 made their first contribution in https://github.com/lancedb/lancedb/pull/531
• @litanlitudan made their first contribution in https://github.com/lancedb/lancedb/pull/533

What are included in Lance 0.8.0

Fast training speed, initial support of pre-filtering, faster post-filtering

https://github.com/lancedb/lance/releases/tag/v0.8.1
https://github.com/lancedb/lance/releases/tag/v0.8.0

Full Changelog: https://github.com/lancedb/lancedb/compare/python-v0.2.6...python-v0.3.0

Up til now filters in LanceDB is applied on the search results. With this new release, we offer early support for applying the filters BEFORE running the search.

Currently limitations:

• Only supports KNN (an error will be raised if you're searching over a table with an ANN index)
• Only supports vector search (not an available option for full-text search yet)
• Opt-in, i.e., requires .where(<filter>, prefilter=True)

All of these will be expected to change in the future.

What's Changed

• [Python] Fix attrs dependency by @hynek in https://github.com/lancedb/lancedb/pull/508
• fix import attr to use import attrs by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/510
• fix: define minimum dependency versions by @wjones127 in https://github.com/lancedb/lancedb/pull/515
• feat(python): expose prefilter to lancedb by @changhiskhan in https://github.com/lancedb/lancedb/pull/522

New Contributors

• @hynek made their first contribution in https://github.com/lancedb/lancedb/pull/508

Full Changelog: https://github.com/lancedb/lancedb/compare/python-v0.2.5...python-v0.2.6

LanceDB python now comes with support for openclip to generate text-to-image embeddings.
You'll need to install torch, pillow, and open-clip packages, or pip install lancedb[clip].

We've updated to Lance v0.7.5 which contains some performance improvements.

Other QoL improvements:

• Added lancedb.__version__
• Improved backwards compat for pydantic 1.x
• Increased timeouts when reading data stored on S3

What's Changed

• fix: increase S3 timeouts by @wjones127 in https://github.com/lancedb/lancedb/pull/494
• multi-modal embedding-function by @changhiskhan in https://github.com/lancedb/lancedb/pull/484
• add health check to wait for all service ready before next step by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/501
• refactor connection string processing by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/500
• ci: fix docs build by @changhiskhan in https://github.com/lancedb/lancedb/pull/496
• feat: add lancedb.version by @changhiskhan in https://github.com/lancedb/lancedb/pull/504
• feat: improve pydantic 1.x compat by @changhiskhan in https://github.com/lancedb/lancedb/pull/503
• fix: upgrade lance to 0.7.5 and add tests for searching empty dataset by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/505

Full Changelog: https://github.com/lancedb/lancedb/compare/python-v0.2.4...python-v0.2.5

What's Changed

• chore: upgrade lance to 0.7.4 by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/491
• fix: increase S3 timeouts by @wjones127 in https://github.com/lancedb/lancedb/pull/494
• multi-modal embedding-function by @changhiskhan in https://github.com/lancedb/lancedb/pull/484
• add health check to wait for all service ready before next step by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/501
• refactor connection string processing by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/500
• ci: fix docs build by @changhiskhan in https://github.com/lancedb/lancedb/pull/496
• feat: add lancedb.version by @changhiskhan in https://github.com/lancedb/lancedb/pull/504
• feat: improve pydantic 1.x compat by @changhiskhan in https://github.com/lancedb/lancedb/pull/503
• fix: upgrade lance to 0.7.5 and add tests for searching empty dataset by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/505

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.2.5...v0.2.6

What's Changed

• chore: make crate more discoverable by @wjones127 in https://github.com/lancedb/lancedb/pull/443
• [python] Temporary update feature by @changhiskhan in https://github.com/lancedb/lancedb/pull/457
• [Docs] Small typo fixes by @AyushExel in https://github.com/lancedb/lancedb/pull/460
• [python] Use pydantic for embedding function persistence by @changhiskhan in https://github.com/lancedb/lancedb/pull/467
• Fix up docs by @changhiskhan in https://github.com/lancedb/lancedb/pull/477
• [Rust] schema coerce and vector column inference by @eddyxu in https://github.com/lancedb/lancedb/pull/476
• feat: upgrade lance to 0.7.3 by @eddyxu in https://github.com/lancedb/lancedb/pull/481
• [Python]Fix record_batch_generator by @AyushExel in https://github.com/lancedb/lancedb/pull/483
• [Python] Pydantic vector field with default value by @eddyxu in https://github.com/lancedb/lancedb/pull/474
• progagate uri query string to lance; add aws integration tests by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/486
• chore: upgrade lance to 0.7.4 by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/491

Full Changelog: https://github.com/lancedb/lancedb/compare/python-v0.2.2...python-v0.2.4

What's Changed

• [python] improve restore functionality by @changhiskhan in https://github.com/lancedb/lancedb/pull/451
• [python] Support schema evolution in local LanceDB by @changhiskhan in https://github.com/lancedb/lancedb/pull/452
• Revert "Bump version: 0.2.4 → 0.2.5" by @changhiskhan in https://github.com/lancedb/lancedb/pull/454
• Revert "Updating package-lock.json" by @changhiskhan in https://github.com/lancedb/lancedb/pull/455
• chore: make crate more discoverable by @wjones127 in https://github.com/lancedb/lancedb/pull/443
• [python] Temporary update feature by @changhiskhan in https://github.com/lancedb/lancedb/pull/457
• [Docs] Small typo fixes by @AyushExel in https://github.com/lancedb/lancedb/pull/460
• [python] Use pydantic for embedding function persistence by @changhiskhan in https://github.com/lancedb/lancedb/pull/467
• Fix up docs by @changhiskhan in https://github.com/lancedb/lancedb/pull/477
• [Rust] schema coerce and vector column inference by @eddyxu in https://github.com/lancedb/lancedb/pull/476
• feat: upgrade lance to 0.7.3 by @eddyxu in https://github.com/lancedb/lancedb/pull/481
• [Python]Fix record_batch_generator by @AyushExel in https://github.com/lancedb/lancedb/pull/483
• [Python] Pydantic vector field with default value by @eddyxu in https://github.com/lancedb/lancedb/pull/474
• progagate uri query string to lance; add aws integration tests by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/486

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.2.4...v0.2.5

You can now add new columns without having to rewrite the data.
This operation is reversible via LanceTable.restore

What's Changed

• [python] improve restore functionality by @changhiskhan in https://github.com/lancedb/lancedb/pull/451
• [python] Support schema evolution in local LanceDB by @changhiskhan in https://github.com/lancedb/lancedb/pull/452
• Revert "Bump version: 0.2.4 → 0.2.5" by @changhiskhan in https://github.com/lancedb/lancedb/pull/454
• Revert "Updating package-lock.json" by @changhiskhan in https://github.com/lancedb/lancedb/pull/455

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.2.4...python-v0.2.2

What's Changed

• [DOCS] Major Docs Revamp by @AyushExel in https://github.com/lancedb/lancedb/pull/435
• Clean up docs testing - exclude by glob instead of by file by @TevinWang in https://github.com/lancedb/lancedb/pull/450
• bump lance version to 0.6.5 for lancedb release by @QianZhu in https://github.com/lancedb/lancedb/pull/453

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.2.3...v0.2.4

What's Changed

• [python] Temporary restore feature by @changhiskhan in https://github.com/lancedb/lancedb/pull/428
• Make creating (and adding to) tables via Iterators more flexible & intuitive by @AyushExel in https://github.com/lancedb/lancedb/pull/430
• [DOCS] Update integrations + small typos by @AyushExel in https://github.com/lancedb/lancedb/pull/432
• [DOCS] Major Docs Revamp by @AyushExel in https://github.com/lancedb/lancedb/pull/435
• [Rust] Compute dot distance as negative dot product by @eddyxu in https://github.com/lancedb/lance/pull/1160
• Improve DX of Lance Write API

New Contributors

• @QianZhu made their first contribution in https://github.com/lancedb/lancedb/pull/453

Full Changelog: https://github.com/lancedb/lancedb/compare/python-v0.2.0...python-v0.2.1

What's Changed

• Make creating (and adding to) tables via Iterators more flexible & intuitive by @AyushExel in https://github.com/lancedb/lancedb/pull/430
• [DOCS] Update integrations + small typos by @AyushExel in https://github.com/lancedb/lancedb/pull/432
• use a lambda function to hide the value of credentials when printing a connection/table by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/438
• feat(node): Create empty tables / Arrow Tables by @gsilvestrin in https://github.com/lancedb/lancedb/pull/399
• expose awsRegion to be configurable by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/441

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.2.2...v0.2.3

What's Changed

• fix(node) Electron crashes when creating external buffer by @gsilvestrin in https://github.com/lancedb/lancedb/pull/424

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.2.1...v0.2.2

What's Changed

• Upgrade lance and pass AWS creds when opening a table by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/426
• fix moka version by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/427

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.2.0...v0.2.1

⚠️ Breaking Change: The score column has been renamed to _distance to more accurately describe the semantics (smaller means closer / better).

What's Changed

• fix(node): Handle overflows in the node bridge by @gsilvestrin in https://github.com/lancedb/lancedb/pull/372
• Implement drop table if exists by @changhiskhan in https://github.com/lancedb/lancedb/pull/383
• Improve pydantic integration by @changhiskhan in https://github.com/lancedb/lancedb/pull/384
• make pandas an optional dependency in lancedb as well by @changhiskhan in https://github.com/lancedb/lancedb/pull/385
• add LanceModel to docs by @changhiskhan in https://github.com/lancedb/lancedb/pull/386
• Restrict semver version to 3.0 by @eddyxu in https://github.com/lancedb/lancedb/pull/389
• feat(node): Improve concurrency by @gsilvestrin in https://github.com/lancedb/lancedb/pull/376
• fix(node) Give preference to local index.node lib by @gsilvestrin in https://github.com/lancedb/lancedb/pull/393
• [WIP] Workflow to trigger vectordb-recipes workflow by @unkn-wn in https://github.com/lancedb/lancedb/pull/371
• fix(node) Unit tests hangs and don't exit by @gsilvestrin in https://github.com/lancedb/lancedb/pull/396
• [Docs] add Tables guide by @AyushExel in https://github.com/lancedb/lancedb/pull/381
• [Docs] Allow edit suggestions and analytics by @AyushExel in https://github.com/lancedb/lancedb/pull/394
• [Documentation Code Testing] temp fix for nodejs docs test hang by @TevinWang in https://github.com/lancedb/lancedb/pull/404
• fix(docs) fix minor typo by @westonpace in https://github.com/lancedb/lancedb/pull/408
• implement remote drop table call by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/411
• fix(node) Remove mpsc from JS SDK by @gsilvestrin in https://github.com/lancedb/lancedb/pull/407
  /pull/413
• feat(node) Remote drop table by @gsilvestrin in https://github.com/lancedb/lancedb/pull/412
• [Node] Use index by default by @eddyxu in https://github.com/lancedb/lancedb/pull/422
• [breaking change] make schema a property by @chebbyChefNEQ in https://github.com/lancedb/lancedb/pull/414
• implementation of drop_database by @ashiskumarnaik in https://github.com/lancedb/lancedb/pull/418
• chore: upgrade Lance and rename score to _distance by @wjones127 in https://github.com/lancedb/lancedb/pull/398

New Contributors

• @westonpace made their first contribution in https://github.com/lancedb/lancedb/pull/408
• @ashiskumarnaik made their first contribution in https://github.com/lancedb/lancedb/pull/418

Full Changelog: https://github.com/lancedb/lancedb/compare/v0.1.19...v0.2.0