lance

Bug fix for reading variable length list arrays (welcome @gsilvestrin).

We're working on windows support (welcome to @dnsco) and OPQ implementation for vector index, so stay tuned!

What's Changed

• Windows support by @dnsco in https://github.com/eto-ai/lance/pull/651
• Trigger CI when workflow changes by @changhiskhan in https://github.com/eto-ai/lance/pull/653
• Compute SVD by @eddyxu in https://github.com/eto-ai/lance/pull/658
• Fix offsets when reading arrays by @gsilvestrin in https://github.com/eto-ai/lance/pull/657

New Contributors

• @dnsco made their first contribution in https://github.com/eto-ai/lance/pull/651
• @gsilvestrin made their first contribution in https://github.com/eto-ai/lance/pull/657

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.10...v0.3.11

You can now choose to bypass the ANN index even if it was available and perform vector search using brute-force. This helps with debugging ANN results. Note that SIMD is still applicable during brute-force search.

What's Changed

• [Bug] Fix passing metric type during PQ index building by @eddyxu in https://github.com/eto-ai/lance/pull/644
• [python] Allow user to bypass ANN index and search using brute-force … by @changhiskhan in https://github.com/eto-ai/lance/pull/645
• expand tilde paths in python by @ananis25 in https://github.com/eto-ai/lance/pull/621
• Fix binary encoder handling array buffer slicing by @eddyxu in https://github.com/eto-ai/lance/pull/649

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.9...v0.3.10

By default pyarrow compute Expressions doesn't serialize to sql strings. This patch release enables a limited set of filter pushdowns via python. Supported syntax:

1. field references
2. Operators: > < >= <= = == !=
3. conjunctions / disjunctions

This enables querying via duckdb without needing to load the whole dataset into memory first.

e.g., duckdb.query("SELECT * FROM dataset WHERE id=5")

What's Changed

• [Rust] Handle double equals in filter by @eddyxu in https://github.com/eto-ai/lance/pull/639

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.8...v0.3.9

You can now query lance datasets outside of python using duckdb! Thanks to @dacort for making the lance extension play nice with duckdb. dbt-duckdb-lance anyone? You can find the extension under integration/duckdb_lance.

We're also very excited to release a very substantial performance optimization for random access for non-numeric columns.
Previously, if you wanted to fetch a string or blob column along with nearest neighbor search results, the non-optimized binary decoder take could add up to 5-20x latency overhead, depending on the sparsity of the indices. In this release we've optimized the take performance so this is basically a free operation.

While most of the work in Rust is completed for filter pushdown, we've had to delay the general release for this feature until we're able to overcome some rough edges making pyarrow compute Expressions play nice with datafusion and sqlparser-rs. It'll be worth the wait though we promise!

Cosine similarity is shipped but the recall performance is lower, due to some issues during index creation. We recommend that you stick with the default L2 distance metric until we address this in the coming few releases.

We'd love to hear from you!

What's Changed

• Update extension for v0.7.0 compatibility by @dacort in https://github.com/eto-ai/lance/pull/599
• Remove -j from DuckDB build script by @changhiskhan in https://github.com/eto-ai/lance/pull/601
• a minor preparatory refactor by @changhiskhan in https://github.com/eto-ai/lance/pull/598
• fix gha duckdb trigger paths by @changhiskhan in https://github.com/eto-ai/lance/pull/602
• Use MetricType to specify the metric / distance compute function by @eddyxu in https://github.com/eto-ai/lance/pull/600
• [Python] Specify metric type in Dataset.create_index by @eddyxu in https://github.com/eto-ai/lance/pull/603
• [Rust] Implement a datafusion phyiscal expr Column that can reads nested columns by @eddyxu in https://github.com/eto-ai/lance/pull/610
• benchmark query performance on 768D vectors by @changhiskhan in https://github.com/eto-ai/lance/pull/607
• Parse sql filter clause to create datafusion physical expression by @eddyxu in https://github.com/eto-ai/lance/pull/609
• Schema exclude fields by @eddyxu in https://github.com/eto-ai/lance/pull/613
• Exec filter during Scan by @eddyxu in https://github.com/eto-ai/lance/pull/612
• workaround to prevent the segfault until we figure out the real problem by @changhiskhan in https://github.com/eto-ai/lance/pull/616
• Improve random access on binary encoding by @eddyxu in https://github.com/eto-ai/lance/pull/615
• [Python] Support filter pushdown from Python Dataset API by @eddyxu in https://github.com/eto-ai/lance/pull/618
• refactor benchmark to use cosine similarity by @changhiskhan in https://github.com/eto-ai/lance/pull/611
• Encoding shared slices of arrays. by @eddyxu in https://github.com/eto-ai/lance/pull/620
• Fix plain encoding by @eddyxu in https://github.com/eto-ai/lance/pull/622
• Fix crash with column projection with ann search by @eddyxu in https://github.com/eto-ai/lance/pull/624
• Relax data type matching float numbers in filter pushdown by @eddyxu in https://github.com/eto-ai/lance/pull/625
• python integration tests for vector index by @changhiskhan in https://github.com/eto-ai/lance/pull/623
• Remove filter pushdown from python api for now by @changhiskhan in https://github.com/eto-ai/lance/pull/628
• PlainDecoder take on boolean values by @eddyxu in https://github.com/eto-ai/lance/pull/627
• remove debug prints by @changhiskhan in https://github.com/eto-ai/lance/pull/633
• Scan node to detect channel close and gracefully break the scan. by @eddyxu in https://github.com/eto-ai/lance/pull/635

New Contributors

• @dacort made their first contribution in https://github.com/eto-ai/lance/pull/599

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.7...v0.3.8

Thanks @ananis25 for implementing Lance support for Duration and Null arrow arrays!

We've also completed the core implementation of cosine distance (with SIMD) and refactored the distance functions to be pluggable. Next release will expose this as a public API in Rust and Python

What's Changed

• support arrow-rs duration type in lance by @ananis25 in https://github.com/eto-ai/lance/pull/590
• Don't inherit the index_section if mode is Overwrite by @changhiskhan in https://github.com/eto-ai/lance/pull/592
• [Rust] Cosine Distance by @eddyxu in https://github.com/eto-ai/lance/pull/595
• Refactor L2 distance into a separate mod by @eddyxu in https://github.com/eto-ai/lance/pull/596
• support arrow-rs null array in lance by @ananis25 in https://github.com/eto-ai/lance/pull/594
• refactor Scanner::try_into_stream by @changhiskhan in https://github.com/eto-ai/lance/pull/597

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.6...v0.3.7

Welcome to @ananis25 and @yah01 !

This release enables time travel capability allowing you to check out the latest version as of a certain date and time.
We've refactored the query and index creation code to make room for multiple distance metrics.

What's Changed

• Refactor distance computation to allow pluggable dist function by @eddyxu in https://github.com/eto-ai/lance/pull/589
• Add an argument to checkout a dataset as of a certain timestamp by @ananis25 in https://github.com/eto-ai/lance/pull/585

New Contributors

• @ananis25 made their first contribution in https://github.com/eto-ai/lance/pull/585

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.5...v0.3.6

What's Changed

• Add take() to retrieve rows by indices by @yah01 in https://github.com/eto-ai/lance/pull/562
• add crateio badge by @changhiskhan in https://github.com/eto-ai/lance/pull/580
• quick update for main Rust readme by @changhiskhan in https://github.com/eto-ai/lance/pull/583
• Fix version timestamp issue by @changhiskhan in https://github.com/eto-ai/lance/pull/582
• Decimal128 and Decimal256 support by @changhiskhan in https://github.com/eto-ai/lance/pull/584

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.4...v0.3.5

This is a minor release with bug fixes, documentation and ergonomic improvements for vectors indices.
Welcome our newest contributor @yah01

What's Changed

• Support converting Boolean Array in DuckDB by @eddyxu in https://github.com/eto-ai/lance/pull/554
• [DuckDB] Fix scan column projection in duckdb extension by @eddyxu in https://github.com/eto-ai/lance/pull/555
• [DuckDB] add a missing boolean array check by @eddyxu in https://github.com/eto-ai/lance/pull/556
• Use Datafusion physical ExecutionPlan node as I/O exec node by @eddyxu in https://github.com/eto-ai/lance/pull/558
• Fix missing duckdb symbol on Linux by @eddyxu in https://github.com/eto-ai/lance/pull/560
• Improve take_rows() performance by binary search by @yah01 in https://github.com/eto-ai/lance/pull/564
• Use the released crate for DataFusion by @eddyxu in https://github.com/eto-ai/lance/pull/570
• Update duckdb version. Closes #569 by @changhiskhan in https://github.com/eto-ai/lance/pull/571
• [python] Add version to python package. Closes #561 by @changhiskhan in https://github.com/eto-ai/lance/pull/572
• utility to convert to pyarrow table with vectors by @changhiskhan in https://github.com/eto-ai/lance/pull/574
• Clippy fixes by @eddyxu in https://github.com/eto-ai/lance/pull/579
• Fix IVF_PQ index creation crash if one of the cluster has no value. by @eddyxu in https://github.com/eto-ai/lance/pull/575
• Enforce SIMD alignment during index creation by @changhiskhan in https://github.com/eto-ai/lance/pull/578

New Contributors

• @yah01 made their first contribution in https://github.com/eto-ai/lance/pull/564

Preview

1. We're most of the way there for a DuckDB extension to read Lance datasets natively (i.e., without python)
2. We're integrating datafusion to enable pushdown of filter predicates

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.2...v0.3.4

We discovered two thing making index creation unnecessarily long:

1. Instead of using KMeans++ to initialize, just do random initialization
2. Turn of BLAS on macOS because it turned out to be super slow

On macbook air, index creation goes from 25min on sift1m to 24s.
On Ubuntu, it's roughly a 6x speedup.

What's Changed

• Remove duckdb extension based on c++ codebase. by @eddyxu in https://github.com/eto-ai/lance/pull/530
• Reader honor batch size by @eddyxu in https://github.com/eto-ai/lance/pull/532
• Rust based duckdb extension by @eddyxu in https://github.com/eto-ai/lance/pull/529
• pretty-print: add pretty print by @AsadullahFarooqi in https://github.com/eto-ai/lance/pull/469
• Add unit tests to verify the read of struct by @eddyxu in https://github.com/eto-ai/lance/pull/537
• [DuckDB] Support Struct, and a separate crate for re-usable components to build duckdb extensions by @eddyxu in https://github.com/eto-ai/lance/pull/534
• [Rust] Fix main branch CI failure and bump arrow version by @eddyxu in https://github.com/eto-ai/lance/pull/538
• metrics for vector index by @changhiskhan in https://github.com/eto-ai/lance/pull/535
• [DuckDB] Add list support. by @eddyxu in https://github.com/eto-ai/lance/pull/540
• Update README for vector index copy by @jaichopra in https://github.com/eto-ai/lance/pull/542
• Support writing pandas dataframes directly by @changhiskhan in https://github.com/eto-ai/lance/pull/543
• Improve IVF_PQ index creation performance by @eddyxu in https://github.com/eto-ai/lance/pull/545
• update metrics by @changhiskhan in https://github.com/eto-ai/lance/pull/547
• Fix duckdb CI failure by @eddyxu in https://github.com/eto-ai/lance/pull/546
• Remove Mac accelerate code by @eddyxu in https://github.com/eto-ai/lance/pull/548
• update docs by @changhiskhan in https://github.com/eto-ai/lance/pull/549
• Set intel mac release version to 10.15 by @eddyxu in https://github.com/eto-ai/lance/pull/551
• Test cli in Github Action by @eddyxu in https://github.com/eto-ai/lance/pull/550
• Changhiskhan/nb docs by @changhiskhan in https://github.com/eto-ai/lance/pull/552
• minor fix for GHA and github by @changhiskhan in https://github.com/eto-ai/lance/pull/553

New Contributors

• @AsadullahFarooqi made their first contribution in https://github.com/eto-ai/lance/pull/469

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.1...v0.3.2

We added an index creation tool that's 2x faster than FAISS.
Accessible in python via Dataset.create_index

What's Changed

• Add unit test for Dataset::take_rows by @eddyxu in https://github.com/eto-ai/lance/pull/523
• Create index API from Python by @eddyxu in https://github.com/eto-ai/lance/pull/524
• Implement a kmean optimized for Arrow-backed vectors in pure rust. by @eddyxu in https://github.com/eto-ai/lance/pull/525
• Reimplement IVF_PQ index. by @eddyxu in https://github.com/eto-ai/lance/pull/519
• Use pure rust kmean in IVF and PQ by @eddyxu in https://github.com/eto-ai/lance/pull/526
• update release actions by @changhiskhan in https://github.com/eto-ai/lance/pull/527
• [rust] make scanning in order configurable by @changhiskhan in https://github.com/eto-ai/lance/pull/528

Full Changelog: https://github.com/eto-ai/lance/compare/v0.3.0...v0.3.1

Sayonara C++, bonjour Rust

What started out as a holiday hack has become a full-blown Rust rewrite.
As we say farewell to our much beloved C++ implementation, we welcome a major new feature to Lance: the vector index.

1. Lance's vector index is fast and has a small memory footprint. From disk, we benchmark average latencies of 1ms on vanilla macbook airs for 1M vectors.
2. Your data, vectors, and index can live in harmony under one roof so you don't need to manage a separate index or service.
3. You can choose to manage and retrieve additional features with the vectors with very little performance impact.

What's Changed

• Only increase cursor if file success to write by @eddyxu in https://github.com/eto-ai/lance/pull/435
• GHA to add python 3.11 (and upgrade to duckdb 0.6.1) by @changhiskhan in https://github.com/eto-ai/lance/pull/434
• ScannerStream accepts early stop by @eddyxu in https://github.com/eto-ai/lance/pull/437
• upgrade arrow-rs to 31.0 by @eddyxu in https://github.com/eto-ai/lance/pull/438
• L2 distance by @eddyxu in https://github.com/eto-ai/lance/pull/439
• Create DataFragment and DataFile during Dataset write process by @eddyxu in https://github.com/eto-ai/lance/pull/440
• Rust Dataset Write API by @eddyxu in https://github.com/eto-ai/lance/pull/441
• [Rust] Read Partially from a plain encoded batch by @eddyxu in https://github.com/eto-ai/lance/pull/443
• Get range in var-binary encoding by @eddyxu in https://github.com/eto-ai/lance/pull/444
• Productionlize Flat Index by @eddyxu in https://github.com/eto-ai/lance/pull/442
• Make Scan an ExecNode by @eddyxu in https://github.com/eto-ai/lance/pull/445
• Take record by Row ID by @eddyxu in https://github.com/eto-ai/lance/pull/446
• Implement Take for dictionary decoder. by @eddyxu in https://github.com/eto-ai/lance/pull/447
• Merge two RecordBatch by @eddyxu in https://github.com/eto-ai/lance/pull/449
• Integrate flat index by @eddyxu in https://github.com/eto-ai/lance/pull/448
• Support limit offset as ExecNode by @changhiskhan in https://github.com/eto-ai/lance/pull/450
• Read IVF_PQ index by @eddyxu in https://github.com/eto-ai/lance/pull/451
• Cli to operate on dataset indices by @eddyxu in https://github.com/eto-ai/lance/pull/452
• [RUST] python (re)integration v1 by @changhiskhan in https://github.com/eto-ai/lance/pull/436
• Support writing dictionary values (at the dataset level). by @eddyxu in https://github.com/eto-ai/lance/pull/454
• Replace ObjectReader as a pub trait. by @eddyxu in https://github.com/eto-ai/lance/pull/459
• [Rust] Implement LocalObjectReader that holds an open file to improve performance. by @eddyxu in https://github.com/eto-ai/lance/pull/460
• inherit from pyarrow Dataset/Scanner by @changhiskhan in https://github.com/eto-ai/lance/pull/462
• [RUST] Flat index benchmark by @eddyxu in https://github.com/eto-ai/lance/pull/461
• Generate spotify dataset with embeddings. by @eddyxu in https://github.com/eto-ai/lance/pull/453
• Fix pylance typo and float32 array conversion. by @eddyxu in https://github.com/eto-ai/lance/pull/463
• Write index metadata with a new version by @eddyxu in https://github.com/eto-ai/lance/pull/466
• [rust] fix projection in Dataset:take_rows by @changhiskhan in https://github.com/eto-ai/lance/pull/464
• blas feature flag by @changhiskhan in https://github.com/eto-ai/lance/pull/467
• Sift dataset generation by @eddyxu in https://github.com/eto-ai/lance/pull/472
• Improve scan perf by re-enable prefetching in ScanNode by @eddyxu in https://github.com/eto-ai/lance/pull/473
• Changhiskhan/new docs by @changhiskhan in https://github.com/eto-ai/lance/pull/474
• Fix AVX and NEON L2 distance computation. by @eddyxu in https://github.com/eto-ai/lance/pull/476
• add recall metric computation by @changhiskhan in https://github.com/eto-ai/lance/pull/475
• Fix reader assertion on manifest buffer size by @eddyxu in https://github.com/eto-ai/lance/pull/478
• [Rust] Minimal dataset append support by @eddyxu in https://github.com/eto-ai/lance/pull/482
• Pass nprobes parameter from python by @changhiskhan in https://github.com/eto-ai/lance/pull/480
• add a test_dataset function to compute the recall for lance by @changhiskhan in https://github.com/eto-ai/lance/pull/479
• Split sparse index read into chunks based on optimal I/O size for the media by @eddyxu in https://github.com/eto-ai/lance/pull/483
• Fix codespace prebuild by @eddyxu in https://github.com/eto-ai/lance/pull/485
• Make ObjectReader prefetch size configurable by @eddyxu in https://github.com/eto-ai/lance/pull/486
• Add a refine stage for vector search by @eddyxu in https://github.com/eto-ai/lance/pull/488
• add nprobes as parameter to benchmark by @changhiskhan in https://github.com/eto-ai/lance/pull/484
• refine factor by @changhiskhan in https://github.com/eto-ai/lance/pull/489
• Use ordered buffer in plain decoder by @eddyxu in https://github.com/eto-ai/lance/pull/493
• New rust+pyo3 based pylance by @eddyxu in https://github.com/eto-ai/lance/pull/494
• Fast count rows by @eddyxu in https://github.com/eto-ai/lance/pull/490
• Count rows in python dataset, and setup GHA again by @eddyxu in https://github.com/eto-ai/lance/pull/495
• Sayonara C++ by @eddyxu in https://github.com/eto-ai/lance/pull/497
• [Rust] Dataset Overwrite, and Version Checkout by @eddyxu in https://github.com/eto-ai/lance/pull/496
• Load S3 credentials using default credentials chain by @eddyxu in https://github.com/eto-ai/lance/pull/498
• Fix doc build by @eddyxu in https://github.com/eto-ai/lance/pull/499
• File format spec by @eddyxu in https://github.com/eto-ai/lance/pull/500
• Doc build fix by @eddyxu in https://github.com/eto-ai/lance/pull/501
• Schema evolution document by @eddyxu in https://github.com/eto-ai/lance/pull/503
• update the python readme for pypi by @changhiskhan in https://github.com/eto-ai/lance/pull/504
• Handle null strings for both cases where nullability is set or not. by @eddyxu in https://github.com/eto-ai/lance/pull/509
• update main github readme by @changhiskhan in https://github.com/eto-ai/lance/pull/508
• [python] write_dataset returns new dataset by @changhiskhan in https://github.com/eto-ai/lance/pull/517
• Changhiskhan/list versions by @changhiskhan in https://github.com/eto-ai/lance/pull/516
• Refine Factor is None by default by @eddyxu in https://github.com/eto-ai/lance/pull/518

Full Changelog: https://github.com/eto-ai/lance/compare/v0.2.9...v0.3.0

And also, we've started to implement Lance is Rust. A new kickass vector indexing feature will be coming soon once we do some more cleanup and hook the Rust module back into python.

What's Changed

• [DuckDB] Add macro to check window size by @eddyxu in https://github.com/eto-ai/lance/pull/395
• [pandas] Add pandas extension type for ImageBinary by @changhiskhan in https://github.com/eto-ai/lance/pull/398
• python 3.11 is updating and causing error by @changhiskhan in https://github.com/eto-ai/lance/pull/397
• [RUST] Initialize read support in Rust. by @eddyxu in https://github.com/eto-ai/lance/pull/401
• Add missing logical type conversions by @eddyxu in https://github.com/eto-ai/lance/pull/404
• [RUST] Schema projection by @eddyxu in https://github.com/eto-ai/lance/pull/403
• [RUST] Data file reader by @eddyxu in https://github.com/eto-ai/lance/pull/402
• [Rust] Decoder for dictionary encoding by @eddyxu in https://github.com/eto-ai/lance/pull/406
• [Rust] Support full scan for BooleanArray by @changhiskhan in https://github.com/eto-ai/lance/pull/407
• [Rust] Basic reading support for nested fields. by @eddyxu in https://github.com/eto-ai/lance/pull/408
• Add unit tests for all supported primitive types by @changhiskhan in https://github.com/eto-ai/lance/pull/409
• [RUST] Binary encoder and null support. by @eddyxu in https://github.com/eto-ai/lance/pull/411
• [Rust] Fix Cargo publish by @eddyxu in https://github.com/eto-ai/lance/pull/410
• [RUST] Large binary support by @eddyxu in https://github.com/eto-ai/lance/pull/412
• Add support for fixed size list by @changhiskhan in https://github.com/eto-ai/lance/pull/413
• Jaichopra/nuscenes converter by @jaichopra in https://github.com/eto-ai/lance/pull/364
• Add Support for Fixed Size Binary Full scan by @changhiskhan in https://github.com/eto-ai/lance/pull/414
• Bare minimal scanner in Rust by @eddyxu in https://github.com/eto-ai/lance/pull/415
• Set field IDs. by @eddyxu in https://github.com/eto-ai/lance/pull/417
• [Rust] Read/Write Protobuf-backed struct directly from file or buffers. by @eddyxu in https://github.com/eto-ai/lance/pull/418
• [Rust] Lance File Writer by @eddyxu in https://github.com/eto-ai/lance/pull/419
• [Rust] Write dictionary data by @eddyxu in https://github.com/eto-ai/lance/pull/420
• [RUST] Write List/LargeList/FixedSizeList/FixedSizeBinary by @eddyxu in https://github.com/eto-ai/lance/pull/421
• fix byte range and iterator bug by @changhiskhan in https://github.com/eto-ai/lance/pull/422
• Fix dict order in logical type to be consistent with C++ by @eddyxu in https://github.com/eto-ai/lance/pull/425
• Limits notebook GHA to only run when C++ / Python changes. by @eddyxu in https://github.com/eto-ai/lance/pull/427
• Implement futures::Stream for Scanner by @eddyxu in https://github.com/eto-ai/lance/pull/426
• Append column to RecordBatch by @eddyxu in https://github.com/eto-ai/lance/pull/429
• [Rust] Read batch with rowid as a meta column. by @eddyxu in https://github.com/eto-ai/lance/pull/430
• [RUST] argmin and argmax kernel for numeric array by @eddyxu in https://github.com/eto-ai/lance/pull/432

Full Changelog: https://github.com/eto-ai/lance/compare/v0.2.8...v0.2.9

This release contains the following:

1. A full-fledged ML data quality improvement workflow using Lance showing model performance insights, detecting mislabels, and doing active learning. An experimental integration with Label Studio is demonstrated as well.
2. Critical bug fix affected read/write of dictionary columns
3. Imagenet dataset converter

What's Changed

• [BUG] Fix reading version aux data reading and writing by @eddyxu in https://github.com/eto-ai/lance/pull/384
• [Benchmark] upload scripts for coco / imagenet benchmark dataset by @eddyxu in https://github.com/eto-ai/lance/pull/385
• Closes #387 by @changhiskhan in https://github.com/eto-ai/lance/pull/388
• Data quality notebook and associated code by @changhiskhan in https://github.com/eto-ai/lance/pull/389
• [DUCKDB] Do not build PyTorch by default by @eddyxu in https://github.com/eto-ai/lance/pull/392
• brew pin python by @changhiskhan in https://github.com/eto-ai/lance/pull/391
• fix off by one error using negative indices for diff'ing by @changhiskhan in https://github.com/eto-ai/lance/pull/383
• Fix GHA for duckdb extension by @changhiskhan in https://github.com/eto-ai/lance/pull/394
• [DUCKDB] Add a Derivative macro by @eddyxu in https://github.com/eto-ai/lance/pull/393
• [Benchmark] Create imagenet from raw dataset by @eddyxu in https://github.com/eto-ai/lance/pull/386
• Various fixes for imagenet and fmt changes by @changhiskhan in https://github.com/eto-ai/lance/pull/396

Full Changelog: https://github.com/eto-ai/lance/compare/v0.2.7...v0.2.8

What's Changed

• create and update tarball for pets by @changhiskhan in https://github.com/eto-ai/lance/pull/372
• [C++] Sanity check to verify column does not overlap when merging a new table by @eddyxu in https://github.com/eto-ai/lance/pull/375
• update notebooks so s3 credentials are not required by @changhiskhan in https://github.com/eto-ai/lance/pull/376
• Add function to get version as of a certain date. Also formatting by @changhiskhan in https://github.com/eto-ai/lance/pull/378
• convenience for comparing metrics across versions by @changhiskhan in https://github.com/eto-ai/lance/pull/379
• Changhiskhan/datadiff by @changhiskhan in https://github.com/eto-ai/lance/pull/380
• Refactor dataset diff and compute metric by @changhiskhan in https://github.com/eto-ai/lance/pull/381
• [C++] Attach new schema update when update dataset by @eddyxu in https://github.com/eto-ai/lance/pull/374

Full Changelog: https://github.com/eto-ai/lance/compare/v0.2.6...v0.2.7

What's Changed

• [C++] Remove unused Reader APIs by @eddyxu in https://github.com/eto-ai/lance/pull/344
• [Python] fix timezone issue with version timestamp by @changhiskhan in https://github.com/eto-ai/lance/pull/345
• [C++] add Dataset::Make(string) API by @eddyxu in https://github.com/eto-ai/lance/pull/346
• [DUCKDB] Native duckdb lance reader by @eddyxu in https://github.com/eto-ai/lance/pull/347
• [DUCKDB] Read a special version of dataset by @eddyxu in https://github.com/eto-ai/lance/pull/350
• [DUCKDB] Fix duckdb manylinux build by @eddyxu in https://github.com/eto-ai/lance/pull/351
• [Python] Add colab badge to notebooks by @eddyxu in https://github.com/eto-ai/lance/pull/354
• [Notebook] ML dev cycle for DINO by @eddyxu in https://github.com/eto-ai/lance/pull/355
• [DUCKDB] fix type mapping for other int types by @changhiskhan in https://github.com/eto-ai/lance/pull/359
• [Python] Fix lance.dataset open local related path by @eddyxu in https://github.com/eto-ai/lance/pull/365
• [C++] Store relative path for data files by @eddyxu in https://github.com/eto-ai/lance/pull/368
• [C++] Add RAII util (defer) to auto cleanup / close resources after exiting the scope by @eddyxu in https://github.com/eto-ai/lance/pull/369
• [Python] Convert of ImageNet 1K into Lance dataset by @eddyxu in https://github.com/eto-ai/lance/pull/366
• [Python] Imagenet data quality analytics notebook by @eddyxu in https://github.com/eto-ai/lance/pull/370

Full Changelog: https://github.com/eto-ai/lance/compare/v0.2.5...v0.2.6

What's Changed

• [DOC] Fix notebook build by @eddyxu in https://github.com/eto-ai/lance/pull/339
• [Python] lance.write_dataset takes pandas DataFrame by @eddyxu in https://github.com/eto-ai/lance/pull/342
• [DOC] update readme docs to cater for import pathways from df/parquet by @jaichopra in https://github.com/eto-ai/lance/pull/340
• [Python] Improve PyTorch dataset ergonomic by @eddyxu in https://github.com/eto-ai/lance/pull/336
• [C++] Add columns from in-memory table by @eddyxu in https://github.com/eto-ai/lance/pull/337
• [Python] append column with a in-memory Pyarrow Table by @eddyxu in https://github.com/eto-ai/lance/pull/338
• [C++][Python] Add timestamp to each manifest version. by @eddyxu in https://github.com/eto-ai/lance/pull/343

Full Changelog: https://github.com/eto-ai/lance/compare/v0.2.4...v0.2.5

Support Schema Evolution via Append Column.

What's Changed

• [Notebook] fixes for notebook backing the blog post by @changhiskhan in https://github.com/eto-ai/lance/pull/316
• [C++] Append column by @eddyxu in https://github.com/eto-ai/lance/pull/299
• [Python] Append columns by @eddyxu in https://github.com/eto-ai/lance/pull/318
• [Use column projection during update by @eddyxu in https://github.com/eto-ai/lance/pull/322
• update to duckdb 0.6 by @changhiskhan in https://github.com/eto-ai/lance/pull/312
• [Python] Support add column via Expression. by @eddyxu in https://github.com/eto-ai/lance/pull/324
• [Python] Expose projection for append column by @eddyxu in https://github.com/eto-ai/lance/pull/325
• [C++] Support column projection during add_columns via expression by @eddyxu in https://github.com/eto-ai/lance/pull/326
• [Python] Pytorch Dataset uses Fragment instead of files and support versions by @eddyxu in https://github.com/eto-ai/lance/pull/327
• [C++] Move writer API a private API by @eddyxu in https://github.com/eto-ai/lance/pull/329
• [C++] Refectory Metadata class to eliminate protobuf reference. by @eddyxu in https://github.com/eto-ai/lance/pull/328
• [C++] Performance profiling and improvement by @eddyxu in https://github.com/eto-ai/lance/pull/333
• [C++] Upgrade lq cmd tool to be able to inspect new versioned format by @eddyxu in https://github.com/eto-ai/lance/pull/334

Full Changelog: https://github.com/eto-ai/lance/compare/v0.2.3...v0.2.4

What's Changed

• [C++] Project schema via field Ids and Schema intersection by @eddyxu in https://github.com/eto-ai/lance/pull/305
• when writing in batches, handle all na arrays properly by @changhiskhan in https://github.com/eto-ai/lance/pull/306
• [C++] Use LanceFragment to build I/O exec plan by @eddyxu in https://github.com/eto-ai/lance/pull/307
• [CI] Fix Github Action warning to upgrade nodejs 12 based actions by @eddyxu in https://github.com/eto-ai/lance/pull/309
• Update README.md by @changhiskhan in https://github.com/eto-ai/lance/pull/310
• Temporarily pin duckdb to 0.5.1 by @changhiskhan in https://github.com/eto-ai/lance/pull/313
• Notebook for new blog post on versioning by @changhiskhan in https://github.com/eto-ai/lance/pull/311
• [C++] Fix reading dictionary values from manifest files by @eddyxu in https://github.com/eto-ai/lance/pull/314

Full Changelog: https://github.com/eto-ai/lance/compare/v0.2.2...v0.2.3

What's Changed

• [DOC] Update README.md by @jaichopra in https://github.com/eto-ai/lance/pull/294
• [DUCKDB] Script to upload lance extension zip by @changhiskhan in https://github.com/eto-ai/lance/pull/295
• [C++] Scan Node reads multiple files by @eddyxu in https://github.com/eto-ai/lance/pull/300
• [Python] Add lance.util.duckdb to help install the extension transparently by @changhiskhan in https://github.com/eto-ai/lance/pull/301
• [Python] Notebook fixes by @changhiskhan in https://github.com/eto-ai/lance/pull/303
• [Python] Make dataset conversion a feature by @changhiskhan in https://github.com/eto-ai/lance/pull/304

Full Changelog: https://github.com/eto-ai/lance/compare/v0.2.1...v0.2.2

Fixed bug affecting writes of fixed size list arrays as well as datagen code for Coco.
Updated to Arrow 10.0 newly released.

What's Changed

• remove duplicate test_mac.sh by @changhiskhan in https://github.com/eto-ai/lance/pull/284
• Fix build on intel mac by @eddyxu in https://github.com/eto-ai/lance/pull/286
• [C++] Fix write fixed list array bug by @eddyxu in https://github.com/eto-ai/lance/pull/288
• Upgrade Apache Arrow to 10.0 by @eddyxu in https://github.com/eto-ai/lance/pull/266
• temporary hack to fix pytorch loader until it can handle a versioned … by @changhiskhan in https://github.com/eto-ai/lance/pull/293
• fix image_id alignment in coco datagen by @changhiskhan in https://github.com/eto-ai/lance/pull/289

Full Changelog: https://github.com/eto-ai/lance/compare/v0.2.0...v0.2.1