Highlights

• Lance Dataset versioning support
• Duckdb Extension supports building against PyTorch with Cuda
• Revamp README and documentation.

What's Changed

• Fetch Dataset Versions by @eddyxu in https://github.com/eto-ai/lance/pull/272
• Readability improvement for metadata class by @Renkai in https://github.com/eto-ai/lance/pull/275
• [DuckDB] Enable DuckDB extension to build/run on CUDA-enabled PyTorch by @changhiskhan in https://github.com/eto-ai/lance/pull/273
• [Python] Support multi-versioned dataset by @eddyxu in https://github.com/eto-ai/lance/pull/278
• [Document] Add logo/README refresh by @jaichopra in https://github.com/eto-ai/lance/pull/279
• [Python] Fetch dataset versions. by @eddyxu in https://github.com/eto-ai/lance/pull/280
• [Python] Support group size and rows_per_file customization via write_dataset API by @eddyxu in https://github.com/eto-ai/lance/pull/281
• [Python] use new write API in python benchmark by @eddyxu in https://github.com/eto-ai/lance/pull/282

Full Changelog: https://github.com/eto-ai/lance/compare/v0.1.5...v0.2.0

What's Changed

• Add model inference notebook by @changhiskhan in https://github.com/eto-ai/lance/pull/244
• update README.md to simplify comms, prior to blog post by @jaichopra in https://github.com/eto-ai/lance/pull/248
• Jaichopra/rebrand lance by @jaichopra in https://github.com/eto-ai/lance/pull/249
• Exclude jupyter notebook from github language stats by @changhiskhan in https://github.com/eto-ai/lance/pull/251
• linguist fix by @changhiskhan in https://github.com/eto-ai/lance/pull/253
• restore skipped test since extension types are working on mac by @changhiskhan in https://github.com/eto-ai/lance/pull/256
• ingestion example by @changhiskhan in https://github.com/eto-ai/lance/pull/252
• Update README.md by @jaichopra in https://github.com/eto-ai/lance/pull/261
• [CI] pin arrow 9.0 in GHA by @eddyxu in https://github.com/eto-ai/lance/pull/268
• Update README.md by @jaichopra in https://github.com/eto-ai/lance/pull/264
• Pandas extension dtype for image by @changhiskhan in https://github.com/eto-ai/lance/pull/267
• Reorganize tutorial notebooks by @changhiskhan in https://github.com/eto-ai/lance/pull/265
• Merge two Schemas by @eddyxu in https://github.com/eto-ai/lance/pull/263
• Versioning support with Appending Dataset by @eddyxu in https://github.com/eto-ai/lance/pull/262
• Change datagen to use public https image urls by @changhiskhan in https://github.com/eto-ai/lance/pull/271

Full Changelog: https://github.com/eto-ai/lance/compare/v0.1.4...v0.1.5

What's Changed

• CLI to inspect lance dataset by @eddyxu in https://github.com/eto-ai/lance/pull/231
• Generate primary key for Oxford Pet dataset by @eddyxu in https://github.com/eto-ai/lance/pull/233
• Fix datagen test by @eddyxu in https://github.com/eto-ai/lance/pull/234
• Add discord link and fix typo in README by @eddyxu in https://github.com/eto-ai/lance/pull/236
• Improve VarBinaryDecoder::Take performance by accumulating small batches by @eddyxu in https://github.com/eto-ai/lance/pull/239

Full Changelog: https://github.com/eto-ai/lance/compare/v0.1.3...v0.1.4

What's Changed

• EDA Howtos by @eddyxu in https://github.com/eto-ai/lance/pull/186
• Apply Limit cross multiple files in the dataset. by @eddyxu in https://github.com/eto-ai/lance/pull/226
• Fix false assertion during BinaryEncoding by @eddyxu in https://github.com/eto-ai/lance/pull/227

Full Changelog: https://github.com/eto-ai/lance/compare/v0.1.2...v0.1.3

1. Lance now supports projection for nested column (e.g., "annotations.name")
2. There's also a fast path for CountRows to get the record count by looking at metadata
3. Finally, Lance now supports writing optional key-value metadata (pa.Table.schema.metadata)

What's Changed

• GH release automation improvement by @changhiskhan in https://github.com/eto-ai/lance/pull/212
• Refactor benchmark and dataset generation by @changhiskhan in https://github.com/eto-ai/lance/pull/205
• install fsspec and s3fs by @changhiskhan in https://github.com/eto-ai/lance/pull/213
• Add ArrayFromJSON and TableFromJSON help functions for easy testing. by @eddyxu in https://github.com/eto-ai/lance/pull/214
• [C++] Merge two schemas by @eddyxu in https://github.com/eto-ai/lance/pull/217
• Support nested column projection by @eddyxu in https://github.com/eto-ai/lance/pull/220
• Implement CountRows for fast path of countings by @eddyxu in https://github.com/eto-ai/lance/pull/221
• Write schema metadata to Manifest by @eddyxu in https://github.com/eto-ai/lance/pull/222

Full Changelog: https://github.com/eto-ai/lance/compare/v0.1.1...v0.1.2

Fix up Mac wheel to enable extension types for MacOS

What's Changed

• run quickstart notebook on every commit by @changhiskhan in https://github.com/eto-ai/lance/pull/209
• update README.md to reflect new URI and bindings by @jaichopra in https://github.com/eto-ai/lance/pull/211
• [Python] Exclude relocating packages in homebrew by @eddyxu in https://github.com/eto-ai/lance/pull/210

New Contributors

• @jaichopra made their first contribution in https://github.com/eto-ai/lance/pull/211

Full Changelog: https://github.com/eto-ai/lance/compare/v0.1.0...v0.1.1

Highlights

1. Documentation is now live and a Quickstart Notebook is available
2. Lance is now integrated with pytorch and supports multiple workers.
3. Vision-specific extension types like Box2d provides vectorized iou and Image types that make it easy to perform IO and go between bytes, PIL, numpy, and tensors.

What's Changed

• Setting BatchSize via ScanBuilder by @eddyxu in https://github.com/eto-ai/lance/pull/135
• Move Expression based schema project to Schema class by @eddyxu in https://github.com/eto-ai/lance/pull/137
• Refactor I/O exec nodes by @eddyxu in https://github.com/eto-ai/lance/pull/136
• Simplify RecordBatchReader to use Project.next() by @eddyxu in https://github.com/eto-ai/lance/pull/139
• Convert bdd100k dataset in python benchmarks by @eddyxu in https://github.com/eto-ai/lance/pull/131
• Fix the condition of Scan advancing batch id by @eddyxu in https://github.com/eto-ai/lance/pull/143
• Initial PyTorch Dataset support by @eddyxu in https://github.com/eto-ai/lance/pull/134
• Example training code over oxford pet dataset by @eddyxu in https://github.com/eto-ai/lance/pull/144
• Test writing fixed size list and fixed size binary via WriteTable by @eddyxu in https://github.com/eto-ai/lance/pull/151
• Fix fixed size length calculation by @eddyxu in https://github.com/eto-ai/lance/pull/152
• Provide binary to profiling scans by @eddyxu in https://github.com/eto-ai/lance/pull/149
• Multi-worker support in Pytorch Dataset by @eddyxu in https://github.com/eto-ai/lance/pull/147
• Vision specific extension types by @changhiskhan in https://github.com/eto-ai/lance/pull/146
• lance dataset that overrides Dataset.scanner and Dataset.head by @changhiskhan in https://github.com/eto-ai/lance/pull/158
• Pickle Image by @changhiskhan in https://github.com/eto-ai/lance/pull/160
• Only load manifest once within the dataset and share Manifest amount the readers by @eddyxu in https://github.com/eto-ai/lance/pull/155
• Improve ergonomic of the Pytorch dataset and Generate embeddings for oxford pet by @eddyxu in https://github.com/eto-ai/lance/pull/157
• Fix PlainEncoder to read empty page by @eddyxu in https://github.com/eto-ai/lance/pull/164
• Convert coco annotations from the list of structs to struct of lists by @eddyxu in https://github.com/eto-ai/lance/pull/166
• Convert coco bounding box format to [x0,y0,x1,y1] format. by @eddyxu in https://github.com/eto-ai/lance/pull/169
• Image Array by @changhiskhan in https://github.com/eto-ai/lance/pull/168
• Fix writing and reading extension type by @eddyxu in https://github.com/eto-ai/lance/pull/172
• Coco improvements by @changhiskhan in https://github.com/eto-ai/lance/pull/174
• Support partitioning and group size control in coco dataset generation. by @eddyxu in https://github.com/eto-ai/lance/pull/175
• Extension type improvements to support 3d types by @changhiskhan in https://github.com/eto-ai/lance/pull/173
• Support converting PIL from Image in pytorch Dataset by @eddyxu in https://github.com/eto-ai/lance/pull/176
• Minor fix for 3d extension types by @changhiskhan in https://github.com/eto-ai/lance/pull/177
• MS coco dataset training by @eddyxu in https://github.com/eto-ai/lance/pull/163
• Change version import to relative import by @eddyxu in https://github.com/eto-ai/lance/pull/181
• [python] Mix of minor improvements by @changhiskhan in https://github.com/eto-ai/lance/pull/182
• Automatically build document and publish to Github Pages by @eddyxu in https://github.com/eto-ai/lance/pull/180
• [benchmarks] simplify the datagen code and remove partitioning for now by @changhiskhan in https://github.com/eto-ai/lance/pull/183
• Fix PlainDecoder handle empty filtered array by @eddyxu in https://github.com/eto-ai/lance/pull/187
• [python] minor improvements by @changhiskhan in https://github.com/eto-ai/lance/pull/190
• Fix bug that attempt to partitioned columns which does not exist in the file. by @eddyxu in https://github.com/eto-ai/lance/pull/189
• Pass filter indices via Limit and Return empty array in GetListArray by @eddyxu in https://github.com/eto-ai/lance/pull/191
• Exclude filter columns from projection by @eddyxu in https://github.com/eto-ai/lance/pull/194
• action to bump version for new release by @changhiskhan in https://github.com/eto-ai/lance/pull/199
• [C++] [BUG] Adjust offset when the batch size is set for reading by @eddyxu in https://github.com/eto-ai/lance/pull/201
• GH action to upload wheels and also make reusable yml by @changhiskhan in https://github.com/eto-ai/lance/pull/200
• [Python] Test projection in Python Torch Dataset by @eddyxu in https://github.com/eto-ai/lance/pull/202
• Fix typo of calculating offsets for slicing index by @eddyxu in https://github.com/eto-ai/lance/pull/206
• Changhiskhan/tutorial by @changhiskhan in https://github.com/eto-ai/lance/pull/167

Full Changelog: https://github.com/eto-ai/lance/compare/v0.0.5...v0.1.0

What's Changed

• Update python dataset generation to use boolean and dictionary encoding. by @eddyxu in https://github.com/eto-ai/lance/pull/124
• Basic python extension types for image, point2d, and box2d by @changhiskhan in https://github.com/eto-ai/lance/pull/102
• Fix offset calculation in VarBinaryEncoder for slices by @eddyxu in https://github.com/eto-ai/lance/pull/128
• Support pyarrow write_dataset with partitions by @eddyxu in https://github.com/eto-ai/lance/pull/125
• Add encoding for FixedSizeBinary and FixedSizeList by @eddyxu in https://github.com/eto-ai/lance/pull/129
• Support parsing fixed size list in schema by @eddyxu in https://github.com/eto-ai/lance/pull/130

Full Changelog: https://github.com/eto-ai/lance/compare/v0.0.4...v0.0.5

Add benchmarks vs parquet and raw json data
Fixes #112, which caused lance datasets to be written with duplicated Arrow buffers
Adds support for large binary, boolean, and temporal types.

What's Changed

• Restore the custom build script for pyarrow by @changhiskhan in https://github.com/eto-ai/lance/pull/84
• Add support for Arrow temporal types, large binarys, and null type by @eddyxu in https://github.com/eto-ai/lance/pull/85
• Fix warnings for CMP0135 by @eddyxu in https://github.com/eto-ai/lance/pull/86
• Remove the primary key for now by @changhiskhan in https://github.com/eto-ai/lance/pull/88
• reproduce large binary write bug by @changhiskhan in https://github.com/eto-ai/lance/pull/91
• Basic extension type handling by @changhiskhan in https://github.com/eto-ai/lance/pull/90
• Make FileReader pass schema as shared_ptr to Project by @eddyxu in https://github.com/eto-ai/lance/pull/100
• Fix compile warnings for testing by @eddyxu in https://github.com/eto-ai/lance/pull/103
• Fix length calculation when reading list array with offset by @eddyxu in https://github.com/eto-ai/lance/pull/104
• Add read/write support for time / datetime types by @eddyxu in https://github.com/eto-ai/lance/pull/106
• Set batch size when writing the file by @eddyxu in https://github.com/eto-ai/lance/pull/111
• Coco benchmarks for lance and parquet formats by @changhiskhan in https://github.com/eto-ai/lance/pull/97
• Minor fixes for benchmarks by @changhiskhan in https://github.com/eto-ai/lance/pull/115
• Project skips scan if the filter covers all the columns by @eddyxu in https://github.com/eto-ai/lance/pull/117
• Rebind filter after lance schema projection. by @eddyxu in https://github.com/eto-ai/lance/pull/118
• Plain encoding supports writing boolean array by @eddyxu in https://github.com/eto-ai/lance/pull/120
• Update README.md by @changhiskhan in https://github.com/eto-ai/lance/pull/123
• PlainEncoding uses raw value pointer to write data to disk. by @eddyxu in https://github.com/eto-ai/lance/pull/116

Full Changelog: https://github.com/eto-ai/lance/compare/v0.0.3...v0.0.4