zarrs

Added

• Add zarrs_icechunk to ecosystem docs

Fixed

• Fix data_key encoding on windows (it contained '//')
• Fix clippy::needless_lifetimes lint

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/zarrs-v0.17.0...zarrs-v0.17.1

Highlights / Major Changes

• zarrs has been split into 3 core crates: zarrs, zarrs_metadata, and zarrs_storage
  • zarrs_storage and zarrs_metadata are re-exported as the storage and metadata modules
• Store implementations have been moved into separate crates: zarrs_{filesystem,http,object_store,opendal,zip}
  • zarrs_filesystem is re-exported as the filesystem module with the filesystem feature (enabled by default)
• Direct IO support for Linux in FilesystemStore
• Implicit groups are no longer supported
  • It is the responsibility of the zarrs consumer to explicitly write group metadata when creating a hierarchy
• Codecs must now be Arc'd instead of Box'd
• Fixes a performance regression introduced in 0.16 when reading sharded arrays
• Check the full release notes for all changes. This release has many breaking changes due to items being removed, moved, renamed, and deprecated.

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.16.4...zarrs-v0.17.0

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.16.4...zarrs-v0.17.0-beta.3

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.16.4...zarrs-v0.17.0-beta.2

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.16.4...zarrs-v0.17.0-beta.1

Highlights / Major Changes

• zarrs has been split into 5 crates: zarrs, zarrs_metadata, zarrs_storage, zarrs_opendal, and zarrs_object_store
  • zarrs_storage and zarrs_metadata are re-exported as the storage and metadata modules
  • opendal and object_store support has been moved to the zarrs_opendal and zarrs_object_store crates
• Chunk caching support via ArrayChunkCacheExt
• Direct IO support for Linux in FilesystemStore
• Implicit groups are no longer supported
  • It is the responsibility of the zarrs consumer to explicitly write group metadata when creating a hierarchy
• Codecs must now be Arc'd instead of Box'd
• Check the full release notes for all changes. This release has many breaking changes due to items being removed, moved, renamed, and deprecated.

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.16.4...zarrs-v0.17.0-beta.0

Fixed

• Reduce maximum supported opendal version from 0.49 to 0.48
  • This reverts a change in 0.16.3 that was not semver compatible

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.16.3...v0.16.4

Changed

• Bump derive_more to 1.0.0
• Bump maximum supported opendal version from 0.48 to 0.49
• Box the metadata in PluginMetadataInvalidError

Fixed

• Fix cargo test with --no-default-features
• Fix new clippy warnings in nightly

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.16.2...v0.16.3

Added

• Add the experimental gdeflate bytes-to-bytes codec
• Add Array::chunk_key()
• Add ArrayShardedExt::inner_chunk_grid_shape()

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.16.1...v0.16.2

Changed

• Bump maximum supported opendal version from 0.47 to 0.48

Fixed

• Fixed handling of empty shards with a variable length data type with sharding partial decoder

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.16.0...v0.16.1

Highlights

• Add experimental support for the string and binary data types and the vlen and vlen_v2 codecs
• Cleanup the Array API for retrieving elements (Vec<T>) and ndarrays
• Support more Zarr V2 array configurations and make various experimental codecs numcodecs compatible

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.15.1...v0.16.0

Added

• Add CITATION.cff

Changed

• Implement From<&String> for DimensionName
• Cleanup macro usage in array

Fixed

• Fix unnecessary allocations in _elements variants of array store methods

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.15.0...v0.15.1

Highlights

• Zarr V2 support (a Zarr V3 compatible subset)
• Codec and array optimisations
• AsyncToSyncStorageAdapter: use an async store (e.g. HTTP, S3, etc.) in a sync context
• Snappy codec support for the blosc codec

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.14.0...v0.15.0

Removed

• Breaking: Remove store_locks module, [Async]ReadableWritableStorageTraits::mutex(), and new_with_locks constructors from stores
  • DefaultStoreLocks could result in a deadlock
  • It is now the responsibility of zarrs consumers to ensure that:
    • Array::store_chunk_subset is not called concurrently on the same chunk
    • Array::store_array_subset is not called concurrently on regions sharing chunks
  • Chunk locking may be revisited in a future release

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.13.2...v0.14.0

Fixed

• Change the store lock used by stores to DisabledStoreLocks rather than DefaultStoreLocks if not set explicitly
  • Fixes an occasional deadlock that needs further investigation
  • This change only impacts consumers of zarrs that use Array::{store_chunk_subset,store_array_subset}.
  • new_with_locks store constructors can be used to explicitly choose DefaultStoreLocks
  • See the updated docs on parallel writing

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.13.2...v0.13.3

Changed

• Make the bz2 and pcodec codecs public
• The "name" of experimental codecs in array metadata now points to a URI to avoid potential future incompatibilities with other implementations
• Improve docs of several experimental codecs

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.13.1...v0.13.2

Added

• Added the array_sharded_ext::{ArrayShardedExt,ArrayShardedReadableExt} extension traits for Array to simplify working with sharded arrays
  • Abstracts the chunk grid to an "inner chunk grid" to simplify inner chunk retrieval.
  • Shard indexes are cached in a ArrayShardedReadableExtCache
  • Retrieval and chunk grid methods have fallbacks for unsharded arrays. For example, an inner chunk in an unsharded array is just a chunk
  • Sync API only, AsyncArrayShardedReadableExt and AsyncArrayShardedReadableExtCache are planned for a future release
• Added ChunkGridTraits::chunks_subset() with default implementation

Changed

• Allow float fill values to be created from int fill value metadata
• Make chunk_grid::{regular,rectangular} public
• Support 8 and 16 bit integer data types with zfp codec by promoting to 32 bit

Fixed

• Fix compute_encoded_size() for BitroundCodec incorrectly indicating various data types were unsupported
• Fix a link in chunk grid docs
• Fix incorrect minimum dependency versions and add CI check
• Fix clippy unexpected_cfgs warning with recent nightly

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.13.0...v0.13.1

Added

• Add "experimental codec store metadata if encode only" option to global config
• Add "store empty chunks" option to global config and CodecOptions
• Add ArrayMetadataOptions
• Add small example to README.md and crate root documentation
• Add uint8/int8 support to the bitround codec
• Add len() and is_empty() methods to array subset iterator producers
• Add Array::chunk_origin()

Changed

• Breaking: Bump MSRV to 1.75 (28 December, 2023)
• Bump pco (pcodec) to 0.2.1
• Breaking: Add CodecTraits::create_metadata_opt()
• Breaking: Rename data_type::IncompatibleFillValueErrorMetadataError to IncompatibleFillValueMetadataError
• Add a useful warning if the shard index references out-of-bounds bytes in a chunk encoded with the sharding codec
• Breaking: Require ndarray::Array parameters in various async array methods to implement Send

Fixed

• Fix implementation of from_chunkgrid_regular_configuration macro leading to recursion
• Fix arithmetic overflow possible with the bitround codec with integer data types
• Address todo!()s in retrieve_chunks_into_array_view_opt methods

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.12.5...v0.13.0

Added

• Implement TryFrom<&str> for {Array,Group}Metadata

Changed

• Make array::codec::array_to_bytes::bytes::{Endianness::is_native(),NATIVE_ENDIAN,reverse_endianness} public

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.12.4...v0.12.5

Fixed

• Remove unnecessary copy in OpendalStore::set
• Fixed zfp codec encode not truncating compressed data
• zfp codec compute_encoded_size() now correctly outputs a bounded size instead of an unbounded size

CHANGELOG.md

Full Changelog: https://github.com/LDeakin/zarrs/compare/v0.12.3...v0.12.4