cubed | Python Ecosystem Directory

cubed - 0.15.0 Latest Release

Published by tomwhite 5 months ago

What's Changed

Test on Python 3.12 by @tomwhite in https://github.com/cubed-dev/cubed/pull/453
Test NumPy 2 by @tomwhite in https://github.com/cubed-dev/cubed/pull/454
Expose Lithops wait_dur_sec to speed up tests by @tomwhite in https://github.com/cubed-dev/cubed/pull/456
Simplify executor naming by @tomwhite in https://github.com/cubed-dev/cubed/pull/457
Treat warnings as errors when running tests by @tomwhite in https://github.com/cubed-dev/cubed/pull/458
Re-interpret 'fusable' to mean with predecessor operations by @tomwhite in https://github.com/cubed-dev/cubed/pull/460
Don't include hidden arrays when counting max_total_source_arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/463
Limited implementation of map_overlap by @tomwhite in https://github.com/cubed-dev/cubed/pull/462
Limited implementation of pad by @tomwhite in https://github.com/cubed-dev/cubed/pull/461
Implement roll by @tomwhite in https://github.com/cubed-dev/cubed/pull/465
Fix map_overlap in case of single chunk in overlap dimension by @tomwhite in https://github.com/cubed-dev/cubed/pull/466
Fix case of Lithops 3.4.0 hanging by @tomwhite in https://github.com/cubed-dev/cubed/pull/471
Fix unify_chunks to return regular chunks in all cases by @tomwhite in https://github.com/cubed-dev/cubed/pull/470
Improve test coverage by @tomwhite in https://github.com/cubed-dev/cubed/pull/472
Change development status to beta by @tomwhite in https://github.com/cubed-dev/cubed/pull/474
Top level array namespace by @tomwhite in https://github.com/cubed-dev/cubed/pull/473
Add notebooks for Pangeo examples by @tomwhite in https://github.com/cubed-dev/cubed/pull/476

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.14.3...0.15.0

cubed - 0.14.3

Published by tomwhite 6 months ago

What's Changed

Take broadcast trick into account for array chunk memory by @tomwhite in https://github.com/cubed-dev/cubed/pull/447
Fix Documentation link by @norlandrhagen in https://github.com/cubed-dev/cubed/pull/451
Include j2 templates in the distribution by @tomwhite in https://github.com/cubed-dev/cubed/pull/450
Add links to blog posts by @TomNicholas in https://github.com/cubed-dev/cubed/pull/452
Add a groupby_blockwise function for use in Flox by @tomwhite in https://github.com/cubed-dev/cubed/pull/448

New Contributors

@norlandrhagen made their first contribution in https://github.com/cubed-dev/cubed/pull/451

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.14.2...0.14.3

cubed - 0.14.2

Published by tomwhite 6 months ago

What's Changed

Allow networkx 3.3 now that pydot issue has been fixed by @tomwhite in https://github.com/cubed-dev/cubed/pull/445
Change drop_axis to default to None rather than an empty list in map_blocks by @tomwhite in https://github.com/cubed-dev/cubed/pull/446
Add a groupby_reduction function for use in Flox by @tomwhite in https://github.com/cubed-dev/cubed/pull/442

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.14.1...0.14.2

cubed - 0.14.1

Published by tomwhite 7 months ago

What's Changed

Providing filesystem credentials through storage_options by @balanz24 in https://github.com/cubed-dev/cubed/pull/436
Only use oindex for advanced indexing (int arrays) by @tomwhite in https://github.com/cubed-dev/cubed/pull/440
Deprecate misspelled aggegrate_func in favour of aggregate_func by @tomwhite in https://github.com/cubed-dev/cubed/pull/444

New Contributors

@balanz24 made their first contribution in https://github.com/cubed-dev/cubed/pull/436

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.14.0...0.14.1

cubed - 0.14.0

Published by tomwhite 7 months ago

What's Changed

Handle zero-length slices in index function by @tomwhite in https://github.com/cubed-dev/cubed/pull/401
Run unit tests on Python 3.11 on Linux by @tomwhite in https://github.com/cubed-dev/cubed/pull/405
Update examples to use Python 3.11 by @tomwhite in https://github.com/cubed-dev/cubed/pull/406
Remove unused lazy_full function (replaced by virtual_full) by @tomwhite in https://github.com/cubed-dev/cubed/pull/409
Delegate dtype functions to backend array API by @tomwhite in https://github.com/cubed-dev/cubed/pull/410
Add a 'processes' executor that runs using local processes by @tomwhite in https://github.com/cubed-dev/cubed/pull/411
Mark stack operation as not fusable by @tomwhite in https://github.com/cubed-dev/cubed/pull/415
Track source array names in PrimitiveOperation by @tomwhite in https://github.com/cubed-dev/cubed/pull/416
Fix Modal deprecations by @tomwhite in https://github.com/cubed-dev/cubed/pull/417
Make the new multiple-inputs optimization function the default by @tomwhite in https://github.com/cubed-dev/cubed/pull/412
Add cloudpickle dependency following #411 ('processes' executor) by @tomwhite in https://github.com/cubed-dev/cubed/pull/421
Documentation corrections following optimization improvements by @tomwhite in https://github.com/cubed-dev/cubed/pull/423
Add name property to all executors by @tomwhite in https://github.com/cubed-dev/cubed/pull/424
Build Lithops images for every push to main by @tomwhite in https://github.com/cubed-dev/cubed/pull/422
Make Lithops example names consistent with benchmarks and other executors by @tomwhite in https://github.com/cubed-dev/cubed/pull/425
Delete existing Lithops image before rebuilding by @tomwhite in https://github.com/cubed-dev/cubed/pull/427
Rename block_function to key_function in blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/426
Use executor name rather than class name by @tomwhite in https://github.com/cubed-dev/cubed/pull/430
Remove unused sync version of Modal executor by @tomwhite in https://github.com/cubed-dev/cubed/pull/431
Change total_nbytes to total_nbytes_written by @tomwhite in https://github.com/cubed-dev/cubed/pull/433
Fix deprecation warning on logger.warn by @tomwhite in https://github.com/cubed-dev/cubed/pull/434

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.13.0...0.14.0

cubed - 0.13.0

Published by tomwhite 8 months ago

What's Changed

Introduce finalize_dag method in Plan that caches DAG optimization by @tomwhite in https://github.com/cubed-dev/cubed/pull/340
Add VirtualInMemoryArray that keeps small arrays in memory by @tomwhite in https://github.com/cubed-dev/cubed/pull/336
Fix Modal test following #336 by @tomwhite in https://github.com/cubed-dev/cubed/pull/341
Change the DAG to have separate nodes for operations and arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/337
Allow more control over optimizations applied by @tomwhite in https://github.com/cubed-dev/cubed/pull/342
Handle the case where an array is created by calling map_blocks with no input arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/343
Add a test for visualizing the quad means computation by @tomwhite in https://github.com/cubed-dev/cubed/pull/345
Show nbytes for each array and total on visualization by @tomwhite in https://github.com/cubed-dev/cubed/pull/344
Fuse multiple inputs by @tomwhite in https://github.com/cubed-dev/cubed/pull/346
Upgrade to sphinx-notes/pages v3 by @tomwhite in https://github.com/cubed-dev/cubed/pull/347
Minor improvements to DAG visualization by @tomwhite in https://github.com/cubed-dev/cubed/pull/348
Remove deprecated code by @tomwhite in https://github.com/cubed-dev/cubed/pull/349
Add generated doc for primitive and core ops to design page by @tomwhite in https://github.com/cubed-dev/cubed/pull/351
Support arbitrary block index functions in blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/350
Refactor pipeline to reduce coupling between runtime and primitive layers by @tomwhite in https://github.com/cubed-dev/cubed/pull/352
Implement stack using general_blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/353
Implement reshape_chunks using general_blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/354
More control over operation fusion by @tomwhite in https://github.com/cubed-dev/cubed/pull/355
Minor memory fixes by @tomwhite in https://github.com/cubed-dev/cubed/pull/358
Use Dask's 'broadcast trick' to save memory for single-valued arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/359
Add a show_hidden option to visualize by @tomwhite in https://github.com/cubed-dev/cubed/pull/360
Reinstate slow tests by @tomwhite in https://github.com/cubed-dev/cubed/pull/361
Add function_nargs to BlockwiseSpec by @tomwhite in https://github.com/cubed-dev/cubed/pull/363
Add missing api doc by @tomwhite in https://github.com/cubed-dev/cubed/pull/364
Implement partial_reduce and tree_reduce using generalized blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/365
Quad means validation test by @tomwhite in https://github.com/cubed-dev/cubed/pull/367
Take allowed_mem into account when fusing primitive operations by @tomwhite in https://github.com/cubed-dev/cubed/pull/366
Fuse operations with different numbers of tasks by @tomwhite in https://github.com/cubed-dev/cubed/pull/368
Show number of tasks prominently on graph visualization by @tomwhite in https://github.com/cubed-dev/cubed/pull/371
Introduce event objects for all callback methods by @tomwhite in https://github.com/cubed-dev/cubed/pull/373
Minor Modal improvements by @tomwhite in https://github.com/cubed-dev/cubed/pull/375
Run tests using JAX as the backend array API (on CPU) by @tomwhite in https://github.com/cubed-dev/cubed/pull/374
Introduce max_total_num_input_blocks as a heuristic to control fusion by @tomwhite in https://github.com/cubed-dev/cubed/pull/376
Rename array_name to name in runtime following #337 by @tomwhite in https://github.com/cubed-dev/cubed/pull/377
Enable backup tasks by default for all executors that support them by @tomwhite in https://github.com/cubed-dev/cubed/pull/378
Remove unused array_names parameter in executors by @tomwhite in https://github.com/cubed-dev/cubed/pull/380
Introduce compute_id for each computation by @tomwhite in https://github.com/cubed-dev/cubed/pull/382
Add a Rich progress bar by @tomwhite in https://github.com/cubed-dev/cubed/pull/383
Change timeline visualization default format to SVG by @tomwhite in https://github.com/cubed-dev/cubed/pull/384
Preserve laziness in fuse_multiple to avoid materializing array blocks in _partial_reduce by @tomwhite in https://github.com/cubed-dev/cubed/pull/386
Fix Modal unit tests following change in #382 by @tomwhite in https://github.com/cubed-dev/cubed/pull/387
Add debug log statements to optimizer to see why nodes are fused by @tomwhite in https://github.com/cubed-dev/cubed/pull/388
Allow optional path in from_zarr and to_zarr by @tomwhite in https://github.com/cubed-dev/cubed/pull/391
Remove CUBED_MODAL_FORCE_BUILD env var … by @tomwhite in https://github.com/cubed-dev/cubed/pull/393
Add introductory slides to docs by @tomwhite in https://github.com/cubed-dev/cubed/pull/394
Use donfig to control default spec by @tomwhite in https://github.com/cubed-dev/cubed/pull/392
small changes to coiled/aws examples by @rsignell in https://github.com/cubed-dev/cubed/pull/369
Skip vecdot test due to zero-size array failures by @tomwhite in https://github.com/cubed-dev/cubed/pull/399
Show both operation ID and name in visualization by @tomwhite in https://github.com/cubed-dev/cubed/pull/396
Add support for minimum_workers in Coiled by @tomwhite in https://github.com/cubed-dev/cubed/pull/398
Create a single set of example scripts that can run on any executor by @tomwhite in https://github.com/cubed-dev/cubed/pull/395

New Contributors

@rsignell made their first contribution in https://github.com/cubed-dev/cubed/pull/369

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.12.0...0.13.0

cubed - 0.12.0

Published by tomwhite 11 months ago

What's Changed

Improve separation between layers by @tomwhite in https://github.com/tomwhite/cubed/pull/314
Fix Modal deprecation of f.call(...) by @tomwhite in https://github.com/tomwhite/cubed/pull/320
Disable the Hypthosis deadline when running Array API tests. by @tomwhite in https://github.com/tomwhite/cubed/pull/325
Allow dtype to be passed to blockwise and reduction functions by @tomwhite in https://github.com/tomwhite/cubed/pull/321
Array API backend by @tomwhite in https://github.com/tomwhite/cubed/pull/317
Update examples to use Python 3.9 by @tomwhite in https://github.com/tomwhite/cubed/pull/330
Re-write arange to use map_direct, which avoids writing a Zarr array by @tomwhite in https://github.com/tomwhite/cubed/pull/332

Full Changelog: https://github.com/tomwhite/cubed/compare/0.11.0...0.12.0

cubed - 0.11.0

Published by tomwhite about 1 year ago

What's Changed

Introduce VirtualEmptyArray and hide virtual arrays in the DAG visualization by @tomwhite in https://github.com/tomwhite/cubed/pull/290
Introduce RetryingFunctionExecutor by @tomwhite in https://github.com/tomwhite/cubed/pull/292
Add support for batching inputs to avoid overwhelming the backend service by @tomwhite in https://github.com/tomwhite/cubed/pull/291
Support version 2022.12 of the Array API by @tomwhite in https://github.com/tomwhite/cubed/pull/293
Fix array __abs__ method for complex inputs by @tomwhite in https://github.com/tomwhite/cubed/pull/294
Be explicit when opening Zarr arrays by @tomwhite in https://github.com/tomwhite/cubed/pull/296
Fix bug when running with Kerchunk by @tomwhite in https://github.com/tomwhite/cubed/pull/297
Remove limited implementation of __setitem__ from Array by @tomwhite in https://github.com/tomwhite/cubed/pull/302
Introduce VirtualFullArray by @tomwhite in https://github.com/tomwhite/cubed/pull/303
[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/tomwhite/cubed/pull/305
Fix case where shape is an int for empty by @tomwhite in https://github.com/tomwhite/cubed/pull/306
Update Coiled example by @tomwhite in https://github.com/tomwhite/cubed/pull/309
Add nansum and nanmean functions by @tomwhite in https://github.com/tomwhite/cubed/pull/308
Add moveaxis by @tomwhite in https://github.com/tomwhite/cubed/pull/313
Update minimum Python version to 3.9 by @tomwhite in https://github.com/tomwhite/cubed/pull/298

Full Changelog: https://github.com/tomwhite/cubed/compare/0.10.0...0.11.0

cubed - 0.10.0

Published by tomwhite about 1 year ago

What's Changed

Scaling docs by @TomNicholas in https://github.com/tomwhite/cubed/pull/252
Remove PipelineExecutor classes by @tomwhite in https://github.com/tomwhite/cubed/pull/253
Surface rechunk intermediates in DAG by @tomwhite in https://github.com/tomwhite/cubed/pull/257
Add type hints for runtime by @tomwhite in https://github.com/tomwhite/cubed/pull/258
Fix modal import by @tomwhite in https://github.com/tomwhite/cubed/pull/261
Add compute_arrays_in_parallel option to Modal and Python async executors by @tomwhite in https://github.com/tomwhite/cubed/pull/263
Use new merge_chunks function in reduction by @tomwhite in https://github.com/tomwhite/cubed/pull/256
Remove vendored rechunker API, since we only use the algorithm by @tomwhite in https://github.com/tomwhite/cubed/pull/264
Add unit tests for virtual_offsets function by @tomwhite in https://github.com/tomwhite/cubed/pull/265
[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/tomwhite/cubed/pull/266
Lithops compute arrays in parallel by @tomwhite in https://github.com/tomwhite/cubed/pull/259
Remove duplicate copy of projected_mem and num_tasks from DAG by @tomwhite in https://github.com/tomwhite/cubed/pull/271
Allow step size > 1 by @tomwhite in https://github.com/tomwhite/cubed/pull/270
Fix reserved_mem in create_lazy_zarr_arrays by @tomwhite in https://github.com/tomwhite/cubed/pull/273
Dask tests by @tomwhite in https://github.com/tomwhite/cubed/pull/269
Coiled Functions executor by @TomNicholas in https://github.com/tomwhite/cubed/pull/260
Ensure runtime memory is not less than allowed by @tomwhite in https://github.com/tomwhite/cubed/pull/274
In-process Python executor should not retry by @tomwhite in https://github.com/tomwhite/cubed/pull/275
Refactor pipeline mappable by @tomwhite in https://github.com/tomwhite/cubed/pull/272
Generalise async executor code by @tomwhite in https://github.com/tomwhite/cubed/pull/278
Dask distributed async executor by @tomwhite in https://github.com/tomwhite/cubed/pull/279
[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/tomwhite/cubed/pull/283
Dask check_runtime_memory by @tomwhite in https://github.com/tomwhite/cubed/pull/281
Fixes for converting strings to bytes by @TomNicholas in https://github.com/tomwhite/cubed/pull/289

Full Changelog: https://github.com/tomwhite/cubed/compare/0.9.0...0.10.0

cubed - 0.9.0

Published by tomwhite over 1 year ago

What's Changed

Fix Modal deprecated API by @tomwhite in https://github.com/tomwhite/cubed/pull/214
Modal GCP by @tomwhite in https://github.com/tomwhite/cubed/pull/215
Add html array repr by @TomNicholas in https://github.com/tomwhite/cubed/pull/216
Add to pre-commit.ci by @tomwhite in https://github.com/tomwhite/cubed/pull/218
Skip brew update when setting up GraphViz on mac CI to improve run time by @tomwhite in https://github.com/tomwhite/cubed/pull/227
Set 'reraise' to True for retries by @tomwhite in https://github.com/tomwhite/cubed/pull/225
Run mypy in CI by @tomwhite in https://github.com/tomwhite/cubed/pull/229
Print irregular chunking pattern when error raised by @TomNicholas in https://github.com/tomwhite/cubed/pull/231
[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/tomwhite/cubed/pull/232
Allow rechunk to take a dict for chunks by @tomwhite in https://github.com/tomwhite/cubed/pull/226
Improve Modal backups implementation by @tomwhite in https://github.com/tomwhite/cubed/pull/238
Test that plan generation scales to very large workloads by @tomwhite in https://github.com/tomwhite/cubed/pull/240
Don't materialize input iterables to map_unordered by @tomwhite in https://github.com/tomwhite/cubed/pull/242
Lithops retries and backups by @tomwhite in https://github.com/tomwhite/cubed/pull/217
Add type hints for primitive functions by @tomwhite in https://github.com/tomwhite/cubed/pull/236
Add links to Xarray integration and blog post to docs by @tomwhite in https://github.com/tomwhite/cubed/pull/243
Account for reserved_mem when creating arrays by @tomwhite in https://github.com/tomwhite/cubed/pull/245
Use np.unravel_index rather than iterating over np.ndindex by @tomwhite in https://github.com/tomwhite/cubed/pull/246
Add matmul example that validates result by @tomwhite in https://github.com/tomwhite/cubed/pull/244
Document strong consistency by @tomwhite in https://github.com/tomwhite/cubed/pull/250
Re-enable rechunker's consolidate_reads by @tomwhite in https://github.com/tomwhite/cubed/pull/249
Introduce VirtualOffsetsArray that is never materialized by @tomwhite in https://github.com/tomwhite/cubed/pull/248
Avoid divide by zero from projected mem by @tomwhite in https://github.com/tomwhite/cubed/pull/251

New Contributors

@pre-commit-ci made their first contribution in https://github.com/tomwhite/cubed/pull/232

Full Changelog: https://github.com/tomwhite/cubed/compare/0.8.0...0.9.0

cubed - 0.8.0

Published by tomwhite over 1 year ago

What's Changed

Lithops gcf example by @TomNicholas in https://github.com/tomwhite/cubed/pull/183
Name Modal Stub objects to avoid warning by @tomwhite in https://github.com/tomwhite/cubed/pull/189
Convert str to bytes by @TomNicholas in https://github.com/tomwhite/cubed/pull/188
Reinstate Lithops AWS examples that were accidently removed in #174 by @tomwhite in https://github.com/tomwhite/cubed/pull/185
Improvements and fixes to Modal examples by @tomwhite in https://github.com/tomwhite/cubed/pull/190
More examples on GCF by @tomwhite in https://github.com/tomwhite/cubed/pull/198
Fix measure reserved by @tomwhite in https://github.com/tomwhite/cubed/pull/200
Include 'cubed' in the directories created in the working directory by @tomwhite in https://github.com/tomwhite/cubed/pull/202
Add user guide and break out getting started into separate pages by @tomwhite in https://github.com/tomwhite/cubed/pull/203
Show array variable name in visualization by @tomwhite in https://github.com/tomwhite/cubed/pull/192
Runtime refactoring by @tomwhite in https://github.com/tomwhite/cubed/pull/206
Make Zarr array creation lazy by @tomwhite in https://github.com/tomwhite/cubed/pull/207
Make executors create Zarr arrays by @tomwhite in https://github.com/tomwhite/cubed/pull/209
Make Lithops retries apply per-task by @tomwhite in https://github.com/tomwhite/cubed/pull/211
Move Modal examples to 'aws' directory by @tomwhite in https://github.com/tomwhite/cubed/pull/212
Publish packages to PyPI using GitHub Actions by @tomwhite in https://github.com/tomwhite/cubed/pull/213

Full Changelog: https://github.com/tomwhite/cubed/compare/0.7.0...0.8.0

cubed - 0.7.0

Published by tomwhite over 1 year ago

What's Changed

History analysis by @tomwhite in https://github.com/tomwhite/cubed/pull/170
Rename memory settings by @tomwhite in https://github.com/tomwhite/cubed/pull/180
Docs on installing from conda + optional dependencies by @TomNicholas in https://github.com/tomwhite/cubed/pull/178
Rename peak_memory to peak_measured_mem in line with other memory settings by @tomwhite in https://github.com/tomwhite/cubed/pull/182

Full Changelog: https://github.com/tomwhite/cubed/compare/0.6.3...0.7.0

cubed - 0.6.3

Published by tomwhite over 1 year ago

What's Changed

Update examples by @tomwhite in https://github.com/tomwhite/cubed/pull/174
Run tests on macOS and Windows by @tomwhite in https://github.com/tomwhite/cubed/pull/176

Full Changelog: https://github.com/tomwhite/cubed/compare/0.6.2...0.6.3

cubed - 0.6.2

Published by tomwhite over 1 year ago

What's Changed

Make lithops optional for running tests by @tomwhite in https://github.com/tomwhite/cubed/pull/169
Optional dependency specification and pyproject.toml cleanup by @TomNicholas in https://github.com/tomwhite/cubed/pull/168
Use requirements file from Array API tests repo by @tomwhite in https://github.com/tomwhite/cubed/pull/172
Fix modal tests (env must not contain beam or lithops dependencies) by @tomwhite in https://github.com/tomwhite/cubed/pull/173

Full Changelog: https://github.com/tomwhite/cubed/compare/0.6.1...0.6.2

cubed - 0.6.1

Published by tomwhite over 1 year ago