xpartition

Tool for writing large xarray datasets to zarr stores with independent processes

MIT License

Downloads
200
Stars
28
Committers
2
View Code on GitHub
Ecosystems: Python
xpartition - v0.2.2 Latest Release

Published by spencerkclark 3 months ago

This minor release enables compatibility with NumPy version 2.0, thanks to @jpdunc23 in #24. It also includes an earlier bug fix to prevent possible coordinate corruption during writes with the PartitionMapper approach (#20). This was a similar issue to what was addressed in #17 for the traditional ds.partition.write approach.

xpartition - v0.2.1

Published by spencerkclark over 1 year ago

xpartition - v0.2.0

Published by spencerkclark over 3 years ago

xpartition - v0.1.0

Published by spencerkclark over 3 years ago

Package Rankings
Top 17.91% on Pypi.org
Badges
Extracted from project README
Build Status codecov PyPI
Related Projects
cluster_tools

Distributed segmentation for bio-image-analysis

20 Jan 2018 35
cubed

Bounded-memory serverless distributed N-dimensional array processing

05 May 2022 95
shardedstore

Provides a sharded Zarr store

07 May 2022 4
dask-notebooks

Dask tutorials for Big Data Analysis and Machine Learning as Jupyter notebooks

06 Feb 2018 5
granular

Fast dataset format and loader

27 Jun 2024 17
once-per-worker

Create dask.delayed objects that only run once per worker

20 Apr 2021 7
zarr-python

An implementation of chunked, compressed, N-dimensional arrays for Python.

15 Dec 2015 1,336
adaptive-scheduler

Run many functions (adaptively) on many cores (>10k-100k) using mpi4py.futures, ipyparallel, loky...

03 May 2019 26
any2dataset

Turn any collection of files into a dataset

06 Jan 2023 42
dask-worker-pools

Assign tasks to pools of workers in dask

10 Mar 2022 5
dask-memusage

A low-impact profiler to figure out how much memory each task in Dask is using

25 Sep 2018 24
xyzpy

Efficiently generate and analyse high dimensional data.

24 May 2016 67
ipython-cluster-helper

Tool to easily start up an IPython cluster on different schedulers.

08 Mar 2013 148
submitit

Python 3.8+ toolbox for submitting jobs to Slurm

24 Apr 2020 1,266
spython

Making sbatch more user-friendly (for python users of Jean-Zay).

27 Apr 2023 8