A low-impact profiler to figure out how much memory each task in Dask is using
MIT License
Decorators that logs stats.
Profile the dask distributed scheduler with py-spy and viztracer
Python memory tracing.
A Python package for easy multiprocessing, but faster than multiprocessing
A Minimal Cluster Computing Framework in plain Python
Making sbatch more user-friendly (for python users of Jean-Zay).
Dask tutorials for Big Data Analysis and Machine Learning as Jupyter notebooks
Create dask.delayed objects that only run once per worker
Horizon chart for CPU/GPU/Neural Engine utilization monitoring on Apple M1/M2 and nVidia GPUs on ...
Python 3.8+ toolbox for submitting jobs to Slurm
Tool for writing large xarray datasets to zarr stores with independent processes
Tool to easily start up an IPython cluster on different schedulers.
Run many functions (adaptively) on many cores (>10k-100k) using mpi4py.futures, ipyparallel, loky...
Code & Slides for the EuroPython 2024 Talk "Profile, Optimize, Repeat: One Core Is All You Need™"
Assign tasks to pools of workers in dask