What's Changed

• [docs][fix] Update example to use offload_model by @anj-s in https://github.com/facebookresearch/fairscale/pull/806
• Switch default branch from master to main by @tmarkstrum in https://github.com/facebookresearch/fairscale/pull/807
• [FairScale] Remove refs to "cpu_offload" in code comments by @rohan-varma in https://github.com/facebookresearch/fairscale/pull/814
• [chore] Remove deprecated THCudaCheck by @anj-s in https://github.com/facebookresearch/fairscale/pull/818
• [feat] layer memory tracking by @QuentinDuval in https://github.com/facebookresearch/fairscale/pull/808
• [chore] Add log for the new experimental memory tracker feature. by @anj-s in https://github.com/facebookresearch/fairscale/pull/819
• [chore] Update the PyTorch version that we run CPU tests with by @anj-s in https://github.com/facebookresearch/fairscale/pull/809
• [chore] Update the PyTorch version that we run benchmarks with. by @anj-s in https://github.com/facebookresearch/fairscale/pull/823
• Extend auto shard capabilities to work around torch.fx edge cases. by @EugenHotaj in https://github.com/facebookresearch/fairscale/pull/817
• [fix] Update golden data for account for the speed regression by @anj-s in https://github.com/facebookresearch/fairscale/pull/825
• [chore] Fix main breakage temporarily by relaxing constraints by @anj-s in https://github.com/facebookresearch/fairscale/pull/828
• Use correct node names for param counting in auto_shard. by @EugenHotaj in https://github.com/facebookresearch/fairscale/pull/830
• [chore] Update requirements file to reflect latest config by @anj-s in https://github.com/facebookresearch/fairscale/pull/832
• [fix]: Fixes an issue with pre_backward hook registering by @min-xu-ai in https://github.com/facebookresearch/fairscale/pull/833
• [feature] Skip creating the CPU grad tensor when training by @anj-s in https://github.com/facebookresearch/fairscale/pull/821
• [test] improve a test's coverage by @min-xu-ai in https://github.com/facebookresearch/fairscale/pull/798
• [fix] Decouple move_params_to_cpu from the mixed_precision. by @anj-s in https://github.com/facebookresearch/fairscale/pull/822
• [fix] fix test on main by @min-xu-ai in https://github.com/facebookresearch/fairscale/pull/835
• [feature] Add the low level SSD APIs by @anj-s in https://github.com/facebookresearch/fairscale/pull/829
• [feat] [FSDP]: add experimental support to shared weights by @min-xu-ai in https://github.com/facebookresearch/fairscale/pull/836
• update nightly torch and test the flaky test by @min-xu-ai in https://github.com/facebookresearch/fairscale/pull/837
• [chore] Fix broken main due to updated github URL requirements by @anj-s in https://github.com/facebookresearch/fairscale/pull/838
• [chore] Update Sphinx version in docs requirements file by @vtantia in https://github.com/facebookresearch/fairscale/pull/841
• [feat] experimental MEVO layer by @min-xu-ai in https://github.com/facebookresearch/fairscale/pull/840
• [feat] Gossip/SlowMo by @blefaudeux in https://github.com/facebookresearch/fairscale/pull/378
• [feature]Add support for SSD offload with FSDP for eval workloads by @anj-s in https://github.com/facebookresearch/fairscale/pull/839
• [chore] 0.4.2 release by @anupambhatnagar in https://github.com/facebookresearch/fairscale/pull/846
• CI config changes by @anupambhatnagar in https://github.com/facebookresearch/fairscale/pull/847
• Setup pre-commit github action and apply pre-commit to all files by @anupambhatnagar in https://github.com/facebookresearch/fairscale/pull/849
• Allow sharded grad scaler to cpu offload with FSDP by @anupambhatnagar in https://github.com/facebookresearch/fairscale/pull/831
• Update changelog, removed meta.yml and requirements cleanup by @anupambhatnagar in https://github.com/facebookresearch/fairscale/pull/853
• [feature] Add a OffloadConfig object to specify offloading params to disk. by @anj-s in https://github.com/facebookresearch/fairscale/pull/855
• [POC] Testing Manual dispatch by @anupambhatnagar in https://github.com/facebookresearch/fairscale/pull/859
• [fix] [MEVO]: make mevo work with eval and optim_state checkpointing by @min-xu-ai in https://github.com/facebookresearch/fairscale/pull/851
• [chore] 0.4.3 release by @min-xu-ai in https://github.com/facebookresearch/fairscale/pull/860

New Contributors

• @rohan-varma made their first contribution in https://github.com/facebookresearch/fairscale/pull/814
• @EugenHotaj made their first contribution in https://github.com/facebookresearch/fairscale/pull/817
• @vtantia made their first contribution in https://github.com/facebookresearch/fairscale/pull/841

Full Changelog: https://github.com/facebookresearch/fairscale/compare/v0.4.1...v0.4.3

Released version 0.4.1 for FairScale.

[0.3.4] - 2021-04-13

Added

• FSDP: Add no broadcast optim state option (#560)

Fixed

• ShardedDDP: Properly handle .eval() mode (#587)
• ShardedDDP: Handle model being moved back to CPU prior to state consolidation (#573)
• FSDP: much faster state consolidation (#595)
• FSDP: Add gradient pre-divide to prevent overflow with large world sizes (#565)
• Offload: (experimental) Fix activation offloading to CPU (#588

[0.3.0] - 2021-02-22

Added

• FullyShardedDataParallel (FSDP) (#413)
• ShardedDDP fp16 grad reduction option (#402)
• Expose experimental algorithms within the pip package (#410)

Fixed

• Catch corner case when the model is too small with respect to the world size, and shards are empty (#406)
• Memory leak in checkpoint_wrapper (#412)

Fixed

• ShardedDDP and OSS handle model trainability changes during training (#369)
• ShardedDDP state dict load/save bug (#386)
• ShardedDDP handle train/eval modes (#393)
• AdaScale handling custom scaling factors (#401)

Added

• ShardedDDP manual reduce option for checkpointing (#389)

Added

• Checkpointing model wrapper (#376)
• Faster OSS, flatbuffers (#371)
• Small speedup in OSS clipgradnorm (#363)

Fixed

• Bug in ShardedDDP with 0.1.5 depending the init (KeyError / OSS)
• Much refactoring in Pipe (#357, #358, #360, #362, #370, #373)
• Better pip integration / resident pytorch (#375)

Added

• Pytorch compatibility for OSS checkpoints (#310)
• Elastic checkpoints for OSS, world size can vary in between save and loads (#310)
• Tensor views for OSS bucketing, reduced CPU use (#300)
• Bucket calls in ShardedDDP, for faster inter node communications (#327)
• FlattenParamWrapper, which flattens module parameters into a single tensor seamlessly (#317)
• AMPnet experimental support (#304)

Fixed

• ShardedDDP properly handles device changes via .to() (#353)
• Add a new interface for AdaScale, AdaScaleWrapper, which makes it compatible with OSS (#347)

Fixed

• Missing cu files in the pip package

Same as 0.1.2, but with the correct numbering in the source code (see init.py)

Added

• AdaScale: Added gradient accumulation feature (#202)
• AdaScale: Added support of torch.lr_scheduler (#229)

Fixed

• AdaScale: smoothing factor value fixed when using gradient accumulation (#235)
• Pipe: documentation on balancing functions (#243)
• ShardedDDP: handle typical NLP models
• ShardedDDP: better partitioning when finetuning