megablocks

What's Changed

• Update dependencies and package organization. by @tgale96 in https://github.com/stanford-futuredata/megablocks/pull/52
• Remove errant "*" in README by @tgale96 in https://github.com/stanford-futuredata/megablocks/pull/54
• Update Megatron-LM scripts and integration for latest Docker container. by @tgale96 in https://github.com/stanford-futuredata/megablocks/pull/55
• Update setup.py to support multiple device capabilities by @simon-mo in https://github.com/stanford-futuredata/megablocks/pull/56
• enable arg enabled normalization of routing weights by @vchiley in https://github.com/stanford-futuredata/megablocks/pull/58
• More customizable norm for expert weights by @snarayan21 in https://github.com/stanford-futuredata/megablocks/pull/60
• Update README.md by @eltociear in https://github.com/stanford-futuredata/megablocks/pull/63
• enable custom activation functions by @vchiley in https://github.com/stanford-futuredata/megablocks/pull/65
• Skip updating load balancing loss on eval by @sedrick-keh-tri in https://github.com/stanford-futuredata/megablocks/pull/69
• Change router weight norm from in-place by @sashaDoubov in https://github.com/stanford-futuredata/megablocks/pull/70
• add mem optimized grouped glu by @vchiley in https://github.com/stanford-futuredata/megablocks/pull/66
• Add cast to tensor for DTensor inputs for groupedmlp by @eracah in https://github.com/stanford-futuredata/megablocks/pull/71
• Dtensor to all paths by @mvpatel2000 in https://github.com/stanford-futuredata/megablocks/pull/73
• Refactor dtesnor by @mvpatel2000 in https://github.com/stanford-futuredata/megablocks/pull/74
• Mem opt glu bkwd by @mvpatel2000 in https://github.com/stanford-futuredata/megablocks/pull/72
• Add dmlp registry args by @j316chuck in https://github.com/stanford-futuredata/megablocks/pull/75
• Fix default to be sparse by @mvpatel2000 in https://github.com/stanford-futuredata/megablocks/pull/76
• Fix moe_normalize_expert_weights when top_k=1 by @152334H in https://github.com/stanford-futuredata/megablocks/pull/87
• Updt triton pin by @vchiley in https://github.com/stanford-futuredata/megablocks/pull/89

New Contributors

• @simon-mo made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/56
• @snarayan21 made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/60
• @eltociear made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/63
• @sedrick-keh-tri made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/69
• @eracah made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/71
• @j316chuck made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/75
• @152334H made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/87

Full Changelog: https://github.com/stanford-futuredata/megablocks/compare/v0.5.0...v0.5.1

What's New

Several improvements to avoid CPU <> GPU device synchronizations, GLU support, and support for some new models 👀

What's Changed

• Update version by @mvpatel2000 in https://github.com/stanford-futuredata/megablocks/pull/36
• Avoid duplicate .cpu() call by @mvpatel2000 in https://github.com/stanford-futuredata/megablocks/pull/37
• Have megablocks rely on torch default precision by @mvpatel2000 in https://github.com/stanford-futuredata/megablocks/pull/39
• Add GLU support by @sashaDoubov in https://github.com/stanford-futuredata/megablocks/pull/38
• Enable generic dimentionality for input by @vchiley in https://github.com/stanford-futuredata/megablocks/pull/41
• Removing an extra size call by @bcui19 in https://github.com/stanford-futuredata/megablocks/pull/43
• Fix bug in topology kernel for ffn_hidden_size>4096. by @tgale96 in https://github.com/stanford-futuredata/megablocks/pull/47

New Contributors

• @sashaDoubov made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/38
• @bcui19 made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/43

Full Changelog: https://github.com/stanford-futuredata/megablocks/compare/v0.4.0...v0.5.0

What's Changed

• Unpack saved context once by @mvpatel2000 in https://github.com/stanford-futuredata/megablocks/pull/33
• Refactoring class hierarchy for FSDP wrapping by @tgale96 in https://github.com/stanford-futuredata/megablocks/pull/34

Full Changelog: https://github.com/stanford-futuredata/megablocks/compare/v0.3.3...v0.4.0

What's Changed

• Enable running MegaBlocks MoE without bias by @vchiley in https://github.com/stanford-futuredata/megablocks/pull/31

Full Changelog: https://github.com/stanford-futuredata/megablocks/compare/v0.3.2...v0.3.3

What's Changed

• Support for bfloat16
• Optimizations for top_k > 1
• Support for fully-sharded data parallelism
• Support tensor model parallelism when expert_parallel_world_size > num_experts
• Optimizations for activation memory
• Support activation quantization (thanks @dblalock!)
• Optimizations for SM90 (Hopper)
• Lots of bug fixes, cleanup and small optimizations

New Contributors

• @vchiley made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/9
• @deepakn94 made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/16
• @b-chu made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/19

Full Changelog: https://github.com/stanford-futuredata/megablocks/compare/v0.1...v0.3.2

What's Changed

• Fix activation quantization by @dblalock in https://github.com/stanford-futuredata/megablocks/pull/30

Full Changelog: https://github.com/stanford-futuredata/megablocks/compare/v0.3...v0.3.1

What's Changed

• Support for bfloat16
• Optimizations for top_k > 1
• Support for fully-sharded data parallelism
• Support tensor model parallelism when expert_parallel_world_size > num_experts
• Optimizations for activation memory
• Support activation quantization (thanks @dblalock!)
• Optimizations for SM90 (Hopper)
• Lots of bug fixes, cleanup and small optimizations

New Contributors

• @vchiley made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/9
• @deepakn94 made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/16
• @b-chu made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/19
• @dblalock made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/23
• @mvpatel2000 made their first contribution in https://github.com/stanford-futuredata/megablocks/pull/25

Full Changelog: https://github.com/stanford-futuredata/megablocks/compare/v0.1...v0.3

Initial release documenting repository state prior to MLSys'23 camera-ready publication.