soft-mixture-of-experts
PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)
MIT License
Stars
30
Ecosystems:
PyTorch
Bot releases are hidden (Show)
What's Changed
- FIx dispatch_weights softmax dimension by @Chris-Nicholls in https://github.com/fkodom/soft-mixture-of-experts/pull/1
New Contributors
- @Chris-Nicholls made their first contribution in https://github.com/fkodom/soft-mixture-of-experts/pull/1
Full Changelog: https://github.com/fkodom/soft-mixture-of-experts/compare/0.1.0...0.2.0
soft-mixture-of-experts
- 0.1.0
Published by fkodom about 1 year ago
First full release for PyPI
-
SoftMoElayer - Transformer layers with Soft MoE
- ViT models, with and without Soft MoE
- ablations from paper
soft-mixture-of-experts
- 0.1.0rc1
Published by fkodom about 1 year ago
First prerelease candidate for PyPI