adamw_bfloat16

AdamW optimizer for bfloat16 models in pytorch 🔥.

MIT License

Stars
19
View Code on GitHub
Ecosystems: Python

Bot releases are hidden (Show)

adamw_bfloat16 - v0.2.0: add implementation based on torch.compile Latest Release

Published by arogozhnikov 11 months ago

  • new implementation is faster, but not cudagraph-compatible
  • old implementation is moved to cudagraph.py
  • requires torch >= 2.0
adamw_bfloat16 - v0.1.0: basic implementation of AdamW

Published by arogozhnikov almost 3 years ago

Initial implementation of AdamW for pytorch supports cuda graphs
and has a built-in mechanism for control of learning rate, because external are unlikely to make a friendship with cuda graphs

Related Projects
quantized-training

Explore training for quantized models

16 Jul 2024 7
lion-pytorch

🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly bet...

15 Feb 2023 2,018
minimal-opt

08 Jul 2022 58
Complex-YOLOv4-Pytorch

The PyTorch Implementation based on YOLOv4 of the paper: "Complex-YOLO: Real-time 3D Object Detec...

03 Jul 2020 1,234
Adan-pytorch

Implementation of the Adan (ADAptive Nesterov momentum algorithm) Optimizer in Pytorch

25 Aug 2022 247
FlexiGen

Running large language models on a single GPU for throughput-oriented scenarios.

15 Feb 2023 9,156
ml-design-patterns

Software Architecture for ML engineers

14 Jun 2021 373
minimal-gpt-neox-20b

09 Mar 2022 126
smoothquant

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

17 Nov 2022 1,199
lion-tf

A TensorFlow implementation of the Lion optimizer

16 Feb 2023 10
diffusers-torchao

End-to-end recipes for optimizing diffusion models with torchao and diffusers (inference and FP8 ...

05 Aug 2024 166
ali-pytorch

Adversarially Learned Inference in Pytorch

24 Mar 2017 30
GaLore

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

07 Mar 2024 1,179
esgd

ESGD-M is a stochastic non-convex second order optimizer, suitable for training deep learning mod...

22 Jan 2022 56
MAE-pytorch

Unofficial PyTorch implementation of Masked Autoencoders Are Scalable Vision Learners

13 Nov 2021 2,591