GradientAccumulator

Accumulated Gradients for TensorFlow 2

MIT License

Downloads
686
Stars
50
View Code on GitHub Visit Website
Ecosystems: Keras, Tensorflow

Bot releases are hidden (Show)

GradientAccumulator - v0.5.2 Latest Release

Published by andreped about 1 year ago

New feature

The main feature of this patch release is that AccumBN can now be used as drop-in replacement for any BatchNormalization layer, even for pretrained networks. Old weights are sufficiently transferred and documentations have been updated to include how to do this.

import tensorflow as tf
from gradient_accumulator import GradientAccumulateModel
from gradient_accumulator.layers import AccumBatchNormalization
from gradient_accumulator.utils import replace_batchnorm_layers

accum_steps = 4

# replace BN layer with AccumBatchNormalization
model = tf.keras.applications.MobileNetV2(input_shape(28, 28, 3))
model = replace_batchnorm_layers(model, accum_steps=accum_steps)

# add gradient accumulation to existing model
model = GradientAccumulateModel(accum_steps=accum_steps, inputs=model.input, outputs=model.output)

What's Changed

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.5.1...v0.5.2

GradientAccumulator - v0.5.1

Published by andreped over 1 year ago

Announcement

This patch release adds support for all tf versions 2.2-2.12 and Python 3.6-3.11. The model wrapper should work as intended for all combinations, whereas the optimizer is only compatible with tf>=2.8 and with poorer performance for tf>=2.10.

What's Changed

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.5.0...v0.5.1

GradientAccumulator - v0.5.0

Published by andreped over 1 year ago

New feature!

  • Multi-GPU support has now been added! Support has been added for both optimizer and model wrappers.
  • Note that only SGD works with the model wrapper, due to challenges controlling the optimizer state during gradient accumulatiom

What's Changed

New Contributors

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.4.2...v0.5.0

GradientAccumulator - v0.4.2

Published by andreped over 1 year ago

What's Changed

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.4.1...v0.4.2

GradientAccumulator - v0.4.1

Published by andreped over 1 year ago

What's Changed

New API

You can now use gradient accumulation with the AccumBatchNormalization layer:

from gradient_accumulator import GradientAccumulateModel, AccumBatchNormalization
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# define model and add accum BN layer
model = Sequential()
model.add(Dense(32, activation="relu"))
model.add(AccumBatchNormalization(accum_steps=8))
model.add(Dense(10))

# add gradient accumulation to the rest of the model
model = GradientAccumulateModel(accum_steps=8, inputs=model.input, outputs=model.output)

More information about remarks and usage can be found at gradientaccumulator.readthedocs.io

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.4.0...v0.4.1

GradientAccumulator - v0.4.0

Published by andreped over 1 year ago

What's Changed

  • Added custom AccumBatchNormalization layer with gradient accumulation support.
  • Added more unit tests -> code coverage = 99%
  • Made proper documentations which is hosted at gradientaccumulator.readthedocs.io/
  • Reduced runtime on several unit tests to make CI jobs faster
  • Added fix for protobuf for tfds in CIs
  • Reworked README - moved most stuff to the Documentations + added CI section w/badges
  • Header image by @jpdefrutos in https://github.com/andreped/GradientAccumulator/pull/51

New Contributors

New API feature

from gradient_accumulator import AccumBatchNormalization

layer = AccumBatchNormalization(accum_steps=4)

Can be used as a regular Keras BatchNormalization layer, but with reduced functionality.

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.3.2...v0.4.0

GradientAccumulator - v0.3.2

Published by andreped over 1 year ago

What's changed

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.3.1...v0.3.2

New Contributors

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.3.1...v0.3.2

How to install?

pip install gradient-accumulator==0.3.2

New API feature

Custom Batch Normalization layer

from gradient_accumulator.layers import AccumBatchNormalization

model = Sequential()
model.add(AccumBatchNormalization())
GradientAccumulator - v0.3.1

Published by andreped over 1 year ago

What's changed

  • Simplified imports - can now directly import accumulators without middle step
  • Renamed GAModelWrapper -> GradientAccumulateModel
  • Renamed GAOptimizerWrapper -> GradientAccumulateOptimizer
  • Updated README and all CIs accordingly
  • Deprecated tensorflow==2.2, due to tensorflow-addons incompatiblity. Now tf >= 2.3 supported.

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.3.0...v0.3.1

How to install?

pip install gradient-accumulator==0.3.1

New API!

Model wrapper:

from gradient_accumulator import GradientAccumulateModel

model = Model(...)
model = GradientAccumulateModel(accum_steps=4, inputs=model.input, outputs=model.output)

Optimizer wrapper:

from gradient_accumulator import GradientAccumulateModel

opt = tf.keras.optimizers.SGD(1e-2)
opt = GradientAccumulateOptimizer(accum_steps=4, optimizer=opt)
GradientAccumulator - v0.3.0

Published by andreped over 1 year ago

What's changed

How to install?

pip install gradient-accumulator==0.3.0

How to use?

Method Usage
GAModelWrapper model = GAModelWrapper(accum_steps=4, inputs=model.input, outputs=model.output)
GAOptimizerWrapper opt = GAOptimizerWrapper(accum_steps=4, optimizer=tf.keras.optimizers.Adam(1e-3))

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.2.2...v0.3.0

GradientAccumulator - v0.2.2

Published by andreped almost 2 years ago

This is a minor patch release.

What's changed:

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.2.1...v0.2.2

GradientAccumulator - v0.2.1

Published by andreped about 2 years ago

This is a minor patch release.

What's changed:

  • Fixed typo by renaming use_acg to use_agc.
GradientAccumulator - v0.2.0

Published by andreped over 2 years ago

What's Changed

New Contributors

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.1.5...v0.2.0

GradientAccumulator - v0.1.5

Published by andreped over 2 years ago

Changes:

  • Added mixed precision support (only float16 currently, which is compatible with NVIDIA GPUs)
  • Added adaptive gradient clipping support (normalization-free approach which works with GA)
  • Added CI test for AGC

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.1.4...v0.1.5

GradientAccumulator - v0.1.4

Published by andreped over 2 years ago

Zenodo DOI release and updated README to contain updated documentation regarding installation and usage.

Changes:

  • Renamed n_gradients to accum_steps.
  • Added citation policy and Zenodo citation

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.1.3...v0.1.4

GradientAccumulator - v0.1.3

Published by andreped over 2 years ago

GradientAccumulator is now available on PyPI :
https://pypi.org/project/gradient-accumulator/#files

Changes:

  • Added experimental mixed precision support
  • Added support for TF >= 2.2
  • Added support for Python >3.6
  • Added pytest to CI for unit testing
  • Added CI test for mixed precision
  • Added CI test for multi-input-output models
  • Added CI test for optimizer invariance
  • Added CI test for basic mnist training
  • Added CI test to verify that we get expected result for GA vs regular batch training

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.1.2...v0.1.3

GradientAccumulator - v0.1.2

Published by andreped over 2 years ago

Changes:

  • Fixed critical bug regarding gradient updates (use MEAN reduction, instead of SUM reduction)
  • Now, GA yields identical results compared to regular batch training
  • Added unit tests with pytest to yield AssertionError if results are different
  • Added compatibility with sample_weight - now GAModelWrapper should be fully compatible with model.compile/fit

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.1.1...v0.1.2

GradientAccumulator - v0.1.1

Published by andreped over 2 years ago

Changes:

  • Swapped optimizer wrapper solution with Model wrapper solution
  • Enables adding gradient accumulation support for "any" tf.keras.Model by simply overloading the train_step method
  • Added convenience class GAModelWrapper that handles all this for you - just provide the model!
  • Solution should also be more compatible with older TF versions, as train_step overloading was added already in TF 2.2.

Full Changelog: https://github.com/andreped/GradientAccumulator/compare/v0.1.0...v0.1.1

GradientAccumulator - v0.1.0

Published by andreped over 2 years ago

First release of the GradientAccumulator package that enables usage of accumulated gradients in TensorFlow 2.x by simply wrapping an optimizer.

Currently, compatible with Python 3.7-3.9, tested with TensorFlow 2.8.0 and 2.9.1, and cross-platform compatible (Windows, Ubuntu, and macOS).

Full Changelog: https://github.com/andreped/GradientAccumulator/commits/v0.1.0

Package Rankings
Top 11.93% on Pypi.org
Badges
Extracted from project README
Pip Downloads PyPI version License DOI codecov Documentation Status CI
Related Projects
GPflow

Gaussian processes in TensorFlow

14 Jan 2016 1,794
DLTK

Deep Learning Toolkit for Medical Image Analysis

02 May 2017 1,426
Gradient-Centralization-TensorFlow

Instantly improve your training performance of TensorFlow models with just 2 lines of code!

15 Feb 2021 105
adapt

Awesome Domain Adaptation Python Toolbox

25 Jun 2020 302
tf-explain

Interpretability Methods for tf.keras models with Tensorflow 2.x

15 Jul 2019 1,009