New features
Add a function of "double_backward" simplifying the training loop (#661)

Bug fixes
Fix issue with setting of param_group for the DPOptimizer wrapper (issue 649) (#660)
Fix issue of DDP optimizer for FGC. The step function incorrectly called "original_optimizer.original_optimizer" (#662)
Replace "opt_einsum.contract" by "torch.einsum"(#663)

Bug fixes
Make the import of opt_einsum.contract (linear.py) explicit (#658).

New features
Fast Gradient Clipping and Ghost Clipping (https://github.com/pytorch/opacus/pull/656)

Bug fixes
Fix gradient shape error for DPMultiheadAttention (issue 650) (https://github.com/pytorch/opacus/pull/651)
Pass kwargs from make_private to _prepare_optimizer (https://github.com/pytorch/opacus/pull/648)
Fix BatchMemoryManager length (https://github.com/pytorch/opacus/pull/641)
Fix GPU-CPU device mismatch error in util filter_dilated_rows (https://github.com/pytorch/opacus/pull/633)
Fix Opacus's runtime error with an empty batch (issue 612) (https://github.com/pytorch/opacus/pull/631)

Bug fixes

• Fix DP MultiheadAttention (#598)
• Fix: make prv accountant robust to larger epsilons (#606)
• Fix the corner case when the optimizer has no trainable parameters (#619)

Highlight: Upgraded to PyTorch 1.13+ as required dependency

New features

• Added clipping schedulers (#556)
• Util to check per sample gradients (#532)

Bug fixes

• Align DataLoader interface with vanilla PyTorch (#543)
• Fix GDP accountant epsilon retrieval changing internal state (#541)
• Add option to specify number of steps in UniformSampler (#550)
• Fix privacy computation script (#565)

New features

• Implement the PRVAccountant based on the paper Numerical Composition of Differential Privacy (#493)
• Support nn.EmbeddingBag (#519)

Bug fixes

• Fix benchmarks (#503, #507, #508)
• Align make_private_with_epsilon with make_private (#509, #526)
• Test fixes (#513, #515, #527, #533)
• Summed discriminator losses to perform one backprop step (#474)
• Fixed issue with missing argument in MNIST example (#520)
• Functorch gradients: investigation and fix (#510)
• Support empty batches (#530)

We're glad to present Opacus v1.2, which contains some major updates to per sample gradient computation mechanisms and includes all the good stuff from the recent PyTorch releases.

Highlights

Functorch - per sample gradients for all

With the recent release of functorch it's now easy to compute per sample gradients for any module, without any limitations we've had to set before.

Here's the new default behaviour:

1. First, we check if the input module contains any layers known to be incompatible with the DP-SGD (e.g. BatchNorm). Note, that these restrictions are fundamental to how DP-SGD works and will always be revelant
2. Then, for each layer we select a method of computing per sample gradients. For performance reasons, we still use old manually written grad samplers for the layers we support and fall back to the generic functorch-based grad sampler for all other layers.

You can also force functorch-based grad sampler for every layer by passing grad_sample_mode="functorch" to PrivacyEngine.make_private() or force_functorch=False to GradSampleModule's constructor.

If you're using functorch for your training pipeline already, consider using GradSampleModuleNoOp (grad_sample_mode="no_op") . As suggested by the name, is performs no action and expects client to compute per sample gradients themselves. See our CIFAR-10 example for code demonstration.

Note, that this functionality is still in beta and we haven't fully explored it's limitations. Please report any weird behaviour or inconsistencies you encounter to out github issues, we greatly appreciate the feedback.

ExpandedWeights - yet another way to compute per sample gradients

One more exciting feature now available in core PyTorch is ExpandedWeights. This feature uses old Opacus' approach of manually-written vectorized per sample gradient computations, but achieves much better performance.

To activate ExpandedWeights pass grad_sample_mode="ew" to PrivacyEngine.make_private() or use GradSampleModuleExpandedWeights

Summary: 3 different ways to compute per sample gradients

With the recent updates, Opacus now supports 3 different ways to compute per sample gradients. Below is the quick comparison. For more details refer to the grad sample README.md

TL;DR: If you want stable implementation, use GradSampleModule (grad_sample_mode="hooks").
If you want to experiment with the new functionality, you have two options. Try
GradSampleModuleExpandedWeights(grad_sample_mode="ew") for better performance and grad_sample_mode=functorch
if your model is not supported by GradSampleModule.

Please switch back to GradSampleModule(grad_sample_mode="hooks") if you encounter strange errors or unexpexted behaviour.
We'd also appreciate it if you report these to us

† Note, that performance differences are unstable and can vary a lot depending on the exact model and batch size.
Numbers above are averaged over benchmarks with small models consisting of convolutional and linear layers.
Note, that performance differences are only observed on GPU training, CPU performance seem to be almost identical
for all approaches.

†† Layers that produce joint computations on batch samples (e.g. BatchNorm) are not allowed under any approach

Other improvements

• Fix utils.unfold2d with non-symmetric pad/dilation/kernel_size/stride (#443)
• Add support for "same" and "valid" padding for hooks-based grad sampler for convolution layers
• Improve model validation to support frozen layers and catch copied parameters (#489)
• Remove annoying logging from set_to_none (#471)
• Improved documentation (#480, #478, #482, #485, #486, #487, #488)
• Imtegration test improvements (#407, #479, #481. #473)

Improvements

• Checkpoint support (#429)
• Support for layers with mix of frozen and trainable params (#437)
• Optimized einsum (#440)
• Improved parameter sanity check (#439)

Bug Fixes

• Fix unfold2d (#443)
• Switch CI to latest PyTorch version (#434)
• Typos and editing (#430, #438, #449)

Misc

• Tutorials on distributed training (#428)

Bug fixes

• Support tied parameters (#417)
• Fix callsite sensitiveness of zero_grad() (#422, #423)
• Improve microbenchmark argument parsing and tests (#425)
• Fix opacus nn.functional import (#426)

Miscellaneous

• Add microbenchmarks (#412, #416)
• Add more badges to readme (#424)

Bug fixes

• Fix accountant when using number of steps instead of epochs
• Add params check when converting BatchNorm to GroupNorm (#390)
• Fix typo in gdp accountant mechansim name (#386)
• Fix linter errors (#392)
• Add friendly and detailed message for unsupported layers (#401)
• Run linter on nightly workflow (#399)
• Add warning for Gaussian DP accounting (#400)
• Clone replacement modules on the same device as original (#356)
• Implementing 3D dilation (#408)
• fix(batch_memory_manager): Ensures split_idxs use native python types (#410)

Miscellaneous

• Migrate nightly CircleCI flows to scheduled pipelines (#402)
• Migrate from ubuntu 16.04 to 20.04 on CircleCI (#403)

v1.1.0

New Feature

• Add support for GDP accounting in get_noise_multiplier (#303)

Bug fixes

• Conservative search for target epsilon in get_noise_multiplier (#348)
• Warn and ignore "drop_last" when set in DPDataLoader (#357)
• Fix per-layer clipping in distributed (#347)

Miscellaneous

• Update code of conduct and file headers
• Add "Support Ukraine" banner to opacus website homepage
• Lint fixes

Bug fixes

• DPOptimizer
  • Passes through .defaults field to match pytorch Optimizer (#329)
  • Better exception message in .step() when p.grad_sample=None (#331)
  • Correct closure call after applying DP noise (#330)
• Proper gradient scaling in DDP mode
• Corrections of typos and errors in tutorials

Miscellaneous

• Opacus can be installed with conda: added recipe in conda-forge (#326)
• Formatting change in accordance with black-22.1.0

Bug fixes

• Hidden states of RNN is passed to device (#314)
• Validate and fix trainable modules only (#316)

Miscellaneous

• Minor corrections and typo fixes in links, documentation, and tutorials.

We are excited to announce the release of Opacus 1.0. This release packs in lot of new features and bug fixes, and most importantly, brings forth new APIs that are simpler, more modular, and easily extensible.

We have bumped up the major version number from 0 to 1 and have introduced breaking changes; although, the major version bump also indicates a step-function upgrade in the capabilities.

What's new?

With this release we're introducing a slightly different approach to the user-facing library API. While heavily based on the old API, updated API better represents abstractions and algorithms used in DP in ML, enabling private training exactly as it's described in the papers, with no assumptions or simplifications. And in doing so we maintain our focus on high performance training.

Clearer semantics

Previously, PrivacyEngine accepted model as an argument, and then needed to be explicitly attached to optimizer. While simple, it wasn't very clear. The new syntax brings abundant clarity with an explicit make_private() method.

privacy_engine = PrivacyEngine(
    model,
    sample_rate=0.01,
    alphas=[10, 100],
    noise_multiplier=1.3,
    max_grad_norm=1.0,
)
privacy_engine.attach(optimizer)

privacy_engine = PrivacyEngine()
model, optimizer, data_loader = privacy_engine.make_private(
    module=model,
    optimizer=optimizer,
    data_loader=data_loader,
    noise_multiplier=1.1,
    max_grad_norm=1.0,
)

To avoid mutually exclusive method parameters, we're now providing separate method to initialize training loop if epsilon is to be provided instead of noise_multiplier

model, optimizer, data_loader = privacy_engine.make_private_with_epsilon(
    module=model,
    optimizer=optimizer,
    data_loader=data_loader,
    epochs=EPOCHS,
    target_epsilon=EPSILON,
    target_delta=DELTA,
    max_grad_norm=MAX_GRAD_NORM,
)

Increased focus on data handling

You might have noticed that we are now passing data loader to make_private in addition to module and optimizer. This is intentional. Batch sampling is an important component of DP-SGD (e.g. privacy accounting relies on amplification by sampling) and Poisson sampling is quite tricky to get right, so now Opacus takes control of three PyTorch training objects: model, optimizer, and data loader.

More modularised components

This release makes more functionalities modular, allowing for easy extensibility, while embracing cleaner semantics:

• model is wrapped with GradSampleModule, which computes per sample gradients.
• optimizer is wrapped with DPOptimizer, which does gradient clipping and noise addition.
• data loader is transformed to a DPDataLoader, which performs uniform-with-replacement batch sampling, as required by privacy accountant.
• Module validation and fix follows the same pattern as GradSampleModule resulting in compartmentalized validation code that is easily extensible and over-rideable.

Privacy analysis

Privacy analysis functions are now promoted into an Accounant class allowing for a more generic API. This has already allowed us to implement two accountants: RDP (default and recommended one) and Gaussian DP accountant; and will enable you to add more without having to worry about messing with the core library.

- eps, alpha = privacy_engine.get_privacy_spent(delta=target_delta)
+ eps = privacy_engine.get_epsilon(delta=target_delta)

Working around device memory

Training with Opacus consumes more memory as it needs to keep track of per-sample gradients. Opacus 0.x featured the concept of virtual steps - you could decouple the logical batch size (that defined how often model weights are updated and how much DP noise is added) and physical batch size (that defined the maximum physical batch size processed by the model at any one time). While the concept is extremely useful, it suffers from serious flaws when used with Poisson sampling. Opacus 1.0 introduces a BatchMemoryManager for your dataloader, which takes care of the logical and physical batch sizes internally.

Dynamic privacy parameters

Opacus now supports changes to the privacy parameters during training, and adjusts the privacy accounting accordingly.
Use various schedulers provided in opacus.scheduler module to adjust the amount of noise during training (the implementation mimics the interface of lr_schedulers).
For all the other parameters Opacus supports subsequent calls to make_private method, while maintaining consistent privacy accounting.

Designed to be extensible

Opacus 1.0 is designed to be flexible and extensible.

• GradSampleModule supports user-provided grad samplers for custom modules.
• DPOptimizer can easily be extended with additional or alternative functionality.
• Support for user-provided privacy accountants via optimizer hooks.
• Support for custom model validation rules.

PEP 3102

Almost all functions are now PEP 3102 compliant; meaning they only accept keyword arguments. You no longer have to memorize or be confused by the position of the arguments to be passed to a functions. This also makes the API future proof as adding non-default arguments becomes easier.

Lightning Support

Now you can add DP training to PyTorch Ligthning code. The lightning framework allows you to make the code cleaner and avoid boilerplate; simply add make_private call to configure_optimizers() method of your LightningModel. A Lightning version of MNIST task is available as a guide at examples/mnist_lightning.py.

Tutorials

We have updated all the existing tutorials and also added some new tutorials to aid migration. While the changes to the library has been significant, we expect user facing changes to be minimal and simple. Please feel free to reach out to us on our forum if you need help.

New features and bug fixes

We have also added new features and fixed some bugs along the way. Some of the notable ones are:

• Robustness against floating point attacks (#260)
• Fixing weird einsum behaviour (#242)
• Revival of compute privacy script (#251)
• Faster unfolding in Conv grad_sampler (#256)
• batch_first support for SequenceBias layer (#274)

New Features

• DDP support for faster distributed training (#196)
• Support of GRU and RNN. Refactored LSTM implementation. (#222)
• PyTorch Lightning Demo (#244)

Bug fixes

• Improve nn.Linear grad sampler memory consumption (#192)
• Update Opacus to stop using deprecated torch.set_deterministic (#197)
• Fix optimizer.step after engine.detach()
• Test fixes

Miscellaneous

• Better validation error reporting (#199)
• grad sampler type checking (#241)

New features

• Major refactoring - per-sample gradient computation is separated into its own module - GradSampleModule (#175)
• Improved RDP to (eps, delta)-DP conversion (#162)
• Multi-GPU support (#166)

Bug fixes

• Handle empty batches in Poisson sampling (#164)
• Fixed memory leak from no_grad execution (#180)

v0.13.0

New features

• PackedSequence support for DPLSTM (#150) (thanks @touqir14 !)

Miscellaneous

• Pytest moved to dev installation (#144)

v0.12.0

This version introduces a mildly-breaking change: the privacy engine will now support sampling with variable batch size, just like in the Abadi et al. paper. To accommodate this feature, we have made batch_size a kwarg (no longer positional). We are also enforcing that all kwargs must not be specified positionally. If you had code that passed kwargs positionally, you will find an error (which will be very simple to fix).

New features

• Enforce kwargs to Privacy Engine (#136).
• Fix batch construction and privacy engine (#128). (thanks @ConstanceBeguier!)
• Compute required sigma to reach (epsilon, delta) budget (#126)
• Friendly user message for unused parameters (#118).
• Print helpful message when models are not in train mode (#113)

Bug fixes

• Now the Opacus package has a __version__ attribute.
• Fix immer security issue, fix website errors
• Updated setup.py version requirements to support 3.6.8 for Windows (#108) (thanks @madhavajay!)

Miscellaneous

• Rewrote the grad_sample tests to use Hypothesis (#125). (thanks @touqir14!)

v0.11.0

New features

• Extend DPLSTM to support multilayer, dropout (#101)
• Modifications to Char LSTM name classification example
• Introduce issue templates for GitHub (#102)
• Added support for Conv3D layers

Bug fixes

• Linter fixes for Conv3D (#105)

Miscellaneous

• Make TorchCSPRNG an optional dependency (#106)
• Removed unnecessary calls to zero_grad from examples and tutorials (#96)

v0.10.1

Bug fixes

• Fix PyPI deployment (#91).

Miscellaneous

• Refactor grad sample tests (#90).
• Avoid storing activations in certain scenarios (#87)