yet-another-retnet

What's Changed

• Bug fix: decay mask for bf16, bf32 by @fkodom in https://github.com/fkodom/yet-another-retnet/pull/25

Full Changelog: https://github.com/fkodom/yet-another-retnet/compare/0.5.0...0.5.1

Significant efficiency improvements to the chunkwise formulation, thanks to @leor-c 🎉

What's Changed

• a more efficient computation of the state in the chunkwise formulation by @leor-c in https://github.com/fkodom/yet-another-retnet/pull/22

Full Changelog: https://github.com/fkodom/yet-another-retnet/compare/0.4.2...0.5.0

What's Changed

• Slightly more efficient / cleaner implementation of the chunkwise relative pos. enc. by @leor-c in https://github.com/fkodom/yet-another-retnet/pull/21

New Contributors

• @leor-c made their first contribution in https://github.com/fkodom/yet-another-retnet/pull/21

Full Changelog: https://github.com/fkodom/yet-another-retnet/compare/0.4.1...0.4.2

What's Changed

• Performance Tuning by @fkodom in https://github.com/fkodom/yet-another-retnet/pull/20
• float32 -> 32-true by @fkodom in https://github.com/fkodom/yet-another-retnet/pull/16
• CPU benchmark support by @fkodom in https://github.com/fkodom/yet-another-retnet/pull/19

Full Changelog: https://github.com/fkodom/yet-another-retnet/compare/0.4.0...0.4.1

What's Changed

• Fixed issue where the dimensions of the Group norm seems to be incorrect by @draguve in https://github.com/fkodom/yet-another-retnet/pull/11

New Contributors

• @draguve made their first contribution in https://github.com/fkodom/yet-another-retnet/pull/11

Full Changelog: https://github.com/fkodom/yet-another-retnet/compare/0.3.1...0.4.0

What's Changed

• bug fix: change F.relu to F.silu by @Dongyeongkim in https://github.com/fkodom/yet-another-retnet/pull/7

New Contributors

• @Dongyeongkim made their first contribution in https://github.com/fkodom/yet-another-retnet/pull/7

Full Changelog: https://github.com/fkodom/yet-another-retnet/compare/0.3.0...0.3.1

More streamlined support for training

• example training script
• RetNet.forward is no longer just a wrapper for RetNet.forward_parallel. It accepts inputs, labels Tensors, and returns a loss value.

  class RetNet:
      ...
      def forward(self, inputs: Tensor, labels: Tensor) -> Tensor:
          pred = self.forward_parallel(inputs)
          criterion = nn.CrossEntropyLoss()
          return criterion(rearrange(pred, "b n c -> (b n) c"), labels.flatten())
  

• include example TorchData datapipe -- top 100 project gutenberg books
• example streaming text generation with trained RetNet

What's Changed

• Chunkwise Formulation by @fkodom in https://github.com/fkodom/yet-another-retnet/pull/4

Full Changelog: https://github.com/fkodom/yet-another-retnet/compare/0.1.3...0.2.0

Set default layer_norm_eps=1e-6, as updated in the official implementation:
https://github.com/microsoft/torchscale/commit/2c29de0fb3e5e559181f0fb4854330c5b35961cd

Remove extra complex conjugation from the relative position embedding.
Reference: https://github.com/microsoft/torchscale/issues/49

Bug fix for automatic PyPI version resolver 😞

First stable public release

• End-to-end RetNet module
• Layer modules (e.g. RetNetDecoderLayer, MultiScaleRetention)
• Low-level retention ops (e.g. retention_parallel, retention_recurrent)
• Inference benchmarks reproduced

First release candidate for PyPI