yet-another-retnet

A simple but robust PyTorch implementation of RetNet from "Retentive Network: A Successor to Transformer for Large Language Models" (https://arxiv.org/pdf/2307.08621.pdf)

MIT License

Stars
80
View Code on GitHub
Ecosystems: PyTorch, Python

Statistics for this project are still being loaded, please check back later.

Related Projects
Megatron-LM

Ongoing research training transformer models at scale

21 Mar 2019 8,839
bottleneck-transformer-pytorch

Implementation of Bottleneck Transformer in Pytorch

28 Jan 2021 669
minGPT-flax

GPT implementation in Flax

30 Sep 2021 16
dilated-attention-pytorch

(Unofficial) Implementation of dilated attention from "LongNet: Scaling Transformers to 1,000,000...

09 Jul 2023 47
se3-transformer-pytorch

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. This specific repo...

09 Jan 2021 253
RetNet

An implementation of "Retentive Network: A Successor to Transformer for Large Language Models"

19 Jul 2023 1,160
memorizing-transformers-pytorch

Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and ...

21 Mar 2022 624
minGPT

A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training

17 Aug 2020 19,926
RETRO-pytorch

Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

15 Jan 2022 850
block-recurrent-transformer-pytorch

Implementation of Block Recurrent Transformer - Pytorch

07 Feb 2023 212
rotary-embedding-torch

Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch

29 Jun 2021 535
kanformer

Naively combining transformers and Kolmogorov-Arnold Networks to learn and experiment

05 May 2024 12
torchinfo

View model summaries in PyTorch!

16 Mar 2020 2,523
flash-attention

Fast and memory-efficient exact attention

19 May 2022 11,791
RepVGG

RepVGG: Making VGG-style ConvNets Great Again

31 Dec 2020 3,313