An implementation of Transformer with Expire-Span, a circuit for learning which memories to retain
MIT License
An attempt to merge ESBN with Transformers, to endow Transformers with the ability to emergently ...
Implementation of Memformer, a Memory-augmented Transformer, in Pytorch
Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and ...
An implementation of local windowed attention for language modeling
Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Fun...
Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning
Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architectu...
Implementation of Infini-Transformer in Pytorch
Pytorch implementation of Compressive Transformers, from Deepmind
Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attenti...
Implementation of Block Recurrent Transformer - Pytorch
Implementation of Feedback Transformer in Pytorch
A variant of Transformer-XL where the memory is updated not with a queue, but with attention
Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch
Implementation of Hierarchical Transformer Memory (HTM) for Pytorch