rela-transformer

Implementation of a Transformer using ReLA (Rectified Linear Attention) from https://arxiv.org/abs/2104.07012

MIT License

Downloads
81
Stars
49
Committers
1
View Code on GitHub
Ecosystems: Python

Commit Statistics

Past Year

All Time

Total Commits
14
Total Committers
1
Avg. Commits Per Committer
14.0
Bot Commits
0

Issue Statistics

Past Year

All Time

Total Pull Requests
0
0
Merged Pull Requests
0
0
Total Issues
0
1
Time to Close Issues
N/A
N/A
Package Rankings
Top 25.46% on Pypi.org
Related Projects
transformer-in-transformer

Implementation of Transformer in Transformer, pixel level attention paired with patch level atten...

02 Mar 2021 300
feedback-transformer-pytorch

Implementation of Feedback Transformer in Pytorch

02 Feb 2021 104
lie-transformer-pytorch

Implementation of Lie Transformer, Equivariant Self-Attention, in Pytorch

22 Dec 2020 87
memory-transformer-xl

A variant of Transformer-XL where the memory is updated not with a queue, but with attention

10 Jul 2020 45
linformer

Implementation of Linformer for Pytorch

28 Jun 2020 248
h-transformer-1d

Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning

28 Jul 2021 153
RQ-Transformer

Implementation of RQ Transformer, proposed in the paper "Autoregressive Image Generation using Re...

11 Mar 2022 85
local-attention

An implementation of local windowed attention for language modeling

05 Jul 2020 373
FLASH-pytorch

Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time"

28 Mar 2022 345
point-transformer-pytorch

Implementation of the Point Transformer layer, in Pytorch

18 Dec 2020 587
performer-pytorch

An implementation of Performer, a linear attention-based transformer, in Pytorch

03 Oct 2020 1,084
molecule-attention-transformer

Pytorch reimplementation of Molecule Attention Transformer, which uses a transformer to tackle th...

30 Nov 2020 58
fast-transformer-pytorch

Implementation of Fast Transformer in Pytorch

23 Aug 2021 171
long-short-transformer

Implementation of Long-Short Transformer, combining local and global inductive biases for attenti...

07 Jul 2021 116
block-recurrent-transformer-pytorch

Implementation of Block Recurrent Transformer - Pytorch

07 Feb 2023 212