Projects
memory-efficient-attention-pytorch
Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"
Python - Released: 03 Mar 2022 - 356
point-transformer-pytorch
Implementation of the Point Transformer layer, in Pytorch
Python - Released: 18 Dec 2020 - 587
taylor-series-linear-attention
Explorations into the recently proposed Taylor Series Linear Attention
Python - Released: 23 Dec 2023 - 88
conformer
Implementation of the convolutional module from the Conformer paper, for use in Transformers
Python - Released: 26 Jul 2020 - 359
recurrent-memory-transformer-pytorch
Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch
Python - Released: 24 Apr 2023 - 393
pytorch-custom-utils
Just some miscellaneous utility functions / decorators / modules related to Pytorch and Accelerate to help speed up implementation of new AI research
Python - Released: 12 Dec 2023 - 116
titok-pytorch
Implementation of TiTok, proposed by Bytedance in "An Image is Worth 32 Tokens for Reconstruction and Generation"
Python - Released: 17 Jun 2024 - 160
PEER-pytorch
Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind
Python - Released: 09 Jul 2024 - 109
g-mlp-pytorch
Implementation of gMLP, an all-MLP replacement for Transformers, in Pytorch
Python - Released: 18 May 2021 - 422
classifier-free-guidance-pytorch
Implementation of Classifier Free Guidance in Pytorch, with emphasis on text conditioning, and flexibility to include multiple text embedding models
Python - Released: 03 Dec 2022 - 271
deformable-attention
Implementation of Deformable Attention in Pytorch from the paper "Vision Transformer with Deformable Attention"
Python - Released: 17 Mar 2022 - 275
infini-transformer-pytorch
Implementation of Infini-Transformer in Pytorch
Python - Released: 01 May 2024 - 102
mixture-of-attention
Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
Python - Released: 21 Apr 2023 - 101
bit-diffusion
Implementation of Bit Diffusion, Hinton's group's attempt at discrete denoising diffusion, in Pytorch
Python - Released: 17 Aug 2022 - 330
gateloop-transformer
Implementation of GateLoop Transformer in Pytorch and Jax
Python - Released: 06 Nov 2023 - 86
product-key-memory
Standalone Product Key Memory module in Pytorch - for augmenting Transformer models
Python - Released: 06 Jun 2020 - 72
agent-attention-pytorch
Implementation of Agent Attention in Pytorch
Python - Released: 18 Dec 2023 - 85
spear-tts-pytorch
Implementation of Spear-TTS - multi-speaker text-to-speech attention network, in Pytorch
Python - Released: 19 Jun 2023 - 254
axial-attention
Implementation of Axial attention - attending to multi-dimensional data efficiently
Python - Released: 28 May 2020 - 348
FLASH-pytorch
Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time"
Python - Released: 28 Mar 2022 - 345
soft-moe-pytorch
Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch
Python - Released: 04 Aug 2023 - 239
se3-transformer-pytorch
Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. This specific repository is geared towards integration with eventual Alphafold2 replication.
Python - Released: 09 Jan 2021 - 254
segformer-pytorch
Implementation of Segformer, Attention + MLP neural network for segmentation, in Pytorch
Python - Released: 06 Jun 2021 - 338
equiformer-pytorch
Implementation of the Equiformer, SE3/E3 equivariant attention network that reaches new SOTA, and adopted for use by EquiFold for protein folding
Python - Released: 29 Oct 2022 - 242