Projects
transframer-pytorch
Implementation of Transframer, Deepmind's U-net + Transformer architecture for up to 30 seconds video generation, in Pytorch
Python - Released: 17 Aug 2022 - 66
adjacent-attention-network
Graph neural network message passing reframed as a Transformer with local attention
Python - Released: 10 Dec 2020 - 65
memory-transformer-xl
A variant of Transformer-XL where the memory is updated not with a queue, but with attention
Python - Released: 10 Jul 2020 - 45
coco-lm-pytorch
Implementation of COCO-LM, Correcting and Contrasting Text Sequences for Language Model Pretraining, in Pytorch
Python - Released: 02 Mar 2021 - 45
token-shift-gpt
Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing
Python - Released: 17 Aug 2021 - 47
retrieval-augmented-ddpm
Implementation of Retrieval-Augmented Denoising Diffusion Probabilistic Models in Pytorch
Python - Released: 01 May 2022 - 64
neural-plexer-pytorch
Implementation of Nvidia's NeuralPlexer, for end-to-end differentiable design of functional small-molecules and ligand-binding proteins, in Pytorch
Python - Released: 04 Apr 2023 - 51
flash-genomics-model
My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other hierarchical methods)
Python - Released: 18 May 2023 - 51
coordinate-descent-attention
Implementation of an Attention layer where each head can attend to more than just one token, using coordinate descent to pick topk
Python - Released: 31 Mar 2023 - 46
ITTR-pytorch
Implementation of the Hybrid Perception Block and Dual-Pruned Self-Attention block from the ITTR paper for Image to Image Translation using Transformers
Python - Released: 01 Apr 2022 - 33
remixer-pytorch
Implementation of the Remixer Block from the Remixer paper, in Pytorch
Python - Released: 24 Sep 2021 - 35
tranception-pytorch
Implementation of Tranception, an attention network, paired with retrieval, that is SOTA for protein fitness prediction
Python - Released: 02 Jun 2022 - 31
ReST-EM-pytorch
Implementations and explorations into the ReST𝐸𝑀 algorithm in the new deepmind paper "Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models"
Python - Released: 05 Jan 2024 - 39
autoregressive-linear-attention-cuda
CUDA implementation of autoregressive linear attention, with all the latest research findings
Python - Released: 07 Feb 2023 - 43
triangle-multiplicative-module
Implementation of the Triangle Multiplicative module, used in Alphafold2 as an efficient way to mix rows or columns of a 2d feature map, as a standalone package for Pytorch
Python - Released: 19 Jul 2021 - 29
metaformer-gpt
Implementation of Metaformer, but in an autoregressive manner
Python - Released: 18 Jun 2022 - 23
panoptic-transformer
Another attempt at a long-context / efficient transformer by me
Python - Released: 22 Nov 2021 - 37
AoA-pytorch
A Pytorch implementation of Attention on Attention module (both self and guided variants), for Visual Question Answering
Python - Released: 07 Nov 2020 - 40
robocat-pytorch
Implementation of Deepmind's RoboCat, Self-Improving Foundation Agent for Robotic Manipulation, in Pytorch
Python - Released: 20 Jun 2023 - 32
decomp-opt-pytorch
Implementation of DecompOpt - Controllable and Decomposed Diffusion Models for Structure-based Molecular Optimization - ICLR 2024
Python - Released: 21 Jan 2024 - 28
ESBN-pytorch
Usable implementation of Emerging Symbol Binding Network (ESBN), in Pytorch
Python - Released: 01 Jan 2021 - 23
learning-to-expire-pytorch
An implementation of Transformer with Expire-Span, a circuit for learning which memories to retain
Python - Released: 11 Oct 2020 - 33