LinearAttentionArena

kronecker-attention-pytorch

Implementation of Kronecker Attention in Pytorch

dilated-attention-pytorch

(Unofficial) Implementation of dilated attention from "LongNet: Scaling Transformers to 1,000,000...

nanotron

Minimalistic large language model 3D-parallelism training

SmallInitEmb

LayerNorm(SmallInit(Embedding)) in a Transformer to improve convergence

FLASH-pytorch

Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time"

msa-transformer

MSA Transformer reproduction code

PyDyNet

NumPy实现类PyTorch的动态计算图和神经网络框架(MLP, CNN, RNN, Transformer)

g-mlp-pytorch

Implementation of gMLP, an all-MLP replacement for Transformers, in Pytorch

sosp21_exp

RWKV-v2-RNN-Pile

RWKV-v2-RNN trained on the Pile. See https://github.com/BlinkDL/RWKV-LM for details.

MPL_Lightning

Lightning implementation of Meta Pseudo Label

point-transformer-pytorch

Implementation of the Point Transformer layer, in Pytorch

essential-BYOL

An essential implementation of BYOL in PyTorch + PyTorch Lightning

minimal-gpt-neox-20b

CO2A