perceiver-ar-pytorch

Implementation of Perceiver AR, Deepmind's new long-context attention network based on Perceiver architecture, in Pytorch

MIT License

Downloads
187
Stars
86
Committers
2
View Code on GitHub
Ecosystems: Python

Commit Statistics

Past Year

All Time

Total Commits
2
19
Total Committers
2
2
Avg. Commits Per Committer
1.0
9.5
Bot Commits
0
0

Issue Statistics

Past Year

All Time

Total Pull Requests
0
2
Merged Pull Requests
0
1
Total Issues
0
8
Time to Close Issues
N/A
about 1 month
Package Rankings
Top 16.4% on Pypi.org
Related Projects
long-short-transformer

Implementation of Long-Short Transformer, combining local and global inductive biases for attenti...

07 Jul 2021 116
block-recurrent-transformer-pytorch

Implementation of Block Recurrent Transformer - Pytorch

07 Feb 2023 212
bidirectional-cross-attention

A simple cross attention that updates both the source and target in one step

27 Mar 2022 143
hourglass-transformer-pytorch

Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

08 Nov 2021 74
local-attention

An implementation of local windowed attention for language modeling

05 Jul 2020 373
PaLM-rlhf-pytorch

Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architectu...

09 Dec 2022 7,595
g-mlp-pytorch

Implementation of gMLP, an all-MLP replacement for Transformers, in Pytorch

18 May 2021 422
omninet-pytorch

Implementation of OmniNet, Omnidirectional Representations from Transformers, in Pytorch

02 Mar 2021 53
g-mlp-gpt

GPT, but made only out of MLPs

20 May 2021 86
EasyContext

Memory optimization and training recipes to extrapolate language models' context length to 1 mill...

05 Apr 2024 574
Perceiver

Implementation of Perceiver, General Perception with Iterative Attention in TensorFlow

12 Apr 2021 86
perceiver-pytorch

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

05 Mar 2021 1,049
feedback-transformer-pytorch

Implementation of Feedback Transformer in Pytorch

02 Feb 2021 104
dilated-attention-pytorch

(Unofficial) Implementation of dilated attention from "LongNet: Scaling Transformers to 1,000,000...

09 Jul 2023 47
perceiver-io

Unofficial implementation of Perceiver IO

28 Aug 2021 117