Empirical Experiments in "Feature Learning in Infinite-width Neural Networks"

This repo contains code to replicate our experiments (Word2Vec, MAML) in our paper

Feature Learning in Infinite-Width Neural Networks Greg Yang, Edward Hu

In short, the code here will allow you to train feature learning infinite-width neural networks on Word2Vec and on Omniglot (via MAML).

Our results on Word2Vec:

Our Results on MAML:

Please see the README in individual folders for more details.

This is the 4th paper in the Tensor Programs series ([0][1][2][3]). Also see here for code in previous papers for calculating the GP and NTK limits of wide neural networks.