Modern LLM Study Group Resources

This repository is for the fastai Modern LLM paper reading study group. Here you can find the papers we covered along with any extra resources.

We are working our way through the seminal LLM papers starting with the GPT-3 paper, Language Models are Few-Shot Learners.

The plan is to read our way through all the modern LLM methods mentioned by Andrej Karpathy in his The State of GPT talk, along with any new developments since then.

The study group is coordinated through the fastai discord in the #cluster-of-stars text channel and currently meets weekly on Fridays at 2300 UTC (7pm Eastern) in the #fastai-study-groups voice channel.

LLM Paper Reading List

Each paper has its own ReadMe with a direct link, summary, further reading (for most papers), and some supporting materials in the section references folder.

Intro to Modern LLMs

1. Language Models are Few-Shot Learners
2. Finetuned Language Models Are Zero-Shot Learners
3. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
4. Training language models to follow instructions with human feedback
5. LoRA: Low-Rank Adaptation of Large Language Models
6. Evaluating Large Language Models Trained on Code

Retrieval, Chain of Thought, & Tool Use

7. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
8. Atlas: Few-shot Learning with Retrieval Augmented Language Models
9. In-Context Retrieval-Augmented Language Models
10. ReAct: Synergizing Reasoning and Acting in Language Models
11. Toolformer: Language Models Can Teach Themselves to Use Tools
12. SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking
13. Chain of Papers: Multiple Chain of Thought Papers
14. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
15. Self-RAG: Learning to Retrieve, Generate and Critique through Self-Reflections
16. TeacherLM: Teaching to Fish Rather Than Giving the Fish: Language Modeling Likewise

Pretraining Data

17. The Pile: An 800GB Dataset of Diverse Text for Language Modeling
18. TinyStories: How Small Can Language Models Be and Still Speak Coherent English
19. LLaMA: Open and Efficient Foundation Language Models
20. D4: Improving LLM Pretraining via Document De-Duplication and Diversification
21. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
22. Training Data for the Price of a Sandwich: Common Crawl Impact on Generative AI
23. How to Train Data-Efficient LLMs

Synthetic Data

24. Textbooks Are All You Need and Textbooks Are All You Need II: phi-1.5 technical report
25. Cosmopedia

RLHF (Reinforcement Learning from Human Feedback)

26. Training Language Models to Follow Instructions with Human Feedback
27. Constitutional AI: Harmlessness from AI Alignment
28. Direct Preference Optimization: Your Language Model is Secretly a Reward Model
29. KTO: Model Alignment as Prospect Theoretic Optimization
30. ORPO: Monolithic Preference Optimization without Reference Model
31. RewardBench: Evaluating Reward Models for Language Modeling

Finetuning

32. Orca: Progressive Learning from Complex Explanation Traces of GPT-4
33. QLoRA: Efficient Finetuning of Quantized LLMs