Recommended Papers for Foundation Model Research

This post lists papers and resources that seem good to read first when starting foundation model research. I will update it from time to time. If you know of other papers worth reading, please let me know.

NLP

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  • Showed that reinforcement learning alone, without SFT data (DeepSeek-R1-Zero), can greatly improve the reasoning ability of LLMs.
• The Llama 3 Herd of Models
  • The technical report for Llama 3. It describes in detail how state-of-the-art LLMs were trained at the time of publication.
• Language Models are Few-Shot Learners
  • Scaled up the model and dataset, improving few-shot, one-shot, and zero-shot performance.
• Scaling Laws for Neural Language Models
  • A comprehensive analysis of scaling laws for language models.
• Language Models are Unsupervised Multitask Learners
  • Showed that a language model can handle a variety of tasks without fine-tuning. The training data is WebText.
• Improving Language Understanding by Generative Pre-Training
  • The first paper in the GPT series. It uses a decoder-only Transformer for next-word prediction. GPT-1 is fine-tuned separately for each task.
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  • The paper that proposed BERT. It uses an encoder-only Transformer to produce embeddings that take bidirectional context into account. It is pre-trained with MLM and NSP.
• Attention Is All You Need
  • The paper that proposed the Transformer.
• Sequence to Sequence Learning with Neural Networks
  • Showed that sequence transduction tasks can be learned end-to-end with an encoder-decoder architecture.

Vision & Language

• Visual Instruction Tuning
  • The paper that proposed LLaVA. It showed that incorporating visual information into an LLM enables multimodal instruction following.
• Learning Transferable Visual Models From Natural Language Supervision
  • Proposed CLIP, which is trained with contrastive learning on a dataset of image-text pairs from the web. As an application of CLIP, it showed that the model can perform zero-shot image classification.
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Proposed ViT, which splits an image into patches and feeds them into a Transformer encoder.

Audio

• Moshi: a speech-text foundation model for real-time dialogue
  • Proposed a modeling approach for full-duplex spoken dialogue models.
• Robust Speech Recognition via Large-Scale Weak Supervision
  • The paper that proposed Whisper (a speech-to-text model).
• SoundStream: An End-to-End Neural Audio Codec
  • Proposed a neural audio codec that combines a VQ-VAE with a residual vector quantizer.
• Generative Spoken Language Modeling from Raw Audio
  • The paper that proposed GSLM. It proposed building a spoken language model by converting speech into discrete tokens and training a language model on them.

HPC

• How To Scale Your Model
  • Summarizes how to think about training and running inference for LLMs efficiently. Using tools such as arithmetic intensity and the roofline model, it explains how to get the most out of TPUs and GPUs.

Robotics

• π0: A Vision-Language-Action Flow Model for General Robot Control
  • Proposed a high-performing VLA model that can serve as a policy for multiple different robots.

Evals

• Successful language model evals
  • Summarizes ideas on how to build good benchmarks.