Schedule

Course Overview & Language Modeling Basics (August 26)

Slides:

Course Oveview

Pretraining - Architectures and Methods (September 9)

Slides:

Transformers / Pretraining / Finetuning

Reading Material

  • Attention is all you need (2017) [link]

  • BERT, Pre-training of Deep Bidirectional Transformers for Language Understanding [link]

Optional readings:

  • The Illustrated Transformer [link]

  • The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) [link]

  • T5 [link]

  • The Illustrated GPT2 [link]

  • What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? [link]

  • BART [link]

  • RoBERTa, A Robustly Optimized BERT Pretraining Approach [link]

Efficiency - Training (LoRA) and Inference(Quantization) (September 16)

Slides:

Efficiency

Reading Material

  • LoRA: Low-Rank Adaptation of Large Language Models (2021) [link]

  • LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale [link]

Optional readings:

  • QLoRA: Efficient Finetuning of Quantized LLMs [link]

  • LoRA+: Efficient Low Rank Adaptation of Large Models [link]

Inference Algorithms (In-Context Learning and Chain-of-Thought) (September 23)

Slides:

Inference Methods

Reading Material

  • Language Models are Few-Shot Learners [link]

  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models [link]

Optional readings:

  • Making Pre-trained Language Models Better Few-shot Learners (2021) [link]

  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022) [link]

  • Data Distributional Properties Drive Emergent In-Context Learning in Transformers (2022) [link]

  • Towards understanding chain-of-thought prompting: An empirical study of what matters (2022) [link]

  • List of recent CoT papers (2024) [link]

Instruction Following (September 30)

Slides:

Instruction Following

Reading Material

  • Finetuned Language Models Are Zero-Shot Learners [link]

  • Training language models to follow instructions with human feedback [link]

Optional readings:

  • Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2 [link]

  • The Llama 3 Herd of Models (Sec 4 and the relevant portion of Sec 5) [link]

  • Fundamental Limitations of Alignment in Large Language Models [link]

  • Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback [link]

Scaling (October 7)

Slides:

Scaling

Reading Material

  • Training Compute-Optimal Large Language Models [link]

  • Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters [link]

Optional readings:

  • A Hitchhiker’s Guide to Scaling Law Estimation [link]
  • Scaling Laws for Predicting Downstream Performance in LLMs [link]
  • Scaling Laws for Multilingual Language Models [link]

Beyond RL/HF (October 14)

Slides:

BeyondRLHF

Reading Material

  • DPO [link]

  • Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision [link]

Optional readings:

  • Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing [link]

  • Awesome RLHF reading list [link]

Ethics and Safety (October 21)

Slides:

Ethics/Safety

Reading Material

  • Taxonomy of Risks posed by Language Models [link]

  • Jailbroken: How Does LLM Safety Training Fail? [link]

Optional readings:

  • Ethics in AI (UW Course reading list) [link]

Retrieval / Long Context (October 28)

Slides:

Retrieval/LongContext

Reading Material

  • Reliable, Adaptable, and Attributable Language Models with Retrieval [link]

  • How to Train Long-Context Language Models (Effectively) [link]

Optional readings:

  • ACL 2023 Tutorial: Retrieval-based Language Models and Applications [link]

  • Scaling Retrieval-Based Language Models with a Trillion-Token Datastore [link]

Tokenization (November 4)

Slides:

Tokenization

Reading Material

  • Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP [link]

Optional readings:

  • The Foundations of Tokenization: Statistical and Computational Concerns [link]

Multimodal Language Models (November 18)

Slides:

Multimodal

Reading Material

Optional readings: