Table of contents

1. Schedule

Schedule

Course Overview & Language Modeling Basics (August 26)

Slides:

Course Oveview

Slides:

Transformers / Pretraining / Finetuning

Reading Material

• Attention is all you need (2017) [link]

• BERT, Pre-training of Deep Bidirectional Transformers for Language Understanding [link]

Optional readings:

• The Illustrated Transformer [link]

• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) [link]

• T5 [link]

• The Illustrated GPT2 [link]

• What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? [link]

• BART [link]

• RoBERTa, A Robustly Optimized BERT Pretraining Approach [link]