Logistics
Table of contents
Introduction
In this course, we’ll read and discuss the latest language modeling and representation learning methods in natural language processing. This includes prominent deep learning architectures including transformers, methods of self-supervised learning and transfer learning, contrastive learning, large language models and the power of scale, emergent properties of large language models, parameter efficient fine-tuning methods, learning from few training examples and task instructions, methods for making large language models more efficient, applications to other fields, and other recent topics in contemporary NLP.
Learning Resources
Textbook
No required textbook. But if you are interested in textbooks or book chapters:
- A Primer on Neural Network Models for Natural Language Processing.
- Natural Language Processing with Transformers
- On the Opportunities and Risks of Foundation Models
We will be reading research papers from premier conferences in the field E.g., ACL, EMNLP, NAACL, ICLR, NeurIPS, ICML, …
Communication
For Submitting assignments: Canvas (https://osu.instructure.com/courses/171551).
For announcements: Teams
Please email me if you have not been added to either of these platforms.
Content
For much of the semester, each class will involve the presentation and discussion of recent important papers. The objective of the course is to instill a holistic view of the latest developments in Language Modeling and broadly NLP, and help the participants understand their broad implications.
Presenters
Each paper will be presented by a group of students each with an assigned “role”. This role defines the lens through which they read the paper and determines what they prepare for the group in-class discussion. Here are the roles we will experiment with:
Stakeholder ✍️: Act as if you’re the authors of this paper. Describes their motivation, problem definition, method and experimental findings of this paper. (time budget: 15 minutes)
Scientific Reviewer 🔎: Act like you’re a reviewer of this work. Be critical of the work, though not necessarily negative. You can follow the guidelines for NeurIPS reviewers (under “Review Content”), taking note of the example reviews included therein. (time budget: 10 minutes)
Archaeologist 🏺: Determine where this paper sits in the context of previous and subsequent work. Find and report on one prior paper that substantially influenced the current paper and one newer paper that cites this current paper. (time budget: 10 minutes)
Visionary 🔭: Propose an imaginary follow-up research project or a new application – not just based on the current but only possible due to the existence and success of the current paper. (time budget: 10 minutes)
The presentation of each role will be done individually or in a group of two depending on the complexity of the paper. In case of a group presentation the presenters may decide how to divide the work among themselves but it should be roughly equal.
Who presents what role and when? In a given class session, two papers centered around a theme will be discussed. The students will each be given a random role (determined at least 10 days before the presentation). Each role (irrespective of how many students are assigned to it) should aim for specified time budgets for each role. You’re encouraged to have slides for your role, though it is not mandatory. If you do so, I would recommend less than 7-10 slides to make sure stay within our time budget.
What slides? To minimize time spent context switching or fighting with screen sharing/projector dongles, we will have a shared pool of slides. Each role group are encouraged to title their slides with “[role emoji]: [student name]” (as in “🏺: Jane, John”) so that the slides are quickly identified during the session. If you choose to make slides, you’re not expected to prepare a full-blown presentation – they’re encouraged for visual aid and facilitating the presentation.
Non-Presenters
If you aren’t in the presenting group during a given class period: Come up with one question / discussion point about the paper (either something you’re confused about or something you’d like to hear discussed more). Submit this question to Canvas (a submission link will be provided before the class)
During the class: While only a subset of the class will participate in presenting a paper, the rest of the class is expected to come to class ready to participate in the discussions.
Grading
This is a seminar level course and instead of exams, grades will be based on leading/participating in class discussions and a final project.
Foundations Homework (5%)
Paper presentation and discussions (40%)
- 20%: Paper presentation
- Each student will lead the presentation of up to 4 sessions (depending on the size of the class). The students will be encouraged to think of themselves as the author of the paper presenting it at a conference venue. The purpose of this is to discuss the main insights and findings of the paper and connect the paper with other papers and lectures discussed in class. The presenter is also encouraged to prepare a few discussion points/questions after the presentation.
10% Active participation Each student, when not presenting, will engage in discussions about the paper. They will act as audience or reviewers of the paper. They will discuss strengths, weaknesses and possible extensions/solutions
- 10% Turn in questions and occasional quiz sheets.
- Quizes will be based on small group discussions and will be distributed occasionally in some (not all) of the sessions and must be turned in by next day after the class.
Class project (55%)
Students must complete a final research project on a topic of their choice related to the class. The students can team up with other students but the team size is limited to 2 students. (In rare cases and depending on the scope of the proposed project, a group of size 3 may be also allowed). See here for how to choose a project.
- 10%: proposal (due September 30)
- Students should submit a 1-2 page proposal for their project by week 4-5. The proposal should: state and motivate the problem, and position the proposed project within related work. The project should propose either a novel research, a novel investigation of existing methods, an extension of prior work for a specific purpose, or a new application. It should also include a brief description of the approach as well as the experimental plan (e.g., baselines, datasets, etc) to validate the effectiveness of the approach.
- 10%: project progress report
- 2-3 page document due by week 10-11 (around the time of mid-term). It should describe the project goal and related work, initial results, and the plan continuing the project.
- 10%: code
- Your project code should be clean, readable, with clear running instructions, and the results should be fully reproducible
- 10%: final project presentation
- We will dedicate the final session of the class to presentations. Depending on the size of the class this can be either a poster presentation or oral presentation. We may need to extend the class time to fully accomodate all presentations.
- 20%: final project report
- 6-8 page conference format report (e.g., ACL/EMNLP Links to an external site.) detailing the project motivation, related work, proposed approach, results, and discussion. You can think of this as a conference paper. Negative results will not be penalized, but should be accompanied with detailed analysis of why the proposed methods didn’t work and provide some additional insights into the problem.