So you know how the transformer works, and you know basic ML/DL, and you want to learn more about LLMs. One way to go is looking into the various "algorithmic" stuff (optimization algorithms, RL, DPO, etc). Lot's of materials on that. But the interesting stuff is (in my opinion at least) not there.
This is an attempt to collect a list of academic (or academic-like) materials that explore LLMs from other directions, and focus on the non-ML-algorithmic aspects.
-
David Chiang's Theory of Neural Networks course.
- This is not primarily LLMs, but does have substantial section on Transformers. Formal/Theory. More of a book than a course.
-
Chenyan Xiong and Daphne Ipolito's Large Language Models: Methods and Applications (fall 2024).
- Nice collection of topics.
-
Danqi Chen and Sanjeev Arora's Deep Dive into Large Language Models Fall 2024.
- Covers many aspects of creating LLMs, across the entire spectrum.
-
Diyi Yang's Human Centered LLMs
- Data, Training, Evaluation, Risks, Creativity and more.
-
Dawn Song and Dan Hendrycks's Understanding Large Language Models: Foundations and Safety
- Nice collection of topics.
-
Graham Neubig's Advanced NLP (Fall 2024)
- Includes also non-LLM materials, but some good sections on LLMs.
-
Daniel Khashabi's NLP: Self-supervised Models
- Lot's of the basics stuff, but worth looking from #12 onwards.
-
Tatsunori Hashimoto and Percy Liang's Language Models from Scratch
- This is to a large extent a technical/ML class, but has some Deep-ish dives into the nitty-gritty of LLM training that are not usually covered in other courses, so it made the cut into this list.
-
Yejin Choi's Natural Language Processing -- LLM Edition
- Week 5 onwards has some less-discussed contenet about decoding, alignment, etc.
-
Tal Linzen's Natural Language Understanding
- Evaluation, interpretability, heuristics, comaprisons to humans, etc, from a more linguistic standpoint. Not so much about creating LLMs, but about assessing them.
-
Robin Jia's Science of Large Language Models seminar.
- Has a nice reading list arranged according to a nice progression of topics. I wish it was a class and not a seminar.
-
Danqi Chen's Understanding Large Language Models.
- From 2022, but many section still relevant. I wish to see the 2025 edition.
-
Michael Hahn's Aligning Language Models with Human Preferences: Methods and Challenges
-
Tal Linzen's Language models: cognitive plausibility and sample efficiency
- Stanford's CS25 has a sequence of interesting invited talks.