Normcore LLM Reads

normcore-llm.md

Anti-hype LLM reading list

Goals: Add links that are reasonable and good explanations of how stuff works. No hype and no vendor content if possible. Practical first-hand accounts of models in prod eagerly sought.

Foundational Concepts

Pre-Transformer Models

• The Illustrated Word2vec - A Gentle Intro to Word Embeddings in Machine Learning (YouTube)
• Transformers as Support Vector Machines
• Survey of LLMS
• Deep Learning Systems
• Fundamental ML Reading List

Building Blocks

• What are embeddings
• Concepts from Operating Systems that Found their way into LLMS
• Talking about Large Language Models
• Language Modeling is Compression
• Vector Search - Long-Term Memory in AI
• Eight things to know about large language models
• The Bitter Lesson
• The Hardware Lottery
• The Scaling Hypothesis
• Tokenization
• LLM Course

Foundational Deep Learning Papers (in semi-chronological order)

• Seq2Seq
• Attention is all you Need
• BERT
• GPT-1
• Scaling Laws for Neural Language Models
• T5
• GPT-2: Language Models are Unsupervised Multi-Task Learners
• InstructGPT: Training Language Models to Follow Instructions
• GPT-3: Language Models are Few-Shot Learners

The Transformer Architecture

• Transformers from Scratch
• Transformer Math
• Five Years of GPT Progress
• Lost in the Middle: How Language Models Use Long Contexts

Attention

• Self-attention and transformer networks
• Attention
• Understanding and Coding the Attention Mechanism
• Attention Mechanisms
• Keys, Queries, and Values

GPT

• What is ChatGPT doing and why does it work
• My own notes from a few months back.
• Karpathy's The State of GPT (YouTube)
• OpenAI Cookbook

Significant OSS Models

• Llama2
• Mistral7B
  • Mixtral
• Phi2
• Falcon7B

LLMs in 2023

• Catching up on the weird world of LLMS
• How open are open architectures?
• Building an LLM from Scratch
• Large Language Models in 2023 and Slides
• Timeline of Transformer Models
• Large Language Model Evolutionary Tree

Training Data

• What's in my Big Data
• "The “it” in AI models is the dataset."
• Extracting Training Data from ChatGPT

Pre-Training

• Why host your own LLM?
• How to train your own LLMs
• Hugging Face Resources on Training Your Own
• Training Compute-Optimal Large Language Models
• Opt-175B Logbook

RLHF and DPO

• RLHF
  • Supervised Fine-tuning
  • How Abilities in LLMs Are Affected by SFT
• Instruction-tuning for LLMs: Survey
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• RLHF and DPO Compared

Fine-Tuning and Compression

• The Complete Guide to LLM Fine-tuning
• LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Language - Really great overview of SOTA fine-tuning techniques
• On the Structural Pruning of Large Language Models
• Quantiztion
  • A Gentle Introduction to 8-bit matrix multiplication
  • Which Quantization Method is Right for You?
  • Survey of Quantization for Inference
• PEFT
  • Fine-tuning with LoRA and QLoRA
  • Adapters
  • Motivation for Parameter-Efficient Fine-tuning

Small and Local LLMs

• How is LlamaCPP Possible?
• How to beat GPT-4 with a 13-B Model
• Efficient LLM Inference on CPUs
• Tiny Language Models Come of Age
• Efficiency LLM Spectrum
• TinyML at MIT

Deployment and Production

• Building LLM Applications for Production
• Challenges and Applications of Large Language Models
• All the Hard Stuff Nobody talks about when building products with LLMs
• Scaling Kubernetes to run ChatGPT
• Numbers every LLM Developer should know
• Against LLM Maximalism
• A Guide to Inference and Performance
• (InThe)WildChat: 570K ChatGPT Interaction Logs In The Wild
• The State of Production LLMs in 2023
• Machine Learning Engineering for successful training of large language models and multi-modal models.
• Fine-tuning RedPajama on Slack Data

LLM Inference and K-V Cache

• LLM Inference Performance Engineering: Best Practices
• How to Make LLMs go Fast
• Transformer Inference Arithmetic
• Which serving technology to use for LLMs?
• Speeding up the K-V cache
• Large Transformer Model Inference Optimization

Prompt Engineering and RAG

• On Prompt Engineering
• Prompt Engineering Versus Blind Prompting
• Building RAG-Based Applications for Production
• Full Fine-Tuning, PEFT, or RAG?
• Prompt Engineering Guide

GPUs

• The Best GPUS for Deep Learning 2023
• Making Deep Learning Go Brr from First Principles
• Everything about Distributed Training and Efficient Finetuning
• Training LLMs at Scale with AMD MI250 GPUs
• GPU Programming

Evaluation

• Evaluating ChatGPT
• ChatGPT: Jack of All Trades, Master of None
• What's Going on with the Open LLM Leaderboard
• Challenges in Evaluating AI Systems
• LLM Evaluation Papers
• Evaluating LLMs is a MineField

Eval Frameworks

• HELM
• LM Eval Harness
• LmSys Chatbot Arena

UX

• Generative Interfaces Beyond Chat (YouTube)
• Why Chatbots are not the Future
• The Future of Search is Boutique
• As a Large Language Model, I
• Natural Language is an Unnatural Interface

What's Next?

Thanks to everyone who added suggestions on Twitter, Mastodon, and Bluesky.

Do you think it's worth annotating with dates of the articles / papers / videos?

Why Chatbots Are Not the Future

Great one!

It seems that the gzip approach, altough really cool, was 'optimistic' and thus overhyped, see: https://kenschutte.com/gzip-knn-paper/ (basiccaly they confused k in k-nn and top-k accuracy, reporting top-2 accuracy). More recent studies found that it is, as expected, on 'bag of words' performance level Gzip versus bag-of-words for text classification.

I don't know if you intend to (or are even interested) but I am on the look out for "usecases for normies".

Yeah that was my read on it as well but I'm also very interested in it as a general theoretical approach and baseline, even if this particular implementation doesn't work.

Do you think it's worth annotating with dates of the articles / papers / videos?

Maybe would be helpful but I explicitly picked stuff that I thought wouldn't age and/or where the recency didn't matter because the fundamentals are timeless.

Patterns for Building LLM-based Systems & Products

In my opinion, this is a super in depth article that covers many of the categories and deserves a place in the reading list.

Against LLM maximalism (disclaimer: I work at Explosion)

Patterns for Building LLM-based Systems & Products In my opinion, this is a super in depth article that covers many of the categories and deserves a place in the reading list.

detailed blog

The Illustrated Transformer - maybe redundant from some of the other transformers content here, but is very well-written and strikes a good balance between prose and visual aids.

ChatGPT Sucks at Being a Testing Expert
https://www.satisfice.com/download/appendix-chatgpt-sucks-at-being-a-testing-expert?wpdmdl=487569

This is a careful analysis of an attempt to demonstrate ChatGPT’s usefulness to help testers.

I liked this article a lot: https://lilianweng.github.io/posts/2023-06-23-agent/

My writing as an extension to "Why you should host your LLM?" article, with some adding on operation perspective: https://tulip4attoo.substack.com/p/why-you-should-host-your-llm-from

Maybe Foundational Papers should include the first instruction-tuned model FLAN (no RLHF)：
Finetuned Language Models Are Zero-Shot Learners: https://arxiv.org/abs/2109.01652

I gave a talk about prompt engineering for normal people and turned it into a pretty decent article, might be useful for the list too? https://timbornholdt.com/blog/prompt-engineering-how-to-think-like-an-ai

The Stochastic Parrots paper presents many things that anyone should be cognisant of when deciding whether or not to use an LLM:

Bender, Emily M., Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜. In Proceedings of FAccT 2021, pp.610-623.

We are exploring the landing and commercialization scenarios of AI Agent at https://askgen.ie, and currently we think customer support is a good scenario

@livc Are you guys hiring?

LLMs for designing 2D game levels: Level Generation Through Large Language Models
LLMs for Neural Architecture Search : LLMatic: Neural Architecture Search via Large Language Models and Quality-Diversity Optimization

I found "Five years of GPT progress" to be a useful overview of the influential papers on GPT.
https://finbarr.ca/five-years-of-gpt-progress/
May work as a high-level summary for "Foundational Papers" section.

p.s. Thank you for compiling the list!

Hi! This is great! Would be even more useful to write the year/month of publication next to each item, to get a sense of which links are more up-to-date and which are more historical.

I really like this list, sad I just discovered this 😎 .

I am not sure if this would complement your Background section, but I wrote this as a primer on LLMs last month: https://willthompson.name/what-we-know-about-llms-primer.

But I don't know, might not be very orthogonal to your other sources here 🤷 .

An overview of vector database. The author highlight the differences between the various vector databases out there as visually as possible.

https://thedataquarry.com/posts/vector-db-1/

An overview of vector database. The author highlight the differences between the various vector databases out there as visually as possible.

https://thedataquarry.com/posts/vector-db-1/

This is really helpful, thank you!

GPT in 60 Lines of NumPy https://jaykmody.com/blog/gpt-from-scratch/

I keep coming back to this list. However I feel like it miss a good discussion about current stuff not working. I keep failling to implement working stuff, despite lenghty theoretical works, and when I scratch the veneer I keep getting the same answer: "technology is not ready yet".

Also: Bash One-Liners for LLMs

Awesome list (and comments), but "graph" is missing

RAGs Do Not Reduce Hallucinations in LLMs — A Math Deep Dive