LLM resources

llm.md

Readings

• Mastering LLM Techniques: Inference Optimization
• LLM Inference Series: 1. Introduction
• How Transformers Work: A Detailed Exploration of Transformer Architecture
• DeepSpeed Deep Dive
• Transformers Explained Visually
• The Illustrated Transformer
• All you need to know about ‘Attention’ and ‘Transformers’ — In-depth Understanding — Part 1
• All you need to know about ‘Attention’ and ‘Transformers’ — In-depth Understanding — Part 2
• Build your own Transformer from scratch using Pytorch
• A collection of resources to study Transformers in depth
• Numbers every LLM Developer should know
• LLM Inference Series: 3. KV caching explained
• The GPT-3 Architecture, on a Napkin
• Transformer Inference Arithmetic
• A guide to LLM inference and performance
• LLM Inference Sizing and Performance Guidance

CUDA

• CUDA semantics
• How to Implement Performance Metrics in CUDA C/C++
• Improving Network Performance of HPC Systems Using NVIDIA Magnum IO NVSHMEM and GPUDirect Async
• NCCL vs NVSHMEM
• Deep Dive into PyTorch Device Copy Operations

Tools and Libraries

• NVIDIA Collective Communications Library (NCCL)
• OpenAI Tokenizer
• CUDA Occupancy Calculator

Visualization

• LLM Visualization
• TRANSFORMER EXPLAINER