dwarkeshsp

Timestamps

(00:00:00) – How batch size affects token cost and speed
(00:32:09) – How MoE models are laid out across a GPU racks
(00:47:12) – How pipeline parallelism moves model layers across racks
(01:03:37) – Why Ilya said, “As we now know, pipelining is not wise.”
(01:18:59) – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal
(01:33:02) – Deducing long context memory costs from API pricing
(02:04:02) – Convergent evolution between neural nets and cryptography

dwarkeshsp

"""
Simple RL training script for teaching a model to add.
Demonstrates REINFORCE and GRPO algorithms in a minimal implementation.

If you want to run this script, put it inside of nanochat/scripts/ and run it with:
python -m scripts.simple_rl

First add "matplotlib>=3.9.0" to pyproject.toml and run 'uv sync'

I wrote a separate script to download the weights for the model: 

dwarkeshsp

#!/bin/bash
set -e

echo "=== NanoChat Weights Download Script ==="
echo ""

# Setup directories
export NANOCHAT_BASE_DIR="$HOME/.cache/nanochat"
TOKENIZER_DIR="$NANOCHAT_BASE_DIR/tokenizer"
CHECKPOINT_DIR="$NANOCHAT_BASE_DIR/chatsft_checkpoints/d20"

dwarkeshsp

#!/usr/bin/env python3
"""
Human quality transcripts from audio files using 
AssemblyAI for transcription and Google's Gemini for enhancement.

Requirements:
- AssemblyAI API key (https://www.assemblyai.com/)
- Google API key (https://aistudio.google.com/)
- Python packages: assemblyai, google-generativeai, pydub

dwarkeshsp

dwarkeshsp

" Specify a directory for plugins
call plug#begin('~/.vim/plugged')

Plug 'neoclide/coc.nvim', {'branch': 'release'}
Plug 'Xuyuanp/nerdtree-git-plugin'
Plug 'tiagofumo/vim-nerdtree-syntax-highlight'
Plug 'ryanoasis/vim-devicons'
Plug 'airblade/vim-gitgutter'
Plug 'ctrlpvim/ctrlp.vim' " fuzzy find files
Plug 'scrooloose/nerdcommenter'