Eric Czech eric-czech

Chinchilla Approach 2 Extrapolation Error Demo

Create an application comparing Chinchilla Approach 2 token extrapolations against a known, analytical ground truth.

All details below reference Hoffmann et al. 2022, "Training Compute-Optimal Large Language Models" (https://arxiv.org/abs/2203.15556).

Implementation:

Define the Chinchilla loss surface: L(N, D) = E + A/N^α + B/D^β with parameters α=0.34, β=0.28, A=406.4, B=410.7, E=1.69 (Appendix D)

Extract IsoFLOP scaling ladder data (loss values, token counts, FLOPs, params) from the Marin Delphi suite without rerunning analysis jobs.

	"""Count rows in bolinas-dna HF datasets (training + validation)."""

	from datasets import load_dataset

	DATASETS = [
	("CDS train", "bolinas-dna/genomes-v5-genome_set-animals-intervals-v5_255_128", "train"),
	("Upstream train", "bolinas-dna/genomes-v5-genome_set-animals-intervals-v1_255_128", "train"),
	("Downstream train", "bolinas-dna/genomes-v5-genome_set-animals-intervals-v15_255_128", "train"),
	("CDS val", "bolinas-dna/genomes-v5-validation-intervals-v5_255_255", "validation"),
	("Upstream val", "bolinas-dna/genomes-v5-validation-intervals-v1_255_255", "validation"),