Marin Delphi Scaling Ladder IsoFLOP Extraction

marin_delphi_isoflop_extraction.md

Marin Delphi Scaling Ladder IsoFLOP Extraction

Extract IsoFLOP scaling ladder data (loss values, token counts, FLOPs, params) from the Marin Delphi suite without rerunning analysis jobs.

Context

• Delphi Scaling Setup #3292
• Delphi parent issue #1337
• Discord discussion
• Post

Reproduction

Checkout the commit used to generate the accompanying CSV:

git checkout 5c7a3fe6a0b3ef893ce611c5bed51d03ff099d77

Source the environment (for SSL certs and W&B credentials):

source .env

Run the extraction:

import csv
import dataclasses

from experiments.isoflop_sweep import MARIN_SCALING_SUITES, IsoFlopAnalysisConfig, load_isoflop_records
from marin.execution.executor import Executor

SUITE_NAME = "nemotron-completed-adamh"
METRIC_KEYS = [
    "eval/paloma/macro_loss",
    "eval/paloma/c4_en/bpb",
]
OUTPUT_CSV = "scratch/marin_delphi_isoflop_records.csv"

# Resolve GCS paths via Executor (no jobs are executed)
training_steps, _ = MARIN_SCALING_SUITES[SUITE_NAME]
ex = Executor(prefix="gs://marin-us-central2", executor_info_base_path="/tmp")
for s in training_steps:
    ex.compute_version(s, is_pseudo_dep=False)
paths = [ex.output_paths[s] for s in training_steps]

# Load records for each metric and flatten into rows with metric name + value
rows: list[dict] = []
for metric_key in METRIC_KEYS:
    config = IsoFlopAnalysisConfig(training_runs=paths, output_path="/tmp/unused", metric_key=metric_key)
    records = load_isoflop_records(config)
    for r in records:
        d = dataclasses.asdict(r)
        d["value"] = d.pop("metric")
        d["metric"] = metric_key
        rows.append(d)

# Write CSV
fieldnames = ["tokens", "metric", "value", "flops", "params", "label"]
with open(OUTPUT_CSV, "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=fieldnames)
    w.writeheader()
    for row in rows:
        w.writerow(row)

print(f"Wrote {len(rows)} records to {OUTPUT_CSV}")

Set MARIN_PREFIX=gs://marin-us-central2 in your environment or pass it inline.

No jobs are executed. The Executor is only used to resolve hashed GCS output paths. load_isoflop_records reads tracker_metrics.jsonl files from GCS (or backfills from W&B if missing).

Output Schema

Field	Description
tokens	Total tokens trained on
metric	Metric name (eval/paloma/macro_loss or eval/paloma/c4_en/bpb)
value	Metric value (loss or bits-per-byte)
flops	Total training FLOPs (bucketed via round_flops_to_bucket)
params	Model parameter count
label	Experiment label (adamh_scaling_v6)

156 records (78 per metric) spanning 7 FLOP budgets (~2.9e18 to ~3.1e20) with 5-14 model sizes per budget.

marin_delphi_isoflop_records.csv