Created
May 15, 2026 18:59
-
-
Save belisarius222/9b6a2e3da05fc9b5ba1acb13fd2e5dcf to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> | |
| <title>End-to-End Hierarchical Memory — A system design for billion-token attention</title> | |
| <link rel="preconnect" href="https://fonts.googleapis.com" /> | |
| <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin /> | |
| <link href="https://fonts.googleapis.com/css2?family=Instrument+Serif:ital@0;1&family=Newsreader:ital,opsz,wght@0,6..72,300..700;1,6..72,300..700&family=JetBrains+Mono:ital,wght@0,400;0,500;0,700;1,400&display=swap" rel="stylesheet" /> | |
| <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.9/katex.min.css" /> | |
| <script defer src="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.9/katex.min.js"></script> | |
| <script defer src="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.9/contrib/auto-render.min.js" onload="renderMathInElement(document.body, {delimiters: [{left: '$$', right: '$$', display: true}, {left: '$', right: '$', display: false}, {left: '\\[', right: '\\]', display: true}, {left: '\\(', right: '\\)', display: false}]});"></script> | |
| <style> | |
| :root { | |
| --bg: #FAF7F2; | |
| --bg-shade: #F2EDE3; | |
| --bg-deep: #ECE5D8; | |
| --fg: #1A1816; | |
| --fg-soft: #2B2825; | |
| --accent: #A03A2E; | |
| --accent-soft: #C95E50; | |
| --accent-bg: #F5E4DE; | |
| --muted: #6B6660; | |
| --muted-soft: #908A82; | |
| --rule: #2B2825; | |
| --subtle: #E8E2D8; | |
| --subtle-2: #DDD5C5; | |
| --font-display: 'Instrument Serif', Georgia, serif; | |
| --font-body: 'Newsreader', Georgia, 'Times New Roman', serif; | |
| --font-mono: 'JetBrains Mono', 'IBM Plex Mono', 'Consolas', monospace; | |
| --measure: 68ch; | |
| --measure-wide: 88ch; | |
| } | |
| * { box-sizing: border-box; } | |
| html { | |
| font-size: 18px; | |
| scroll-behavior: smooth; | |
| scroll-padding-top: 2rem; | |
| } | |
| body { | |
| margin: 0; | |
| background: var(--bg); | |
| color: var(--fg); | |
| font-family: var(--font-body); | |
| font-weight: 400; | |
| line-height: 1.55; | |
| font-feature-settings: "kern", "liga", "calt"; | |
| -webkit-font-smoothing: antialiased; | |
| text-rendering: optimizeLegibility; | |
| } | |
| /* Skip link for accessibility */ | |
| .skip-link { | |
| position: absolute; | |
| top: -40px; | |
| left: 0; | |
| background: var(--fg); | |
| color: var(--bg); | |
| padding: 8px 16px; | |
| z-index: 100; | |
| text-decoration: none; | |
| } | |
| .skip-link:focus { top: 0; } | |
| /* ============ TYPOGRAPHY ============ */ | |
| h1, h2, h3, h4 { | |
| font-family: var(--font-display); | |
| font-weight: 400; | |
| color: var(--fg); | |
| line-height: 1.08; | |
| letter-spacing: -0.005em; | |
| } | |
| h1 { | |
| font-size: clamp(3rem, 7vw, 5.5rem); | |
| margin: 0 0 0.4em; | |
| } | |
| h2 { | |
| font-size: clamp(2rem, 4vw, 3rem); | |
| margin: 3.5rem 0 1rem; | |
| position: relative; | |
| } | |
| h3 { | |
| font-size: 1.6rem; | |
| margin: 2.4rem 0 0.6rem; | |
| } | |
| h4 { | |
| font-family: var(--font-body); | |
| font-weight: 600; | |
| font-size: 1.05rem; | |
| margin: 1.8rem 0 0.4rem; | |
| text-transform: uppercase; | |
| letter-spacing: 0.08em; | |
| color: var(--accent); | |
| } | |
| p { margin: 0 0 1em; } | |
| em, .ital { | |
| font-family: var(--font-body); | |
| font-style: italic; | |
| } | |
| strong { font-weight: 600; color: var(--fg); } | |
| a { | |
| color: var(--accent); | |
| text-decoration: underline; | |
| text-decoration-thickness: 1px; | |
| text-underline-offset: 3px; | |
| } | |
| a:hover { color: var(--accent-soft); } | |
| code, .mono { | |
| font-family: var(--font-mono); | |
| font-size: 0.88em; | |
| background: var(--subtle); | |
| padding: 0.08em 0.3em; | |
| border-radius: 2px; | |
| } | |
| pre { | |
| font-family: var(--font-mono); | |
| font-size: 0.82rem; | |
| line-height: 1.5; | |
| background: var(--bg-deep); | |
| color: var(--fg); | |
| padding: 1.2rem 1.4rem; | |
| overflow-x: auto; | |
| border-left: 3px solid var(--accent); | |
| margin: 1.4rem 0; | |
| } | |
| pre code { | |
| background: none; | |
| padding: 0; | |
| border-radius: 0; | |
| font-size: inherit; | |
| } | |
| /* ============ LAYOUT ============ */ | |
| .container { | |
| max-width: 1400px; | |
| margin: 0 auto; | |
| padding: 0 2rem; | |
| display: grid; | |
| grid-template-columns: minmax(0, 1fr); | |
| gap: 0; | |
| } | |
| @media (min-width: 1100px) { | |
| .container { | |
| grid-template-columns: 240px minmax(0, 1fr); | |
| gap: 4rem; | |
| } | |
| } | |
| header.masthead { | |
| grid-column: 1 / -1; | |
| padding: 5rem 0 2.5rem; | |
| border-bottom: 1px solid var(--rule); | |
| margin-bottom: 3rem; | |
| } | |
| .masthead-meta { | |
| font-family: var(--font-mono); | |
| font-size: 0.72rem; | |
| text-transform: uppercase; | |
| letter-spacing: 0.2em; | |
| color: var(--muted); | |
| margin-bottom: 1.5rem; | |
| } | |
| .masthead-meta span { color: var(--accent); } | |
| .masthead-title { | |
| max-width: 18ch; | |
| } | |
| .masthead-title em { | |
| color: var(--accent); | |
| font-style: italic; | |
| } | |
| .masthead-subtitle { | |
| font-family: var(--font-body); | |
| font-style: italic; | |
| font-size: clamp(1.1rem, 1.6vw, 1.4rem); | |
| color: var(--fg-soft); | |
| max-width: 50ch; | |
| line-height: 1.4; | |
| margin-top: 1rem; | |
| } | |
| /* Sidebar TOC */ | |
| nav.toc { | |
| grid-column: 1; | |
| } | |
| @media (min-width: 1100px) { | |
| nav.toc { | |
| position: sticky; | |
| top: 2rem; | |
| align-self: start; | |
| max-height: calc(100vh - 4rem); | |
| overflow-y: auto; | |
| padding-top: 0.5rem; | |
| } | |
| } | |
| nav.toc .toc-label { | |
| font-family: var(--font-mono); | |
| font-size: 0.7rem; | |
| text-transform: uppercase; | |
| letter-spacing: 0.2em; | |
| color: var(--muted); | |
| margin-bottom: 1rem; | |
| padding-bottom: 0.5rem; | |
| border-bottom: 1px solid var(--subtle-2); | |
| } | |
| nav.toc ol { | |
| list-style: none; | |
| padding: 0; | |
| margin: 0; | |
| counter-reset: section; | |
| font-size: 0.85rem; | |
| line-height: 1.4; | |
| } | |
| nav.toc li { | |
| counter-increment: section; | |
| margin-bottom: 0.5rem; | |
| } | |
| nav.toc a { | |
| color: var(--fg-soft); | |
| text-decoration: none; | |
| display: block; | |
| padding: 0.15rem 0; | |
| } | |
| nav.toc a:hover { color: var(--accent); } | |
| nav.toc li::before { | |
| content: counter(section, decimal-leading-zero); | |
| font-family: var(--font-mono); | |
| font-size: 0.7rem; | |
| color: var(--accent); | |
| margin-right: 0.6rem; | |
| } | |
| main { | |
| grid-column: 1; | |
| max-width: var(--measure); | |
| } | |
| @media (min-width: 1100px) { | |
| main { grid-column: 2; } | |
| } | |
| section { margin-bottom: 3rem; } | |
| section > h2 { | |
| padding-top: 1rem; | |
| } | |
| .section-number { | |
| display: block; | |
| font-family: var(--font-mono); | |
| font-size: 0.75rem; | |
| letter-spacing: 0.2em; | |
| text-transform: uppercase; | |
| color: var(--accent); | |
| margin-bottom: 0.5rem; | |
| font-weight: 500; | |
| } | |
| /* ============ COMPONENTS ============ */ | |
| .lede { | |
| font-size: 1.18rem; | |
| line-height: 1.5; | |
| color: var(--fg-soft); | |
| max-width: 60ch; | |
| font-style: italic; | |
| border-left: 3px solid var(--accent); | |
| padding-left: 1.4rem; | |
| margin: 0 0 2rem; | |
| } | |
| .callout { | |
| background: var(--bg-shade); | |
| border-left: 3px solid var(--accent); | |
| padding: 1.2rem 1.4rem; | |
| margin: 1.8rem 0; | |
| font-size: 0.96rem; | |
| } | |
| .callout p:last-child { margin-bottom: 0; } | |
| .callout-label { | |
| font-family: var(--font-mono); | |
| font-size: 0.7rem; | |
| text-transform: uppercase; | |
| letter-spacing: 0.18em; | |
| color: var(--accent); | |
| margin-bottom: 0.4rem; | |
| display: block; | |
| } | |
| /* Definition lists */ | |
| dl.defs { | |
| display: grid; | |
| grid-template-columns: minmax(0, 1fr); | |
| gap: 0; | |
| margin: 1.4rem 0; | |
| border-top: 1px solid var(--subtle-2); | |
| } | |
| @media (min-width: 720px) { | |
| dl.defs { | |
| grid-template-columns: minmax(120px, 22%) minmax(0, 1fr); | |
| gap: 0 2rem; | |
| } | |
| } | |
| dl.defs dt { | |
| font-family: var(--font-mono); | |
| font-size: 0.86rem; | |
| font-weight: 500; | |
| color: var(--accent); | |
| padding: 0.9rem 0 0.3rem; | |
| border-bottom: 1px solid var(--subtle-2); | |
| } | |
| @media (min-width: 720px) { | |
| dl.defs dt { | |
| padding: 0.9rem 0; | |
| border-bottom: 1px solid var(--subtle-2); | |
| } | |
| } | |
| dl.defs dd { | |
| margin: 0; | |
| padding: 0 0 0.9rem; | |
| font-size: 0.96rem; | |
| line-height: 1.5; | |
| border-bottom: 1px solid var(--subtle-2); | |
| } | |
| @media (min-width: 720px) { | |
| dl.defs dd { padding: 0.9rem 0; } | |
| } | |
| /* Tables */ | |
| table { | |
| width: 100%; | |
| border-collapse: collapse; | |
| font-size: 0.92rem; | |
| margin: 1.6rem 0; | |
| font-family: var(--font-body); | |
| } | |
| table.dim-table { font-family: var(--font-mono); font-size: 0.82rem; } | |
| thead th { | |
| text-align: left; | |
| font-family: var(--font-mono); | |
| font-size: 0.72rem; | |
| text-transform: uppercase; | |
| letter-spacing: 0.12em; | |
| color: var(--muted); | |
| font-weight: 500; | |
| padding: 0.6rem 0.8rem 0.6rem 0; | |
| border-bottom: 1.5px solid var(--rule); | |
| } | |
| tbody td { | |
| padding: 0.7rem 0.8rem 0.7rem 0; | |
| border-bottom: 1px solid var(--subtle-2); | |
| vertical-align: top; | |
| } | |
| tbody tr:hover { background: var(--bg-shade); } | |
| /* Math display */ | |
| .math-block { | |
| background: var(--bg-shade); | |
| padding: 1.2rem 1.4rem; | |
| margin: 1.4rem 0; | |
| border-left: 3px solid var(--subtle-2); | |
| overflow-x: auto; | |
| } | |
| .math-caption { | |
| font-size: 0.85rem; | |
| color: var(--muted); | |
| font-style: italic; | |
| margin-top: 0.5rem; | |
| max-width: 60ch; | |
| } | |
| /* Figure */ | |
| figure { | |
| margin: 2.4rem 0; | |
| max-width: 100%; | |
| } | |
| figure svg { display: block; width: 100%; height: auto; max-width: 100%; } | |
| figcaption { | |
| font-size: 0.86rem; | |
| color: var(--muted); | |
| font-style: italic; | |
| margin-top: 0.8rem; | |
| max-width: 65ch; | |
| line-height: 1.5; | |
| } | |
| figcaption strong { | |
| font-style: normal; | |
| color: var(--fg-soft); | |
| } | |
| /* Rejected list */ | |
| .rejected { | |
| list-style: none; | |
| padding: 0; | |
| margin: 1.4rem 0; | |
| } | |
| .rejected li { | |
| padding: 1.2rem 0; | |
| border-top: 1px solid var(--subtle-2); | |
| } | |
| .rejected li:last-child { border-bottom: 1px solid var(--subtle-2); } | |
| .rejected .strike { | |
| font-family: var(--font-display); | |
| font-size: 1.4rem; | |
| color: var(--muted); | |
| text-decoration: line-through; | |
| text-decoration-color: var(--accent); | |
| text-decoration-thickness: 2px; | |
| margin-bottom: 0.4rem; | |
| display: block; | |
| } | |
| .rejected .reason { | |
| font-size: 0.95rem; | |
| color: var(--fg-soft); | |
| max-width: 60ch; | |
| } | |
| /* Drop cap for first section paragraph */ | |
| .dropcap::first-letter { | |
| font-family: var(--font-display); | |
| float: left; | |
| font-size: 4.4em; | |
| line-height: 0.85; | |
| padding: 0.05em 0.08em 0 0; | |
| color: var(--accent); | |
| } | |
| /* Footnote-ish marginal note */ | |
| .marginal { | |
| font-size: 0.84rem; | |
| color: var(--muted); | |
| font-style: italic; | |
| border-left: 1px solid var(--subtle-2); | |
| padding-left: 1rem; | |
| margin: 1.2rem 0; | |
| max-width: 55ch; | |
| } | |
| /* Footer */ | |
| footer { | |
| grid-column: 1 / -1; | |
| margin-top: 5rem; | |
| padding: 2.5rem 0; | |
| border-top: 1px solid var(--rule); | |
| font-family: var(--font-mono); | |
| font-size: 0.78rem; | |
| color: var(--muted); | |
| letter-spacing: 0.05em; | |
| } | |
| footer p { margin: 0.3rem 0; } | |
| /* Inline math should be small */ | |
| .katex { font-size: 1em !important; } | |
| .katex-display { margin: 0 !important; } | |
| /* Section divider rule */ | |
| .rule { | |
| height: 1px; | |
| background: var(--rule); | |
| margin: 4rem 0 2rem; | |
| grid-column: 1 / -1; | |
| } | |
| /* Pull-quote */ | |
| .pull { | |
| font-family: var(--font-display); | |
| font-size: 2rem; | |
| line-height: 1.15; | |
| color: var(--fg); | |
| margin: 2.4rem 0; | |
| max-width: 26ch; | |
| border-top: 1px solid var(--rule); | |
| border-bottom: 1px solid var(--rule); | |
| padding: 1.6rem 0; | |
| } | |
| .pull em { color: var(--accent); } | |
| /* Focus styles for accessibility */ | |
| :focus-visible { | |
| outline: 2px solid var(--accent); | |
| outline-offset: 3px; | |
| border-radius: 2px; | |
| } | |
| </style> | |
| </head> | |
| <body> | |
| <a class="skip-link" href="#main">Skip to main content</a> | |
| <div class="container"> | |
| <header class="masthead"> | |
| <div class="masthead-meta"> | |
| <span>Specification</span> · System design · Hierarchical attention at <span>10⁹ tokens</span> | |
| </div> | |
| <h1 class="masthead-title">End-to-End <em>Hierarchical Memory</em></h1> | |
| <p class="masthead-subtitle">A jointly-learned recursive chunking and traversal system for billion-token attention, with no handcrafted indexing structure.</p> | |
| </header> | |
| <nav class="toc" aria-label="Table of contents"> | |
| <div class="toc-label">Contents</div> | |
| <ol> | |
| <li><a href="#sec-preamble">The principle</a></li> | |
| <li><a href="#sec-glance">At a glance</a></li> | |
| <li><a href="#sec-notation">Notation & dimensions</a></li> | |
| <li><a href="#sec-compiler">The compiler</a></li> | |
| <li><a href="#sec-tree">The latent tree</a></li> | |
| <li><a href="#sec-large-model">Large model & query head</a></li> | |
| <li><a href="#sec-traversal">Traversal</a></li> | |
| <li><a href="#sec-multires">Multi-resolution read</a></li> | |
| <li><a href="#sec-training">Training</a></li> | |
| <li><a href="#sec-absent">What is absent</a></li> | |
| <li><a href="#sec-config">1B-token configuration</a></li> | |
| <li><a href="#sec-open">Open problems</a></li> | |
| </ol> | |
| </nav> | |
| <main id="main"> | |
| <section id="sec-preamble" aria-labelledby="h-preamble"> | |
| <span class="section-number">§ 01</span> | |
| <h2 id="h-preamble">The principle</h2> | |
| <p class="lede">One mechanism, applied recursively, trained jointly with the model that queries it. Nothing else.</p> | |
| <p class="dropcap">The recurring temptation in designing memory for a billion-token model is to bolt structure onto the system before training begins. A directory of documents. A list of anchor types: function definitions, section headings, citations, theorem statements. A lexical sidecar — BM25, exact-string search — to catch the queries the learned router will miss. A memory controller that decides when to retrieve based on a hand-authored rulebook. Each of these is individually reasonable. Each fixes a real failure mode. And taken together they reproduce, one floor up, the same handcrafted preprocessing layer that tokenization is one floor down.</p> | |
| <p>The diagnosis from <em>H-Net</em> is that handcrafted preprocessing loses to learned representation, given enough data and compute, because the handcrafted layer encodes assumptions that the learned layer would discover anyway — and discover better, because gradient sees the downstream consequences and the handcraft does not. The same diagnosis applies to a billion-token memory system. If modality boundaries matter for routing, the compiler should discover that. If rare identifiers need surface-form-sensitive route keys, the routing module should learn dual representations. If certain query types need deeper traversal, the model should learn to allocate it.</p> | |
| <p>The bet of this design is that one mechanism — dynamic chunking, applied recursively from raw bytes up to a single root, trained end-to-end with the large model that consumes its output — is sufficient, and that every additional engineered component compromises the bet without saving us from the failure modes it claims to address.</p> | |
| <div class="callout"> | |
| <span class="callout-label">The single commitment</span> | |
| <p>Everything in the system that could be handcrafted is instead a function of weights that receive gradient from the downstream task. Chunk boundaries, parent compressions, when to query memory, how deep to traverse, which resolution to read from — all of it learned, jointly, in one differentiable graph.</p> | |
| </div> | |
| <p>What follows is the smallest specification that makes that commitment concrete enough to build.</p> | |
| </section> | |
| <section id="sec-glance" aria-labelledby="h-glance"> | |
| <span class="section-number">§ 02</span> | |
| <h2 id="h-glance">At a glance</h2> | |
| <p>The system has two components, trained jointly as one model:</p> | |
| <ol> | |
| <li><strong>A compiler.</strong> A byte-level encoder followed by a stack of dynamic-chunking layers, applied recursively. Raw bytes enter at the bottom; latent K/V nodes are produced at every level of the resulting tree, up to a single root.</li> | |
| <li><strong>A large model.</strong> A standard sequence model (Transformer, Mamba, or hybrid) that emits memory queries from its hidden states, traverses the compiler-produced tree, and consumes the retrieved working set via cross-attention layers interleaved with its main blocks.</li> | |
| </ol> | |
| <p>The compiler is paid once per ingested document. The large model is run once per generated token. Gradient flows freely between them through soft traversal during training, and the entire system is optimized against a single downstream objective with a small number of auxiliary losses that exist to make the tree learnable, not to inject prior structure into it.</p> | |
| <figure aria-labelledby="fig1-caption"> | |
| <svg viewBox="0 0 880 480" xmlns="http://www.w3.org/2000/svg" role="img" aria-labelledby="fig1-title fig1-caption"> | |
| <title id="fig1-title">Architecture overview</title> | |
| <defs> | |
| <marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"> | |
| <path d="M 0 0 L 10 5 L 0 10 z" fill="#A03A2E" /> | |
| </marker> | |
| <marker id="arrow-soft" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"> | |
| <path d="M 0 0 L 10 5 L 0 10 z" fill="#6B6660" /> | |
| </marker> | |
| </defs> | |
| <!-- LEFT COLUMN: compiler --> | |
| <text x="40" y="32" font-family="JetBrains Mono, monospace" font-size="11" fill="#A03A2E" letter-spacing="2" font-weight="500">COMPILER (PAID ONCE)</text> | |
| <!-- Raw bytes --> | |
| <rect x="40" y="50" width="280" height="40" fill="#ECE5D8" stroke="#2B2825" stroke-width="1" /> | |
| <text x="180" y="76" text-anchor="middle" font-family="Newsreader, serif" font-style="italic" font-size="14" fill="#1A1816">raw bytes x₁, x₂, …, x_L</text> | |
| <!-- Byte encoder --> | |
| <rect x="40" y="110" width="280" height="34" fill="#F5E4DE" stroke="#A03A2E" stroke-width="1" /> | |
| <text x="180" y="132" text-anchor="middle" font-family="JetBrains Mono, monospace" font-size="12" fill="#1A1816">E_byte (Mamba-2 stack)</text> | |
| <!-- Level 0 leaves --> | |
| <g> | |
| <rect x="40" y="164" width="56" height="24" fill="#FAF7F2" stroke="#2B2825" stroke-width="1" /> | |
| <rect x="100" y="164" width="56" height="24" fill="#FAF7F2" stroke="#2B2825" stroke-width="1" /> | |
| <rect x="160" y="164" width="56" height="24" fill="#FAF7F2" stroke="#2B2825" stroke-width="1" /> | |
| <rect x="220" y="164" width="56" height="24" fill="#FAF7F2" stroke="#2B2825" stroke-width="1" /> | |
| <text x="288" y="180" font-family="JetBrains Mono, monospace" font-size="13" fill="#6B6660">…</text> | |
| </g> | |
| <text x="40" y="208" font-family="JetBrains Mono, monospace" font-size="10" fill="#6B6660" letter-spacing="1">DC STAGE 0 → LEAF NODES</text> | |
| <!-- Level 1 --> | |
| <g> | |
| <rect x="50" y="226" width="80" height="24" fill="#FAF7F2" stroke="#2B2825" stroke-width="1" /> | |
| <rect x="140" y="226" width="80" height="24" fill="#FAF7F2" stroke="#2B2825" stroke-width="1" /> | |
| <rect x="230" y="226" width="80" height="24" fill="#FAF7F2" stroke="#2B2825" stroke-width="1" /> | |
| </g> | |
| <text x="40" y="270" font-family="JetBrains Mono, monospace" font-size="10" fill="#6B6660" letter-spacing="1">DC STAGE 1</text> | |
| <!-- Level 2 --> | |
| <g> | |
| <rect x="80" y="288" width="100" height="24" fill="#FAF7F2" stroke="#2B2825" stroke-width="1" /> | |
| <rect x="200" y="288" width="100" height="24" fill="#FAF7F2" stroke="#2B2825" stroke-width="1" /> | |
| </g> | |
| <text x="40" y="332" font-family="JetBrains Mono, monospace" font-size="10" fill="#6B6660" letter-spacing="1">DC STAGE k (recurse)</text> | |
| <!-- Vertical dots indicating recursion --> | |
| <text x="180" y="354" text-anchor="middle" font-family="JetBrains Mono, monospace" font-size="16" fill="#A03A2E" letter-spacing="3">⋮ ⋮ ⋮</text> | |
| <!-- Root --> | |
| <rect x="140" y="378" width="80" height="28" fill="#A03A2E" /> | |
| <text x="180" y="397" text-anchor="middle" font-family="JetBrains Mono, monospace" font-size="12" fill="#FAF7F2">root</text> | |
| <!-- Stage arrows --> | |
| <line x1="180" y1="92" x2="180" y2="108" stroke="#A03A2E" stroke-width="1.5" marker-end="url(#arrow)" /> | |
| <line x1="180" y1="146" x2="180" y2="162" stroke="#A03A2E" stroke-width="1.5" marker-end="url(#arrow)" /> | |
| <line x1="180" y1="190" x2="180" y2="224" stroke="#A03A2E" stroke-width="1.5" marker-end="url(#arrow)" /> | |
| <line x1="180" y1="252" x2="180" y2="286" stroke="#A03A2E" stroke-width="1.5" marker-end="url(#arrow)" /> | |
| <!-- Vertical bracket label for compiler --> | |
| <line x1="20" y1="50" x2="20" y2="406" stroke="#2B2825" stroke-width="1" /> | |
| <line x1="20" y1="50" x2="28" y2="50" stroke="#2B2825" stroke-width="1" /> | |
| <line x1="20" y1="406" x2="28" y2="406" stroke="#2B2825" stroke-width="1" /> | |
| <!-- RIGHT COLUMN: large model --> | |
| <text x="500" y="32" font-family="JetBrains Mono, monospace" font-size="11" fill="#A03A2E" letter-spacing="2" font-weight="500">LARGE MODEL (PER TOKEN)</text> | |
| <!-- Stack of blocks --> | |
| <g> | |
| <rect x="500" y="60" width="220" height="36" fill="#F5E4DE" stroke="#A03A2E" stroke-width="1" /> | |
| <text x="610" y="83" text-anchor="middle" font-family="JetBrains Mono, monospace" font-size="12" fill="#1A1816">block + memory query</text> | |
| <rect x="500" y="106" width="220" height="36" fill="#ECE5D8" stroke="#2B2825" stroke-width="1" /> | |
| <text x="610" y="129" text-anchor="middle" font-family="Newsreader, serif" font-style="italic" font-size="13" fill="#1A1816">cross-attn over working set</text> | |
| <rect x="500" y="152" width="220" height="36" fill="#F5E4DE" stroke="#A03A2E" stroke-width="1" /> | |
| <text x="610" y="175" text-anchor="middle" font-family="JetBrains Mono, monospace" font-size="12" fill="#1A1816">block + memory query</text> | |
| <rect x="500" y="198" width="220" height="36" fill="#ECE5D8" stroke="#2B2825" stroke-width="1" /> | |
| <text x="610" y="221" text-anchor="middle" font-family="Newsreader, serif" font-style="italic" font-size="13" fill="#1A1816">cross-attn over working set</text> | |
| <text x="610" y="252" text-anchor="middle" font-family="JetBrains Mono, monospace" font-size="14" fill="#A03A2E" letter-spacing="3">⋮</text> | |
| <rect x="500" y="262" width="220" height="36" fill="#F5E4DE" stroke="#A03A2E" stroke-width="1" /> | |
| <text x="610" y="285" text-anchor="middle" font-family="JetBrains Mono, monospace" font-size="12" fill="#1A1816">final block</text> | |
| <rect x="500" y="316" width="220" height="36" fill="#1A1816" stroke="#1A1816" stroke-width="1" /> | |
| <text x="610" y="339" text-anchor="middle" font-family="JetBrains Mono, monospace" font-size="12" fill="#FAF7F2">next-token prediction</text> | |
| </g> | |
| <!-- Bracket --> | |
| <line x1="740" y1="60" x2="740" y2="352" stroke="#2B2825" stroke-width="1" /> | |
| <line x1="740" y1="60" x2="732" y2="60" stroke="#2B2825" stroke-width="1" /> | |
| <line x1="740" y1="352" x2="732" y2="352" stroke="#2B2825" stroke-width="1" /> | |
| <!-- Query arrows from large model to tree --> | |
| <path d="M 500 124 Q 400 124 350 220" stroke="#6B6660" stroke-width="1" fill="none" stroke-dasharray="3,3" marker-end="url(#arrow-soft)" /> | |
| <text x="370" y="158" font-family="Newsreader, serif" font-style="italic" font-size="11" fill="#6B6660">query →</text> | |
| <path d="M 350 290 Q 400 250 500 216" stroke="#A03A2E" stroke-width="1" fill="none" marker-end="url(#arrow)" /> | |
| <text x="385" y="278" font-family="Newsreader, serif" font-style="italic" font-size="11" fill="#A03A2E">← latents</text> | |
| <!-- Bottom caption: end-to-end --> | |
| <rect x="40" y="430" width="680" height="36" fill="none" stroke="#A03A2E" stroke-width="1" stroke-dasharray="4,4" /> | |
| <text x="380" y="453" text-anchor="middle" font-family="Newsreader, serif" font-style="italic" font-size="14" fill="#A03A2E">one differentiable graph · one downstream objective · gradient through all of it</text> | |
| </svg> | |
| <figcaption id="fig1-caption"> | |
| <strong>Figure 1.</strong> The compiler (left) converts raw bytes into a recursive tree of latent K/V nodes via stacked dynamic-chunking stages. The large model (right) emits memory queries from selected blocks, traverses the tree, and consumes the retrieved working set through interleaved cross-attention. During training, traversal is soft and gradient flows through routing decisions; at inference, traversal is hard top-<em>m</em> beam search. There is no separate retrieval system, no lexical index, no controller policy — the boundary between "memory" and "model" exists only in the diagram, not in the optimizer. | |
| </figcaption> | |
| </figure> | |
| </section> | |
| <section id="sec-notation" aria-labelledby="h-notation"> | |
| <span class="section-number">§ 03</span> | |
| <h2 id="h-notation">Notation and dimensions</h2> | |
| <p>Every symbol used in the remainder of the document is defined here, along with its dimension and the role it plays. The intent is that the section be readable in isolation as a glossary: later sections refer back to it without redefinition.</p> | |
| <h3>Inputs and indexing</h3> | |
| <dl class="defs"> | |
| <dt>$x_t$</dt> | |
| <dd>The byte at position $t$ in the raw input. An integer in $[0, 256)$. The model never sees text in any other form; tokenization is not a preprocessing step.</dd> | |
| <dt>$L_0$</dt> | |
| <dd>Input length in bytes. The target deployment scale is $L_0 \approx 4 \times 10^9$, corresponding to roughly $10^9$ tokens at typical UTF-8 densities.</dd> | |
| <dt>$S$</dt> | |
| <dd>Number of dynamic-chunking stages in the compiler. Each stage compresses its input sequence by a learned, content-dependent factor. The compiler is structurally identical at every stage; only the dimensions and the per-stage targets differ.</dd> | |
| <dt>$s$</dt> | |
| <dd>Stage index, $0 \le s < S$. Stage $0$ operates on byte-encoder outputs; stage $S-1$ produces the root.</dd> | |
| <dt>$L_s$</dt> | |
| <dd>Sequence length at the input of stage $s$. We have $L_0 \approx 4 \times 10^9$ at the bottom; $L_S = 1$ at the root.</dd> | |
| <dt>$D_s$</dt> | |
| <dd>Hidden dimension at stage $s$. Monotone non-decreasing in $s$, following the H-Net guideline that inner stages can afford and benefit from wider representations.</dd> | |
| </dl> | |
| <h3>Compiler-side vectors</h3> | |
| <dl class="defs"> | |
| <dt>$\hat{x}_t^s$</dt> | |
| <dd>The encoder output at stage $s$, position $t$. A vector in $\mathbb{R}^{D_s}$. These are what the stage-$s$ routing module scores for boundary placement. The byte encoder produces $\hat{x}_t^0$ from the raw bytes.</dd> | |
| <dt>$p_t^s \in [0,1]$</dt> | |
| <dd>The routing module's boundary probability at stage $s$, position $t$. Interpreted as the router's confidence that a chunk boundary occurs immediately before position $t$. Set to $1.0$ at $t = 1$ by definition.</dd> | |
| <dt>$b_t^s \in \{0, 1\}$</dt> | |
| <dd>The hard boundary indicator at stage $s$, position $t$, equal to $\mathbf{1}\{p_t^s \ge 0.5\}$. Boundaries select which encoder outputs survive to become inputs at stage $s+1$.</dd> | |
| <dt>$W_q^s, W_k^s$</dt> | |
| <dd>Routing projection matrices at stage $s$, each in $\mathbb{R}^{D_s \times D_s}$. Used to compute the query/key vectors whose cosine similarity drives the boundary probability.</dd> | |
| </dl> | |
| <h3>Tree nodes</h3> | |
| <dl class="defs"> | |
| <dt>$u$</dt> | |
| <dd>A node in the latent tree. Nodes are produced by chunking stages; leaves come from stage $0$, parents from stages $1, \ldots, S-1$.</dd> | |
| <dt>$r$</dt> | |
| <dd>The number of latent slots stored per node. Each slot is a route-key/value pair, so a node is a small bundle of latent tokens rather than a single embedding. Fixed across all nodes in the system.</dd> | |
| <dt>$K_u \in \mathbb{R}^{r \times d_K}$</dt> | |
| <dd>The node's route keys. Used at query time to decide whether a traversal should enter the subtree rooted at $u$. Quantized to int8 in storage; promoted to fp16 for scoring.</dd> | |
| <dt>$V_u \in \mathbb{R}^{r \times d_V}$</dt> | |
| <dd>The node's value slots. These are what the large model attends to once the node is in the working set. $d_V$ matches the large model's residual-stream dimension, so retrieved values can be projected directly into the cross-attention.</dd> | |
| <dt>$d_K$</dt> | |
| <dd>Route key dimension. Strictly smaller than $d_V$, because scoring is done at every visited node and we want scoring to be cheap. A typical choice is $d_K = 256$.</dd> | |
| <dt>$d_V$</dt> | |
| <dd>Value dimension. Matches large-model residual width, $d_V = D_M$, so that no projection is needed when the values enter cross-attention.</dd> | |
| <dt>$\mathrm{children}(u)$</dt> | |
| <dd>The set of nodes at stage $s-1$ that were grouped together by stage $s$'s chunking to produce $u$. Empty for leaves.</dd> | |
| <dt>$\mathrm{raw\_ref}(u)$</dt> | |
| <dd>For a leaf, a pointer to the byte span covered by the leaf. Implemented as a $(\text{document\_id}, \text{byte\_start}, \text{byte\_end})$ tuple. This lets the large model dereference the raw bytes when value latents are not sufficient for the task.</dd> | |
| </dl> | |
| <h3>Large-model side</h3> | |
| <dl class="defs"> | |
| <dt>$D_M$</dt> | |
| <dd>The large model's residual-stream dimension. Equal to $d_V$ by construction.</dd> | |
| <dt>$h_t \in \mathbb{R}^{D_M}$</dt> | |
| <dd>The large model's hidden state at position $t$, after some block at which a memory query is emitted. Not every block emits queries; the model has a fixed set of memory-query layers, set as a hyperparameter.</dd> | |
| <dt>$Q_\theta$</dt> | |
| <dd>The memory query head. A small MLP that maps $h_t$ to a query vector $q \in \mathbb{R}^{d_K}$. There is one query head per memory-query layer; weights are not shared across layers.</dd> | |
| <dt>$q$</dt> | |
| <dd>The memory query, $q = Q_\theta(h_t) \in \mathbb{R}^{d_K}$. Used to score nodes during traversal.</dd> | |
| <dt>$s(q, u)$</dt> | |
| <dd>The score the traversal assigns to node $u$ given query $q$. Defined as $s(q, u) = \max_{j \in \{1, \ldots, r\}} q^\top K_u[j]$. The max-over-slots structure lets a parent expose several semantic directions of its subtree at once, which matters for heterogeneous parents.</dd> | |
| <dt>$m$</dt> | |
| <dd>Beam width. The number of nodes retained at each level of the traversal. A typical inference value is $m = 8$; soft traversal during training uses softmax over all candidates at each level, which is equivalent to $m = \text{fanout}$ with weighted gradient.</dd> | |
| <dt>$\mathcal{F}_\ell$</dt> | |
| <dd>The frontier at level $\ell$ of the traversal, i.e. the set of currently selected nodes at depth $\ell$. The traversal starts with $\mathcal{F}_0 = \{\text{root}\}$ and proceeds downward.</dd> | |
| <dt>$\mathcal{W}(q)$</dt> | |
| <dd>The working set returned by traversal for query $q$. A union of value slots from selected nodes at every level traversed, plus optional dereferenced raw spans for selected leaves.</dd> | |
| </dl> | |
| <h3>Training</h3> | |
| <dl class="defs"> | |
| <dt>$\mathcal{L}_\text{task}$</dt> | |
| <dd>The downstream loss. For language modeling, the next-byte negative log-likelihood. This is the only loss whose gradient signal is conceptually load-bearing; the others exist to make the tree learnable.</dd> | |
| <dt>$\mathcal{L}_\text{route}$</dt> | |
| <dd>The routing loss enforcing that a parent's score is at least the best descendant's score, minus a margin. Prevents the tree from hiding relevant subtrees behind a parent whose key happens to be orthogonal to the query.</dd> | |
| <dt>$\mathcal{L}_\text{value}$</dt> | |
| <dd>The value loss enforcing that attention over a parent's value slots approximates attention over its children's value slots. Allows the model to terminate traversal early when a parent suffices.</dd> | |
| <dt>$\mathcal{L}_\text{ratio}^s$</dt> | |
| <dd>The per-stage compression-ratio loss, in the form of H-Net's ratio loss. One per chunking stage. Targets an average chunking ratio $N_s$ but only weakly — the model is free to allocate compression non-uniformly within the stage as long as the average matches.</dd> | |
| <dt>$\mathcal{L}_\text{budget}$</dt> | |
| <dd>A small penalty on the number of nodes scored and expanded per query. Discourages the traversal from expanding wastefully when narrower beams suffice. Tied loosely to actual compute cost.</dd> | |
| <dt>$N_s$</dt> | |
| <dd>Target average compression ratio at stage $s$. A hyperparameter, but a soft one: the ratio loss is a guidance term, not a constraint.</dd> | |
| <dt>$\tau$</dt> | |
| <dd>The temperature used to anneal the traversal from soft (high $\tau$, near-uniform weighting of all candidates) to hard (low $\tau$, near-one-hot top-$m$ selection). Scheduled to decrease over training.</dd> | |
| <dt>$\alpha, \beta, \gamma, \delta$</dt> | |
| <dd>Loss weights for the auxiliary losses. Tuned but typically small (in the range $10^{-2}$ to $10^{-1}$) so they do not dominate the task gradient once the tree is functioning.</dd> | |
| </dl> | |
| <h3>Dimensional sanity table</h3> | |
| <p>The following table is the canonical reference for what shape every object has, for the specific 1B-token configuration described in <a href="#sec-config">§ 11</a>. Other configurations adjust these but preserve the structural relationships.</p> | |
| <table class="dim-table" aria-label="Dimensions for 1B-token configuration"> | |
| <thead> | |
| <tr><th>Object</th><th>Shape</th><th>Notes</th></tr> | |
| </thead> | |
| <tbody> | |
| <tr><td>$x_t$</td><td>scalar in $[0, 256)$</td><td>raw byte</td></tr> | |
| <tr><td>$\hat{x}_t^0$</td><td>$\mathbb{R}^{512}$</td><td>byte-encoder output, $D_0 = 512$</td></tr> | |
| <tr><td>$\hat{x}_t^s$ for $s \ge 1$</td><td>$\mathbb{R}^{D_s}$</td><td>$D_s$ monotone, ramps to $D_M$ at root</td></tr> | |
| <tr><td>$p_t^s, b_t^s$</td><td>scalar</td><td>per position, per stage</td></tr> | |
| <tr><td>$W_q^s, W_k^s$</td><td>$\mathbb{R}^{D_s \times D_s}$</td><td>per stage</td></tr> | |
| <tr><td>$K_u$</td><td>$\mathbb{R}^{r \times d_K} = \mathbb{R}^{8 \times 256}$</td><td>per node, int8 storage</td></tr> | |
| <tr><td>$V_u$</td><td>$\mathbb{R}^{r \times d_V} = \mathbb{R}^{8 \times 2048}$</td><td>per node, int8 or fp8 storage</td></tr> | |
| <tr><td>$h_t$</td><td>$\mathbb{R}^{2048}$</td><td>large-model hidden state, $D_M = 2048$</td></tr> | |
| <tr><td>$q$</td><td>$\mathbb{R}^{256}$</td><td>memory query, $d_K = 256$</td></tr> | |
| <tr><td>$s(q, u)$</td><td>scalar</td><td>node score</td></tr> | |
| <tr><td>$\mathcal{W}(q)$</td><td>set of $\mathbb{R}^{2048}$ vectors</td><td>working set, typically a few hundred vectors</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <section id="sec-compiler" aria-labelledby="h-compiler"> | |
| <span class="section-number">§ 04</span> | |
| <h2 id="h-compiler">The compiler</h2> | |
| <p>The compiler maps a sequence of raw bytes into a tree of latent K/V nodes. It is the same mechanism applied $S$ times in succession: a byte-level encoder feeds the first chunking stage; each subsequent stage takes the output of the previous one and applies the same operation. The recursion is what makes the system structurally minimal — there is exactly one kind of compression in the whole system, and it is the kind that H-Net validated empirically.</p> | |
| <h3>The byte encoder</h3> | |
| <p>The byte encoder $E_\text{byte}$ is a stack of four Mamba-2 layers operating on byte embeddings of dimension $D_0$. We use Mamba rather than Transformer layers here for the reasons established by H-Net's ablations: state-space models compress fine-grained data more effectively than attention does, and we want compression as the inductive bias at the lowest level of the hierarchy. Mamba layers also avoid the quadratic cost on byte-length sequences, which would otherwise make ingestion the bottleneck.</p> | |
| <p>The encoder takes byte ids $x_1, \ldots, x_{L_0}$ and produces a sequence $\hat{x}^0_1, \ldots, \hat{x}^0_{L_0}$ with each $\hat{x}^0_t \in \mathbb{R}^{D_0}$. No further preprocessing is applied between bytes and this output.</p> | |
| <h3>The dynamic chunking layer</h3> | |
| <p>Each stage $s$ contains a routing module, a downsampler, and a small compressor that produces the node K/V slots. The routing and downsampling logic is taken directly from H-Net; the K/V production is what's new, and it replaces H-Net's "main network" pathway because our downstream is the latent tree, not a same-resolution decoder.</p> | |
| <h4>Routing module</h4> | |
| <p>The routing module measures similarity between adjacent encoder outputs and emits a boundary probability at each position. The intuition is the H-Net one: when context shifts semantically, consecutive vectors become less similar; a boundary should be placed where they are least similar.</p> | |
| <div class="math-block"> | |
| $$q_t = W_q^s \hat{x}_t^s, \quad k_t = W_k^s \hat{x}_t^s, \quad p_t^s = \tfrac{1}{2}\Big(1 - \tfrac{q_t^\top k_{t-1}}{\|q_t\| \, \|k_{t-1}\|}\Big), \quad b_t^s = \mathbf{1}\{p_t^s \ge 0.5\}$$ | |
| </div> | |
| <p class="math-caption">Eq. 4.1. Scaled cosine similarity between adjacent positions, mapped to a boundary probability in $[0,1]$. $p_1^s = 1$ by definition to ensure the sequence begins with a boundary. The hard indicator $b_t^s$ is used to select which positions are retained at the next stage; the soft probability $p_t^s$ is what gradient flows through.</p> | |
| <h4>Downsampler</h4> | |
| <p>The downsampler retains exactly the positions where $b_t^s = 1$ and discards the others. The retained encoder outputs become the children of one parent node — specifically, a parent is constructed from each run of positions between consecutive boundaries.</p> | |
| <p>Concretely: if positions $t_1 < t_2 < \cdots < t_k$ are the boundary positions in the stage-$s$ output, then the segment $[t_i, t_{i+1})$ becomes one parent at stage $s+1$. Its children are the stage-$s$ nodes (or, at $s = 0$, the encoder vectors) at those positions.</p> | |
| <h4>Slot compressor</h4> | |
| <p>For each parent segment, a small Perceiver-style cross-attention block produces $r$ latent slots. Let $Z \in \mathbb{R}^{(rb) \times D_s}$ be the flattened concatenation of the $r$ slots of each of the $b$ children (or the bare encoder outputs at $s = 0$). Let $S_0 \in \mathbb{R}^{r \times D_{s+1}}$ be a learned set of $r$ summary queries shared across all parents at stage $s+1$.</p> | |
| <div class="math-block"> | |
| $$S = \mathrm{CrossAttn}(S_0, Z) \in \mathbb{R}^{r \times D_{s+1}}$$ | |
| $$K_u = \mathrm{norm}(S W_K) \in \mathbb{R}^{r \times d_K}, \quad V_u = S W_V \in \mathbb{R}^{r \times d_V}$$ | |
| </div> | |
| <p class="math-caption">Eq. 4.2. The slot compressor reads the children's latent slots (or, for leaves, encoder outputs) with $r$ learned queries, producing $r$ slot vectors. The route keys and value slots are linear projections of those slot vectors. $\mathrm{norm}$ is RMSNorm applied per slot — necessary because scores are inner products and we want the magnitude not to dominate.</p> | |
| <p>The compressor is small: one cross-attention block with one or two heads, a feedforward layer of dimension $4 D_{s+1}$, and post-norm. It is shared across all parents at the same stage. It is not shared across stages — each stage has its own compressor weights, because the relevant compression statistics differ between, say, byte→word and chapter→book.</p> | |
| <h4>Smoothing</h4> | |
| <p>During training, the discrete boundary indicator $b_t^s$ breaks gradient flow. We import H-Net's smoothing module wholesale: the boundary probability $p_t^s$ is used to interpolate between adjacent latents during compression, so that low-confidence boundaries do not propagate hard errors to the next stage. This is critical — the H-Net ablations show training collapses without it — and it transfers directly to our setting because the failure mode is identical.</p> | |
| <h3>Recursion</h3> | |
| <p>Stage $s+1$ operates on the parent latents produced by stage $s$. To do so, the parent slots are flattened back into a sequence of $r L_{s+1}$ vectors of dimension $D_{s+1}$, where $L_{s+1}$ is the number of parent nodes at stage $s$'s output. A small Mamba-2 stack then processes this sequence to produce $\hat{x}^{s+1}$, and the next routing module is applied. The same dynamic-chunking layer is applied again. And so on, until a single root remains.</p> | |
| <p>The number of stages $S$ is not hand-tuned to match the expected depth of the corpus. It is instead set generously — typically $S = 10$ or $12$ — and the ratio losses are configured so that compression terminates at the root naturally. In practice, the model often hits a regime where late stages produce only one or two boundaries, which is the signal that no further compression is meaningful.</p> | |
| <div class="callout"> | |
| <span class="callout-label">A note on signal propagation</span> | |
| <p>With more chunking stages than H-Net's two, the signal-propagation tricks from H-Net's §2.3 become correspondingly more important. We adopt all of them: post-network RMSNorm at each stage; projection on the residual pathway only, not on the main path; learning-rate modulation where outer stages get higher learning rates proportional to $\sqrt{L_s / L_{s+1}}$ and inverse to $\sqrt{D_s / D_{s+1}}$; and residual connections initialized close to zero. These are not optional decoration — they are the difference between a stable training run and a collapse.</p> | |
| </div> | |
| </section> | |
| <section id="sec-tree" aria-labelledby="h-tree"> | |
| <span class="section-number">§ 05</span> | |
| <h2 id="h-tree">The latent tree</h2> | |
| <p>The compiler's output is a tree. Not a forest, not a sharded collection, not a database — a single tree, with one root, where every node has the same structural type. The absence of a shard directory at the top is deliberate: a directory by project, document, time, or modality would be a handcrafted partitioning of the very structure the compiler is supposed to discover. If the compiler learns that, say, code and prose belong in separate subtrees, that fact will appear in the route keys of the relevant ancestors. We do not need to tell it.</p> | |
| <h3>Node structure</h3> | |
| <p>Every node $u$ in the tree stores the same fields:</p> | |
| <dl class="defs"> | |
| <dt>$K_u$</dt> | |
| <dd>Route keys, shape $\mathbb{R}^{r \times d_K}$. Used for scoring during traversal.</dd> | |
| <dt>$V_u$</dt> | |
| <dd>Value slots, shape $\mathbb{R}^{r \times d_V}$. Used by cross-attention when $u$ is selected into the working set.</dd> | |
| <dt>$\mathrm{children}(u)$</dt> | |
| <dd>A list of pointers to child nodes. Empty if and only if $u$ is a leaf.</dd> | |
| <dt>$\mathrm{raw\_ref}(u)$</dt> | |
| <dd>A pointer to a byte range in the raw corpus. Present on every node, not just leaves: a parent's raw_ref is the union of its children's, computable in one pass at compile time. This lets the model dereference raw bytes for any node, not just leaves — useful when a mid-level summary points the model at a region but the model wants the exact text.</dd> | |
| </dl> | |
| <p>There is no <code>node_type</code> field. There are no anchor flags, modality flags, or quality scores stored as metadata. A leaf differs from a parent only in whether its children list is empty.</p> | |
| <h3>Storage and layout</h3> | |
| <p>The tree is large but tractable. For a 1B-token corpus with the configuration in <a href="#sec-config">§ 11</a>, there are roughly $4 \times 10^7$ leaves and a comparable number of internal nodes summed across all levels. Each node stores $r \cdot (d_K + d_V) = 8 \cdot (256 + 2048) = 18{,}432$ scalars. At int8 quantization that is ~18 KB per node, or ~700 GB for the whole tree. This is a server-memory or fast-disk scale, not a GPU-memory scale.</p> | |
| <p>Nodes are stored in <strong>contiguous tensor pages</strong> grouped by tree level. All nodes at level $s$ live in one large array indexed by node id. Children of a node are stored as integer offsets into the level-$(s-1)$ array, not as pointers. This is the only piece of engineered structure in the system, and it is engineered for hardware, not for semantics: a level-keyed array layout lets traversal kernels batch scoring across many queries against many candidate children with regular memory access patterns. Pointer-chasing would make GPU traversal hopelessly slow.</p> | |
| <p>Route keys are kept in fast memory (HBM if the GPU has room, otherwise pinned host memory accessed by RDMA). Value slots can be cooler — they are only fetched for nodes actually selected into the working set, which is a small fraction of nodes visited.</p> | |
| <h3>Why one structural type</h3> | |
| <p>The LLA paper makes this point and it is worth keeping: a node is a node. A parent is not a "summary node," a "task node," or a "document node." It is a latent K/V bundle that the compiler produced from its children. The semantics of what a parent represents — whether it covers a paragraph, a chapter, a function, a conversation, or a transition between two unrelated documents — is a property of its learned weights and emerges from training. It is never asserted by the structure.</p> | |
| <p>This is the same commitment H-Net makes at the byte/word/phrase level, extended one level higher. Chunks are not declared; they emerge. Parents are not declared; they emerge. The structural type system has exactly one entry.</p> | |
| </section> | |
| <section id="sec-large-model" aria-labelledby="h-large-model"> | |
| <span class="section-number">§ 06</span> | |
| <h2 id="h-large-model">Large model and query head</h2> | |
| <p>The large model is a standard autoregressive sequence model. We assume a Transformer or hybrid Mamba-Transformer of around 2–8B parameters, residual-stream dimension $D_M$, and an arbitrary depth chosen for the compute budget. Nothing about the memory system requires a specific main-network architecture — the choice is orthogonal, and we follow the H-Net finding that hybrid Mamba-Transformer main networks scale slightly better than pure Transformer.</p> | |
| <p>What is new is the <strong>memory interface</strong>: a small number of designated layers in the large model emit memory queries and receive working sets back. Between those layers, the model is identical to a standard isotropic model with no memory.</p> | |
| <h3>Memory-query layers</h3> | |
| <p>We designate $M$ layers across the large model's depth as memory-query layers. At each such layer, after the block has produced its output hidden state $h_t \in \mathbb{R}^{D_M}$ for the current token, three things happen in sequence:</p> | |
| <ol> | |
| <li><strong>Query emission.</strong> A small per-layer MLP $Q_\theta^{(\ell)}$ maps $h_t$ to a memory query $q^{(\ell)} = Q_\theta^{(\ell)}(h_t) \in \mathbb{R}^{d_K}$. The query lives in the same space as the tree's route keys.</li> | |
| <li><strong>Traversal.</strong> The query traverses the tree (see <a href="#sec-traversal">§ 07</a>) and returns a working set $\mathcal{W}(q^{(\ell)})$ of value vectors.</li> | |
| <li><strong>Cross-attention.</strong> A cross-attention sub-layer reads from $\mathcal{W}(q^{(\ell)})$ using $h_t$ as query, and the result is added to the residual stream.</li> | |
| </ol> | |
| <div class="math-block"> | |
| $$h_t \leftarrow h_t + \mathrm{CrossAttn}\!\left(h_t, \mathcal{W}(q^{(\ell)})\right)$$ | |
| </div> | |
| <p class="math-caption">Eq. 6.1. The retrieved working set is consumed by a standard cross-attention sub-layer added to the residual stream. The cross-attention uses $h_t$ as the query and the working-set value vectors $\mathcal{W}(q^{(\ell)}) \subset \mathbb{R}^{d_V}$ as both keys and values, with a separate small projection learned to produce keys from those values. Standard multi-head attention; nothing exotic.</p> | |
| <p>Memory-query layers are spaced roughly uniformly through the model's depth. For a 32-layer Transformer, $M = 4$ memory-query layers placed at depths 8, 16, 24, and 31 is a reasonable starting choice. The total per-token cost of memory access is then $M$ traversals plus $M$ cross-attentions, each bounded by the size of the working set.</p> | |
| <h3>Why multiple memory queries per token</h3> | |
| <p>A single memory query per token is sufficient in principle but wasteful in practice. Different layers of the large model represent different aspects of the current token: early layers capture surface form and immediate syntax, middle layers represent semantic content, late layers represent task-relevant abstractions. Letting each of these emit its own query, into the same shared tree, allows the model to retrieve at the resolution that matters for that layer. The route keys learn to support all of these query types simultaneously because the gradients from all of them flow back through the tree.</p> | |
| <h3>When the model queries</h3> | |
| <p>There is no policy controller deciding when retrieval is necessary. The memory-query layers always fire — every token, every memory-query layer, one query. The model learns to emit <em>uninformative</em> queries when retrieval is not useful, and the resulting working sets contribute little to the residual stream because the cross-attention learns near-zero weights. This is more wasteful per token than a gated controller would be, but it has one major advantage: there is no separate gating policy to train, no hand-authored "when to retrieve" rulebook, and no extra hyperparameter for a retrieval threshold. Gradient settles all of it.</p> | |
| <p>If profiling shows that uninformative queries are a meaningful fraction of inference cost, a learned gate can be added later as a strict refinement — but only after measuring that the always-on baseline has a real problem. The bitter-lesson default is to let the model figure it out.</p> | |
| <div class="marginal"> | |
| The same argument applies to "memory writes." We do not include them. The tree is built by the compiler from ingested corpus; in-context information from the current generation is handled by the large model's own KV cache for its short attention window, exactly as in a normal Transformer. If the corpus grows during deployment, the compiler runs on the new material and inserts new nodes into the relevant subtree. There is no learned write head, because writes are an operational concern, not a modeling one. | |
| </div> | |
| </section> | |
| <section id="sec-traversal" aria-labelledby="h-traversal"> | |
| <span class="section-number">§ 07</span> | |
| <h2 id="h-traversal">Traversal</h2> | |
| <p>Given a query $q \in \mathbb{R}^{d_K}$, the traversal walks down the tree from the root, scoring children at each level and keeping a bounded frontier. This is what makes per-query memory access sub-linear in the corpus size — the model never enumerates leaves, never scans flat sequences, and never iterates over the corpus as a whole.</p> | |
| <h3>Node scoring</h3> | |
| <p>The score the traversal assigns to a node is the maximum inner product between the query and any of the node's $r$ route keys:</p> | |
| <div class="math-block"> | |
| $$s(q, u) = \max_{j \in \{1, \ldots, r\}} q^\top K_u[j]$$ | |
| </div> | |
| <p class="math-caption">Eq. 7.1. Node score. Using max-over-slots instead of a single dot product is what lets a heterogeneous parent expose several relevant directions of its subtree at once. A parent covering a transition between two different topics can have one slot aligned with each, and a query matching either topic will activate the parent.</p> | |
| <h3>Inference: hard beam search</h3> | |
| <p>At inference time, traversal is straightforward top-$m$ beam search:</p> | |
| <pre><code># Hard traversal (inference) | |
| frontier = {root} | |
| working_set = ∅ | |
| for level ℓ in 1 .. depth(tree): | |
| candidates = ⋃_{u ∈ frontier} children(u) | |
| scores = [s(q, c) for c in candidates] | |
| frontier = top_m(candidates, scores) | |
| working_set ∪= {V_u : u ∈ frontier} | |
| return working_set ∪ {V_u : u is an ancestor of any selected leaf}</code></pre> | |
| <p>The final working set is the union of value slots from every level traversed. This is what makes the read multi-resolution (<a href="#sec-multires">§ 08</a>); the cross-attention sees coarse latents from upper levels and fine latents from leaves in a single attention call, and learns its own attention pattern over them.</p> | |
| <h3>Cost analysis</h3> | |
| <p>At each level, the traversal scores at most $m \cdot b_\text{eff}$ candidates, where $b_\text{eff}$ is the effective fanout (the average number of children per parent at that level). With fixed-shape per-stage compression targeting ratios $N_s$, $b_\text{eff} \approx N_s$. Tree depth is roughly $\log_{\bar{N}}(L_0)$, where $\bar{N}$ is the geometric mean of the $N_s$.</p> | |
| <p>For the 1B-token configuration, depth is around $10$–$12$ and per-level fanout averages $4$–$8$, giving on the order of $m \cdot b_\text{eff} \cdot r \cdot \text{depth} = 8 \cdot 8 \cdot 8 \cdot 11 \approx 5{,}600$ slot dot products per memory query. With $M = 4$ memory queries per token, that is $\sim 22{,}000$ dot products per generated token attributable to memory traversal — comfortably below the cost of one attention step over a $128$K-token KV cache, and exponentially below the cost of dense attention over the corpus.</p> | |
| <h3>Training: soft traversal with annealed temperature</h3> | |
| <p>Hard top-$m$ is not differentiable in the selection step. If the model picks the wrong children early in training, the right branch never receives gradient, and the routing module has no way to learn that it should have routed differently. This is the same problem H-Net's smoothing module solves at the byte level, and we apply the same fix at the traversal level.</p> | |
| <p>During training, traversal uses a soft variant: instead of selecting the top $m$ children at each level, we compute a softmax over all children's scores with temperature $\tau$, and the working set at that level becomes a weighted combination of every candidate's value slots.</p> | |
| <div class="math-block"> | |
| $$w_c = \frac{\exp(s(q, c) / \tau)}{\sum_{c' \in \text{candidates}} \exp(s(q, c') / \tau)}$$ | |
| $$V_\text{level} = \sum_c w_c \, V_c, \qquad \mathcal{F}_\ell = \text{candidates}$$ | |
| </div> | |
| <p class="math-caption">Eq. 7.2. Soft traversal during training. Every candidate at each level contributes to the working set, weighted by softmax over its score. Gradient flows back through the weights into the route keys of every candidate, including the ones that would have been pruned under hard top-$m$. Pruning to a frontier still happens — we keep only candidates with weight above a small threshold $\epsilon$ — but the frontier is variable-size and the cutoff is soft.</p> | |
| <p>The temperature $\tau$ is annealed from high (effectively averaging) to low (effectively one-hot) over training. The schedule mirrors H-Net's logic: early training cannot tolerate hard decisions because routing has not yet learned to make them well, and late training cannot tolerate soft decisions because the cost of evaluating every candidate is prohibitive at inference and we want training to match inference behavior.</p> | |
| <p>A reasonable schedule: $\tau = 1.0$ for the first 10% of training, linearly anneal to $\tau = 0.1$ over the next 60%, then anneal to a target of $\tau = 0.01$ for the remainder. The exact schedule is a hyperparameter and will probably need tuning, but the shape — long high-temperature warmup, gradual sharpening, hard-ish endpoint — is borrowed from regimes that have worked elsewhere (Gumbel-softmax training, DeepSeek's DSA dense-then-sparse warmup, H-Net's smoothing module).</p> | |
| <h3>Straight-through estimator at the frontier cutoff</h3> | |
| <p>Even with soft routing, the model needs to learn to make confident, sparse decisions by the end of training. To encourage this, we apply a straight-through estimator at the frontier cutoff: in the forward pass, candidates whose weight falls below $\epsilon$ are zeroed out; in the backward pass, gradient flows as if the cutoff were absent. This is the same trick H-Net uses on its routing module's confidence scores, applied here at the level of beam selection rather than boundary placement.</p> | |
| </section> | |
| <section id="sec-multires" aria-labelledby="h-multires"> | |
| <span class="section-number">§ 08</span> | |
| <h2 id="h-multires">Multi-resolution read</h2> | |
| <p>A query that has traversed the tree returns a working set $\mathcal{W}(q)$ containing value slots from every level visited, not just the leaves. The cross-attention layer in the large model sees this entire collection at once and decides what to weight, in a standard attention call.</p> | |
| <p>Concretely, if a beam-$m$ traversal visited depth $D$, the working set contains up to $m \cdot r$ slot vectors per level, totaling up to $m \cdot r \cdot D$ vectors. For $m = 8$, $r = 8$, $D = 11$, that is $\le 704$ vectors per query — small enough for ordinary attention.</p> | |
| <p>The point of including upper-level latents is that the large model often needs both broad context and specific evidence in the same step. A query about a particular function in a codebase wants to see the function's exact bytes <em>and</em> the module-level summary that contextualizes them. A query about a citation wants the cited sentence <em>and</em> the surrounding section. Multi-resolution makes both available simultaneously; the cross-attention's softmax learns the right mix.</p> | |
| <h3>Raw-byte dereferencing</h3> | |
| <p>Each node carries a $\mathrm{raw\_ref}$ pointer. For a small subset of leaves (typically the top-scoring few), the working set is augmented with the byte-encoder representations of those raw spans, recomputed on demand by running $E_\text{byte}$ over the indicated bytes.</p> | |
| <p>This is the escape hatch for tasks where the latent representation cannot substitute for the bytes themselves: verbatim quotation, code editing, exact-string comparison. It is not a separate retrieval system — the leaves <em>are</em> the entry points to raw spans, and the decision to dereference is just one more piece of behavior the model learns. A learned gate, parameterized as a small MLP on each selected leaf's value slot, decides whether to dereference that leaf or to use its latent slots alone. The gate is trained against the same downstream task loss, so the model learns to dereference exactly when latent slots are insufficient.</p> | |
| <p>Dereferencing is the most expensive operation in the system — running $E_\text{byte}$ over a 1KB span on demand is non-trivial — so the gate's job is partly to keep the dereference rate low. The budget loss (see <a href="#sec-training">§ 09</a>) penalizes the expected number of dereferences per query.</p> | |
| <div class="callout"> | |
| <span class="callout-label">Why raw-byte access is not a "lexical sidecar"</span> | |
| <p>It would be tempting to add a parallel lexical index (BM25, exact-string) so that queries like "find every mention of <code>parse_config</code>" could be served by a different retrieval mechanism. We do not. The reason is exactly the bitter-lesson reason: if rare-identifier lookup is a real failure mode, training data including such queries will produce gradient pressure on the route keys to support surface-form-sensitive retrieval. The compiler's encoder operates on raw bytes from the start, so the substrate already contains the necessary information — what needs to learn is the route keys' alignment with surface-form queries. A lexical sidecar would absorb that gradient pressure and prevent the learned system from ever developing the capability.</p> | |
| </div> | |
| </section> | |
| <section id="sec-training" aria-labelledby="h-training"> | |
| <span class="section-number">§ 09</span> | |
| <h2 id="h-training">Training</h2> | |
| <p>The whole system — byte encoder, $S$ chunking stages, slot compressors, large-model backbone, query heads, cross-attention layers — is one differentiable computation graph trained against one downstream objective with a handful of auxiliary losses. The auxiliary losses exist to make the tree learnable; they do not inject a priori structure into it.</p> | |
| <h3>The objective</h3> | |
| <div class="math-block"> | |
| $$\mathcal{L} = \mathcal{L}_\text{task} + \alpha \sum_{s=0}^{S-1} \mathcal{L}^s_\text{ratio} + \beta\, \mathcal{L}_\text{route} + \gamma\, \mathcal{L}_\text{value} + \delta\, \mathcal{L}_\text{budget}$$ | |
| </div> | |
| <p class="math-caption">Eq. 9.1. The full training objective. $\mathcal{L}_\text{task}$ is next-byte prediction. The four auxiliary terms exist for specific learnability reasons described below; their weights are small enough that $\mathcal{L}_\text{task}$ dominates once the tree has begun to function. There is no anchor loss, no boundary-supervision loss, no reconstruction loss against handcrafted segments.</p> | |
| <h3>Task loss</h3> | |
| <p>The downstream task is next-byte prediction over the corpus. Standard cross-entropy on byte logits. Importantly, the task is evaluated <em>in context</em>: training examples are long-form sequences that require information from far back in the corpus, and the only way to access that information is through the memory tree. If the task were simple short-window language modeling, the tree would receive no gradient pressure to be useful and would collapse into noise. Training data therefore mixes:</p> | |
| <ul> | |
| <li>Long-form continuations within documents (the standard regime)</li> | |
| <li>Cross-document references requiring retrieval ("the paper discussed earlier in this batch said…")</li> | |
| <li>Needle-in-haystack queries with controlled needle placement at various depths</li> | |
| <li>Rare-identifier retrieval requiring surface-form-sensitive route keys</li> | |
| <li>Multi-hop tasks requiring two or more sequential memory queries</li> | |
| <li>Aggregation tasks requiring root-level statistics across all leaves</li> | |
| </ul> | |
| <p>The training data is engineered to require the tree, not to bypass it. Every failure mode that a non-end-to-end design would mitigate with a sidecar is instead a training-data category here.</p> | |
| <h3>Ratio loss (per stage)</h3> | |
| <p>For each chunking stage $s$, we apply the H-Net ratio loss with target compression $N_s$. Let $F_s$ be the realized fraction of positions retained at stage $s$ (i.e. $1/N_s$ in expectation) and $G_s$ be the mean boundary probability at stage $s$:</p> | |
| <div class="math-block"> | |
| $$\mathcal{L}^s_\text{ratio} = \frac{N_s}{N_s - 1}\Big((N_s - 1) F_s G_s + (1 - F_s)(1 - G_s)\Big)$$ | |
| </div> | |
| <p class="math-caption">Eq. 9.2. Per-stage ratio loss, taken directly from H-Net §2.2.3. Pushes the average compression toward $1/N_s$ while remaining differentiable in $G_s$, since $F_s$ is not. The loss attains minimum when $F_s = G_s = 1/N_s$, with continuous feedback through $G_s$ that drags $F_s$ to match.</p> | |
| <h3>Route loss</h3> | |
| <p>The single most important auxiliary loss. It prevents the tree from hiding relevant subtrees behind unhelpful parents. For each parent $u$ in a training batch, let $S^*(q, u) = \max_{v \in \mathrm{desc}(u)} s(q, v)$ be the best score any descendant of $u$ achieves on query $q$. We want the parent's own score to be no worse than this, up to a small margin:</p> | |
| <div class="math-block"> | |
| $$\mathcal{L}_\text{route} = \mathbb{E}_q \left[ \max\!\left(0,\; \gamma_0 + S^*(q, u) - s(q, u)\right) \right] + \mathcal{L}_\text{neg}$$ | |
| </div> | |
| <p class="math-caption">Eq. 9.3. The hinge term encourages $s(q, u) \ge S^*(q, u) - \gamma_0$ for some margin $\gamma_0$, taken in expectation over queries $q$ drawn from the actual large-model query distribution during training. Without an additional negative term, a degenerate solution sets every parent's keys to point toward every query, so we add contrastive negatives.</p> | |
| <div class="math-block"> | |
| $$\mathcal{L}_\text{neg} = -\log \frac{\exp(s(q, u^+) / \tau_\text{neg})}{\exp(s(q, u^+) / \tau_\text{neg}) + \sum_{u^-} \exp(s(q, u^-) / \tau_\text{neg})}$$ | |
| </div> | |
| <p class="math-caption">Eq. 9.4. Contrastive negatives. $u^+$ is a parent that actually contains relevant content for query $q$; $u^-$ are unrelated parents sampled from elsewhere in the tree, preferably hard negatives (siblings, same-document distractors). $\tau_\text{neg}$ is a fixed temperature, not annealed.</p> | |
| <p>The route loss is what makes the routing condition the LLA paper relies on actually hold. Without it, gradient from the task loss alone is insufficient to ensure recall — task gradient only sees the branches that were actually traversed, so branches pruned by miscalibrated routing never get the signal that they should have been kept.</p> | |
| <h3>Value loss</h3> | |
| <p>The value loss enforces that attention over a parent's value slots is a good approximation to attention over its children's value slots. Let $A_C(q) = \mathrm{Attn}(q, K_\text{children}, V_\text{children})$ be the attention output over a node's children, and $A_p(q) = \mathrm{Attn}(q, K_u, V_u)$ be the attention over the parent itself.</p> | |
| <div class="math-block"> | |
| $$\mathcal{L}_\text{value} = \mathbb{E}_q \left[ \| A_p(q) - A_C(q) \|_2^2 \right]$$ | |
| </div> | |
| <p class="math-caption">Eq. 9.5. The parent's attention output should approximate the children's. This is what allows the traversal to stop at a parent when its value slots are sufficient, instead of always descending to leaves. The expectation is over queries from the training distribution; the children's attention output is computed on a small sub-sample of children rather than all of them when fanout is large.</p> | |
| <h3>Budget loss</h3> | |
| <p>A small penalty on the expected number of nodes scored and expanded per query, plus the expected number of raw-span dereferences. This is what keeps the model from learning to expand wastefully wide beams or dereference on every leaf.</p> | |
| <div class="math-block"> | |
| $$\mathcal{L}_\text{budget} = \alpha_1 \cdot \mathbb{E}_q[\text{nodes scored}] + \alpha_2 \cdot \mathbb{E}_q[\text{leaves dereferenced}]$$ | |
| </div> | |
| <p class="math-caption">Eq. 9.6. The two terms have separate coefficients because their compute costs differ by orders of magnitude. The expected counts are differentiable through the soft traversal weights during training; at inference they are just the realized counts under hard top-$m$.</p> | |
| <h3>Annealing schedule</h3> | |
| <p>Training has three regimes, marked by the traversal temperature $\tau$:</p> | |
| <ol> | |
| <li><strong>Soft regime ($\tau \approx 1$):</strong> The first ~10% of training. Traversal is effectively averaging over all candidates at each level. Auxiliary losses dominate; task loss is large but produces useful gradient anyway because all branches receive it. The tree gets a chance to organize itself before being asked to make hard decisions.</li> | |
| <li><strong>Annealing regime ($\tau: 1 \to 0.1$):</strong> The middle ~60% of training. Temperature decreases linearly. The model is being weaned off averaging. The route loss is doing the bulk of the work to make sure that when hard decisions start being made, they preserve recall.</li> | |
| <li><strong>Hard regime ($\tau \approx 0.01$):</strong> The final ~30% of training. Traversal closely matches inference behavior. Task loss dominates and refines the model.</li> | |
| </ol> | |
| <h3>The bootstrap problem</h3> | |
| <p>At the start of training, the routing module's boundaries are essentially random, the slot compressors produce meaningless latents, the route keys point in arbitrary directions, and the large model's queries are noise. Task gradient through this chain is extremely weak. There is a real risk that nothing learns, and the system collapses to a state where the cross-attention learns to ignore the working set entirely (because it is noise) and the rest of the system never receives gradient.</p> | |
| <p>We mitigate this with a curriculum, not with a separate pretraining phase:</p> | |
| <ol> | |
| <li><strong>Stage warmup.</strong> Train with $S = 1$ active stage and a small corpus for the first few thousand steps. The model is forced to use the (shallow) memory because the corpus is in-context. The byte encoder and stage-0 compressor begin learning useful representations.</li> | |
| <li><strong>Stage activation.</strong> Add chunking stages one at a time over the next phase of training, each time increasing the effective corpus that can be addressed. The stages already trained continue learning; the new ones initialize from a small perturbation of the previous-stage compressor's weights.</li> | |
| <li><strong>Full corpus.</strong> Once all $S$ stages are active and the tree spans the target corpus, training proceeds with the full objective.</li> | |
| </ol> | |
| <p>This is a curriculum on the data and architecture, not a separate pretraining objective. The losses are the same at every step; only the corpus size and active stage count change. Everything stays end-to-end.</p> | |
| </section> | |
| <section id="sec-absent" aria-labelledby="h-absent"> | |
| <span class="section-number">§ 10</span> | |
| <h2 id="h-absent">What is deliberately absent</h2> | |
| <p>A version of this system that included the following components would be a recognizable retrieval-augmented model. It would probably work earlier in training, fail in fewer evaluations, and require less data. It would also be a different system with a different ceiling. The bitter-lesson bet is that the ceiling matters more than the floor.</p> | |
| <ul class="rejected"> | |
| <li> | |
| <span class="strike">Top-level shard directory by project / document / time / modality</span> | |
| <p class="reason">A handcrafted partitioning of the corpus before training. If these distinctions matter for routing, the upper-level route keys will discover them; if they do not, hand-coding them hurts. We use one root.</p> | |
| </li> | |
| <li> | |
| <span class="strike">Anchor list for function definitions, headers, citations, theorem statements, table rows, tool outputs</span> | |
| <p class="reason">A handcrafted ontology of "things that matter." This is tokenization at the document-structure level. H-Net Figure 4 shows the model discovering semantically coherent units without supervision; we extend the same commitment upward.</p> | |
| </li> | |
| <li> | |
| <span class="strike">BM25, exact-string, or other lexical sidecar index</span> | |
| <p class="reason">A parallel retrieval path that absorbs gradient pressure away from the learned route keys. If rare-identifier lookup is a failure mode, the route keys should learn surface-form-sensitive representations under training pressure. A sidecar prevents that learning from ever needing to happen.</p> | |
| </li> | |
| <li> | |
| <span class="strike">Memory controller with a rule-based "when to retrieve" policy</span> | |
| <p class="reason">A handcrafted policy where a learned one should live. The query head emits a query every memory-query layer, and the cross-attention learns to ignore useless retrievals. Adding a controller adds a separate optimization problem with its own failure modes.</p> | |
| </li> | |
| <li> | |
| <span class="strike">Map-reduce fallback for global aggregation queries</span> | |
| <p class="reason">Concedes an entire class of queries to a database execution path. If aggregation is in the training data, the root-level latents must learn to support it. They may or may not — but routing aggregation away from the learned system guarantees they never will.</p> | |
| </li> | |
| <li> | |
| <span class="strike">Variable per-node slot counts (4 / 8 / 16 / 32 buckets)</span> | |
| <p class="reason">A handcrafted allocation policy for node capacity. We use fixed $r$ everywhere. If certain regions need more capacity, that pressure shows up in training as poor reconstruction or routing recall in those regions, and the appropriate response is to raise $r$ globally, not to learn a bucket assignment.</p> | |
| </li> | |
| <li> | |
| <span class="strike">Variable per-stage fanout learned by the compiler</span> | |
| <p class="reason">This one is genuinely tempting and we considered it. The reason against: variable fanout breaks tensor-friendly batched traversal, and the engineering cost is severe at billion-token scale. The ratio loss provides differentiable target compression; that is enough to let the model allocate compression non-uniformly within each stage. Fully variable fanout is a v2 question, not a v1 commitment.</p> | |
| </li> | |
| <li> | |
| <span class="strike">Local per-leaf prediction losses (next-byte, span continuation)</span> | |
| <p class="reason">Redundant with the task loss. The task loss is already next-byte prediction across the corpus; per-leaf next-byte losses are the same gradient signal arriving more locally. The H-Net machinery is already there to propagate gradient through the chunking stages without needing a separate per-leaf objective.</p> | |
| </li> | |
| <li> | |
| <span class="strike">Reconstruction loss against handcrafted boundaries</span> | |
| <p class="reason">A supervised signal toward boundaries we have decided are right. The ratio loss provides unsupervised pressure toward target compression; the route loss provides gradient toward boundaries that preserve relevance. The model should discover where to chunk from those pressures plus the task loss, not from a list of where we think boundaries should be.</p> | |
| </li> | |
| <li> | |
| <span class="strike">Boundary-confidence metadata stored as a separate node field</span> | |
| <p class="reason">The confidence is already implicit in the route-key inner product. A separate field is bookkeeping for a quantity the scoring already captures.</p> | |
| </li> | |
| <li> | |
| <span class="strike">Query-time rechunking of hot subtrees</span> | |
| <p class="reason">A v2 feature. Imposing it now would constitute a learned write path with separate optimization concerns. The persistent tree is rebuilt by re-running the compiler on changed regions during corpus updates, which is an operational matter handled outside the model.</p> | |
| </li> | |
| <li> | |
| <span class="strike">A separate "memory mode" inference path distinct from the standard forward pass</span> | |
| <p class="reason">There is no separate inference path. Every forward pass uses the memory tree. Every training step uses it. The system has no mode where memory is bypassed; bypassing it is a learned no-op when the cross-attention sets near-zero weights on the working set, not a structural alternative.</p> | |
| </li> | |
| </ul> | |
| <p>Each of these would be defensible in isolation. The argument for excluding all of them simultaneously is that the bet of the system is incoherent if any of them is included: each one is a place where we have stopped trusting the learned representation to handle a particular case. If we stop trusting it in twelve places, we have built a retrieval-augmented model with a tree-shaped retrieval component, which is a different and probably weaker thing than what we are trying to build here.</p> | |
| <p class="pull">The whole bet is that <em>one mechanism, trained jointly,</em> beats twelve careful exceptions.</p> | |
| </section> | |
| <section id="sec-config" aria-labelledby="h-config"> | |
| <span class="section-number">§ 11</span> | |
| <h2 id="h-config">Concrete configuration for 1B tokens</h2> | |
| <p>The dimensions and counts below are a single coherent instantiation of the design at the target scale. They are not the only valid choice — there is a wide design space and the system should tolerate exploration — but they are the specific numbers we would build first.</p> | |
| <h3>Corpus</h3> | |
| <table class="dim-table" aria-label="Corpus parameters"> | |
| <thead><tr><th>Quantity</th><th>Value</th><th>Notes</th></tr></thead> | |
| <tbody> | |
| <tr><td>Target tokens</td><td>$10^9$</td><td>at typical UTF-8 density</td></tr> | |
| <tr><td>Target bytes $L_0$</td><td>$\approx 4 \times 10^9$</td><td>language-dependent</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>Compiler</h3> | |
| <table class="dim-table" aria-label="Compiler parameters"> | |
| <thead><tr><th>Component</th><th>Value</th><th>Notes</th></tr></thead> | |
| <tbody> | |
| <tr><td>$E_\text{byte}$ depth</td><td>4 Mamba-2 layers</td><td>byte-level encoder</td></tr> | |
| <tr><td>$D_0$</td><td>$512$</td><td>byte-encoder width</td></tr> | |
| <tr><td>Number of stages $S$</td><td>$10$</td><td>generously set; late stages may produce near-trivial output</td></tr> | |
| <tr><td>Per-stage compressor</td><td>1 cross-attn + 1 FFN</td><td>shared across all parents at the same stage</td></tr> | |
| <tr><td>Stage 0 target $N_0$</td><td>$\sim 6$</td><td>byte → word-ish (matches H-Net's natural rate)</td></tr> | |
| <tr><td>Stages 1–9 target $N_s$</td><td>$\sim 4$ each</td><td>each level a 4× compression on average</td></tr> | |
| <tr><td>$D_s$ schedule</td><td>$512 \to 768 \to 1024 \to \cdots \to 2048$</td><td>monotone non-decreasing</td></tr> | |
| <tr><td>Slots per node $r$</td><td>$8$</td><td>fixed</td></tr> | |
| <tr><td>Route key dim $d_K$</td><td>$256$</td><td>smaller than $d_V$ for cheap scoring</td></tr> | |
| <tr><td>Value dim $d_V$</td><td>$2048$</td><td>= $D_M$, matches large model</td></tr> | |
| <tr><td>Compiler parameters (total)</td><td>$\sim 250$M</td><td>across all $S$ stages</td></tr> | |
| </tbody> | |
| </table> | |
| <p>With these ratios the expected tree shape is:</p> | |
| <pre><code>level approx. count role | |
| ───── ───────────── ─────────────────────────── | |
| 0 4.0 × 10⁹ bytes (no nodes, encoder only) | |
| 0 6.7 × 10⁸ stage-0 leaf nodes (word-ish) | |
| 1 1.7 × 10⁸ stage-1 parents (phrase-ish) | |
| 2 4.2 × 10⁷ stage-2 parents (sentence-ish) | |
| 3 1.0 × 10⁷ stage-3 parents (paragraph-ish) | |
| 4 2.6 × 10⁶ stage-4 parents (section-ish) | |
| 5 6.5 × 10⁵ stage-5 parents | |
| 6 1.6 × 10⁵ stage-6 parents | |
| 7 4.1 × 10⁴ stage-7 parents | |
| 8 1.0 × 10⁴ stage-8 parents | |
| 9 2.5 × 10³ stage-9 parents | |
| 10 1 root</code></pre> | |
| <p>The "role" annotations are descriptive guesses, not declared semantics. In practice the model may discover a different mapping: the H-Net visualizations in Figure 4 of that paper show stage-2 chunks corresponding to multi-word semantic groups like "the backbone," which would land at our stage 1 or 2 by analogy. The point is that we do not assert these labels — they emerge.</p> | |
| <h3>Large model</h3> | |
| <table class="dim-table" aria-label="Large-model parameters"> | |
| <thead><tr><th>Component</th><th>Value</th><th>Notes</th></tr></thead> | |
| <tbody> | |
| <tr><td>Backbone</td><td>hybrid Mamba-Transformer</td><td>20 Mamba-2 + 12 Transformer interleaved</td></tr> | |
| <tr><td>Residual dim $D_M$</td><td>$2048$</td><td>= $d_V$ by construction</td></tr> | |
| <tr><td>Depth</td><td>$32$ blocks</td><td>before counting cross-attn layers</td></tr> | |
| <tr><td>Memory-query layers $M$</td><td>$4$</td><td>placed at depths 8, 16, 24, 31</td></tr> | |
| <tr><td>Per-query MLP $Q_\theta^{(\ell)}$</td><td>2-layer MLP, hidden $1024$</td><td>one per memory-query layer</td></tr> | |
| <tr><td>Cross-attn heads</td><td>$8$</td><td>standard multi-head attention</td></tr> | |
| <tr><td>Large-model parameters</td><td>$\sim 3$B</td><td>without memory components</td></tr> | |
| <tr><td>Total trainable parameters</td><td>$\sim 3.3$B</td><td>compiler + large model + interface</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>Inference budget</h3> | |
| <table class="dim-table" aria-label="Per-token inference cost"> | |
| <thead><tr><th>Quantity</th><th>Value</th><th>Notes</th></tr></thead> | |
| <tbody> | |
| <tr><td>Memory-query layers per token</td><td>$M = 4$</td><td></td></tr> | |
| <tr><td>Beam width $m$</td><td>$8$</td><td></td></tr> | |
| <tr><td>Avg. fanout per level</td><td>$\sim 4$</td><td></td></tr> | |
| <tr><td>Tree depth</td><td>$\sim 11$</td><td></td></tr> | |
| <tr><td>Slot dot products per query</td><td>$m \cdot b \cdot r \cdot \text{depth} \approx 5{,}600$</td><td></td></tr> | |
| <tr><td>Working-set size per query</td><td>$\le m \cdot r \cdot \text{depth} \approx 700$</td><td></td></tr> | |
| <tr><td>Slot dot products per generated token</td><td>$\approx 2.2 \times 10^4$</td><td>$M$ queries × per-query cost</td></tr> | |
| <tr><td>Raw-byte dereferences per query (target)</td><td>$\le 2$</td><td>controlled by budget loss</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>Storage</h3> | |
| <table class="dim-table" aria-label="Storage requirements"> | |
| <thead><tr><th>Component</th><th>Value</th><th>Notes</th></tr></thead> | |
| <tbody> | |
| <tr><td>Total nodes</td><td>$\sim 9 \times 10^8$</td><td>across all levels</td></tr> | |
| <tr><td>Bytes per node (int8, $K + V$)</td><td>$8 \cdot (256 + 2048) = 18{,}432$ bytes</td><td>before metadata</td></tr> | |
| <tr><td>Total node storage</td><td>$\sim 17$ TB</td><td>int8 quantization</td></tr> | |
| <tr><td>Route-key-only hot storage</td><td>$\sim 1.8$ TB</td><td>route keys only, kept in fast tier</td></tr> | |
| <tr><td>Raw corpus</td><td>$\sim 4$ GB</td><td>original bytes for dereferencing</td></tr> | |
| </tbody> | |
| </table> | |
| <p>The storage numbers are honestly large. This is one of the open problems (see <a href="#sec-open">§ 12</a>); compression beyond int8 (binary quantization, product quantization for route keys, learned hash codes) is a research direction we have not specified here. The hot-tier route-key footprint is the binding constraint for actual inference latency, and bringing it under 1 TB would qualitatively improve serving.</p> | |
| <h3>Training</h3> | |
| <table class="dim-table" aria-label="Training parameters"> | |
| <thead><tr><th>Component</th><th>Value</th><th>Notes</th></tr></thead> | |
| <tbody> | |
| <tr><td>Total training bytes</td><td>$\sim 500$B</td><td>multiple passes over training corpus</td></tr> | |
| <tr><td>Initial stage count active</td><td>$1$</td><td>warmup phase</td></tr> | |
| <tr><td>Schedule for activating stages</td><td>+1 stage every ~50B bytes</td><td>over first ~500B bytes of training</td></tr> | |
| <tr><td>Temperature schedule $\tau$</td><td>$1.0 \to 0.1 \to 0.01$</td><td>linear, knots at 10% and 70% of training</td></tr> | |
| <tr><td>Loss weight $\alpha$ (ratio)</td><td>$0.03$</td><td>per H-Net</td></tr> | |
| <tr><td>Loss weight $\beta$ (route)</td><td>$0.1$</td><td>tuned</td></tr> | |
| <tr><td>Loss weight $\gamma$ (value)</td><td>$0.05$</td><td>tuned</td></tr> | |
| <tr><td>Loss weight $\delta$ (budget)</td><td>$0.01$</td><td>tuned, small</td></tr> | |
| <tr><td>Optimizer</td><td>AdamW</td><td>with H-Net LR modulation per stage</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <section id="sec-open" aria-labelledby="h-open"> | |
| <span class="section-number">§ 12</span> | |
| <h2 id="h-open">Open problems</h2> | |
| <p>This is the most speculative system in the document, and several of its premises are not validated. A complete specification has to be honest about what is at risk.</p> | |
| <h3>Training stability across many chunking stages</h3> | |
| <p>H-Net was validated at $S = 2$ and the authors did not attempt $S = 3$. We are proposing $S \approx 10$. The signal-propagation tricks from H-Net §2.3 (norm balance, residual projection, LR modulation) scale in principle, but their interaction effects at this depth are unknown. The stage-warmup curriculum is designed to mitigate the worst case, but if late-stage chunking is structurally harder to train than early-stage chunking — which seems plausible, because the inputs to late stages are themselves outputs of trained earlier stages and inherit their noise — then the system may struggle to compress its top levels meaningfully.</p> | |
| <p>The fallback is to allow the late stages to learn near-trivial compression (single-child parents, effectively identity passes) without penalizing them too hard. The ratio loss is soft enough to allow this. The risk is that the resulting tree's upper levels do nothing useful, in which case the system reduces to a shallow tree with poor logarithmic scaling.</p> | |
| <h3>Routing recall degrading with depth</h3> | |
| <p>Each level of routing has some recall against an optimal oracle. Recall compounds multiplicatively: if level $\ell$ keeps the correct branch with probability $\rho_\ell$, total recall to a target leaf is $\prod_\ell \rho_\ell$. With $\rho_\ell = 0.95$ at each of 11 levels, total recall is only $0.57$. The route loss directly trains against this failure mode, but how well it can be pushed empirically is unknown.</p> | |
| <p>Mitigations within the design: wider beam $m$, more route slots $r$ per node, and aggressive negative mining for the contrastive route loss. Beyond a point, the only honest answer is empirical measurement — and the place where this design is most likely to fall short of dense attention.</p> | |
| <h3>Bootstrap from a cold start</h3> | |
| <p>The curriculum described in §9 aims to avoid the cold-start collapse, but it has not been validated. The hardest case is when the corpus is small at warmup and the model can essentially solve the task without using memory at all — in which case the memory components never receive useful gradient. The mitigation is to choose warmup corpora large enough that in-context attention is insufficient, forcing the model to use the (shallow) tree from step zero. Specifying what "large enough" means here is an empirical question.</p> | |
| <h3>Aggregation queries</h3> | |
| <p>"Count every occurrence of $X$ across the corpus" requires information aggregated across all leaves. The bitter-lesson commitment is that the root and near-root latents must learn to represent such aggregates if aggregation tasks are in the training data. Whether 8 slots at the root are sufficient to encode useful corpus-wide statistics is an empirical question. They may not be, in which case either $r$ must increase substantially at upper levels (violating uniform $r$) or aggregation is intrinsically not what this architecture does well. The honest position is that we will find out.</p> | |
| <h3>Hardware efficiency of tree traversal</h3> | |
| <p>Per-query compute is logarithmic in corpus size, but tree-walk kernels are not what GPUs are good at. Batched traversal across many queries with regular memory access patterns is implementable but not trivial — the level-keyed page layout in §5 is necessary but not sufficient. There is a real possibility that wall-clock latency dominates theoretical compute by a factor of 5–10× compared to a more cache-friendly access pattern, eroding the win over flat sparse attention. Resolving this is an engineering problem with no theoretical answer; it has to be measured.</p> | |
| <h3>Storage at full int8</h3> | |
| <p>17 TB of node storage for a 1B-token corpus is feasible on dense storage but expensive to serve hot. Bringing the hot-tier footprint under 1 TB likely requires aggressive quantization of route keys — binary quantization with rotation, product quantization, or learned hash codes — and we have not specified how those interact with the route and value losses. If quantization-aware training proves difficult, this is the constraint that will most directly limit deployment scale.</p> | |
| <h3>Comparison baselines</h3> | |
| <p>The design has not been compared empirically against the closest alternatives: a flat sparse attention model like DeepSeek's DSA scaled up; a retrieval-augmented model with a learned dense retriever; a fine-tuned model with extended dense context (e.g., a 1M-token Transformer with sliding window plus selected global attention). At the scale and training compute proposed, each of these is a serious competitor and the bitter-lesson argument is theoretical until we have data. The case for this design is that its <em>ceiling</em> is higher because its substrate is more learned; but ceilings are claims, not measurements.</p> | |
| <div class="callout"> | |
| <span class="callout-label">The wager, stated plainly</span> | |
| <p>The system bets that a single recursive learned mechanism, trained end-to-end against a downstream objective on training data engineered to exercise long-range retrieval, will scale with compute and data better than any system whose retrieval substrate is partly handcrafted. The bet may lose. If it loses, the losing mode will most likely be (a) training instability across deep chunking, or (b) routing recall that cannot be pushed high enough by the auxiliary losses. The bet is worth making because the alternative — handcrafted retrieval components with a fixed ceiling — has a known endgame, and this one does not.</p> | |
| </div> | |
| </section> | |
| </main> | |
| <footer> | |
| <p>End-to-End Hierarchical Memory · System design specification</p> | |
| <p>Set in Instrument Serif, Newsreader, and JetBrains Mono. Mathematics typeset with KaTeX.</p> | |
| </footer> | |
| </div> | |
| </body> | |
| </html> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment