Created
September 20, 2026 18:55
-
-
Save hammer/f1b6c40490210b2fcc387030f39d9317 to your computer and use it in GitHub Desktop.
Looped transformer for Marin's grug templates: proposed issue and replication plan for arXiv 2609.19107
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>Looped transformer for Marin's grug templates: proposed issue and replication plan for arXiv 2609.19107</title> | |
| <style> | |
| :root { | |
| --ink: #1a1a2e; --ink-secondary: #555770; --ink-faint: #8a8a9a; | |
| --accent: #8b2500; --accent-light: #c4530a; | |
| --surface: #faf9f6; --surface-raised: #f0eeea; --surface-code: #f5f3ef; | |
| --rule: #d4d0c8; --link: #8b2500; --link-hover: #c4530a; --ref-bg: #f7f5f1; | |
| --content-width: 650px; --sidenote-width: 230px; --sidenote-gap: 30px; | |
| } | |
| @media (prefers-color-scheme: dark) { | |
| :root:not([data-theme="light"]) { | |
| --ink: #d8d5cf; --ink-secondary: #9e9bab; --ink-faint: #6e6b7b; | |
| --accent: #d4764e; --accent-light: #e8956e; | |
| --surface: #1a1a24; --surface-raised: #242430; --surface-code: #20202c; | |
| --rule: #33333f; --link: #d4764e; --link-hover: #e8956e; --ref-bg: #1e1e2a; | |
| } | |
| } | |
| :root[data-theme="dark"] { | |
| --ink: #d8d5cf; --ink-secondary: #9e9bab; --ink-faint: #6e6b7b; | |
| --accent: #d4764e; --accent-light: #e8956e; | |
| --surface: #1a1a24; --surface-raised: #242430; --surface-code: #20202c; | |
| --rule: #33333f; --link: #d4764e; --link-hover: #e8956e; --ref-bg: #1e1e2a; | |
| } | |
| * { margin: 0; padding: 0; box-sizing: border-box; } | |
| body { | |
| background: var(--surface); color: var(--ink); | |
| font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; | |
| font-size: 16px; line-height: 1.7; | |
| -webkit-font-smoothing: antialiased; | |
| } | |
| .page { | |
| max-width: calc(var(--content-width) + var(--sidenote-width) + var(--sidenote-gap) + 80px); | |
| margin: 0 auto; padding: 3rem 40px 4rem; position: relative; | |
| } | |
| @media (max-width: 1060px) { | |
| .page { max-width: 100%; padding: 2rem 1.5rem 3rem; } | |
| } | |
| .content { max-width: var(--content-width); } | |
| .paper-header { max-width: var(--content-width); margin-bottom: 2.5rem; padding-bottom: 2rem; } | |
| .paper-header h1 { | |
| font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif; | |
| font-size: 2rem; font-weight: 600; line-height: 1.25; color: var(--ink); | |
| text-wrap: balance; margin-bottom: 0.6rem; letter-spacing: -0.01em; | |
| } | |
| .paper-meta { font-size: 0.875rem; color: var(--ink-secondary); line-height: 1.5; } | |
| .paper-meta .author { font-weight: 500; } | |
| .paper-meta .mumwelt-link { color: var(--ink-secondary); text-decoration: none; border-bottom: 1px dotted var(--ink-faint); } | |
| .paper-meta .mumwelt-link:hover { color: var(--accent); border-bottom-color: var(--accent); } | |
| .prompt-box { | |
| max-width: var(--content-width); margin-bottom: 2.5rem; position: relative; | |
| } | |
| .prompt-label { | |
| position: absolute; left: -0.6em; top: 50%; transform: translateY(-50%); | |
| font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif; | |
| font-size: 5rem; font-weight: 700; color: var(--ink); opacity: 0.08; | |
| line-height: 1; pointer-events: none; user-select: none; | |
| } | |
| .prompt-text { | |
| font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif; | |
| font-style: italic; font-size: 1.15rem; line-height: 1.55; color: var(--ink); | |
| position: relative; | |
| } | |
| .abstract { margin-bottom: 2.5rem; max-width: var(--content-width); } | |
| .abstract-label { | |
| font-size: 0.7rem; font-weight: 600; text-transform: uppercase; | |
| letter-spacing: 0.1em; color: var(--ink-secondary); margin-bottom: 0.5rem; | |
| } | |
| .abstract p { font-size: 0.92rem; line-height: 1.75; color: var(--ink); } | |
| h1, h2, h3 { | |
| font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif; | |
| font-weight: 600; color: var(--ink); text-wrap: balance; | |
| } | |
| h2 { font-size: 1.4rem; margin-top: 2.5rem; margin-bottom: 0.75rem; letter-spacing: -0.005em; } | |
| h3 { font-size: 1.1rem; margin-top: 1.75rem; margin-bottom: 0.5rem; } | |
| p { margin-bottom: 1rem; max-width: var(--content-width); } | |
| a { color: var(--link); text-decoration: none; border-bottom: 1px solid transparent; | |
| transition: border-color 0.15s, color 0.15s; } | |
| a:hover { color: var(--link-hover); border-bottom-color: var(--link-hover); } | |
| .date-label { cursor: default; border-bottom: 1px dotted var(--ink-faint); } | |
| strong { font-weight: 600; } | |
| .sidenote-checkbox { display: none; } | |
| .sidenote-toggle { display: none; } | |
| .sidenote { | |
| float: right; clear: right; width: var(--sidenote-width); | |
| margin-right: calc(-1 * (var(--sidenote-width) + var(--sidenote-gap))); | |
| margin-top: 0.2rem; margin-bottom: 1rem; | |
| font-size: 0.8rem; line-height: 1.5; color: var(--ink-secondary); | |
| } | |
| .sidenote-number { font-size: 0.7rem; font-weight: 600; color: var(--accent); margin-right: 0.3em; } | |
| @media (max-width: 1060px) { | |
| .sidenote-toggle { | |
| display: inline; cursor: pointer; color: var(--accent); | |
| font-size: 0.78rem; font-weight: 600; user-select: none; | |
| } | |
| .sidenote { | |
| float: none; display: none; width: 100%; margin: 0.4rem 0 0.75rem 0; | |
| font-size: 0.84rem; padding: 0.6rem 0.9rem; background: var(--surface-raised); | |
| border-radius: 4px; border-left: 2px solid var(--accent); | |
| } | |
| .sidenote-checkbox:checked + .sidenote { display: block; } | |
| .sidenote-number { display: none; } | |
| } | |
| blockquote { border-left: 2px solid var(--rule); padding-left: 1.25rem; | |
| margin: 1.25rem 0; color: var(--ink-secondary); font-style: italic; } | |
| code { font-family: 'SF Mono', Menlo, Consolas, monospace; font-size: 0.85em; | |
| background: var(--surface-code); padding: 0.15em 0.35em; border-radius: 3px; } | |
| pre { background: var(--surface-code); border: 1px solid var(--rule); border-radius: 4px; | |
| padding: 1rem 1.25rem; overflow-x: auto; margin: 1.25rem 0; max-width: var(--content-width); } | |
| pre code { background: none; padding: 0; font-size: 0.82rem; line-height: 1.6; } | |
| ul, ol { margin-bottom: 1rem; padding-left: 1.5rem; max-width: var(--content-width); } | |
| li { margin-bottom: 0.35rem; } | |
| li::marker { color: var(--ink-faint); } | |
| table { max-width: var(--content-width); border-collapse: collapse; width: 100%; | |
| margin: 1.25rem 0; font-variant-numeric: tabular-nums; font-size: 0.9rem; } | |
| thead { border-top: 2px solid var(--ink); border-bottom: 1px solid var(--ink); } | |
| th { font-weight: 600; padding: 0.3rem 0.75rem 0.35rem; text-align: left; | |
| line-height: 1.2; white-space: nowrap; font-size: 0.82rem; vertical-align: bottom; } | |
| td { padding: 0.35rem 0.75rem; border: none; vertical-align: top; } | |
| tbody { border-bottom: 1.5px solid var(--ink); } | |
| th:first-child, td:first-child { padding-left: 0; } | |
| th:last-child, td:last-child { padding-right: 0; } | |
| figure { margin: 2rem 0; max-width: var(--content-width); } | |
| figure img { width: 100%; border-radius: 3px; border: 1px solid var(--rule); } | |
| figcaption { font-size: 0.8rem; color: var(--ink-secondary); margin-top: 0.5rem; | |
| line-height: 1.5; font-style: italic; } | |
| footer.provenance { | |
| margin-top: 3rem; padding-top: 1rem; max-width: var(--content-width); | |
| font-size: 0.78rem; line-height: 1.6; color: var(--ink-faint); | |
| } | |
| footer.provenance p { margin-bottom: 0.3rem; } | |
| footer.provenance a { color: var(--ink-faint); } | |
| footer.provenance blockquote { margin: 0; padding: 0; border: none; color: inherit; } | |
| a[data-hover-title] { position: relative; } | |
| .hover-card { | |
| position: absolute; bottom: 100%; left: 50%; transform: translateX(-50%); | |
| width: 320px; max-width: 90vw; padding: 0.65rem 0.8rem; | |
| background: var(--surface-raised); border: 1px solid var(--rule); | |
| border-radius: 6px; box-shadow: 0 4px 12px rgba(0,0,0,0.1); | |
| font-size: 0.78rem; line-height: 1.45; color: var(--ink); | |
| pointer-events: none; z-index: 100; margin-bottom: 6px; | |
| opacity: 0; transition: opacity 0.12s; | |
| } | |
| a[data-hover-title]:hover .hover-card, | |
| a[data-hover-title]:focus .hover-card, | |
| a.cite:hover .hover-card { opacity: 1; } | |
| .hover-card .hc-title { font-weight: 600; margin-bottom: 0.2rem; } | |
| .hover-card .hc-meta { font-size: 0.72rem; color: var(--ink-faint); margin-bottom: 0.25rem; } | |
| .hover-card .hc-status { | |
| display: inline-block; font-size: 0.65rem; font-weight: 600; | |
| text-transform: uppercase; letter-spacing: 0.04em; | |
| padding: 0.1em 0.4em; border-radius: 3px; margin-right: 0.4em; | |
| } | |
| .hc-status-open { background: #fff3e0; color: #e65100; } | |
| .hc-status-closed, .hc-status-merged { background: #e8f5e9; color: #2e7d32; } | |
| @media (prefers-color-scheme: dark) { | |
| :root:not([data-theme="light"]) .hover-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); } | |
| :root:not([data-theme="light"]) .hc-status-open { background: #3a2a10; color: #ffb74d; } | |
| :root:not([data-theme="light"]) .hc-status-closed, | |
| :root:not([data-theme="light"]) .hc-status-merged { background: #1b3a1e; color: #66bb6a; } | |
| } | |
| :root[data-theme="dark"] .hover-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); } | |
| :root[data-theme="dark"] .hc-status-open { background: #3a2a10; color: #ffb74d; } | |
| :root[data-theme="dark"] .hc-status-closed, | |
| :root[data-theme="dark"] .hc-status-merged { background: #1b3a1e; color: #66bb6a; } | |
| .hover-card .hc-desc { color: var(--ink-secondary); } | |
| .cite { | |
| font-size: 0.72rem; vertical-align: super; line-height: 0; | |
| color: var(--accent); font-weight: 600; text-decoration: none; | |
| border-bottom: none !important; position: relative; | |
| } | |
| .cite:hover { color: var(--link-hover); } | |
| .references { margin-top: 3rem; max-width: var(--content-width); } | |
| .references h2 { font-size: 1.15rem; margin-bottom: 1rem; } | |
| .ref-list { list-style: none; padding: 0; counter-reset: ref; } | |
| .ref-list li { | |
| counter-increment: ref; display: flex; align-items: baseline; | |
| gap: 0.5em; font-size: 0.82rem; line-height: 1.55; | |
| margin-bottom: 0.4rem; color: var(--ink-secondary); | |
| } | |
| .ref-list li::before { | |
| content: "[" counter(ref) "]"; flex-shrink: 0; | |
| font-variant-numeric: tabular-nums; color: var(--ink-faint); | |
| font-size: 0.78rem; min-width: 2.2em; | |
| } | |
| .ref-list .ref-body { flex: 1; min-width: 0; } | |
| .ref-list .ref-title { font-weight: 500; color: var(--ink); } | |
| .ref-list .ref-url { | |
| font-family: 'SF Mono', Menlo, Consolas, monospace; font-size: 0.75rem; | |
| color: var(--ink-faint); word-break: break-all; margin-left: 0.4em; | |
| } | |
| .ref-list .ref-url a { color: var(--ink-faint); border-bottom: none; } | |
| .ref-list .ref-url a:hover { color: var(--link-hover); } | |
| .ref-back { | |
| color: var(--accent); text-decoration: none; border-bottom: none !important; | |
| margin-left: 0.3em; font-size: 0.78rem; | |
| } | |
| .ref-back:hover { color: var(--link-hover); } | |
| .status { | |
| display: inline-block; font-size: 0.65rem; font-weight: 600; | |
| text-transform: uppercase; letter-spacing: 0.04em; | |
| padding: 0.15em 0.5em; border-radius: 3px; vertical-align: middle; | |
| } | |
| .status-done { background: #e8f5e9; color: #2e7d32; } | |
| .status-open { background: #fff3e0; color: #e65100; } | |
| .status-blocked { background: #fce4ec; color: #c62828; } | |
| @media (prefers-color-scheme: dark) { | |
| :root:not([data-theme="light"]) .status-done { background: #1b3a1e; color: #66bb6a; } | |
| :root:not([data-theme="light"]) .status-open { background: #3a2a10; color: #ffb74d; } | |
| :root:not([data-theme="light"]) .status-blocked { background: #3a1520; color: #ef9a9a; } | |
| } | |
| :root[data-theme="dark"] .status-done { background: #1b3a1e; color: #66bb6a; } | |
| :root[data-theme="dark"] .status-open { background: #3a2a10; color: #ffb74d; } | |
| :root[data-theme="dark"] .status-blocked { background: #3a1520; color: #ef9a9a; } | |
| .cite-card { | |
| position: absolute; bottom: 100%; left: 50%; transform: translateX(-50%); | |
| width: 360px; max-width: 90vw; padding: 0.65rem 0.8rem; | |
| background: var(--surface-raised); border: 1px solid var(--rule); | |
| border-radius: 6px; box-shadow: 0 4px 12px rgba(0,0,0,0.1); | |
| font-size: 0.78rem; line-height: 1.45; color: var(--ink); | |
| pointer-events: none; z-index: 100; margin-bottom: 6px; | |
| opacity: 0; transition: opacity 0.12s; | |
| font-weight: 400; vertical-align: baseline; text-align: left; | |
| } | |
| a.cite:hover .cite-card { opacity: 1; } | |
| @media (prefers-color-scheme: dark) { | |
| :root:not([data-theme="light"]) .cite-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); } | |
| } | |
| :root[data-theme="dark"] .cite-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); } | |
| @media print { | |
| body { font-size: 11pt; } | |
| .page { max-width: 100%; padding: 0; } | |
| .sidenote { float: right; width: 180px; margin-right: -210px; } | |
| .sidenote-toggle { display: none; } | |
| a { color: inherit; border-bottom: none; } | |
| .hover-card { display: none; } | |
| .cite-card { display: none; } | |
| } | |
| .katex{font:normal 1.21em KaTeX_Main,Times New Roman,serif;line-height:1.2;position:relative;text-indent:0;text-rendering:auto}.katex *{-ms-high-contrast-adjust:none!important;border-color:currentColor}.katex .katex-version:after{content:"0.18.7"}.katex .katex-mathml{border:0;-webkit-clip-path:inset(50%);clip-path:inset(50%);height:1px;overflow:hidden;padding:0;position:absolute;width:1px}.katex .katex-html>.katex-newline{display:block}.katex .katex-base{position:relative;white-space:nowrap;width:-webkit-min-content;width:-moz-min-content;width:min-content}.katex .katex-base,.katex .katex-strut{display:inline-block}.katex .textbf{font-weight:700}.katex .textit{font-style:italic}.katex .textrm{font-family:KaTeX_Main}.katex .textsf{font-family:KaTeX_SansSerif}.katex .texttt{font-family:KaTeX_Typewriter}.katex .mathnormal{font-family:KaTeX_Math;font-style:italic}.katex .mathit{font-family:KaTeX_Main;font-style:italic}.katex .mathrm{font-style:normal}.katex .mathbf{font-family:KaTeX_Main;font-weight:700}.katex .boldsymbol{font-family:KaTeX_Math;font-style:italic;font-weight:700}.katex .amsrm,.katex .mathbb,.katex .textbb{font-family:KaTeX_AMS}.katex .mathcal{font-family:KaTeX_Caligraphic}.katex .mathfrak,.katex .textfrak{font-family:KaTeX_Fraktur}.katex .mathboldfrak,.katex .textboldfrak{font-family:KaTeX_Fraktur;font-weight:700}.katex .mathtt{font-family:KaTeX_Typewriter}.katex .mathscr,.katex .textscr{font-family:KaTeX_Script}.katex .mathsf,.katex .textsf{font-family:KaTeX_SansSerif}.katex .mathboldsf,.katex .textboldsf{font-family:KaTeX_SansSerif;font-weight:700}.katex .mathitsf,.katex .mathsfit,.katex .textitsf{font-family:KaTeX_SansSerif;font-style:italic}.katex .mainrm{font-family:KaTeX_Main;font-style:normal}.katex .vlist-t{border-collapse:collapse;display:inline-table;table-layout:fixed}.katex .vlist-r{display:table-row}.katex .vlist{display:table-cell;position:relative;vertical-align:bottom}.katex .vlist>span{display:block;height:0;position:relative}.katex .vlist>span>span{display:inline-block}.katex .vlist>span>.pstrut{overflow:hidden;width:0}.katex .vlist-t2{margin-right:-2px}.katex .vlist-s{display:table-cell;font-size:1px;min-width:2px;vertical-align:bottom;width:2px}.katex .katex-vbox{align-items:baseline;display:inline-flex;flex-direction:column}.katex .katex-thinbox{display:inline-flex;flex-direction:row;max-width:0;width:0}.katex .msupsub{text-align:left}.katex .mfrac>span>span{text-align:center}.katex .mfrac .frac-line{border-bottom-style:solid;display:inline-block;width:100%}.katex .katex-hdashline,.katex .katex-hline,.katex .katex-overline .overline-line,.katex .katex-rule,.katex .katex-underline .underline-line,.katex .mfrac .frac-line{min-height:1px}.katex .mspace{display:inline-block}.katex .katex-smash{display:inline;line-height:0}.katex .clap,.katex .llap,.katex .rlap{position:relative;width:0}.katex .clap>.katex-inner,.katex .llap>.katex-inner,.katex .rlap>.katex-inner{position:absolute}.katex .clap>.katex-fix,.katex .llap>.katex-fix,.katex .rlap>.katex-fix{display:inline-block}.katex .llap>.katex-inner{right:0}.katex .clap>.katex-inner,.katex .rlap>.katex-inner{left:0}.katex .clap>.katex-inner>span{margin-left:-50%;margin-right:50%}.katex .katex-rule{border:0 solid;display:inline-block;position:relative}.katex .katex-hline,.katex .katex-overline .overline-line,.katex .katex-underline .underline-line{border-bottom-style:solid;display:inline-block;width:100%}.katex .katex-hdashline{border-bottom-style:dashed;display:inline-block;width:100%}.katex .sqrt>.katex-root{margin-left:.2777777778em;margin-right:-.5555555556em}.katex .fontsize-ensurer.reset-size1.size1,.katex .katex-sizing.reset-size1.size1{font-size:1em}.katex .fontsize-ensurer.reset-size1.size2,.katex .katex-sizing.reset-size1.size2{font-size:1.2em}.katex .fontsize-ensurer.reset-size1.size3,.katex .katex-sizing.reset-size1.size3{font-size:1.4em}.katex .fontsize-ensurer.reset-size1.size4,.katex .katex-sizing.reset-size1.size4{font-size:1.6em}.katex .fontsize-ensurer.reset-size1.size5,.katex .katex-sizing.reset-size1.size5{font-size:1.8em}.katex .fontsize-ensurer.reset-size1.size6,.katex .katex-sizing.reset-size1.size6{font-size:2em}.katex .fontsize-ensurer.reset-size1.size7,.katex .katex-sizing.reset-size1.size7{font-size:2.4em}.katex .fontsize-ensurer.reset-size1.size8,.katex .katex-sizing.reset-size1.size8{font-size:2.88em}.katex .fontsize-ensurer.reset-size1.size9,.katex .katex-sizing.reset-size1.size9{font-size:3.456em}.katex .fontsize-ensurer.reset-size1.size10,.katex .katex-sizing.reset-size1.size10{font-size:4.148em}.katex .fontsize-ensurer.reset-size1.size11,.katex .katex-sizing.reset-size1.size11{font-size:4.976em}.katex .fontsize-ensurer.reset-size2.size1,.katex .katex-sizing.reset-size2.size1{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size2.size2,.katex .katex-sizing.reset-size2.size2{font-size:1em}.katex .fontsize-ensurer.reset-size2.size3,.katex .katex-sizing.reset-size2.size3{font-size:1.1666666667em}.katex .fontsize-ensurer.reset-size2.size4,.katex .katex-sizing.reset-size2.size4{font-size:1.3333333333em}.katex .fontsize-ensurer.reset-size2.size5,.katex .katex-sizing.reset-size2.size5{font-size:1.5em}.katex .fontsize-ensurer.reset-size2.size6,.katex .katex-sizing.reset-size2.size6{font-size:1.6666666667em}.katex .fontsize-ensurer.reset-size2.size7,.katex .katex-sizing.reset-size2.size7{font-size:2em}.katex .fontsize-ensurer.reset-size2.size8,.katex .katex-sizing.reset-size2.size8{font-size:2.4em}.katex .fontsize-ensurer.reset-size2.size9,.katex .katex-sizing.reset-size2.size9{font-size:2.88em}.katex .fontsize-ensurer.reset-size2.size10,.katex .katex-sizing.reset-size2.size10{font-size:3.4566666667em}.katex .fontsize-ensurer.reset-size2.size11,.katex .katex-sizing.reset-size2.size11{font-size:4.1466666667em}.katex .fontsize-ensurer.reset-size3.size1,.katex .katex-sizing.reset-size3.size1{font-size:.7142857143em}.katex .fontsize-ensurer.reset-size3.size2,.katex .katex-sizing.reset-size3.size2{font-size:.8571428571em}.katex .fontsize-ensurer.reset-size3.size3,.katex .katex-sizing.reset-size3.size3{font-size:1em}.katex .fontsize-ensurer.reset-size3.size4,.katex .katex-sizing.reset-size3.size4{font-size:1.1428571429em}.katex .fontsize-ensurer.reset-size3.size5,.katex .katex-sizing.reset-size3.size5{font-size:1.2857142857em}.katex .fontsize-ensurer.reset-size3.size6,.katex .katex-sizing.reset-size3.size6{font-size:1.4285714286em}.katex .fontsize-ensurer.reset-size3.size7,.katex .katex-sizing.reset-size3.size7{font-size:1.7142857143em}.katex .fontsize-ensurer.reset-size3.size8,.katex .katex-sizing.reset-size3.size8{font-size:2.0571428571em}.katex .fontsize-ensurer.reset-size3.size9,.katex .katex-sizing.reset-size3.size9{font-size:2.4685714286em}.katex .fontsize-ensurer.reset-size3.size10,.katex .katex-sizing.reset-size3.size10{font-size:2.9628571429em}.katex .fontsize-ensurer.reset-size3.size11,.katex .katex-sizing.reset-size3.size11{font-size:3.5542857143em}.katex .fontsize-ensurer.reset-size4.size1,.katex .katex-sizing.reset-size4.size1{font-size:.625em}.katex .fontsize-ensurer.reset-size4.size2,.katex .katex-sizing.reset-size4.size2{font-size:.75em}.katex .fontsize-ensurer.reset-size4.size3,.katex .katex-sizing.reset-size4.size3{font-size:.875em}.katex .fontsize-ensurer.reset-size4.size4,.katex .katex-sizing.reset-size4.size4{font-size:1em}.katex .fontsize-ensurer.reset-size4.size5,.katex .katex-sizing.reset-size4.size5{font-size:1.125em}.katex .fontsize-ensurer.reset-size4.size6,.katex .katex-sizing.reset-size4.size6{font-size:1.25em}.katex .fontsize-ensurer.reset-size4.size7,.katex .katex-sizing.reset-size4.size7{font-size:1.5em}.katex .fontsize-ensurer.reset-size4.size8,.katex .katex-sizing.reset-size4.size8{font-size:1.8em}.katex .fontsize-ensurer.reset-size4.size9,.katex .katex-sizing.reset-size4.size9{font-size:2.16em}.katex .fontsize-ensurer.reset-size4.size10,.katex .katex-sizing.reset-size4.size10{font-size:2.5925em}.katex .fontsize-ensurer.reset-size4.size11,.katex .katex-sizing.reset-size4.size11{font-size:3.11em}.katex .fontsize-ensurer.reset-size5.size1,.katex .katex-sizing.reset-size5.size1{font-size:.5555555556em}.katex .fontsize-ensurer.reset-size5.size2,.katex .katex-sizing.reset-size5.size2{font-size:.6666666667em}.katex .fontsize-ensurer.reset-size5.size3,.katex .katex-sizing.reset-size5.size3{font-size:.7777777778em}.katex .fontsize-ensurer.reset-size5.size4,.katex .katex-sizing.reset-size5.size4{font-size:.8888888889em}.katex .fontsize-ensurer.reset-size5.size5,.katex .katex-sizing.reset-size5.size5{font-size:1em}.katex .fontsize-ensurer.reset-size5.size6,.katex .katex-sizing.reset-size5.size6{font-size:1.1111111111em}.katex .fontsize-ensurer.reset-size5.size7,.katex .katex-sizing.reset-size5.size7{font-size:1.3333333333em}.katex .fontsize-ensurer.reset-size5.size8,.katex .katex-sizing.reset-size5.size8{font-size:1.6em}.katex .fontsize-ensurer.reset-size5.size9,.katex .katex-sizing.reset-size5.size9{font-size:1.92em}.katex .fontsize-ensurer.reset-size5.size10,.katex .katex-sizing.reset-size5.size10{font-size:2.3044444444em}.katex .fontsize-ensurer.reset-size5.size11,.katex .katex-sizing.reset-size5.size11{font-size:2.7644444444em}.katex .fontsize-ensurer.reset-size6.size1,.katex .katex-sizing.reset-size6.size1{font-size:.5em}.katex .fontsize-ensurer.reset-size6.size2,.katex .katex-sizing.reset-size6.size2{font-size:.6em}.katex .fontsize-ensurer.reset-size6.size3,.katex .katex-sizing.reset-size6.size3{font-size:.7em}.katex .fontsize-ensurer.reset-size6.size4,.katex .katex-sizing.reset-size6.size4{font-size:.8em}.katex .fontsize-ensurer.reset-size6.size5,.katex .katex-sizing.reset-size6.size5{font-size:.9em}.katex .fontsize-ensurer.reset-size6.size6,.katex .katex-sizing.reset-size6.size6{font-size:1em}.katex .fontsize-ensurer.reset-size6.size7,.katex .katex-sizing.reset-size6.size7{font-size:1.2em}.katex .fontsize-ensurer.reset-size6.size8,.katex .katex-sizing.reset-size6.size8{font-size:1.44em}.katex .fontsize-ensurer.reset-size6.size9,.katex .katex-sizing.reset-size6.size9{font-size:1.728em}.katex .fontsize-ensurer.reset-size6.size10,.katex .katex-sizing.reset-size6.size10{font-size:2.074em}.katex .fontsize-ensurer.reset-size6.size11,.katex .katex-sizing.reset-size6.size11{font-size:2.488em}.katex .fontsize-ensurer.reset-size7.size1,.katex .katex-sizing.reset-size7.size1{font-size:.4166666667em}.katex .fontsize-ensurer.reset-size7.size2,.katex .katex-sizing.reset-size7.size2{font-size:.5em}.katex .fontsize-ensurer.reset-size7.size3,.katex .katex-sizing.reset-size7.size3{font-size:.5833333333em}.katex .fontsize-ensurer.reset-size7.size4,.katex .katex-sizing.reset-size7.size4{font-size:.6666666667em}.katex .fontsize-ensurer.reset-size7.size5,.katex .katex-sizing.reset-size7.size5{font-size:.75em}.katex .fontsize-ensurer.reset-size7.size6,.katex .katex-sizing.reset-size7.size6{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size7.size7,.katex .katex-sizing.reset-size7.size7{font-size:1em}.katex .fontsize-ensurer.reset-size7.size8,.katex .katex-sizing.reset-size7.size8{font-size:1.2em}.katex .fontsize-ensurer.reset-size7.size9,.katex .katex-sizing.reset-size7.size9{font-size:1.44em}.katex .fontsize-ensurer.reset-size7.size10,.katex .katex-sizing.reset-size7.size10{font-size:1.7283333333em}.katex .fontsize-ensurer.reset-size7.size11,.katex .katex-sizing.reset-size7.size11{font-size:2.0733333333em}.katex .fontsize-ensurer.reset-size8.size1,.katex .katex-sizing.reset-size8.size1{font-size:.3472222222em}.katex .fontsize-ensurer.reset-size8.size2,.katex .katex-sizing.reset-size8.size2{font-size:.4166666667em}.katex .fontsize-ensurer.reset-size8.size3,.katex .katex-sizing.reset-size8.size3{font-size:.4861111111em}.katex .fontsize-ensurer.reset-size8.size4,.katex .katex-sizing.reset-size8.size4{font-size:.5555555556em}.katex .fontsize-ensurer.reset-size8.size5,.katex .katex-sizing.reset-size8.size5{font-size:.625em}.katex .fontsize-ensurer.reset-size8.size6,.katex .katex-sizing.reset-size8.size6{font-size:.6944444444em}.katex .fontsize-ensurer.reset-size8.size7,.katex .katex-sizing.reset-size8.size7{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size8.size8,.katex .katex-sizing.reset-size8.size8{font-size:1em}.katex .fontsize-ensurer.reset-size8.size9,.katex .katex-sizing.reset-size8.size9{font-size:1.2em}.katex .fontsize-ensurer.reset-size8.size10,.katex .katex-sizing.reset-size8.size10{font-size:1.4402777778em}.katex .fontsize-ensurer.reset-size8.size11,.katex .katex-sizing.reset-size8.size11{font-size:1.7277777778em}.katex .fontsize-ensurer.reset-size9.size1,.katex .katex-sizing.reset-size9.size1{font-size:.2893518519em}.katex .fontsize-ensurer.reset-size9.size2,.katex .katex-sizing.reset-size9.size2{font-size:.3472222222em}.katex .fontsize-ensurer.reset-size9.size3,.katex .katex-sizing.reset-size9.size3{font-size:.4050925926em}.katex .fontsize-ensurer.reset-size9.size4,.katex .katex-sizing.reset-size9.size4{font-size:.462962963em}.katex .fontsize-ensurer.reset-size9.size5,.katex .katex-sizing.reset-size9.size5{font-size:.5208333333em}.katex .fontsize-ensurer.reset-size9.size6,.katex .katex-sizing.reset-size9.size6{font-size:.5787037037em}.katex .fontsize-ensurer.reset-size9.size7,.katex .katex-sizing.reset-size9.size7{font-size:.6944444444em}.katex .fontsize-ensurer.reset-size9.size8,.katex .katex-sizing.reset-size9.size8{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size9.size9,.katex .katex-sizing.reset-size9.size9{font-size:1em}.katex .fontsize-ensurer.reset-size9.size10,.katex .katex-sizing.reset-size9.size10{font-size:1.2002314815em}.katex .fontsize-ensurer.reset-size9.size11,.katex .katex-sizing.reset-size9.size11{font-size:1.4398148148em}.katex .fontsize-ensurer.reset-size10.size1,.katex .katex-sizing.reset-size10.size1{font-size:.2410800386em}.katex .fontsize-ensurer.reset-size10.size2,.katex .katex-sizing.reset-size10.size2{font-size:.2892960463em}.katex .fontsize-ensurer.reset-size10.size3,.katex .katex-sizing.reset-size10.size3{font-size:.337512054em}.katex .fontsize-ensurer.reset-size10.size4,.katex .katex-sizing.reset-size10.size4{font-size:.3857280617em}.katex .fontsize-ensurer.reset-size10.size5,.katex .katex-sizing.reset-size10.size5{font-size:.4339440694em}.katex .fontsize-ensurer.reset-size10.size6,.katex .katex-sizing.reset-size10.size6{font-size:.4821600771em}.katex .fontsize-ensurer.reset-size10.size7,.katex .katex-sizing.reset-size10.size7{font-size:.5785920926em}.katex .fontsize-ensurer.reset-size10.size8,.katex .katex-sizing.reset-size10.size8{font-size:.6943105111em}.katex .fontsize-ensurer.reset-size10.size9,.katex .katex-sizing.reset-size10.size9{font-size:.8331726133em}.katex .fontsize-ensurer.reset-size10.size10,.katex .katex-sizing.reset-size10.size10{font-size:1em}.katex .fontsize-ensurer.reset-size10.size11,.katex .katex-sizing.reset-size10.size11{font-size:1.1996142719em}.katex .fontsize-ensurer.reset-size11.size1,.katex .katex-sizing.reset-size11.size1{font-size:.2009646302em}.katex .fontsize-ensurer.reset-size11.size2,.katex .katex-sizing.reset-size11.size2{font-size:.2411575563em}.katex .fontsize-ensurer.reset-size11.size3,.katex .katex-sizing.reset-size11.size3{font-size:.2813504823em}.katex .fontsize-ensurer.reset-size11.size4,.katex .katex-sizing.reset-size11.size4{font-size:.3215434084em}.katex .fontsize-ensurer.reset-size11.size5,.katex .katex-sizing.reset-size11.size5{font-size:.3617363344em}.katex .fontsize-ensurer.reset-size11.size6,.katex .katex-sizing.reset-size11.size6{font-size:.4019292605em}.katex .fontsize-ensurer.reset-size11.size7,.katex .katex-sizing.reset-size11.size7{font-size:.4823151125em}.katex .fontsize-ensurer.reset-size11.size8,.katex .katex-sizing.reset-size11.size8{font-size:.578778135em}.katex .fontsize-ensurer.reset-size11.size9,.katex .katex-sizing.reset-size11.size9{font-size:.6945337621em}.katex .fontsize-ensurer.reset-size11.size10,.katex .katex-sizing.reset-size11.size10{font-size:.8336012862em}.katex .fontsize-ensurer.reset-size11.size11,.katex .katex-sizing.reset-size11.size11{font-size:1em}.katex .delimsizing.size1{font-family:KaTeX_Size1}.katex .delimsizing.size2{font-family:KaTeX_Size2}.katex .delimsizing.size3{font-family:KaTeX_Size3}.katex .delimsizing.size4{font-family:KaTeX_Size4}.katex .delimsizing.mult .delim-size1>span{font-family:KaTeX_Size1}.katex .delimsizing.mult .delim-size4>span{font-family:KaTeX_Size4}.katex .nulldelimiter{display:inline-block;width:.12em}.katex .delimcenter,.katex .op-symbol{position:relative}.katex .op-symbol.small-op{font-family:KaTeX_Size1}.katex .op-symbol.large-op{font-family:KaTeX_Size2}.katex .katex-accent>.vlist-t,.katex .op-limits>.vlist-t{text-align:center}.katex .katex-accent .accent-body{position:relative}.katex .katex-accent .accent-body:not(.accent-full){width:0}.katex .katex-overlay{display:block}.katex .mtable .vertical-separator{display:inline-block;min-width:1px}.katex .mtable .arraycolsep{display:inline-block}.katex .mtable .col-align-c>.vlist-t{text-align:center}.katex .mtable .col-align-l>.vlist-t{text-align:left}.katex .mtable .col-align-r>.vlist-t{text-align:right}.katex .svg-align{text-align:left}.katex svg{fill:currentColor;stroke:currentColor;display:block;height:inherit;position:absolute;width:100%}.katex svg path{stroke:none}.katex svg{fill-rule:nonzero;fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1}.katex img{border-style:none;max-height:none;max-width:none;min-height:0;min-width:0}.katex .katex-stretchy{display:block;overflow:hidden;position:relative;width:100%}.katex .katex-stretchy:after,.katex .katex-stretchy:before{content:""}.katex .hide-tail{overflow:hidden;position:relative;width:100%}.katex .halfarrow-left{left:0;overflow:hidden;position:absolute;width:50.2%}.katex .halfarrow-right{overflow:hidden;position:absolute;right:0;width:50.2%}.katex .brace-left{left:0;overflow:hidden;position:absolute;width:25.1%}.katex .brace-center{left:25%;overflow:hidden;position:absolute;width:50%}.katex .brace-right{overflow:hidden;position:absolute;right:0;width:25.1%}.katex .x-arrow-pad{padding:0 .5em}.katex .cd-arrow-pad{padding:0 .55556em 0 .27778em}.katex .mover,.katex .munder,.katex .x-arrow{text-align:center}.katex .boxpad{padding:0 .3em}.katex .fbox,.katex .fcolorbox{border:.04em solid;box-sizing:border-box}.katex .cancel-pad{padding:0 .2em}.katex .cancel-lap{margin-left:-.2em;margin-right:-.2em}.katex .katex-sout{border-bottom-style:solid;border-bottom-width:.08em}.katex .angl{border-right:.049em solid;border-top:.049em solid;box-sizing:border-box;margin-right:.03889em}.katex .anglpad{padding:0 .03889em}.katex .reflectbox{display:inline-block;transform:scaleX(-1)}.katex .eqn-num:before{content:"(" counter(katexEqnNo) ")";counter-increment:katexEqnNo}.katex .mml-eqn-num:before{content:"(" counter(mmlEqnNo) ")";counter-increment:mmlEqnNo}.katex .mtr-glue{width:50%}.katex .cd-vert-arrow{display:inline-block;position:relative}.katex .cd-label-left{display:inline-block;position:absolute;right:calc(50% + .3em);text-align:left}.katex .cd-label-right{display:inline-block;left:calc(50% + .3em);position:absolute;text-align:right}.katex-display{display:block;margin:1em 0;text-align:center}.katex-display>.katex{display:block;text-align:center;white-space:nowrap}.katex-display>.katex>.katex-html{display:block;position:relative}.katex-display>.katex>.katex-html>.katex-tag{position:absolute;right:0}.katex-display.leqno>.katex>.katex-html>.katex-tag{left:0;right:auto}.katex-display.fleqn>.katex{padding-left:2em;text-align:left}body{counter-reset:katexEqnNo mmlEqnNo} | |
| </style> | |
| <script>(function() { | |
| function __mumInit() { | |
| // Restore external links that htmlpreview.github.io's loader rewrote to | |
| // in-page anchors (it treats any href containing '#' as a local anchor). | |
| // Runs now (htmlpreview re-executes this script after its rewrite), again | |
| // after delays, and on click as a last line of defense. | |
| function restoreOrigHrefs() { | |
| document.querySelectorAll('a[data-orig-href]').forEach(function(a) { | |
| var orig = a.getAttribute('data-orig-href'); | |
| if (orig && a.getAttribute('href') !== orig) a.setAttribute('href', orig); | |
| }); | |
| } | |
| restoreOrigHrefs(); | |
| setTimeout(restoreOrigHrefs, 100); | |
| setTimeout(restoreOrigHrefs, 1000); | |
| document.addEventListener('click', function(e) { | |
| var t = e.target && e.target.closest ? e.target.closest('a[data-orig-href]') : null; | |
| if (t) { var o = t.getAttribute('data-orig-href'); if (o && t.getAttribute('href') !== o) t.setAttribute('href', o); } | |
| }, true); | |
| var vt = document.getElementById('viewing-time'); | |
| if (vt) { | |
| function pad(n) { return n < 10 ? '0' + n : n; } | |
| function updateTime() { | |
| var d = new Date(); | |
| vt.textContent = d.getUTCFullYear() + '-' + pad(d.getUTCMonth()+1) + '-' + pad(d.getUTCDate()) + ' ' + pad(d.getUTCHours()) + ':' + pad(d.getUTCMinutes()) + ' UTC'; | |
| } | |
| updateTime(); | |
| setInterval(updateTime, 60000); | |
| } | |
| document.querySelectorAll('a[data-hover-title]').forEach(function(a) { | |
| var card = document.createElement('span'); | |
| card.className = 'hover-card'; | |
| var title = a.getAttribute('data-hover-title') || ''; | |
| var status = a.getAttribute('data-hover-status') || ''; | |
| var owner = a.getAttribute('data-hover-owner') || ''; | |
| var desc = a.getAttribute('data-hover-desc') || ''; | |
| var statusCls = 'hc-status hc-status-' + status.toLowerCase().replace(/[^a-z]/g, ''); | |
| var html = '<div class="hc-title">' + title + '</div>'; | |
| var meta = []; | |
| if (status) meta.push('<span class="' + statusCls + '">' + status + '</span>'); | |
| if (owner) meta.push(owner); | |
| if (meta.length) html += '<div class="hc-meta">' + meta.join(' ') + '</div>'; | |
| if (desc) html += '<div class="hc-desc">' + desc + '</div>'; | |
| card.innerHTML = html; | |
| a.appendChild(card); | |
| }); | |
| document.querySelectorAll('a.cite').forEach(function(a) { | |
| var href = a.getAttribute('href') || ''; | |
| if (!href.startsWith('#ref-')) return; | |
| var li = document.getElementById(href.slice(1)); | |
| if (!li) return; | |
| var body = li.querySelector('.ref-body'); | |
| if (!body) return; | |
| var refUrl = body.querySelector('.ref-url a'); | |
| var refHref = refUrl ? refUrl.getAttribute('href') : ''; | |
| var hoverSource = refHref ? document.querySelector('a[data-hover-title][href="' + refHref + '"]') : null; | |
| if (hoverSource) { | |
| var card = document.createElement('span'); | |
| card.className = 'hover-card'; | |
| var t = hoverSource.getAttribute('data-hover-title') || ''; | |
| var s = hoverSource.getAttribute('data-hover-status') || ''; | |
| var o = hoverSource.getAttribute('data-hover-owner') || ''; | |
| var d = hoverSource.getAttribute('data-hover-desc') || ''; | |
| var sc = 'hc-status hc-status-' + s.toLowerCase().replace(/[^a-z]/g, ''); | |
| var h = '<div class="hc-title">' + t + '</div>'; | |
| var m = []; | |
| if (s) m.push('<span class="' + sc + '">' + s + '</span>'); | |
| if (o) m.push(o); | |
| if (m.length) h += '<div class="hc-meta">' + m.join(' ') + '</div>'; | |
| if (d) h += '<div class="hc-desc">' + d + '</div>'; | |
| card.innerHTML = h; | |
| a.appendChild(card); | |
| } else { | |
| var title = body.querySelector('.ref-title'); | |
| if (!title) return; | |
| var card = document.createElement('span'); | |
| card.className = 'cite-card'; | |
| card.textContent = title.textContent; | |
| a.appendChild(card); | |
| } | |
| }); | |
| } | |
| if (document.readyState !== 'loading') { __mumInit(); } | |
| else { document.addEventListener('DOMContentLoaded', __mumInit); } | |
| })();</script> | |
| </head> | |
| <body> | |
| <div class="page"> | |
| <header class="paper-header"> | |
| <h1>Looped transformer for Marin's grug templates: proposed issue and replication plan for arXiv 2609.19107</h1> | |
| <div class="paper-meta">Posed by <span class="author">hammer</span>, answered by <a class="mumwelt-link" href="https://github.com/marin-community/mumwelt">mumwelt</a> · <span class="date-label" title="Generated 2026-09-20 18:55 UTC">Published 2026-09-20</span></div> | |
| </header> | |
| <div class="content"> | |
| <p>Prepared 2026-09-20 for the marin-community/marin repo. Not filed. | |
| Proposed title and labels:</p> | |
| <ul> | |
| <li>Title: | |
| <code>[grug] Looped transformer variant (K>1 core, growth) for arXiv 2609.19107</code></li> | |
| <li>Labels: <code>experiment, agent-generated</code></li> | |
| </ul> | |
| <h2 id="what-the-study-changed-about-the-plan">What the study changed | |
| about the plan</h2> | |
| <ul> | |
| <li>Marin already has three pieces of this work: the dense paper | |
| replication in PR #9293 (K=1 only, no runs yet), the boundary-operator | |
| MoE experiment in #9280 whose first gate failed, and the June re-entrant | |
| MoE attempt in PR #6387 where every looped arm lost. The plan extends | |
| the dense replication and reuses #6387's per-K eval idea.</li> | |
| <li>The dense replication differs from the paper's released code in six | |
| places. Three of them move its own targets: head size, the MLP width | |
| rule, and the uniform init bound. The width rule was verified against | |
| the paper's exact d26 parameter count and the others against the | |
| released source.</li> | |
| <li>By the paper's own fit, tied looping is compute-neutral at the base | |
| size. The paper's value is in the boundary operator at equal FLOPs, in | |
| growth, and in the exponent. That reshaped every gate.</li> | |
| </ul> | |
| <h2 id="what-the-adversarial-review-changed">What the adversarial review | |
| changed</h2> | |
| <p>Two independent reviewers read the draft: a cold reader with no | |
| context, and a technical critic with the code. Findings that | |
| survived:</p> | |
| <ul> | |
| <li>The first gate compared arms at different token counts and its stop | |
| rule could pass on extra FLOPs alone. The first run is now the one | |
| confound-free pair in the family, Untied-2 against Deep Vanilla under | |
| one recipe.</li> | |
| <li>Dormant cores had no weight-decay exclusion, the EMA copy was not | |
| grown, and the checkpointed K went stale after tied growth. All three | |
| are now specified mechanisms with tests.</li> | |
| <li>The re-warm formula went negative before growth, and the core copy | |
| was not resume-safe. Both fixed.</li> | |
| <li>The growth gate and the data-repetition gate had no iso-FLOP | |
| control. Both now have one, and the expected margins are stated so a | |
| fail is interpretable.</li> | |
| <li>The exponent gate relied on a two-degree-of-freedom fit. It is now a | |
| monotone iso-FLOP gap rule, with the fit reported only.</li> | |
| <li>The flat block tuple, the per-call K override, and a single bucket | |
| were all traded for the dumber option.</li> | |
| </ul> | |
| <h2 id="how-to-use-the-implementation-to-replicate-the-paper">How to use | |
| the implementation to replicate the paper</h2> | |
| <p>The issue's launch section and replication map carry the details. The | |
| order of operations is: materialize the FineWeb caches from PR #9293, | |
| run the two-arm operator test at d8, then the base-size arms with a | |
| second seed to measure the noise floor, then the untied growth arm | |
| against its compute-matched control, then the two ladders for the | |
| exponent claim, and last the four-cell data-repetition test. Each later | |
| experiment in the paper maps to one launcher flag combination in the | |
| replication map.</p> | |
| <p>Two things to decide before filing. The six deviations in the dense | |
| replication belong in a comment on #9292 or #9293 regardless of this | |
| plan, since they change its targets. And the fold-in question is the | |
| #9293 owner's call; the decision log states both cases.</p> | |
| <hr /> | |
| <h1 id="proposed-issue-body">Proposed issue body</h1> | |
| <h2 id="tldr">TL;DR</h2> | |
| <p>Plan for <code>experiments/grug/paper_loop/</code>, a copy of | |
| <code>experiments/grug/paper_rep/</code> (PR #9293) that adds the | |
| paper's K>1 core loop, tied or untied cores, and in-training growth | |
| from K=2 to K=4. Two model-config fields (<code>num_loops</code>, | |
| <code>num_cores</code>) and one train-config block (<code>growth</code>) | |
| express all eight paper variants. Three buckets: loop and untied cores, | |
| then growth, then whole-model weight decay and epoch cycling. No code or | |
| runs yet. The first TPU run is the one confound-free pair in the family, | |
| Untied-2 against Deep Vanilla under a shared recipe at d8 (2.4 | |
| Vanilla-d8 units; the paper_rep guide estimates 30 minutes per unit on a | |
| v4-16, unmeasured). If the boundary operator does nothing there, the | |
| direction stops.</p> | |
| <h2 id="description">Description</h2> | |
| <p>arXiv 2609.19107 (Chen, Vegesna, Dahal, Wilson; code at <a | |
| href="https://github.com/qlabs-eng/scaling-exponents">https://github.com/qlabs-eng/scaling-exponents</a>) | |
| defines one model family, its Eq. 2: a prelude of P blocks, a core of C | |
| blocks applied K times, a coda of D blocks. Between core passes and | |
| before the coda a boundary operator | |
| <code>BO(h, e) = rms_norm(h) + alpha * e</code> re-injects the prelude | |
| output e; <code>alpha</code> is a fixed scalar | |
| (<code>injection_scale</code> in <code>paper_rep</code>). Model "d-ell" | |
| has ell stored blocks, width 128 * ell, ell heads of size 128. The split | |
| rule is thirds with the remainder going to the core first, then the | |
| coda: d6 2/2/2, d8 2/3/3, d10 3/4/3, d12 4/4/4, d14 4/5/5, d16 5/6/5, | |
| d18 6/6/6. Executed depth is P + K*C + D. The eight variants:</p> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th>Variant</th> | |
| <th>K</th> | |
| <th>Cores stored</th> | |
| <th>BO</th> | |
| <th>Note</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td>Vanilla</td> | |
| <td>1</td> | |
| <td>1</td> | |
| <td>no</td> | |
| <td>plain stack, ell blocks</td> | |
| </tr> | |
| <tr> | |
| <td>Operator-1</td> | |
| <td>1</td> | |
| <td>1</td> | |
| <td>yes</td> | |
| <td>Vanilla plus BO, same params and FLOPs</td> | |
| </tr> | |
| <tr> | |
| <td>Loop-2</td> | |
| <td>2</td> | |
| <td>1</td> | |
| <td>yes</td> | |
| <td>one core, two passes</td> | |
| </tr> | |
| <tr> | |
| <td>Untied-2</td> | |
| <td>2</td> | |
| <td>2</td> | |
| <td>yes</td> | |
| <td>two distinct cores, same FLOPs as Loop-2</td> | |
| </tr> | |
| <tr> | |
| <td>Deep Vanilla</td> | |
| <td>1</td> | |
| <td>1</td> | |
| <td>no</td> | |
| <td>plain stack of ell + C blocks (11 at d8), Untied-2's depth and | |
| parameter count</td> | |
| </tr> | |
| <tr> | |
| <td>Loop-Grow</td> | |
| <td>2 then 4</td> | |
| <td>1</td> | |
| <td>yes</td> | |
| <td>K raised late in training</td> | |
| </tr> | |
| <tr> | |
| <td>Untied-Grow</td> | |
| <td>2 then 4</td> | |
| <td>4</td> | |
| <td>yes</td> | |
| <td>cores 3 and 4 dormant, then copied from 1 and 2</td> | |
| </tr> | |
| <tr> | |
| <td>Deep Vanilla Grow</td> | |
| <td>2 then 4</td> | |
| <td>4</td> | |
| <td>no</td> | |
| <td>Untied-Grow without BO; no training script released</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <p>Claims (FineWeb, GPT-2 tokenizer, d6 to d18): Operator-1 improves the | |
| loss-vs-compute exponent over Vanilla (gamma 0.1143 vs 0.1112); Loop-2 | |
| and Untied-2 keep that exponent with a better constant; growth raises it | |
| again (Untied-Grow 0.1168; compute multiplier over Vanilla 1.30x at 1e18 | |
| FLOPs, 1.55x at 1e20). Under 10 epochs of a 100M-token subset, the | |
| fitted compute-optimal K rises from 1.4 on fresh data to 6.7.</p> | |
| <p>What Marin already has:</p> | |
| <ul> | |
| <li>#9292 / PR #9293: <code>experiments/grug/paper_rep/</code>, dense, | |
| paper optimizer and init, FineWeb GPT-2 data, Operator-1 (K=1). No | |
| looping, no growth. Runs not started as of 2026-09-20.</li> | |
| <li>#9280 / PR #9281: Operator-1 on the MoE template. Gate 1 failed at | |
| hidden width 512 (3.5460 vs 3.5422 baseline); most paper recipe knobs do | |
| not exist in that template.</li> | |
| <li>PR #6387 (June 2026, closed unmerged, branch | |
| <code>weaver/re-entrant-model-testing</code> at 8f1e197): re-entrant MoE | |
| with a tied core, per-pass FiLM, sampled K, consistency loss. Every arm | |
| lost to dense at width 512 (loop-4: 3.905 vs 3.818). Its | |
| <code>eval_sweep.py</code> (loss at several K from one checkpoint) is | |
| copied and adapted here.</li> | |
| <li>Discord #architecture: dlwh flagged the paper on 2026-09-18 (<a | |
| href="https://discord.com/channels/1354881461060243556/1527756652890161292/1550652125220962378">https://discord.com/channels/1354881461060243556/1527756652890161292/1550652125220962378</a>); | |
| Kaiyue Wen linked SMELT, a looped MoE study (arXiv 2609.01343), on | |
| 2026-09-20 (<a | |
| href="https://discord.com/channels/1354881461060243556/1550935190505914541/1551067464622669914">https://discord.com/channels/1354881461060243556/1550935190505914541/1551067464622669914</a>).</li> | |
| </ul> | |
| <p>Initiating request (Jeff Hammerbacher, 2026-09-20): "carefully study | |
| implementations of looped transformers in other repos ... then carefully | |
| analyze the Marin codebase, in particular the grug models in Levanter. | |
| Then propose a plan to add a grug-style looped transformer to Levanter | |
| ... Finally, describe how to use the new looped transformer | |
| implementation to replicate the experiments and findings from <a | |
| href="https://arxiv.org/abs/2609.19107">https://arxiv.org/abs/2609.19107</a>."</p> | |
| <p>Five other looped-transformer codebases were read for this plan | |
| (Parcae, LoopFormer, Huginn, Mixture-of-Recursions, Looped-GPT). They | |
| add per-pass conditioning, sampled depth, truncated backprop through the | |
| loop, adaptive exit, and per-loop KV caches. The paper's released code | |
| has none of these except a truncated-backprop knob | |
| (<code>num_core_stop_gradients</code>) that every ladder script leaves | |
| at 0. This plan implements the paper's family only; the other mechanisms | |
| are listed under "Not in this pass" with the change each needs.</p> | |
| <p>Six places where <code>paper_rep</code> differs from the released | |
| code (<code>models/layers.py</code>, <code>models/transformer.py</code>, | |
| <code>train.py</code>):</p> | |
| <ol type="1"> | |
| <li>Heads: the code uses <code>n_head = ell</code> (head size 128). | |
| <code>paper_rep</code> uses <code>hidden_dim // 64</code> (head size | |
| 64). Same parameter count, different attention.</li> | |
| <li>MLP width: the code uses | |
| <code>256 * ceil(8 * width / 3 / 256)</code> (2816 at d8). | |
| <code>paper_rep</code> uses <code>3 * width</code> (3072). Only the | |
| code's rule reproduces the released exact count for the d26 Untied-Grow | |
| model (7,424,049,152 parameters); the two rules differ by 3 percent in | |
| parameters and FLOPs at d8.</li> | |
| <li>Init bound: the code draws input matrices from <code>U(-s, s)</code> | |
| with <code>s = UIS * width ** -0.5</code> (0.011 at d8 for Operator-1). | |
| <code>paper_rep</code> uses <code>U(-UIS, UIS)</code> (0.354).</li> | |
| <li>First core pass: the code feeds the raw prelude output e; | |
| <code>paper_rep</code> (and PR #9281's alpha=0.707 arm) feeds | |
| <code>alpha * e</code>. Identical at alpha=1, different for Loop-2 | |
| (alpha=0.707).</li> | |
| <li>Weight decay: the code decays every group, Muon matrices included. | |
| <code>paper_rep</code> decays only the embedding and head. The | |
| data-repetition experiment (Gate 4) sweeps weight decay from 0.05 to 1.6 | |
| on the whole model.</li> | |
| <li>Muon momentum: the code ramps it from 0.85 to 0.95 over the first | |
| 300 steps; <code>paper_muon</code> holds 0.95. Accepted as a difference; | |
| recorded so a gap has a candidate cause.</li> | |
| </ol> | |
| <p>Items 1 to 3 move #9292's targets and should be fixed in PR #9293 | |
| before its arms run. <code>paper_loop</code> implements 4 whatever #9293 | |
| does (the parity test covers alpha=1 only), adds a | |
| <code>muon_weight_decay</code> knob for 5, and reruns its own Vanilla | |
| and Operator-1 baselines so every comparison is within one codebase.</p> | |
| <h2 id="hypothesis-or-goal">Hypothesis or Goal</h2> | |
| <p>At d8 on 1B tokens under each variant's own recipe, the paper's Table | |
| 6 order is Untied-2 3.2563 < Loop-2 3.2704 < Deep Vanilla 3.2772 | |
| < Operator-1 3.3057 < Vanilla 3.3279. Under the shared Vanilla | |
| recipe (Table 6, second column) Untied-2 is 3.2697 and Deep Vanilla | |
| 3.2869: same stored blocks, same executed depth, same tokens, and only | |
| the operator differs. Under the Vanilla scaling exponent, 25 percent | |
| more FLOPs buys about 0.04 of loss at d8 (FineWeb fit: E = 1.7064, gamma | |
| = 0.1112), so the paper's Loop-2 gap over Operator-1 (0.035, at 25 | |
| percent more FLOPs) means tied looping is compute-neutral at d8; the | |
| paper's own fit says the same (compute multipliers 1.10 for Loop-2 and | |
| 1.12 for Operator-1 at 1e18). The paper's claimed value is in the | |
| operator (iso-FLOP), in growth, and in the exponent. In the ladder runs | |
| at d8, Untied-Grow reaches 3.1304 against Untied-2 3.1904 while | |
| consuming 34 percent more FLOPs; the iso-FLOP margin implied by the | |
| paper's multipliers (1.30 vs 1.19) is about 0.015. On a d6 to d12 | |
| ladder, Untied-Grow's iso-FLOP advantage over Vanilla grows with scale. | |
| Absolute losses will differ from the paper because the data slice, | |
| packing order, and Muon internals differ; every rule below is on gaps | |
| between arms trained in this variant on the same data, seed, and | |
| hardware, against a measured seed noise floor. The metric everywhere is | |
| <code>eval/loss</code> on the held-out FineWeb validation tag | |
| (<code>eval/fineweb-val-gpt2/loss</code> in #9292), at least 10M tokens | |
| with <code>max_eval_batches</code> fixed across arms, at the final | |
| step.</p> | |
| <h2 id="plan">Plan</h2> | |
| <h3 | |
| id="bucket-1-loop-and-untied-cores-paper_loopmodelpy-copied-from-paper_rep">Bucket | |
| 1: loop and untied cores (<code>paper_loop/model.py</code>, copied from | |
| <code>paper_rep</code>)</h3> | |
| <p>Config gains two fields:</p> | |
| <div class="sourceCode" id="cb1"><pre | |
| class="sourceCode python"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a>num_loops: <span class="bu">int</span> <span class="op">=</span> <span class="dv">1</span> <span class="co"># K in paper Eq. 2: how many times the core runs per forward.</span></span> | |
| <span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>num_cores: <span class="bu">int</span> <span class="op">=</span> <span class="dv">1</span> <span class="co"># Stored core copies. 1 = one core reused every pass (Loop-K).</span></span> | |
| <span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a> <span class="co"># K = one distinct core per pass (Untied-K). `tied` property = (num_cores == 1).</span></span></code></pre></div> | |
| <p>Validation: <code>num_loops >= 1</code>; | |
| <code>num_cores == 1 or num_cores >= num_loops</code>; | |
| <code>num_cores > num_loops</code> only when a growth config reaches | |
| <code>num_cores</code> (checked at run start). <code>prelude_len</code> | |
| and <code>coda_len</code> are accepted whenever | |
| <code>boundary_operator</code> is on or <code>num_loops > 1</code>; | |
| the paper's App. D.4 control is the P/C/D split looped without the | |
| operator. Blocks are stored as three fields, | |
| <code>prelude: tuple[Block, ...]</code>, | |
| <code>cores: tuple[tuple[Block, ...], ...]</code>, | |
| <code>coda: tuple[Block, ...]</code>, so the forward, growth, the | |
| dormant mask, and per-pass metrics address cores by index and never | |
| slice a flat tuple. A test helper maps a <code>paper_rep</code> | |
| checkpoint onto these fields by block index.</p> | |
| <p>Forward:</p> | |
| <div class="sourceCode" id="cb2"><pre | |
| class="sourceCode python"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a>h <span class="op">=</span> rms_norm(embed(tokens))</span> | |
| <span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> b <span class="kw">in</span> <span class="va">self</span>.prelude: h <span class="op">=</span> b(h)</span> | |
| <span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a>e <span class="op">=</span> h <span class="co"># un-normalized prelude output (the code's "anchor")</span></span> | |
| <span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> k <span class="kw">in</span> <span class="bu">range</span>(cfg.num_loops): <span class="co"># static Python loop; K <= 12 in every paper experiment</span></span> | |
| <span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a> core <span class="op">=</span> <span class="va">self</span>.cores[<span class="dv">0</span>] <span class="cf">if</span> cfg.num_cores <span class="op">==</span> <span class="dv">1</span> <span class="cf">else</span> <span class="va">self</span>.cores[k]</span> | |
| <span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> b <span class="kw">in</span> core: h <span class="op">=</span> b(h) <span class="co"># one eqx.filter_checkpoint per block application</span></span> | |
| <span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> cfg.boundary_operator:</span> | |
| <span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a> h <span class="op">=</span> rms_norm(h) <span class="op">+</span> cfg.injection_scale <span class="op">*</span> e <span class="co"># after every pass, including before the coda</span></span> | |
| <span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> b <span class="kw">in</span> <span class="va">self</span>.coda: h <span class="op">=</span> b(h)</span> | |
| <span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a><span class="cf">return</span> rms_norm(h)</span></code></pre></div> | |
| <p>This matches the released <code>_forward_blocks</code> in the setting | |
| every ladder script uses | |
| (<code>recurrence_rmsnorm=recurrence_only</code>, | |
| <code>decoder_inject</code> on). K is read from the static config only; | |
| an eval at another K uses | |
| <code>dataclasses.replace(model.config, num_loops=K)</code>, as PR | |
| #6387's sweep did. A Python loop instead of <code>lax.scan</code>: K is | |
| small and static, untied cores differ per pass, and per-pass metrics and | |
| named scopes are one line each. The largest planned compile is d12 at | |
| K=4 (24 block applications); a later K=12 arm at d12 (56 applications) | |
| is the first place a scan would be needed, and the core loop is the only | |
| place to put it.</p> | |
| <p>Metrics: <code>core/residual_rms_pass_{k}</code> for k = 1..K, the | |
| RMS of the stream change over pass k (input snapshot after the previous | |
| pass's norm and injection, output snapshot before this pass's norm), and | |
| their mean under the released name <code>rms/res_core</code>. At the end | |
| of training <code>eval/loss_at_loops_{k}</code> from the final | |
| checkpoint: k = 1..2K for tied models (the paper's B.4 evaluates more | |
| passes than trained), k = 1..K for untied.</p> | |
| <p><code>recipes.py</code> adds three recipes (acronyms, all from paper | |
| Table 5: RM residual-branch multiplier, OM output multiplier, WTE | |
| token-embedding init std, UIS uniform init scale, GLR global Muon | |
| learning rate, ELRM and HLRM embedding and head multipliers on GLR, WD | |
| weight decay, WU warmup steps, WDR warmdown fraction, beta1/beta2/eps | |
| AdamW moments). GLR is 0.04 for all three.</p> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th>Recipe</th> | |
| <th>RM</th> | |
| <th>OM</th> | |
| <th>WTE</th> | |
| <th>UIS</th> | |
| <th>ELRM</th> | |
| <th>HLRM</th> | |
| <th>WD</th> | |
| <th>WU</th> | |
| <th>WDR</th> | |
| <th>beta1</th> | |
| <th>beta2</th> | |
| <th>eps</th> | |
| <th>alpha</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td>LOOP2</td> | |
| <td>0.25</td> | |
| <td>1</td> | |
| <td>0.02</td> | |
| <td>0.044</td> | |
| <td>0.32</td> | |
| <td>0.113</td> | |
| <td>0.05</td> | |
| <td>40</td> | |
| <td>1</td> | |
| <td>0.8</td> | |
| <td>0.95</td> | |
| <td>1e-10</td> | |
| <td>0.707</td> | |
| </tr> | |
| <tr> | |
| <td>UNTIED2</td> | |
| <td>0.25</td> | |
| <td>1</td> | |
| <td>0.01</td> | |
| <td>0.354</td> | |
| <td>0.16</td> | |
| <td>0.16</td> | |
| <td>0.071</td> | |
| <td>40</td> | |
| <td>1</td> | |
| <td>0.8</td> | |
| <td>0.99</td> | |
| <td>1e-8</td> | |
| <td>1</td> | |
| </tr> | |
| <tr> | |
| <td>DEEP_VANILLA</td> | |
| <td>0.5</td> | |
| <td>1</td> | |
| <td>0.005</td> | |
| <td>0.5</td> | |
| <td>0.16</td> | |
| <td>0.057</td> | |
| <td>0.1</td> | |
| <td>5</td> | |
| <td>0.8</td> | |
| <td>0.8</td> | |
| <td>0.99</td> | |
| <td>1e-8</td> | |
| <td>none</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <p><code>launch.py</code> takes | |
| <code>--arm {vanilla, op1, loop2, untied2, deep_vanilla, loop_grow, untied_grow}</code>, | |
| <code>--depth {6,...,18}</code>, and overrides <code>--recipe</code> | |
| (train one arm under another arm's recipe), <code>--num-loops</code>, | |
| <code>--crossover</code>, <code>--steps</code>. A table transcribed from | |
| <code>ladder_scripts/*.sh</code> gives each (arm, depth) its step count, | |
| crossover fraction, and GLR (d8: Vanilla 1962; Operator-1 and Loop-2 | |
| 2355; Untied-2 and Deep Vanilla 2796; Loop-Grow 2563 at crossover 0.82; | |
| Untied-Grow 3341 at 0.71; in every script the LR schedule length equals | |
| the step count). The paper's rule | |
| <code>GLR(d) = 0.04 * (N(8) / N(d)) ** exponent</code>, with N the | |
| stored parameter count including embedding and head and a per-arm | |
| exponent from Table 9, is a consistency test on that table. Gate 1 arms | |
| use <code>--steps 1907</code>, the Table 6 budget that #9292 uses. Deep | |
| Vanilla Grow has no released script and is left out.</p> | |
| <p>Tests (CPU): (1) <code>num_loops=1, num_cores=1</code> reproduces | |
| <code>paper_rep.Transformer</code> logits from the same parameters, with | |
| <code>boundary_operator</code> off and on at alpha=1; (2) tied K=2 | |
| equals a hand-unrolled forward; (3) untied K=2: perturbing | |
| <code>cores[1]</code> changes the output; (6) | |
| <code>_compute_flops</code> at K=1 equals <code>paper_rep</code>'s, and | |
| the K=2 to K=1 ratio equals the block-count ratio from the paper | |
| formula; (8) validation rejects <code>num_cores=2, num_loops=3</code> | |
| and <code>num_cores > num_loops</code> without growth; (9) split | |
| table extended to the paper's d10 to d26 rows. | |
| <code>tests/test_grug_variant_contracts.py</code> discovers the | |
| directory and lowers one train step at the defaults. A | |
| <code>docs/reports/grug-archive.md</code> entry records the origin | |
| commit of <code>paper_rep</code>.</p> | |
| <h3 id="bucket-2-growth-paper_looptrainpy">Bucket 2: growth | |
| (<code>paper_loop/train.py</code>)</h3> | |
| <p>One new train-config block:</p> | |
| <div class="sourceCode" id="cb3"><pre | |
| class="sourceCode python"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a>growth: GrowthConfig <span class="op">|</span> <span class="va">None</span> <span class="op">=</span> <span class="va">None</span></span> | |
| <span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="co"># GrowthConfig(num_loops_after=4, crossover_fraction=0.71, lr_rewarm_steps=40)</span></span></code></pre></div> | |
| <p>Growth step g = | |
| <code>int(crossover_fraction * num_train_steps)</code>, the code's rule; | |
| with a constant batch, <code>crossover_fraction = 1 - rho</code> where | |
| rho is the paper's fraction of tokens trained after growth. The run loop | |
| builds two jitted train steps, one per K, from | |
| <code>dataclasses.replace(config, num_loops=K)</code>, and compiles both | |
| ahead of time before step 0 so a K=4 OOM or a long compile shows up at | |
| launch. Driver order at every step: checkpoint hook, then | |
| <code>if step == g: state = grow(state)</code>, then the step for the | |
| current K. <code>grow</code> on an untied model copies | |
| <code>cores[k % K_before]</code> into <code>cores[k]</code> for | |
| <code>k >= K_before</code> (the code's | |
| <code>dep_stack_grow_init</code>) in both <code>params</code> and | |
| <code>ema_params</code>; on a tied model it changes nothing. In both | |
| cases it replaces the static config inside the state with | |
| <code>num_loops=num_loops_after</code>, so the checkpoint written after | |
| g carries the trained K and <code>train/num_loops</code> is read from | |
| the config. A run resumed at step s > g never re-applies the copy; | |
| one resumed at s <= g applies it once when it reaches g. Growth runs | |
| default to <code>ema_beta=None</code>.</p> | |
| <p>Dormant cores: in the K_before train step the optimizer's update tree | |
| is zeroed on <code>cores[K_before:]</code> with a static mask after | |
| <code>optimizer.update</code> and before <code>apply_updates</code>. | |
| This removes the decay term and the already-zero Muon term without | |
| touching the optimizer state, so no group changes shape at g. Levanter's | |
| Muon normalizes by <code>norm + eps</code>, so a zero gradient produces | |
| a zero update and no NaN. The released code keeps the same cores at zero | |
| momentum with a cautious decay that never fires on zero gradients.</p> | |
| <p>Re-warm: the update tree is multiplied by 1 before g and by | |
| <code>clip((step - g) / lr_rewarm_steps, 0, 1)</code> from g on, inside | |
| <code>train_step</code>. The <code>paper_muon</code> optimizer and its | |
| warmup-stable-decay schedule are untouched; multiplying the whole update | |
| also scales the decayed-weight term, which is what the code's LR | |
| multiplier does.</p> | |
| <p>FLOPs across g: <code>log_performance_stats</code> takes one scalar, | |
| so it receives the K_before value and a variant-local callback logs | |
| <code>throughput/flops_per_token_paper</code> per phase and a cumulative | |
| <code>train/total_flops</code>, both recomputed from step and g so they | |
| survive resume, using the paper's per-token formula | |
| (<code>6 * (executed block params + head params) + 12 * heads * head_size * mean attended keys</code>, | |
| the released <code>estimate_flops</code>). That number is the compute | |
| axis for every fit; Levanter's <code>lm_flops_per_token</code> is used | |
| for MFU only. <code>parameter_count</code> stays the stored count | |
| (dormant cores included, the paper's convention); | |
| <code>train/executed_block_params</code> is logged beside it.</p> | |
| <p>Tests: (4) before g the loss gradient wrt <code>cores[2:]</code> is | |
| exactly zero and one <code>paper_muon</code> step with nonzero | |
| <code>muon_weight_decay</code> leaves them unchanged and finite; after | |
| <code>grow</code>, <code>cores[2] == cores[0]</code> and | |
| <code>cores[3] == cores[1]</code>, and the state's config reads K=4; (5) | |
| tied <code>grow</code> changes no parameters and the K=4 forward differs | |
| from K=2; (7) re-warm multiplier is 1 before g, 0 at g, 1 at g + | |
| <code>lr_rewarm_steps</code>; (10) a run checkpointed at g+1 and resumed | |
| has <code>cores[2] != cores[0]</code> after one more step.</p> | |
| <h3 id="bucket-3-whole-model-weight-decay-and-epoch-cycling">Bucket 3: | |
| whole-model weight decay and epoch cycling</h3> | |
| <p><code>muon_weight_decay</code> on <code>PaperMuonConfig</code> | |
| (decoupled, same value as the AdamW groups when set, so the paper's | |
| single WD knob can be reproduced). Epoch cycling: Marin's | |
| <code>train_lm</code> path resolves <code>num_train_epochs</code> from | |
| the token count over a single dataset | |
| (<code>lib/marin/src/marin/experiment/train.py</code>); the grug loader | |
| builds a <code>MixtureDataset</code>. Which path applies to a 100M-token | |
| cache repeated ten times with a fresh shuffle per epoch is settled | |
| before Gate 4 is scheduled.</p> | |
| <h3 id="not-in-this-pass">Not in this pass</h3> | |
| <p>Truncated backprop (one <code>jax.lax.stop_gradient(h)</code> at the | |
| end of pass k, the code's <code>num_core_stop_gradients</code>); | |
| per-pass FiLM or time embeddings (one table lookup before the inner | |
| loop); sampled K (one compiled train step per K in the sample set plus | |
| Python-side dispatch, the cost PR #6387 paid); consistency loss; | |
| adaptive exit; the KL effective-depth probe (paper Sec. 6); DCLM CORE; | |
| HF export of a looped model. Of the Table 4 operator ablations, | |
| norm-only is already expressible | |
| (<code>boundary_operator=True, injection_scale=0</code>); injection-only | |
| and coda-injection-off need two booleans.</p> | |
| <h3 id="gates-and-cost">Gates and cost</h3> | |
| <p>One unit is the Vanilla d8 ladder run: 1962 steps of 524,288 tokens, | |
| 1.06e18 FLOPs by the paper's per-token formula, which with the split | |
| rule and the ladder step counts reproduces the released CSV endpoints | |
| (3.25e17 for Vanilla d6, 1.49e20 for Untied-Grow d18). The paper_rep | |
| guide estimates 30 minutes per unit on a v4-16; unmeasured. Noise floor: | |
| the pooled RMS of the seed differences from the second-seed arms in Gate | |
| 1b, on the metric above. A gap rule passes when the gap exceeds twice | |
| the floor. If the floor exceeds half the paper gap being tested, two | |
| more seeds are run before the gate is judged.</p> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th>Gate</th> | |
| <th>Arms at d8, 1907 steps unless noted (units each)</th> | |
| <th>Units</th> | |
| <th>Pass rule</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td>0</td> | |
| <td>bucket 1 tests</td> | |
| <td>0</td> | |
| <td>green on CPU</td> | |
| </tr> | |
| <tr> | |
| <td>1a</td> | |
| <td>Untied-2 and Deep Vanilla, both under <code>VANILLA_RECIPE</code> | |
| (1.2 each)</td> | |
| <td>2.4</td> | |
| <td>Untied-2 < Deep Vanilla by more than twice the floor (paper | |
| 0.017). Same stored blocks, executed depth, tokens, and recipe; only the | |
| operator differs. Else stop.</td> | |
| </tr> | |
| <tr> | |
| <td>1b</td> | |
| <td>Vanilla, Operator-1 (1.0 each); Loop-2, Untied-2, Deep Vanilla under | |
| own recipes (1.2 each); second seeds of Vanilla (1.0) and Loop-2 | |
| (1.2)</td> | |
| <td>7.8</td> | |
| <td>Operator-1 < Vanilla by more than twice the floor (own recipes, | |
| paper 0.022; same params, FLOPs, and tokens). Reported, not gated, | |
| because FLOPs or params differ: Loop-2 vs Operator-1 (paper 0.035; about | |
| 0.04 is expected from the extra FLOPs alone), Loop-2 vs Untied-2 (paper | |
| 0.014, the cost of tying at iso-FLOP), Loop-2 vs Deep Vanilla (paper | |
| 0.007).</td> | |
| </tr> | |
| <tr> | |
| <td>2</td> | |
| <td>Untied-Grow at 3341 steps, crossover 0.71 (2.4); iso-FLOP control | |
| Untied-2 at 3744 steps (2.4)</td> | |
| <td>4.8</td> | |
| <td>Untied-Grow < control by more than twice the floor. Expected | |
| margin about 0.015 (the paper's per-run gap 0.060 minus about 0.045 the | |
| control gains from 34 percent more compute), so this gate needs a floor | |
| below 0.007; otherwise more seeds, or the same pair at d10 (7.3 + 7.3 | |
| units).</td> | |
| </tr> | |
| <tr> | |
| <td>3</td> | |
| <td>d6 to d12 ladders at ladder step counts: Vanilla (10.5), Untied-Grow | |
| (26.4)</td> | |
| <td>36.9</td> | |
| <td>For each Untied-Grow rung, Vanilla's loss at the same compute by | |
| log-log interpolation on the Vanilla ladder; the gap is non-decreasing | |
| from d6 to d12 and the d12 gap exceeds the d6 gap by more than twice the | |
| floor. Fitted gamma (E frozen from the Vanilla fit, OLS of | |
| <code>log(L - E)</code> on <code>log C</code>) and the compute | |
| multiplier at the Vanilla-d8 loss are reported as descriptive numbers | |
| only; four points leave two residual degrees of freedom.</td> | |
| </tr> | |
| <tr> | |
| <td>4</td> | |
| <td>Operator-1 recipe: d8 K=4 tied and d10 K=1 (matched per-token FLOPs, | |
| 2.83e8 vs 2.77e8), each on fresh 1B tokens (WD 0.05) and on 100M tokens | |
| x 10 epochs (WD 0.8, the paper's K=1-selected value); 1.7 each</td> | |
| <td>6.8</td> | |
| <td>The iso-FLOP advantage of d8-K4 over d10-K1 under repetition exceeds | |
| its advantage on fresh data by more than twice the floor.</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <p>Gate 1a runs before any growth code exists and does not depend on | |
| another PR's runs. Gate 2's control is the fixed-K parent trained to the | |
| grow arm's FLOP count; without it a grow arm only shows that more | |
| compute helps. Gate 3 adds Operator-1 (12.6), Untied-2 (19.8), or | |
| Loop-Grow (18.8) rungs only for the descriptive fit. Extending past d12 | |
| is not planned; the full paper ladders (eight arms, d6 to d18) are about | |
| 1,560 units.</p> | |
| <h2 id="replication-map">Replication map</h2> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th>Paper experiment</th> | |
| <th>Arms and settings</th> | |
| <th>Cost (units)</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td>Table 6 transfer regret, d8, 1B tokens (loss under own recipe vs | |
| under the Vanilla recipe)</td> | |
| <td>Gates 1a and 1b cover 7 of the 10 cells; the remaining | |
| <code>--recipe vanilla</code> cells are Loop-2 and Operator-1</td> | |
| <td>about 2.2</td> | |
| </tr> | |
| <tr> | |
| <td>Table 4 operator ablation</td> | |
| <td>norm-only via <code>injection_scale=0</code> now; injection-only and | |
| coda-injection-off need two booleans</td> | |
| <td>1.2 per cell</td> | |
| </tr> | |
| <tr> | |
| <td>Fig. 3 exponents, FineWeb d6 to d18</td> | |
| <td>Gate 3 is the Vanilla and Untied-Grow d6 to d12 subset; other arms | |
| and depths per the unit table</td> | |
| <td>36.9 first; 1,560 full</td> | |
| </tr> | |
| <tr> | |
| <td>A.2.2 loop count at fixed tokens</td> | |
| <td><code>--arm loop2 --num-loops K</code> for K in {1, 2, 3, 4, 6} at | |
| d8, Loop-2 recipe</td> | |
| <td>about 8</td> | |
| </tr> | |
| <tr> | |
| <td>A.2.3 growth transition</td> | |
| <td><code>--arm untied_grow --crossover f</code> for f in {0.5, 0.6, | |
| 0.71, 0.8, 0.9} at d8, each with its iso-FLOP control</td> | |
| <td>about 24</td> | |
| </tr> | |
| <tr> | |
| <td>Loop-Grow (ladder d8 3.2002 vs Loop-2 3.2323)</td> | |
| <td><code>--arm loop_grow</code> (2563 steps, crossover 0.82) with | |
| control Loop-2 at 2760 steps; the paper's multipliers (1.12 vs 1.10 at | |
| 1e18) imply an iso-FLOP margin near 0.003 at d8, below any seed floor, | |
| so this needs a d12 or larger rung to be informative</td> | |
| <td>3.6 at d8</td> | |
| </tr> | |
| <tr> | |
| <td>B.4 inference-time passes</td> | |
| <td><code>eval/loss_at_loops_{k}</code> from any tied checkpoint; no | |
| training</td> | |
| <td>0</td> | |
| </tr> | |
| <tr> | |
| <td>Sec. 5 data repetition</td> | |
| <td>Gate 4 first; then K in {1, 2, 3, 4, 6, 8, 12} at d8 with WD in | |
| {0.05, 0.2, 0.4, 0.8, 1.2, 1.6}, each cell paired with an iso-FLOP K=1 | |
| model of larger depth</td> | |
| <td>6.8 first; about 80 for the d8 grid</td> | |
| </tr> | |
| <tr> | |
| <td>Sec. 6 KL effective depth</td> | |
| <td>later bucket (per-block logit-lens readout)</td> | |
| <td>later</td> | |
| </tr> | |
| <tr> | |
| <td>Sec. 4.3 FineWeb-Edu d26 extrapolation, CORE 0.3865</td> | |
| <td>1.2e21 FLOPs, about 1,160 units, plus a FineWeb-Edu cache and a CORE | |
| harness</td> | |
| <td>not planned</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <h2 id="status">Status</h2> | |
| <p>Proposed on 2026-09-20. No code written. Open decisions: whether | |
| <code>paper_rep</code> deviations 1 to 3 are fixed in PR #9293 by its | |
| owner or carried into <code>paper_loop</code>; whether Gate 1a runs on | |
| the same v4-16 reservation as the #9292 arms.</p> | |
| <h2 id="links">Links</h2> | |
| <ul> | |
| <li>Logbook: none yet</li> | |
| <li>W&B Report: none yet</li> | |
| <li>Paper: <a | |
| href="https://arxiv.org/abs/2609.19107">https://arxiv.org/abs/2609.19107</a> | |
| ; code: <a | |
| href="https://github.com/qlabs-eng/scaling-exponents">https://github.com/qlabs-eng/scaling-exponents</a> | |
| (<code>ladder_scripts/</code>, <code>plotting_scripts/data/</code>)</li> | |
| <li>Dense replication: #9292, PR #9293. MoE Operator-1: #9280, PR #9281. | |
| Re-entrant MoE: PR #6387, #6390. Cross-layer expert tying: #8032.</li> | |
| <li>Discord: dlwh 2026-09-18 <a | |
| href="https://discord.com/channels/1354881461060243556/1527756652890161292/1550652125220962378">https://discord.com/channels/1354881461060243556/1527756652890161292/1550652125220962378</a> | |
| ; Kaiyue Wen 2026-09-20 <a | |
| href="https://discord.com/channels/1354881461060243556/1550935190505914541/1551067464622669914">https://discord.com/channels/1354881461060243556/1550935190505914541/1551067464622669914</a></li> | |
| </ul> | |
| <h2 id="decision-log">Decision Log</h2> | |
| <ul> | |
| <li>2026-09-20: Build on the dense <code>paper_rep</code> template. The | |
| paper's recipe knobs (RM, OM, UIS, WTE, three-group Muon) exist there | |
| and not in the MoE template; #9280 showed the MoE vehicle cannot | |
| separate the operator from the missing recipe.</li> | |
| <li>2026-09-20: A separate directory stacked on the | |
| <code>paper-replication</code> branch until PR #9293 merges. The case | |
| for folding the loop into <code>paper_rep</code> is real: the PR is | |
| unmerged with no runs, K=1 reproduces it exactly, and deviations 1 to 3 | |
| must otherwise be fixed in two copies. Against it: #9292's gates and its | |
| autonomous agent guide are defined on K=1, one change bucket per PR is | |
| the grug rule, and the CI variant-diff report plus the archive entry | |
| exist for exactly this duplication. If the #9293 owner prefers folding | |
| in, bucket 1 moves as one commit.</li> | |
| <li>2026-09-20: Blocks stored as <code>prelude</code>, | |
| <code>cores</code>, <code>coda</code> fields. A flat tuple would keep | |
| <code>paper_rep</code>'s checkpoint layout at K=1, but every loop-aware | |
| code path would then slice it; a test helper maps old checkpoints by | |
| index instead.</li> | |
| <li>2026-09-20: K lives only in the static config. The two train steps | |
| and the eval sweep all use | |
| <code>dataclasses.replace(config, num_loops=K)</code>; no per-call | |
| override.</li> | |
| <li>2026-09-20: Growth lives in the train config. The model config | |
| describes stored parameters and the default K; when K changes is a | |
| schedule.</li> | |
| <li>2026-09-20: First core pass receives the raw prelude output, | |
| following the released code; the paper's Algorithm 1 implies | |
| <code>alpha * e</code>. The two agree at alpha=1.</li> | |
| <li>2026-09-20: Every gate is an iso-FLOP or FLOP-corrected comparison | |
| inside this variant. The paper's growth runs train on more tokens than | |
| their fixed-K parents, and Loop-2 spends 25 percent more FLOPs than | |
| Operator-1 at equal steps.</li> | |
| <li>2026-09-20: No mechanism outside the paper's family in the first | |
| pass. PR #6387 tested FiLM, sampled K, and a consistency loss and all | |
| lost; the paper's own B.4 found sampled K slightly worse than fixed | |
| K.</li> | |
| </ul> | |
| <h2 id="conclusion">Conclusion</h2> | |
| <p>None yet.</p> | |
| </div> | |
| <footer class="provenance"> | |
| <p><em>Generated: 2026-09-20 18:55 UTC · Viewing: <span id="viewing-time"></span></em></p> | |
| <blockquote> | |
| <p><em>Data: marinmirror — 217761 chunks, built 2026-09-20 · summaries | |
| through 2026-09-07_2026-09-13 (refreshed this run). Also read: the | |
| paper's released code at <a | |
| href="https://github.com/qlabs-eng/scaling-exponents">github.com/qlabs-eng/scaling-exponents</a> | |
| (main), PR <a | |
| href="https://github.com/marin-community/marin/pull/9293">#9293</a>'s | |
| <code>paper-replication</code> branch, PR <a | |
| href="https://github.com/marin-community/marin/pull/6387">#6387</a>'s | |
| <code>weaver/re-entrant-model-testing</code> branch at 8f1e197, and PR | |
| <a href="https://github.com/marin-community/marin/pull/9281">#9281</a>'s | |
| <code>boundary-operator-phase-0</code> branch.</em></p> | |
| <p><em>Query: "carefully study implementations of looped transformers in | |
| other repos ... propose a plan to add a grug-style looped transformer to | |
| Levanter ... review with an adversarial eye ... share as a proposed | |
| issue for the Marin repo ... describe how to replicate <a | |
| href="https://arxiv.org/abs/2609.19107">https://arxiv.org/abs/2609.19107</a>"</em></p> | |
| <p><em>Sub-queries: "Parcae injection.py recurrence vs backprop depth" · | |
| "LoopFormer time embeddings, alignment loss, always-true elif" · "Huginn | |
| iterate_forward, KV cache, stopping criteria" · "Mixture-of-Recursions | |
| expert-choice router" · "Looped-GPT recurrent_refinement, loop_steps | |
| baseline" · "grug base/moe/hero model stacking, ArrayStacked, remat" · | |
| "Marin prior looping work: re-entrant #6387, boundary operator | |
| #9280/#9281, paper_rep #9292/#9293, cross-layer tying #8032" · "grug | |
| conventions: change-grug, grugformer.md, variant contracts, archive" · | |
| "FLOP/MFU/parameter accounting under weight tying" · "arXiv 2609.19107 | |
| architecture, recipes, ladders, growth, data-constrained regime, CORE" · | |
| "SMELT 2609.01343" · "Discord #architecture thread on the paper" · | |
| "Levanter epoch cycling and Muon zero-gradient safety"</em></p> | |
| </blockquote> | |
| </footer> | |
| </div> | |
| </body> | |
| </html> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment