Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Select an option

  • Save hammer/f1b6c40490210b2fcc387030f39d9317 to your computer and use it in GitHub Desktop.

Select an option

Save hammer/f1b6c40490210b2fcc387030f39d9317 to your computer and use it in GitHub Desktop.
Looped transformer for Marin's grug templates: proposed issue and replication plan for arXiv 2609.19107
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Looped transformer for Marin&#x27;s grug templates: proposed issue and replication plan for arXiv 2609.19107</title>
<style>
:root {
--ink: #1a1a2e; --ink-secondary: #555770; --ink-faint: #8a8a9a;
--accent: #8b2500; --accent-light: #c4530a;
--surface: #faf9f6; --surface-raised: #f0eeea; --surface-code: #f5f3ef;
--rule: #d4d0c8; --link: #8b2500; --link-hover: #c4530a; --ref-bg: #f7f5f1;
--content-width: 650px; --sidenote-width: 230px; --sidenote-gap: 30px;
}
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) {
--ink: #d8d5cf; --ink-secondary: #9e9bab; --ink-faint: #6e6b7b;
--accent: #d4764e; --accent-light: #e8956e;
--surface: #1a1a24; --surface-raised: #242430; --surface-code: #20202c;
--rule: #33333f; --link: #d4764e; --link-hover: #e8956e; --ref-bg: #1e1e2a;
}
}
:root[data-theme="dark"] {
--ink: #d8d5cf; --ink-secondary: #9e9bab; --ink-faint: #6e6b7b;
--accent: #d4764e; --accent-light: #e8956e;
--surface: #1a1a24; --surface-raised: #242430; --surface-code: #20202c;
--rule: #33333f; --link: #d4764e; --link-hover: #e8956e; --ref-bg: #1e1e2a;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: var(--surface); color: var(--ink);
font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif;
font-size: 16px; line-height: 1.7;
-webkit-font-smoothing: antialiased;
}
.page {
max-width: calc(var(--content-width) + var(--sidenote-width) + var(--sidenote-gap) + 80px);
margin: 0 auto; padding: 3rem 40px 4rem; position: relative;
}
@media (max-width: 1060px) {
.page { max-width: 100%; padding: 2rem 1.5rem 3rem; }
}
.content { max-width: var(--content-width); }
.paper-header { max-width: var(--content-width); margin-bottom: 2.5rem; padding-bottom: 2rem; }
.paper-header h1 {
font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif;
font-size: 2rem; font-weight: 600; line-height: 1.25; color: var(--ink);
text-wrap: balance; margin-bottom: 0.6rem; letter-spacing: -0.01em;
}
.paper-meta { font-size: 0.875rem; color: var(--ink-secondary); line-height: 1.5; }
.paper-meta .author { font-weight: 500; }
.paper-meta .mumwelt-link { color: var(--ink-secondary); text-decoration: none; border-bottom: 1px dotted var(--ink-faint); }
.paper-meta .mumwelt-link:hover { color: var(--accent); border-bottom-color: var(--accent); }
.prompt-box {
max-width: var(--content-width); margin-bottom: 2.5rem; position: relative;
}
.prompt-label {
position: absolute; left: -0.6em; top: 50%; transform: translateY(-50%);
font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif;
font-size: 5rem; font-weight: 700; color: var(--ink); opacity: 0.08;
line-height: 1; pointer-events: none; user-select: none;
}
.prompt-text {
font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif;
font-style: italic; font-size: 1.15rem; line-height: 1.55; color: var(--ink);
position: relative;
}
.abstract { margin-bottom: 2.5rem; max-width: var(--content-width); }
.abstract-label {
font-size: 0.7rem; font-weight: 600; text-transform: uppercase;
letter-spacing: 0.1em; color: var(--ink-secondary); margin-bottom: 0.5rem;
}
.abstract p { font-size: 0.92rem; line-height: 1.75; color: var(--ink); }
h1, h2, h3 {
font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif;
font-weight: 600; color: var(--ink); text-wrap: balance;
}
h2 { font-size: 1.4rem; margin-top: 2.5rem; margin-bottom: 0.75rem; letter-spacing: -0.005em; }
h3 { font-size: 1.1rem; margin-top: 1.75rem; margin-bottom: 0.5rem; }
p { margin-bottom: 1rem; max-width: var(--content-width); }
a { color: var(--link); text-decoration: none; border-bottom: 1px solid transparent;
transition: border-color 0.15s, color 0.15s; }
a:hover { color: var(--link-hover); border-bottom-color: var(--link-hover); }
.date-label { cursor: default; border-bottom: 1px dotted var(--ink-faint); }
strong { font-weight: 600; }
.sidenote-checkbox { display: none; }
.sidenote-toggle { display: none; }
.sidenote {
float: right; clear: right; width: var(--sidenote-width);
margin-right: calc(-1 * (var(--sidenote-width) + var(--sidenote-gap)));
margin-top: 0.2rem; margin-bottom: 1rem;
font-size: 0.8rem; line-height: 1.5; color: var(--ink-secondary);
}
.sidenote-number { font-size: 0.7rem; font-weight: 600; color: var(--accent); margin-right: 0.3em; }
@media (max-width: 1060px) {
.sidenote-toggle {
display: inline; cursor: pointer; color: var(--accent);
font-size: 0.78rem; font-weight: 600; user-select: none;
}
.sidenote {
float: none; display: none; width: 100%; margin: 0.4rem 0 0.75rem 0;
font-size: 0.84rem; padding: 0.6rem 0.9rem; background: var(--surface-raised);
border-radius: 4px; border-left: 2px solid var(--accent);
}
.sidenote-checkbox:checked + .sidenote { display: block; }
.sidenote-number { display: none; }
}
blockquote { border-left: 2px solid var(--rule); padding-left: 1.25rem;
margin: 1.25rem 0; color: var(--ink-secondary); font-style: italic; }
code { font-family: 'SF Mono', Menlo, Consolas, monospace; font-size: 0.85em;
background: var(--surface-code); padding: 0.15em 0.35em; border-radius: 3px; }
pre { background: var(--surface-code); border: 1px solid var(--rule); border-radius: 4px;
padding: 1rem 1.25rem; overflow-x: auto; margin: 1.25rem 0; max-width: var(--content-width); }
pre code { background: none; padding: 0; font-size: 0.82rem; line-height: 1.6; }
ul, ol { margin-bottom: 1rem; padding-left: 1.5rem; max-width: var(--content-width); }
li { margin-bottom: 0.35rem; }
li::marker { color: var(--ink-faint); }
table { max-width: var(--content-width); border-collapse: collapse; width: 100%;
margin: 1.25rem 0; font-variant-numeric: tabular-nums; font-size: 0.9rem; }
thead { border-top: 2px solid var(--ink); border-bottom: 1px solid var(--ink); }
th { font-weight: 600; padding: 0.3rem 0.75rem 0.35rem; text-align: left;
line-height: 1.2; white-space: nowrap; font-size: 0.82rem; vertical-align: bottom; }
td { padding: 0.35rem 0.75rem; border: none; vertical-align: top; }
tbody { border-bottom: 1.5px solid var(--ink); }
th:first-child, td:first-child { padding-left: 0; }
th:last-child, td:last-child { padding-right: 0; }
figure { margin: 2rem 0; max-width: var(--content-width); }
figure img { width: 100%; border-radius: 3px; border: 1px solid var(--rule); }
figcaption { font-size: 0.8rem; color: var(--ink-secondary); margin-top: 0.5rem;
line-height: 1.5; font-style: italic; }
footer.provenance {
margin-top: 3rem; padding-top: 1rem; max-width: var(--content-width);
font-size: 0.78rem; line-height: 1.6; color: var(--ink-faint);
}
footer.provenance p { margin-bottom: 0.3rem; }
footer.provenance a { color: var(--ink-faint); }
footer.provenance blockquote { margin: 0; padding: 0; border: none; color: inherit; }
a[data-hover-title] { position: relative; }
.hover-card {
position: absolute; bottom: 100%; left: 50%; transform: translateX(-50%);
width: 320px; max-width: 90vw; padding: 0.65rem 0.8rem;
background: var(--surface-raised); border: 1px solid var(--rule);
border-radius: 6px; box-shadow: 0 4px 12px rgba(0,0,0,0.1);
font-size: 0.78rem; line-height: 1.45; color: var(--ink);
pointer-events: none; z-index: 100; margin-bottom: 6px;
opacity: 0; transition: opacity 0.12s;
}
a[data-hover-title]:hover .hover-card,
a[data-hover-title]:focus .hover-card,
a.cite:hover .hover-card { opacity: 1; }
.hover-card .hc-title { font-weight: 600; margin-bottom: 0.2rem; }
.hover-card .hc-meta { font-size: 0.72rem; color: var(--ink-faint); margin-bottom: 0.25rem; }
.hover-card .hc-status {
display: inline-block; font-size: 0.65rem; font-weight: 600;
text-transform: uppercase; letter-spacing: 0.04em;
padding: 0.1em 0.4em; border-radius: 3px; margin-right: 0.4em;
}
.hc-status-open { background: #fff3e0; color: #e65100; }
.hc-status-closed, .hc-status-merged { background: #e8f5e9; color: #2e7d32; }
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) .hover-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); }
:root:not([data-theme="light"]) .hc-status-open { background: #3a2a10; color: #ffb74d; }
:root:not([data-theme="light"]) .hc-status-closed,
:root:not([data-theme="light"]) .hc-status-merged { background: #1b3a1e; color: #66bb6a; }
}
:root[data-theme="dark"] .hover-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); }
:root[data-theme="dark"] .hc-status-open { background: #3a2a10; color: #ffb74d; }
:root[data-theme="dark"] .hc-status-closed,
:root[data-theme="dark"] .hc-status-merged { background: #1b3a1e; color: #66bb6a; }
.hover-card .hc-desc { color: var(--ink-secondary); }
.cite {
font-size: 0.72rem; vertical-align: super; line-height: 0;
color: var(--accent); font-weight: 600; text-decoration: none;
border-bottom: none !important; position: relative;
}
.cite:hover { color: var(--link-hover); }
.references { margin-top: 3rem; max-width: var(--content-width); }
.references h2 { font-size: 1.15rem; margin-bottom: 1rem; }
.ref-list { list-style: none; padding: 0; counter-reset: ref; }
.ref-list li {
counter-increment: ref; display: flex; align-items: baseline;
gap: 0.5em; font-size: 0.82rem; line-height: 1.55;
margin-bottom: 0.4rem; color: var(--ink-secondary);
}
.ref-list li::before {
content: "[" counter(ref) "]"; flex-shrink: 0;
font-variant-numeric: tabular-nums; color: var(--ink-faint);
font-size: 0.78rem; min-width: 2.2em;
}
.ref-list .ref-body { flex: 1; min-width: 0; }
.ref-list .ref-title { font-weight: 500; color: var(--ink); }
.ref-list .ref-url {
font-family: 'SF Mono', Menlo, Consolas, monospace; font-size: 0.75rem;
color: var(--ink-faint); word-break: break-all; margin-left: 0.4em;
}
.ref-list .ref-url a { color: var(--ink-faint); border-bottom: none; }
.ref-list .ref-url a:hover { color: var(--link-hover); }
.ref-back {
color: var(--accent); text-decoration: none; border-bottom: none !important;
margin-left: 0.3em; font-size: 0.78rem;
}
.ref-back:hover { color: var(--link-hover); }
.status {
display: inline-block; font-size: 0.65rem; font-weight: 600;
text-transform: uppercase; letter-spacing: 0.04em;
padding: 0.15em 0.5em; border-radius: 3px; vertical-align: middle;
}
.status-done { background: #e8f5e9; color: #2e7d32; }
.status-open { background: #fff3e0; color: #e65100; }
.status-blocked { background: #fce4ec; color: #c62828; }
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) .status-done { background: #1b3a1e; color: #66bb6a; }
:root:not([data-theme="light"]) .status-open { background: #3a2a10; color: #ffb74d; }
:root:not([data-theme="light"]) .status-blocked { background: #3a1520; color: #ef9a9a; }
}
:root[data-theme="dark"] .status-done { background: #1b3a1e; color: #66bb6a; }
:root[data-theme="dark"] .status-open { background: #3a2a10; color: #ffb74d; }
:root[data-theme="dark"] .status-blocked { background: #3a1520; color: #ef9a9a; }
.cite-card {
position: absolute; bottom: 100%; left: 50%; transform: translateX(-50%);
width: 360px; max-width: 90vw; padding: 0.65rem 0.8rem;
background: var(--surface-raised); border: 1px solid var(--rule);
border-radius: 6px; box-shadow: 0 4px 12px rgba(0,0,0,0.1);
font-size: 0.78rem; line-height: 1.45; color: var(--ink);
pointer-events: none; z-index: 100; margin-bottom: 6px;
opacity: 0; transition: opacity 0.12s;
font-weight: 400; vertical-align: baseline; text-align: left;
}
a.cite:hover .cite-card { opacity: 1; }
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) .cite-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); }
}
:root[data-theme="dark"] .cite-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); }
@media print {
body { font-size: 11pt; }
.page { max-width: 100%; padding: 0; }
.sidenote { float: right; width: 180px; margin-right: -210px; }
.sidenote-toggle { display: none; }
a { color: inherit; border-bottom: none; }
.hover-card { display: none; }
.cite-card { display: none; }
}
.katex{font:normal 1.21em KaTeX_Main,Times New Roman,serif;line-height:1.2;position:relative;text-indent:0;text-rendering:auto}.katex *{-ms-high-contrast-adjust:none!important;border-color:currentColor}.katex .katex-version:after{content:"0.18.7"}.katex .katex-mathml{border:0;-webkit-clip-path:inset(50%);clip-path:inset(50%);height:1px;overflow:hidden;padding:0;position:absolute;width:1px}.katex .katex-html>.katex-newline{display:block}.katex .katex-base{position:relative;white-space:nowrap;width:-webkit-min-content;width:-moz-min-content;width:min-content}.katex .katex-base,.katex .katex-strut{display:inline-block}.katex .textbf{font-weight:700}.katex .textit{font-style:italic}.katex .textrm{font-family:KaTeX_Main}.katex .textsf{font-family:KaTeX_SansSerif}.katex .texttt{font-family:KaTeX_Typewriter}.katex .mathnormal{font-family:KaTeX_Math;font-style:italic}.katex .mathit{font-family:KaTeX_Main;font-style:italic}.katex .mathrm{font-style:normal}.katex .mathbf{font-family:KaTeX_Main;font-weight:700}.katex .boldsymbol{font-family:KaTeX_Math;font-style:italic;font-weight:700}.katex .amsrm,.katex .mathbb,.katex .textbb{font-family:KaTeX_AMS}.katex .mathcal{font-family:KaTeX_Caligraphic}.katex .mathfrak,.katex .textfrak{font-family:KaTeX_Fraktur}.katex .mathboldfrak,.katex .textboldfrak{font-family:KaTeX_Fraktur;font-weight:700}.katex .mathtt{font-family:KaTeX_Typewriter}.katex .mathscr,.katex .textscr{font-family:KaTeX_Script}.katex .mathsf,.katex .textsf{font-family:KaTeX_SansSerif}.katex .mathboldsf,.katex .textboldsf{font-family:KaTeX_SansSerif;font-weight:700}.katex .mathitsf,.katex .mathsfit,.katex .textitsf{font-family:KaTeX_SansSerif;font-style:italic}.katex .mainrm{font-family:KaTeX_Main;font-style:normal}.katex .vlist-t{border-collapse:collapse;display:inline-table;table-layout:fixed}.katex .vlist-r{display:table-row}.katex .vlist{display:table-cell;position:relative;vertical-align:bottom}.katex .vlist>span{display:block;height:0;position:relative}.katex .vlist>span>span{display:inline-block}.katex .vlist>span>.pstrut{overflow:hidden;width:0}.katex .vlist-t2{margin-right:-2px}.katex .vlist-s{display:table-cell;font-size:1px;min-width:2px;vertical-align:bottom;width:2px}.katex .katex-vbox{align-items:baseline;display:inline-flex;flex-direction:column}.katex .katex-thinbox{display:inline-flex;flex-direction:row;max-width:0;width:0}.katex .msupsub{text-align:left}.katex .mfrac>span>span{text-align:center}.katex .mfrac .frac-line{border-bottom-style:solid;display:inline-block;width:100%}.katex .katex-hdashline,.katex .katex-hline,.katex .katex-overline .overline-line,.katex .katex-rule,.katex .katex-underline .underline-line,.katex .mfrac .frac-line{min-height:1px}.katex .mspace{display:inline-block}.katex .katex-smash{display:inline;line-height:0}.katex .clap,.katex .llap,.katex .rlap{position:relative;width:0}.katex .clap>.katex-inner,.katex .llap>.katex-inner,.katex .rlap>.katex-inner{position:absolute}.katex .clap>.katex-fix,.katex .llap>.katex-fix,.katex .rlap>.katex-fix{display:inline-block}.katex .llap>.katex-inner{right:0}.katex .clap>.katex-inner,.katex .rlap>.katex-inner{left:0}.katex .clap>.katex-inner>span{margin-left:-50%;margin-right:50%}.katex .katex-rule{border:0 solid;display:inline-block;position:relative}.katex .katex-hline,.katex .katex-overline .overline-line,.katex .katex-underline .underline-line{border-bottom-style:solid;display:inline-block;width:100%}.katex .katex-hdashline{border-bottom-style:dashed;display:inline-block;width:100%}.katex .sqrt>.katex-root{margin-left:.2777777778em;margin-right:-.5555555556em}.katex .fontsize-ensurer.reset-size1.size1,.katex .katex-sizing.reset-size1.size1{font-size:1em}.katex .fontsize-ensurer.reset-size1.size2,.katex .katex-sizing.reset-size1.size2{font-size:1.2em}.katex .fontsize-ensurer.reset-size1.size3,.katex .katex-sizing.reset-size1.size3{font-size:1.4em}.katex .fontsize-ensurer.reset-size1.size4,.katex .katex-sizing.reset-size1.size4{font-size:1.6em}.katex .fontsize-ensurer.reset-size1.size5,.katex .katex-sizing.reset-size1.size5{font-size:1.8em}.katex .fontsize-ensurer.reset-size1.size6,.katex .katex-sizing.reset-size1.size6{font-size:2em}.katex .fontsize-ensurer.reset-size1.size7,.katex .katex-sizing.reset-size1.size7{font-size:2.4em}.katex .fontsize-ensurer.reset-size1.size8,.katex .katex-sizing.reset-size1.size8{font-size:2.88em}.katex .fontsize-ensurer.reset-size1.size9,.katex .katex-sizing.reset-size1.size9{font-size:3.456em}.katex .fontsize-ensurer.reset-size1.size10,.katex .katex-sizing.reset-size1.size10{font-size:4.148em}.katex .fontsize-ensurer.reset-size1.size11,.katex .katex-sizing.reset-size1.size11{font-size:4.976em}.katex .fontsize-ensurer.reset-size2.size1,.katex .katex-sizing.reset-size2.size1{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size2.size2,.katex .katex-sizing.reset-size2.size2{font-size:1em}.katex .fontsize-ensurer.reset-size2.size3,.katex .katex-sizing.reset-size2.size3{font-size:1.1666666667em}.katex .fontsize-ensurer.reset-size2.size4,.katex .katex-sizing.reset-size2.size4{font-size:1.3333333333em}.katex .fontsize-ensurer.reset-size2.size5,.katex .katex-sizing.reset-size2.size5{font-size:1.5em}.katex .fontsize-ensurer.reset-size2.size6,.katex .katex-sizing.reset-size2.size6{font-size:1.6666666667em}.katex .fontsize-ensurer.reset-size2.size7,.katex .katex-sizing.reset-size2.size7{font-size:2em}.katex .fontsize-ensurer.reset-size2.size8,.katex .katex-sizing.reset-size2.size8{font-size:2.4em}.katex .fontsize-ensurer.reset-size2.size9,.katex .katex-sizing.reset-size2.size9{font-size:2.88em}.katex .fontsize-ensurer.reset-size2.size10,.katex .katex-sizing.reset-size2.size10{font-size:3.4566666667em}.katex .fontsize-ensurer.reset-size2.size11,.katex .katex-sizing.reset-size2.size11{font-size:4.1466666667em}.katex .fontsize-ensurer.reset-size3.size1,.katex .katex-sizing.reset-size3.size1{font-size:.7142857143em}.katex .fontsize-ensurer.reset-size3.size2,.katex .katex-sizing.reset-size3.size2{font-size:.8571428571em}.katex .fontsize-ensurer.reset-size3.size3,.katex .katex-sizing.reset-size3.size3{font-size:1em}.katex .fontsize-ensurer.reset-size3.size4,.katex .katex-sizing.reset-size3.size4{font-size:1.1428571429em}.katex .fontsize-ensurer.reset-size3.size5,.katex .katex-sizing.reset-size3.size5{font-size:1.2857142857em}.katex .fontsize-ensurer.reset-size3.size6,.katex .katex-sizing.reset-size3.size6{font-size:1.4285714286em}.katex .fontsize-ensurer.reset-size3.size7,.katex .katex-sizing.reset-size3.size7{font-size:1.7142857143em}.katex .fontsize-ensurer.reset-size3.size8,.katex .katex-sizing.reset-size3.size8{font-size:2.0571428571em}.katex .fontsize-ensurer.reset-size3.size9,.katex .katex-sizing.reset-size3.size9{font-size:2.4685714286em}.katex .fontsize-ensurer.reset-size3.size10,.katex .katex-sizing.reset-size3.size10{font-size:2.9628571429em}.katex .fontsize-ensurer.reset-size3.size11,.katex .katex-sizing.reset-size3.size11{font-size:3.5542857143em}.katex .fontsize-ensurer.reset-size4.size1,.katex .katex-sizing.reset-size4.size1{font-size:.625em}.katex .fontsize-ensurer.reset-size4.size2,.katex .katex-sizing.reset-size4.size2{font-size:.75em}.katex .fontsize-ensurer.reset-size4.size3,.katex .katex-sizing.reset-size4.size3{font-size:.875em}.katex .fontsize-ensurer.reset-size4.size4,.katex .katex-sizing.reset-size4.size4{font-size:1em}.katex .fontsize-ensurer.reset-size4.size5,.katex .katex-sizing.reset-size4.size5{font-size:1.125em}.katex .fontsize-ensurer.reset-size4.size6,.katex .katex-sizing.reset-size4.size6{font-size:1.25em}.katex .fontsize-ensurer.reset-size4.size7,.katex .katex-sizing.reset-size4.size7{font-size:1.5em}.katex .fontsize-ensurer.reset-size4.size8,.katex .katex-sizing.reset-size4.size8{font-size:1.8em}.katex .fontsize-ensurer.reset-size4.size9,.katex .katex-sizing.reset-size4.size9{font-size:2.16em}.katex .fontsize-ensurer.reset-size4.size10,.katex .katex-sizing.reset-size4.size10{font-size:2.5925em}.katex .fontsize-ensurer.reset-size4.size11,.katex .katex-sizing.reset-size4.size11{font-size:3.11em}.katex .fontsize-ensurer.reset-size5.size1,.katex .katex-sizing.reset-size5.size1{font-size:.5555555556em}.katex .fontsize-ensurer.reset-size5.size2,.katex .katex-sizing.reset-size5.size2{font-size:.6666666667em}.katex .fontsize-ensurer.reset-size5.size3,.katex .katex-sizing.reset-size5.size3{font-size:.7777777778em}.katex .fontsize-ensurer.reset-size5.size4,.katex .katex-sizing.reset-size5.size4{font-size:.8888888889em}.katex .fontsize-ensurer.reset-size5.size5,.katex .katex-sizing.reset-size5.size5{font-size:1em}.katex .fontsize-ensurer.reset-size5.size6,.katex .katex-sizing.reset-size5.size6{font-size:1.1111111111em}.katex .fontsize-ensurer.reset-size5.size7,.katex .katex-sizing.reset-size5.size7{font-size:1.3333333333em}.katex .fontsize-ensurer.reset-size5.size8,.katex .katex-sizing.reset-size5.size8{font-size:1.6em}.katex .fontsize-ensurer.reset-size5.size9,.katex .katex-sizing.reset-size5.size9{font-size:1.92em}.katex .fontsize-ensurer.reset-size5.size10,.katex .katex-sizing.reset-size5.size10{font-size:2.3044444444em}.katex .fontsize-ensurer.reset-size5.size11,.katex .katex-sizing.reset-size5.size11{font-size:2.7644444444em}.katex .fontsize-ensurer.reset-size6.size1,.katex .katex-sizing.reset-size6.size1{font-size:.5em}.katex .fontsize-ensurer.reset-size6.size2,.katex .katex-sizing.reset-size6.size2{font-size:.6em}.katex .fontsize-ensurer.reset-size6.size3,.katex .katex-sizing.reset-size6.size3{font-size:.7em}.katex .fontsize-ensurer.reset-size6.size4,.katex .katex-sizing.reset-size6.size4{font-size:.8em}.katex .fontsize-ensurer.reset-size6.size5,.katex .katex-sizing.reset-size6.size5{font-size:.9em}.katex .fontsize-ensurer.reset-size6.size6,.katex .katex-sizing.reset-size6.size6{font-size:1em}.katex .fontsize-ensurer.reset-size6.size7,.katex .katex-sizing.reset-size6.size7{font-size:1.2em}.katex .fontsize-ensurer.reset-size6.size8,.katex .katex-sizing.reset-size6.size8{font-size:1.44em}.katex .fontsize-ensurer.reset-size6.size9,.katex .katex-sizing.reset-size6.size9{font-size:1.728em}.katex .fontsize-ensurer.reset-size6.size10,.katex .katex-sizing.reset-size6.size10{font-size:2.074em}.katex .fontsize-ensurer.reset-size6.size11,.katex .katex-sizing.reset-size6.size11{font-size:2.488em}.katex .fontsize-ensurer.reset-size7.size1,.katex .katex-sizing.reset-size7.size1{font-size:.4166666667em}.katex .fontsize-ensurer.reset-size7.size2,.katex .katex-sizing.reset-size7.size2{font-size:.5em}.katex .fontsize-ensurer.reset-size7.size3,.katex .katex-sizing.reset-size7.size3{font-size:.5833333333em}.katex .fontsize-ensurer.reset-size7.size4,.katex .katex-sizing.reset-size7.size4{font-size:.6666666667em}.katex .fontsize-ensurer.reset-size7.size5,.katex .katex-sizing.reset-size7.size5{font-size:.75em}.katex .fontsize-ensurer.reset-size7.size6,.katex .katex-sizing.reset-size7.size6{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size7.size7,.katex .katex-sizing.reset-size7.size7{font-size:1em}.katex .fontsize-ensurer.reset-size7.size8,.katex .katex-sizing.reset-size7.size8{font-size:1.2em}.katex .fontsize-ensurer.reset-size7.size9,.katex .katex-sizing.reset-size7.size9{font-size:1.44em}.katex .fontsize-ensurer.reset-size7.size10,.katex .katex-sizing.reset-size7.size10{font-size:1.7283333333em}.katex .fontsize-ensurer.reset-size7.size11,.katex .katex-sizing.reset-size7.size11{font-size:2.0733333333em}.katex .fontsize-ensurer.reset-size8.size1,.katex .katex-sizing.reset-size8.size1{font-size:.3472222222em}.katex .fontsize-ensurer.reset-size8.size2,.katex .katex-sizing.reset-size8.size2{font-size:.4166666667em}.katex .fontsize-ensurer.reset-size8.size3,.katex .katex-sizing.reset-size8.size3{font-size:.4861111111em}.katex .fontsize-ensurer.reset-size8.size4,.katex .katex-sizing.reset-size8.size4{font-size:.5555555556em}.katex .fontsize-ensurer.reset-size8.size5,.katex .katex-sizing.reset-size8.size5{font-size:.625em}.katex .fontsize-ensurer.reset-size8.size6,.katex .katex-sizing.reset-size8.size6{font-size:.6944444444em}.katex .fontsize-ensurer.reset-size8.size7,.katex .katex-sizing.reset-size8.size7{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size8.size8,.katex .katex-sizing.reset-size8.size8{font-size:1em}.katex .fontsize-ensurer.reset-size8.size9,.katex .katex-sizing.reset-size8.size9{font-size:1.2em}.katex .fontsize-ensurer.reset-size8.size10,.katex .katex-sizing.reset-size8.size10{font-size:1.4402777778em}.katex .fontsize-ensurer.reset-size8.size11,.katex .katex-sizing.reset-size8.size11{font-size:1.7277777778em}.katex .fontsize-ensurer.reset-size9.size1,.katex .katex-sizing.reset-size9.size1{font-size:.2893518519em}.katex .fontsize-ensurer.reset-size9.size2,.katex .katex-sizing.reset-size9.size2{font-size:.3472222222em}.katex .fontsize-ensurer.reset-size9.size3,.katex .katex-sizing.reset-size9.size3{font-size:.4050925926em}.katex .fontsize-ensurer.reset-size9.size4,.katex .katex-sizing.reset-size9.size4{font-size:.462962963em}.katex .fontsize-ensurer.reset-size9.size5,.katex .katex-sizing.reset-size9.size5{font-size:.5208333333em}.katex .fontsize-ensurer.reset-size9.size6,.katex .katex-sizing.reset-size9.size6{font-size:.5787037037em}.katex .fontsize-ensurer.reset-size9.size7,.katex .katex-sizing.reset-size9.size7{font-size:.6944444444em}.katex .fontsize-ensurer.reset-size9.size8,.katex .katex-sizing.reset-size9.size8{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size9.size9,.katex .katex-sizing.reset-size9.size9{font-size:1em}.katex .fontsize-ensurer.reset-size9.size10,.katex .katex-sizing.reset-size9.size10{font-size:1.2002314815em}.katex .fontsize-ensurer.reset-size9.size11,.katex .katex-sizing.reset-size9.size11{font-size:1.4398148148em}.katex .fontsize-ensurer.reset-size10.size1,.katex .katex-sizing.reset-size10.size1{font-size:.2410800386em}.katex .fontsize-ensurer.reset-size10.size2,.katex .katex-sizing.reset-size10.size2{font-size:.2892960463em}.katex .fontsize-ensurer.reset-size10.size3,.katex .katex-sizing.reset-size10.size3{font-size:.337512054em}.katex .fontsize-ensurer.reset-size10.size4,.katex .katex-sizing.reset-size10.size4{font-size:.3857280617em}.katex .fontsize-ensurer.reset-size10.size5,.katex .katex-sizing.reset-size10.size5{font-size:.4339440694em}.katex .fontsize-ensurer.reset-size10.size6,.katex .katex-sizing.reset-size10.size6{font-size:.4821600771em}.katex .fontsize-ensurer.reset-size10.size7,.katex .katex-sizing.reset-size10.size7{font-size:.5785920926em}.katex .fontsize-ensurer.reset-size10.size8,.katex .katex-sizing.reset-size10.size8{font-size:.6943105111em}.katex .fontsize-ensurer.reset-size10.size9,.katex .katex-sizing.reset-size10.size9{font-size:.8331726133em}.katex .fontsize-ensurer.reset-size10.size10,.katex .katex-sizing.reset-size10.size10{font-size:1em}.katex .fontsize-ensurer.reset-size10.size11,.katex .katex-sizing.reset-size10.size11{font-size:1.1996142719em}.katex .fontsize-ensurer.reset-size11.size1,.katex .katex-sizing.reset-size11.size1{font-size:.2009646302em}.katex .fontsize-ensurer.reset-size11.size2,.katex .katex-sizing.reset-size11.size2{font-size:.2411575563em}.katex .fontsize-ensurer.reset-size11.size3,.katex .katex-sizing.reset-size11.size3{font-size:.2813504823em}.katex .fontsize-ensurer.reset-size11.size4,.katex .katex-sizing.reset-size11.size4{font-size:.3215434084em}.katex .fontsize-ensurer.reset-size11.size5,.katex .katex-sizing.reset-size11.size5{font-size:.3617363344em}.katex .fontsize-ensurer.reset-size11.size6,.katex .katex-sizing.reset-size11.size6{font-size:.4019292605em}.katex .fontsize-ensurer.reset-size11.size7,.katex .katex-sizing.reset-size11.size7{font-size:.4823151125em}.katex .fontsize-ensurer.reset-size11.size8,.katex .katex-sizing.reset-size11.size8{font-size:.578778135em}.katex .fontsize-ensurer.reset-size11.size9,.katex .katex-sizing.reset-size11.size9{font-size:.6945337621em}.katex .fontsize-ensurer.reset-size11.size10,.katex .katex-sizing.reset-size11.size10{font-size:.8336012862em}.katex .fontsize-ensurer.reset-size11.size11,.katex .katex-sizing.reset-size11.size11{font-size:1em}.katex .delimsizing.size1{font-family:KaTeX_Size1}.katex .delimsizing.size2{font-family:KaTeX_Size2}.katex .delimsizing.size3{font-family:KaTeX_Size3}.katex .delimsizing.size4{font-family:KaTeX_Size4}.katex .delimsizing.mult .delim-size1>span{font-family:KaTeX_Size1}.katex .delimsizing.mult .delim-size4>span{font-family:KaTeX_Size4}.katex .nulldelimiter{display:inline-block;width:.12em}.katex .delimcenter,.katex .op-symbol{position:relative}.katex .op-symbol.small-op{font-family:KaTeX_Size1}.katex .op-symbol.large-op{font-family:KaTeX_Size2}.katex .katex-accent>.vlist-t,.katex .op-limits>.vlist-t{text-align:center}.katex .katex-accent .accent-body{position:relative}.katex .katex-accent .accent-body:not(.accent-full){width:0}.katex .katex-overlay{display:block}.katex .mtable .vertical-separator{display:inline-block;min-width:1px}.katex .mtable .arraycolsep{display:inline-block}.katex .mtable .col-align-c>.vlist-t{text-align:center}.katex .mtable .col-align-l>.vlist-t{text-align:left}.katex .mtable .col-align-r>.vlist-t{text-align:right}.katex .svg-align{text-align:left}.katex svg{fill:currentColor;stroke:currentColor;display:block;height:inherit;position:absolute;width:100%}.katex svg path{stroke:none}.katex svg{fill-rule:nonzero;fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1}.katex img{border-style:none;max-height:none;max-width:none;min-height:0;min-width:0}.katex .katex-stretchy{display:block;overflow:hidden;position:relative;width:100%}.katex .katex-stretchy:after,.katex .katex-stretchy:before{content:""}.katex .hide-tail{overflow:hidden;position:relative;width:100%}.katex .halfarrow-left{left:0;overflow:hidden;position:absolute;width:50.2%}.katex .halfarrow-right{overflow:hidden;position:absolute;right:0;width:50.2%}.katex .brace-left{left:0;overflow:hidden;position:absolute;width:25.1%}.katex .brace-center{left:25%;overflow:hidden;position:absolute;width:50%}.katex .brace-right{overflow:hidden;position:absolute;right:0;width:25.1%}.katex .x-arrow-pad{padding:0 .5em}.katex .cd-arrow-pad{padding:0 .55556em 0 .27778em}.katex .mover,.katex .munder,.katex .x-arrow{text-align:center}.katex .boxpad{padding:0 .3em}.katex .fbox,.katex .fcolorbox{border:.04em solid;box-sizing:border-box}.katex .cancel-pad{padding:0 .2em}.katex .cancel-lap{margin-left:-.2em;margin-right:-.2em}.katex .katex-sout{border-bottom-style:solid;border-bottom-width:.08em}.katex .angl{border-right:.049em solid;border-top:.049em solid;box-sizing:border-box;margin-right:.03889em}.katex .anglpad{padding:0 .03889em}.katex .reflectbox{display:inline-block;transform:scaleX(-1)}.katex .eqn-num:before{content:"(" counter(katexEqnNo) ")";counter-increment:katexEqnNo}.katex .mml-eqn-num:before{content:"(" counter(mmlEqnNo) ")";counter-increment:mmlEqnNo}.katex .mtr-glue{width:50%}.katex .cd-vert-arrow{display:inline-block;position:relative}.katex .cd-label-left{display:inline-block;position:absolute;right:calc(50% + .3em);text-align:left}.katex .cd-label-right{display:inline-block;left:calc(50% + .3em);position:absolute;text-align:right}.katex-display{display:block;margin:1em 0;text-align:center}.katex-display>.katex{display:block;text-align:center;white-space:nowrap}.katex-display>.katex>.katex-html{display:block;position:relative}.katex-display>.katex>.katex-html>.katex-tag{position:absolute;right:0}.katex-display.leqno>.katex>.katex-html>.katex-tag{left:0;right:auto}.katex-display.fleqn>.katex{padding-left:2em;text-align:left}body{counter-reset:katexEqnNo mmlEqnNo}
</style>
<script>(function() {
function __mumInit() {
// Restore external links that htmlpreview.github.io's loader rewrote to
// in-page anchors (it treats any href containing '#' as a local anchor).
// Runs now (htmlpreview re-executes this script after its rewrite), again
// after delays, and on click as a last line of defense.
function restoreOrigHrefs() {
document.querySelectorAll('a[data-orig-href]').forEach(function(a) {
var orig = a.getAttribute('data-orig-href');
if (orig && a.getAttribute('href') !== orig) a.setAttribute('href', orig);
});
}
restoreOrigHrefs();
setTimeout(restoreOrigHrefs, 100);
setTimeout(restoreOrigHrefs, 1000);
document.addEventListener('click', function(e) {
var t = e.target && e.target.closest ? e.target.closest('a[data-orig-href]') : null;
if (t) { var o = t.getAttribute('data-orig-href'); if (o && t.getAttribute('href') !== o) t.setAttribute('href', o); }
}, true);
var vt = document.getElementById('viewing-time');
if (vt) {
function pad(n) { return n < 10 ? '0' + n : n; }
function updateTime() {
var d = new Date();
vt.textContent = d.getUTCFullYear() + '-' + pad(d.getUTCMonth()+1) + '-' + pad(d.getUTCDate()) + ' ' + pad(d.getUTCHours()) + ':' + pad(d.getUTCMinutes()) + ' UTC';
}
updateTime();
setInterval(updateTime, 60000);
}
document.querySelectorAll('a[data-hover-title]').forEach(function(a) {
var card = document.createElement('span');
card.className = 'hover-card';
var title = a.getAttribute('data-hover-title') || '';
var status = a.getAttribute('data-hover-status') || '';
var owner = a.getAttribute('data-hover-owner') || '';
var desc = a.getAttribute('data-hover-desc') || '';
var statusCls = 'hc-status hc-status-' + status.toLowerCase().replace(/[^a-z]/g, '');
var html = '<div class="hc-title">' + title + '</div>';
var meta = [];
if (status) meta.push('<span class="' + statusCls + '">' + status + '</span>');
if (owner) meta.push(owner);
if (meta.length) html += '<div class="hc-meta">' + meta.join(' ') + '</div>';
if (desc) html += '<div class="hc-desc">' + desc + '</div>';
card.innerHTML = html;
a.appendChild(card);
});
document.querySelectorAll('a.cite').forEach(function(a) {
var href = a.getAttribute('href') || '';
if (!href.startsWith('#ref-')) return;
var li = document.getElementById(href.slice(1));
if (!li) return;
var body = li.querySelector('.ref-body');
if (!body) return;
var refUrl = body.querySelector('.ref-url a');
var refHref = refUrl ? refUrl.getAttribute('href') : '';
var hoverSource = refHref ? document.querySelector('a[data-hover-title][href="' + refHref + '"]') : null;
if (hoverSource) {
var card = document.createElement('span');
card.className = 'hover-card';
var t = hoverSource.getAttribute('data-hover-title') || '';
var s = hoverSource.getAttribute('data-hover-status') || '';
var o = hoverSource.getAttribute('data-hover-owner') || '';
var d = hoverSource.getAttribute('data-hover-desc') || '';
var sc = 'hc-status hc-status-' + s.toLowerCase().replace(/[^a-z]/g, '');
var h = '<div class="hc-title">' + t + '</div>';
var m = [];
if (s) m.push('<span class="' + sc + '">' + s + '</span>');
if (o) m.push(o);
if (m.length) h += '<div class="hc-meta">' + m.join(' ') + '</div>';
if (d) h += '<div class="hc-desc">' + d + '</div>';
card.innerHTML = h;
a.appendChild(card);
} else {
var title = body.querySelector('.ref-title');
if (!title) return;
var card = document.createElement('span');
card.className = 'cite-card';
card.textContent = title.textContent;
a.appendChild(card);
}
});
}
if (document.readyState !== 'loading') { __mumInit(); }
else { document.addEventListener('DOMContentLoaded', __mumInit); }
})();</script>
</head>
<body>
<div class="page">
<header class="paper-header">
<h1>Looped transformer for Marin&#x27;s grug templates: proposed issue and replication plan for arXiv 2609.19107</h1>
<div class="paper-meta">Posed by <span class="author">hammer</span>, answered by <a class="mumwelt-link" href="https://github.com/marin-community/mumwelt">mumwelt</a> &middot; <span class="date-label" title="Generated 2026-09-20 18:55 UTC">Published 2026-09-20</span></div>
</header>
<div class="content">
<p>Prepared 2026-09-20 for the marin-community/marin repo. Not filed.
Proposed title and labels:</p>
<ul>
<li>Title:
<code>[grug] Looped transformer variant (K&gt;1 core, growth) for arXiv 2609.19107</code></li>
<li>Labels: <code>experiment, agent-generated</code></li>
</ul>
<h2 id="what-the-study-changed-about-the-plan">What the study changed
about the plan</h2>
<ul>
<li>Marin already has three pieces of this work: the dense paper
replication in PR #9293 (K=1 only, no runs yet), the boundary-operator
MoE experiment in #9280 whose first gate failed, and the June re-entrant
MoE attempt in PR #6387 where every looped arm lost. The plan extends
the dense replication and reuses #6387's per-K eval idea.</li>
<li>The dense replication differs from the paper's released code in six
places. Three of them move its own targets: head size, the MLP width
rule, and the uniform init bound. The width rule was verified against
the paper's exact d26 parameter count and the others against the
released source.</li>
<li>By the paper's own fit, tied looping is compute-neutral at the base
size. The paper's value is in the boundary operator at equal FLOPs, in
growth, and in the exponent. That reshaped every gate.</li>
</ul>
<h2 id="what-the-adversarial-review-changed">What the adversarial review
changed</h2>
<p>Two independent reviewers read the draft: a cold reader with no
context, and a technical critic with the code. Findings that
survived:</p>
<ul>
<li>The first gate compared arms at different token counts and its stop
rule could pass on extra FLOPs alone. The first run is now the one
confound-free pair in the family, Untied-2 against Deep Vanilla under
one recipe.</li>
<li>Dormant cores had no weight-decay exclusion, the EMA copy was not
grown, and the checkpointed K went stale after tied growth. All three
are now specified mechanisms with tests.</li>
<li>The re-warm formula went negative before growth, and the core copy
was not resume-safe. Both fixed.</li>
<li>The growth gate and the data-repetition gate had no iso-FLOP
control. Both now have one, and the expected margins are stated so a
fail is interpretable.</li>
<li>The exponent gate relied on a two-degree-of-freedom fit. It is now a
monotone iso-FLOP gap rule, with the fit reported only.</li>
<li>The flat block tuple, the per-call K override, and a single bucket
were all traded for the dumber option.</li>
</ul>
<h2 id="how-to-use-the-implementation-to-replicate-the-paper">How to use
the implementation to replicate the paper</h2>
<p>The issue's launch section and replication map carry the details. The
order of operations is: materialize the FineWeb caches from PR #9293,
run the two-arm operator test at d8, then the base-size arms with a
second seed to measure the noise floor, then the untied growth arm
against its compute-matched control, then the two ladders for the
exponent claim, and last the four-cell data-repetition test. Each later
experiment in the paper maps to one launcher flag combination in the
replication map.</p>
<p>Two things to decide before filing. The six deviations in the dense
replication belong in a comment on #9292 or #9293 regardless of this
plan, since they change its targets. And the fold-in question is the
#9293 owner's call; the decision log states both cases.</p>
<hr />
<h1 id="proposed-issue-body">Proposed issue body</h1>
<h2 id="tldr">TL;DR</h2>
<p>Plan for <code>experiments/grug/paper_loop/</code>, a copy of
<code>experiments/grug/paper_rep/</code> (PR #9293) that adds the
paper's K&gt;1 core loop, tied or untied cores, and in-training growth
from K=2 to K=4. Two model-config fields (<code>num_loops</code>,
<code>num_cores</code>) and one train-config block (<code>growth</code>)
express all eight paper variants. Three buckets: loop and untied cores,
then growth, then whole-model weight decay and epoch cycling. No code or
runs yet. The first TPU run is the one confound-free pair in the family,
Untied-2 against Deep Vanilla under a shared recipe at d8 (2.4
Vanilla-d8 units; the paper_rep guide estimates 30 minutes per unit on a
v4-16, unmeasured). If the boundary operator does nothing there, the
direction stops.</p>
<h2 id="description">Description</h2>
<p>arXiv 2609.19107 (Chen, Vegesna, Dahal, Wilson; code at <a
href="https://github.com/qlabs-eng/scaling-exponents">https://github.com/qlabs-eng/scaling-exponents</a>)
defines one model family, its Eq. 2: a prelude of P blocks, a core of C
blocks applied K times, a coda of D blocks. Between core passes and
before the coda a boundary operator
<code>BO(h, e) = rms_norm(h) + alpha * e</code> re-injects the prelude
output e; <code>alpha</code> is a fixed scalar
(<code>injection_scale</code> in <code>paper_rep</code>). Model "d-ell"
has ell stored blocks, width 128 * ell, ell heads of size 128. The split
rule is thirds with the remainder going to the core first, then the
coda: d6 2/2/2, d8 2/3/3, d10 3/4/3, d12 4/4/4, d14 4/5/5, d16 5/6/5,
d18 6/6/6. Executed depth is P + K*C + D. The eight variants:</p>
<table>
<thead>
<tr>
<th>Variant</th>
<th>K</th>
<th>Cores stored</th>
<th>BO</th>
<th>Note</th>
</tr>
</thead>
<tbody>
<tr>
<td>Vanilla</td>
<td>1</td>
<td>1</td>
<td>no</td>
<td>plain stack, ell blocks</td>
</tr>
<tr>
<td>Operator-1</td>
<td>1</td>
<td>1</td>
<td>yes</td>
<td>Vanilla plus BO, same params and FLOPs</td>
</tr>
<tr>
<td>Loop-2</td>
<td>2</td>
<td>1</td>
<td>yes</td>
<td>one core, two passes</td>
</tr>
<tr>
<td>Untied-2</td>
<td>2</td>
<td>2</td>
<td>yes</td>
<td>two distinct cores, same FLOPs as Loop-2</td>
</tr>
<tr>
<td>Deep Vanilla</td>
<td>1</td>
<td>1</td>
<td>no</td>
<td>plain stack of ell + C blocks (11 at d8), Untied-2's depth and
parameter count</td>
</tr>
<tr>
<td>Loop-Grow</td>
<td>2 then 4</td>
<td>1</td>
<td>yes</td>
<td>K raised late in training</td>
</tr>
<tr>
<td>Untied-Grow</td>
<td>2 then 4</td>
<td>4</td>
<td>yes</td>
<td>cores 3 and 4 dormant, then copied from 1 and 2</td>
</tr>
<tr>
<td>Deep Vanilla Grow</td>
<td>2 then 4</td>
<td>4</td>
<td>no</td>
<td>Untied-Grow without BO; no training script released</td>
</tr>
</tbody>
</table>
<p>Claims (FineWeb, GPT-2 tokenizer, d6 to d18): Operator-1 improves the
loss-vs-compute exponent over Vanilla (gamma 0.1143 vs 0.1112); Loop-2
and Untied-2 keep that exponent with a better constant; growth raises it
again (Untied-Grow 0.1168; compute multiplier over Vanilla 1.30x at 1e18
FLOPs, 1.55x at 1e20). Under 10 epochs of a 100M-token subset, the
fitted compute-optimal K rises from 1.4 on fresh data to 6.7.</p>
<p>What Marin already has:</p>
<ul>
<li>#9292 / PR #9293: <code>experiments/grug/paper_rep/</code>, dense,
paper optimizer and init, FineWeb GPT-2 data, Operator-1 (K=1). No
looping, no growth. Runs not started as of 2026-09-20.</li>
<li>#9280 / PR #9281: Operator-1 on the MoE template. Gate 1 failed at
hidden width 512 (3.5460 vs 3.5422 baseline); most paper recipe knobs do
not exist in that template.</li>
<li>PR #6387 (June 2026, closed unmerged, branch
<code>weaver/re-entrant-model-testing</code> at 8f1e197): re-entrant MoE
with a tied core, per-pass FiLM, sampled K, consistency loss. Every arm
lost to dense at width 512 (loop-4: 3.905 vs 3.818). Its
<code>eval_sweep.py</code> (loss at several K from one checkpoint) is
copied and adapted here.</li>
<li>Discord #architecture: dlwh flagged the paper on 2026-09-18 (<a
href="https://discord.com/channels/1354881461060243556/1527756652890161292/1550652125220962378">https://discord.com/channels/1354881461060243556/1527756652890161292/1550652125220962378</a>);
Kaiyue Wen linked SMELT, a looped MoE study (arXiv 2609.01343), on
2026-09-20 (<a
href="https://discord.com/channels/1354881461060243556/1550935190505914541/1551067464622669914">https://discord.com/channels/1354881461060243556/1550935190505914541/1551067464622669914</a>).</li>
</ul>
<p>Initiating request (Jeff Hammerbacher, 2026-09-20): "carefully study
implementations of looped transformers in other repos ... then carefully
analyze the Marin codebase, in particular the grug models in Levanter.
Then propose a plan to add a grug-style looped transformer to Levanter
... Finally, describe how to use the new looped transformer
implementation to replicate the experiments and findings from <a
href="https://arxiv.org/abs/2609.19107">https://arxiv.org/abs/2609.19107</a>."</p>
<p>Five other looped-transformer codebases were read for this plan
(Parcae, LoopFormer, Huginn, Mixture-of-Recursions, Looped-GPT). They
add per-pass conditioning, sampled depth, truncated backprop through the
loop, adaptive exit, and per-loop KV caches. The paper's released code
has none of these except a truncated-backprop knob
(<code>num_core_stop_gradients</code>) that every ladder script leaves
at 0. This plan implements the paper's family only; the other mechanisms
are listed under "Not in this pass" with the change each needs.</p>
<p>Six places where <code>paper_rep</code> differs from the released
code (<code>models/layers.py</code>, <code>models/transformer.py</code>,
<code>train.py</code>):</p>
<ol type="1">
<li>Heads: the code uses <code>n_head = ell</code> (head size 128).
<code>paper_rep</code> uses <code>hidden_dim // 64</code> (head size
64). Same parameter count, different attention.</li>
<li>MLP width: the code uses
<code>256 * ceil(8 * width / 3 / 256)</code> (2816 at d8).
<code>paper_rep</code> uses <code>3 * width</code> (3072). Only the
code's rule reproduces the released exact count for the d26 Untied-Grow
model (7,424,049,152 parameters); the two rules differ by 3 percent in
parameters and FLOPs at d8.</li>
<li>Init bound: the code draws input matrices from <code>U(-s, s)</code>
with <code>s = UIS * width ** -0.5</code> (0.011 at d8 for Operator-1).
<code>paper_rep</code> uses <code>U(-UIS, UIS)</code> (0.354).</li>
<li>First core pass: the code feeds the raw prelude output e;
<code>paper_rep</code> (and PR #9281's alpha=0.707 arm) feeds
<code>alpha * e</code>. Identical at alpha=1, different for Loop-2
(alpha=0.707).</li>
<li>Weight decay: the code decays every group, Muon matrices included.
<code>paper_rep</code> decays only the embedding and head. The
data-repetition experiment (Gate 4) sweeps weight decay from 0.05 to 1.6
on the whole model.</li>
<li>Muon momentum: the code ramps it from 0.85 to 0.95 over the first
300 steps; <code>paper_muon</code> holds 0.95. Accepted as a difference;
recorded so a gap has a candidate cause.</li>
</ol>
<p>Items 1 to 3 move #9292's targets and should be fixed in PR #9293
before its arms run. <code>paper_loop</code> implements 4 whatever #9293
does (the parity test covers alpha=1 only), adds a
<code>muon_weight_decay</code> knob for 5, and reruns its own Vanilla
and Operator-1 baselines so every comparison is within one codebase.</p>
<h2 id="hypothesis-or-goal">Hypothesis or Goal</h2>
<p>At d8 on 1B tokens under each variant's own recipe, the paper's Table
6 order is Untied-2 3.2563 &lt; Loop-2 3.2704 &lt; Deep Vanilla 3.2772
&lt; Operator-1 3.3057 &lt; Vanilla 3.3279. Under the shared Vanilla
recipe (Table 6, second column) Untied-2 is 3.2697 and Deep Vanilla
3.2869: same stored blocks, same executed depth, same tokens, and only
the operator differs. Under the Vanilla scaling exponent, 25 percent
more FLOPs buys about 0.04 of loss at d8 (FineWeb fit: E = 1.7064, gamma
= 0.1112), so the paper's Loop-2 gap over Operator-1 (0.035, at 25
percent more FLOPs) means tied looping is compute-neutral at d8; the
paper's own fit says the same (compute multipliers 1.10 for Loop-2 and
1.12 for Operator-1 at 1e18). The paper's claimed value is in the
operator (iso-FLOP), in growth, and in the exponent. In the ladder runs
at d8, Untied-Grow reaches 3.1304 against Untied-2 3.1904 while
consuming 34 percent more FLOPs; the iso-FLOP margin implied by the
paper's multipliers (1.30 vs 1.19) is about 0.015. On a d6 to d12
ladder, Untied-Grow's iso-FLOP advantage over Vanilla grows with scale.
Absolute losses will differ from the paper because the data slice,
packing order, and Muon internals differ; every rule below is on gaps
between arms trained in this variant on the same data, seed, and
hardware, against a measured seed noise floor. The metric everywhere is
<code>eval/loss</code> on the held-out FineWeb validation tag
(<code>eval/fineweb-val-gpt2/loss</code> in #9292), at least 10M tokens
with <code>max_eval_batches</code> fixed across arms, at the final
step.</p>
<h2 id="plan">Plan</h2>
<h3
id="bucket-1-loop-and-untied-cores-paper_loopmodelpy-copied-from-paper_rep">Bucket
1: loop and untied cores (<code>paper_loop/model.py</code>, copied from
<code>paper_rep</code>)</h3>
<p>Config gains two fields:</p>
<div class="sourceCode" id="cb1"><pre
class="sourceCode python"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a>num_loops: <span class="bu">int</span> <span class="op">=</span> <span class="dv">1</span> <span class="co"># K in paper Eq. 2: how many times the core runs per forward.</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>num_cores: <span class="bu">int</span> <span class="op">=</span> <span class="dv">1</span> <span class="co"># Stored core copies. 1 = one core reused every pass (Loop-K).</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a> <span class="co"># K = one distinct core per pass (Untied-K). `tied` property = (num_cores == 1).</span></span></code></pre></div>
<p>Validation: <code>num_loops &gt;= 1</code>;
<code>num_cores == 1 or num_cores &gt;= num_loops</code>;
<code>num_cores &gt; num_loops</code> only when a growth config reaches
<code>num_cores</code> (checked at run start). <code>prelude_len</code>
and <code>coda_len</code> are accepted whenever
<code>boundary_operator</code> is on or <code>num_loops &gt; 1</code>;
the paper's App. D.4 control is the P/C/D split looped without the
operator. Blocks are stored as three fields,
<code>prelude: tuple[Block, ...]</code>,
<code>cores: tuple[tuple[Block, ...], ...]</code>,
<code>coda: tuple[Block, ...]</code>, so the forward, growth, the
dormant mask, and per-pass metrics address cores by index and never
slice a flat tuple. A test helper maps a <code>paper_rep</code>
checkpoint onto these fields by block index.</p>
<p>Forward:</p>
<div class="sourceCode" id="cb2"><pre
class="sourceCode python"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a>h <span class="op">=</span> rms_norm(embed(tokens))</span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> b <span class="kw">in</span> <span class="va">self</span>.prelude: h <span class="op">=</span> b(h)</span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a>e <span class="op">=</span> h <span class="co"># un-normalized prelude output (the code&#39;s &quot;anchor&quot;)</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> k <span class="kw">in</span> <span class="bu">range</span>(cfg.num_loops): <span class="co"># static Python loop; K &lt;= 12 in every paper experiment</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a> core <span class="op">=</span> <span class="va">self</span>.cores[<span class="dv">0</span>] <span class="cf">if</span> cfg.num_cores <span class="op">==</span> <span class="dv">1</span> <span class="cf">else</span> <span class="va">self</span>.cores[k]</span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> b <span class="kw">in</span> core: h <span class="op">=</span> b(h) <span class="co"># one eqx.filter_checkpoint per block application</span></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> cfg.boundary_operator:</span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a> h <span class="op">=</span> rms_norm(h) <span class="op">+</span> cfg.injection_scale <span class="op">*</span> e <span class="co"># after every pass, including before the coda</span></span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> b <span class="kw">in</span> <span class="va">self</span>.coda: h <span class="op">=</span> b(h)</span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a><span class="cf">return</span> rms_norm(h)</span></code></pre></div>
<p>This matches the released <code>_forward_blocks</code> in the setting
every ladder script uses
(<code>recurrence_rmsnorm=recurrence_only</code>,
<code>decoder_inject</code> on). K is read from the static config only;
an eval at another K uses
<code>dataclasses.replace(model.config, num_loops=K)</code>, as PR
#6387's sweep did. A Python loop instead of <code>lax.scan</code>: K is
small and static, untied cores differ per pass, and per-pass metrics and
named scopes are one line each. The largest planned compile is d12 at
K=4 (24 block applications); a later K=12 arm at d12 (56 applications)
is the first place a scan would be needed, and the core loop is the only
place to put it.</p>
<p>Metrics: <code>core/residual_rms_pass_{k}</code> for k = 1..K, the
RMS of the stream change over pass k (input snapshot after the previous
pass's norm and injection, output snapshot before this pass's norm), and
their mean under the released name <code>rms/res_core</code>. At the end
of training <code>eval/loss_at_loops_{k}</code> from the final
checkpoint: k = 1..2K for tied models (the paper's B.4 evaluates more
passes than trained), k = 1..K for untied.</p>
<p><code>recipes.py</code> adds three recipes (acronyms, all from paper
Table 5: RM residual-branch multiplier, OM output multiplier, WTE
token-embedding init std, UIS uniform init scale, GLR global Muon
learning rate, ELRM and HLRM embedding and head multipliers on GLR, WD
weight decay, WU warmup steps, WDR warmdown fraction, beta1/beta2/eps
AdamW moments). GLR is 0.04 for all three.</p>
<table>
<thead>
<tr>
<th>Recipe</th>
<th>RM</th>
<th>OM</th>
<th>WTE</th>
<th>UIS</th>
<th>ELRM</th>
<th>HLRM</th>
<th>WD</th>
<th>WU</th>
<th>WDR</th>
<th>beta1</th>
<th>beta2</th>
<th>eps</th>
<th>alpha</th>
</tr>
</thead>
<tbody>
<tr>
<td>LOOP2</td>
<td>0.25</td>
<td>1</td>
<td>0.02</td>
<td>0.044</td>
<td>0.32</td>
<td>0.113</td>
<td>0.05</td>
<td>40</td>
<td>1</td>
<td>0.8</td>
<td>0.95</td>
<td>1e-10</td>
<td>0.707</td>
</tr>
<tr>
<td>UNTIED2</td>
<td>0.25</td>
<td>1</td>
<td>0.01</td>
<td>0.354</td>
<td>0.16</td>
<td>0.16</td>
<td>0.071</td>
<td>40</td>
<td>1</td>
<td>0.8</td>
<td>0.99</td>
<td>1e-8</td>
<td>1</td>
</tr>
<tr>
<td>DEEP_VANILLA</td>
<td>0.5</td>
<td>1</td>
<td>0.005</td>
<td>0.5</td>
<td>0.16</td>
<td>0.057</td>
<td>0.1</td>
<td>5</td>
<td>0.8</td>
<td>0.8</td>
<td>0.99</td>
<td>1e-8</td>
<td>none</td>
</tr>
</tbody>
</table>
<p><code>launch.py</code> takes
<code>--arm {vanilla, op1, loop2, untied2, deep_vanilla, loop_grow, untied_grow}</code>,
<code>--depth {6,...,18}</code>, and overrides <code>--recipe</code>
(train one arm under another arm's recipe), <code>--num-loops</code>,
<code>--crossover</code>, <code>--steps</code>. A table transcribed from
<code>ladder_scripts/*.sh</code> gives each (arm, depth) its step count,
crossover fraction, and GLR (d8: Vanilla 1962; Operator-1 and Loop-2
2355; Untied-2 and Deep Vanilla 2796; Loop-Grow 2563 at crossover 0.82;
Untied-Grow 3341 at 0.71; in every script the LR schedule length equals
the step count). The paper's rule
<code>GLR(d) = 0.04 * (N(8) / N(d)) ** exponent</code>, with N the
stored parameter count including embedding and head and a per-arm
exponent from Table 9, is a consistency test on that table. Gate 1 arms
use <code>--steps 1907</code>, the Table 6 budget that #9292 uses. Deep
Vanilla Grow has no released script and is left out.</p>
<p>Tests (CPU): (1) <code>num_loops=1, num_cores=1</code> reproduces
<code>paper_rep.Transformer</code> logits from the same parameters, with
<code>boundary_operator</code> off and on at alpha=1; (2) tied K=2
equals a hand-unrolled forward; (3) untied K=2: perturbing
<code>cores[1]</code> changes the output; (6)
<code>_compute_flops</code> at K=1 equals <code>paper_rep</code>'s, and
the K=2 to K=1 ratio equals the block-count ratio from the paper
formula; (8) validation rejects <code>num_cores=2, num_loops=3</code>
and <code>num_cores &gt; num_loops</code> without growth; (9) split
table extended to the paper's d10 to d26 rows.
<code>tests/test_grug_variant_contracts.py</code> discovers the
directory and lowers one train step at the defaults. A
<code>docs/reports/grug-archive.md</code> entry records the origin
commit of <code>paper_rep</code>.</p>
<h3 id="bucket-2-growth-paper_looptrainpy">Bucket 2: growth
(<code>paper_loop/train.py</code>)</h3>
<p>One new train-config block:</p>
<div class="sourceCode" id="cb3"><pre
class="sourceCode python"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a>growth: GrowthConfig <span class="op">|</span> <span class="va">None</span> <span class="op">=</span> <span class="va">None</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="co"># GrowthConfig(num_loops_after=4, crossover_fraction=0.71, lr_rewarm_steps=40)</span></span></code></pre></div>
<p>Growth step g =
<code>int(crossover_fraction * num_train_steps)</code>, the code's rule;
with a constant batch, <code>crossover_fraction = 1 - rho</code> where
rho is the paper's fraction of tokens trained after growth. The run loop
builds two jitted train steps, one per K, from
<code>dataclasses.replace(config, num_loops=K)</code>, and compiles both
ahead of time before step 0 so a K=4 OOM or a long compile shows up at
launch. Driver order at every step: checkpoint hook, then
<code>if step == g: state = grow(state)</code>, then the step for the
current K. <code>grow</code> on an untied model copies
<code>cores[k % K_before]</code> into <code>cores[k]</code> for
<code>k &gt;= K_before</code> (the code's
<code>dep_stack_grow_init</code>) in both <code>params</code> and
<code>ema_params</code>; on a tied model it changes nothing. In both
cases it replaces the static config inside the state with
<code>num_loops=num_loops_after</code>, so the checkpoint written after
g carries the trained K and <code>train/num_loops</code> is read from
the config. A run resumed at step s &gt; g never re-applies the copy;
one resumed at s &lt;= g applies it once when it reaches g. Growth runs
default to <code>ema_beta=None</code>.</p>
<p>Dormant cores: in the K_before train step the optimizer's update tree
is zeroed on <code>cores[K_before:]</code> with a static mask after
<code>optimizer.update</code> and before <code>apply_updates</code>.
This removes the decay term and the already-zero Muon term without
touching the optimizer state, so no group changes shape at g. Levanter's
Muon normalizes by <code>norm + eps</code>, so a zero gradient produces
a zero update and no NaN. The released code keeps the same cores at zero
momentum with a cautious decay that never fires on zero gradients.</p>
<p>Re-warm: the update tree is multiplied by 1 before g and by
<code>clip((step - g) / lr_rewarm_steps, 0, 1)</code> from g on, inside
<code>train_step</code>. The <code>paper_muon</code> optimizer and its
warmup-stable-decay schedule are untouched; multiplying the whole update
also scales the decayed-weight term, which is what the code's LR
multiplier does.</p>
<p>FLOPs across g: <code>log_performance_stats</code> takes one scalar,
so it receives the K_before value and a variant-local callback logs
<code>throughput/flops_per_token_paper</code> per phase and a cumulative
<code>train/total_flops</code>, both recomputed from step and g so they
survive resume, using the paper's per-token formula
(<code>6 * (executed block params + head params) + 12 * heads * head_size * mean attended keys</code>,
the released <code>estimate_flops</code>). That number is the compute
axis for every fit; Levanter's <code>lm_flops_per_token</code> is used
for MFU only. <code>parameter_count</code> stays the stored count
(dormant cores included, the paper's convention);
<code>train/executed_block_params</code> is logged beside it.</p>
<p>Tests: (4) before g the loss gradient wrt <code>cores[2:]</code> is
exactly zero and one <code>paper_muon</code> step with nonzero
<code>muon_weight_decay</code> leaves them unchanged and finite; after
<code>grow</code>, <code>cores[2] == cores[0]</code> and
<code>cores[3] == cores[1]</code>, and the state's config reads K=4; (5)
tied <code>grow</code> changes no parameters and the K=4 forward differs
from K=2; (7) re-warm multiplier is 1 before g, 0 at g, 1 at g +
<code>lr_rewarm_steps</code>; (10) a run checkpointed at g+1 and resumed
has <code>cores[2] != cores[0]</code> after one more step.</p>
<h3 id="bucket-3-whole-model-weight-decay-and-epoch-cycling">Bucket 3:
whole-model weight decay and epoch cycling</h3>
<p><code>muon_weight_decay</code> on <code>PaperMuonConfig</code>
(decoupled, same value as the AdamW groups when set, so the paper's
single WD knob can be reproduced). Epoch cycling: Marin's
<code>train_lm</code> path resolves <code>num_train_epochs</code> from
the token count over a single dataset
(<code>lib/marin/src/marin/experiment/train.py</code>); the grug loader
builds a <code>MixtureDataset</code>. Which path applies to a 100M-token
cache repeated ten times with a fresh shuffle per epoch is settled
before Gate 4 is scheduled.</p>
<h3 id="not-in-this-pass">Not in this pass</h3>
<p>Truncated backprop (one <code>jax.lax.stop_gradient(h)</code> at the
end of pass k, the code's <code>num_core_stop_gradients</code>);
per-pass FiLM or time embeddings (one table lookup before the inner
loop); sampled K (one compiled train step per K in the sample set plus
Python-side dispatch, the cost PR #6387 paid); consistency loss;
adaptive exit; the KL effective-depth probe (paper Sec. 6); DCLM CORE;
HF export of a looped model. Of the Table 4 operator ablations,
norm-only is already expressible
(<code>boundary_operator=True, injection_scale=0</code>); injection-only
and coda-injection-off need two booleans.</p>
<h3 id="gates-and-cost">Gates and cost</h3>
<p>One unit is the Vanilla d8 ladder run: 1962 steps of 524,288 tokens,
1.06e18 FLOPs by the paper's per-token formula, which with the split
rule and the ladder step counts reproduces the released CSV endpoints
(3.25e17 for Vanilla d6, 1.49e20 for Untied-Grow d18). The paper_rep
guide estimates 30 minutes per unit on a v4-16; unmeasured. Noise floor:
the pooled RMS of the seed differences from the second-seed arms in Gate
1b, on the metric above. A gap rule passes when the gap exceeds twice
the floor. If the floor exceeds half the paper gap being tested, two
more seeds are run before the gate is judged.</p>
<table>
<thead>
<tr>
<th>Gate</th>
<th>Arms at d8, 1907 steps unless noted (units each)</th>
<th>Units</th>
<th>Pass rule</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>bucket 1 tests</td>
<td>0</td>
<td>green on CPU</td>
</tr>
<tr>
<td>1a</td>
<td>Untied-2 and Deep Vanilla, both under <code>VANILLA_RECIPE</code>
(1.2 each)</td>
<td>2.4</td>
<td>Untied-2 &lt; Deep Vanilla by more than twice the floor (paper
0.017). Same stored blocks, executed depth, tokens, and recipe; only the
operator differs. Else stop.</td>
</tr>
<tr>
<td>1b</td>
<td>Vanilla, Operator-1 (1.0 each); Loop-2, Untied-2, Deep Vanilla under
own recipes (1.2 each); second seeds of Vanilla (1.0) and Loop-2
(1.2)</td>
<td>7.8</td>
<td>Operator-1 &lt; Vanilla by more than twice the floor (own recipes,
paper 0.022; same params, FLOPs, and tokens). Reported, not gated,
because FLOPs or params differ: Loop-2 vs Operator-1 (paper 0.035; about
0.04 is expected from the extra FLOPs alone), Loop-2 vs Untied-2 (paper
0.014, the cost of tying at iso-FLOP), Loop-2 vs Deep Vanilla (paper
0.007).</td>
</tr>
<tr>
<td>2</td>
<td>Untied-Grow at 3341 steps, crossover 0.71 (2.4); iso-FLOP control
Untied-2 at 3744 steps (2.4)</td>
<td>4.8</td>
<td>Untied-Grow &lt; control by more than twice the floor. Expected
margin about 0.015 (the paper's per-run gap 0.060 minus about 0.045 the
control gains from 34 percent more compute), so this gate needs a floor
below 0.007; otherwise more seeds, or the same pair at d10 (7.3 + 7.3
units).</td>
</tr>
<tr>
<td>3</td>
<td>d6 to d12 ladders at ladder step counts: Vanilla (10.5), Untied-Grow
(26.4)</td>
<td>36.9</td>
<td>For each Untied-Grow rung, Vanilla's loss at the same compute by
log-log interpolation on the Vanilla ladder; the gap is non-decreasing
from d6 to d12 and the d12 gap exceeds the d6 gap by more than twice the
floor. Fitted gamma (E frozen from the Vanilla fit, OLS of
<code>log(L - E)</code> on <code>log C</code>) and the compute
multiplier at the Vanilla-d8 loss are reported as descriptive numbers
only; four points leave two residual degrees of freedom.</td>
</tr>
<tr>
<td>4</td>
<td>Operator-1 recipe: d8 K=4 tied and d10 K=1 (matched per-token FLOPs,
2.83e8 vs 2.77e8), each on fresh 1B tokens (WD 0.05) and on 100M tokens
x 10 epochs (WD 0.8, the paper's K=1-selected value); 1.7 each</td>
<td>6.8</td>
<td>The iso-FLOP advantage of d8-K4 over d10-K1 under repetition exceeds
its advantage on fresh data by more than twice the floor.</td>
</tr>
</tbody>
</table>
<p>Gate 1a runs before any growth code exists and does not depend on
another PR's runs. Gate 2's control is the fixed-K parent trained to the
grow arm's FLOP count; without it a grow arm only shows that more
compute helps. Gate 3 adds Operator-1 (12.6), Untied-2 (19.8), or
Loop-Grow (18.8) rungs only for the descriptive fit. Extending past d12
is not planned; the full paper ladders (eight arms, d6 to d18) are about
1,560 units.</p>
<h2 id="replication-map">Replication map</h2>
<table>
<thead>
<tr>
<th>Paper experiment</th>
<th>Arms and settings</th>
<th>Cost (units)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Table 6 transfer regret, d8, 1B tokens (loss under own recipe vs
under the Vanilla recipe)</td>
<td>Gates 1a and 1b cover 7 of the 10 cells; the remaining
<code>--recipe vanilla</code> cells are Loop-2 and Operator-1</td>
<td>about 2.2</td>
</tr>
<tr>
<td>Table 4 operator ablation</td>
<td>norm-only via <code>injection_scale=0</code> now; injection-only and
coda-injection-off need two booleans</td>
<td>1.2 per cell</td>
</tr>
<tr>
<td>Fig. 3 exponents, FineWeb d6 to d18</td>
<td>Gate 3 is the Vanilla and Untied-Grow d6 to d12 subset; other arms
and depths per the unit table</td>
<td>36.9 first; 1,560 full</td>
</tr>
<tr>
<td>A.2.2 loop count at fixed tokens</td>
<td><code>--arm loop2 --num-loops K</code> for K in {1, 2, 3, 4, 6} at
d8, Loop-2 recipe</td>
<td>about 8</td>
</tr>
<tr>
<td>A.2.3 growth transition</td>
<td><code>--arm untied_grow --crossover f</code> for f in {0.5, 0.6,
0.71, 0.8, 0.9} at d8, each with its iso-FLOP control</td>
<td>about 24</td>
</tr>
<tr>
<td>Loop-Grow (ladder d8 3.2002 vs Loop-2 3.2323)</td>
<td><code>--arm loop_grow</code> (2563 steps, crossover 0.82) with
control Loop-2 at 2760 steps; the paper's multipliers (1.12 vs 1.10 at
1e18) imply an iso-FLOP margin near 0.003 at d8, below any seed floor,
so this needs a d12 or larger rung to be informative</td>
<td>3.6 at d8</td>
</tr>
<tr>
<td>B.4 inference-time passes</td>
<td><code>eval/loss_at_loops_{k}</code> from any tied checkpoint; no
training</td>
<td>0</td>
</tr>
<tr>
<td>Sec. 5 data repetition</td>
<td>Gate 4 first; then K in {1, 2, 3, 4, 6, 8, 12} at d8 with WD in
{0.05, 0.2, 0.4, 0.8, 1.2, 1.6}, each cell paired with an iso-FLOP K=1
model of larger depth</td>
<td>6.8 first; about 80 for the d8 grid</td>
</tr>
<tr>
<td>Sec. 6 KL effective depth</td>
<td>later bucket (per-block logit-lens readout)</td>
<td>later</td>
</tr>
<tr>
<td>Sec. 4.3 FineWeb-Edu d26 extrapolation, CORE 0.3865</td>
<td>1.2e21 FLOPs, about 1,160 units, plus a FineWeb-Edu cache and a CORE
harness</td>
<td>not planned</td>
</tr>
</tbody>
</table>
<h2 id="status">Status</h2>
<p>Proposed on 2026-09-20. No code written. Open decisions: whether
<code>paper_rep</code> deviations 1 to 3 are fixed in PR #9293 by its
owner or carried into <code>paper_loop</code>; whether Gate 1a runs on
the same v4-16 reservation as the #9292 arms.</p>
<h2 id="links">Links</h2>
<ul>
<li>Logbook: none yet</li>
<li>W&amp;B Report: none yet</li>
<li>Paper: <a
href="https://arxiv.org/abs/2609.19107">https://arxiv.org/abs/2609.19107</a>
; code: <a
href="https://github.com/qlabs-eng/scaling-exponents">https://github.com/qlabs-eng/scaling-exponents</a>
(<code>ladder_scripts/</code>, <code>plotting_scripts/data/</code>)</li>
<li>Dense replication: #9292, PR #9293. MoE Operator-1: #9280, PR #9281.
Re-entrant MoE: PR #6387, #6390. Cross-layer expert tying: #8032.</li>
<li>Discord: dlwh 2026-09-18 <a
href="https://discord.com/channels/1354881461060243556/1527756652890161292/1550652125220962378">https://discord.com/channels/1354881461060243556/1527756652890161292/1550652125220962378</a>
; Kaiyue Wen 2026-09-20 <a
href="https://discord.com/channels/1354881461060243556/1550935190505914541/1551067464622669914">https://discord.com/channels/1354881461060243556/1550935190505914541/1551067464622669914</a></li>
</ul>
<h2 id="decision-log">Decision Log</h2>
<ul>
<li>2026-09-20: Build on the dense <code>paper_rep</code> template. The
paper's recipe knobs (RM, OM, UIS, WTE, three-group Muon) exist there
and not in the MoE template; #9280 showed the MoE vehicle cannot
separate the operator from the missing recipe.</li>
<li>2026-09-20: A separate directory stacked on the
<code>paper-replication</code> branch until PR #9293 merges. The case
for folding the loop into <code>paper_rep</code> is real: the PR is
unmerged with no runs, K=1 reproduces it exactly, and deviations 1 to 3
must otherwise be fixed in two copies. Against it: #9292's gates and its
autonomous agent guide are defined on K=1, one change bucket per PR is
the grug rule, and the CI variant-diff report plus the archive entry
exist for exactly this duplication. If the #9293 owner prefers folding
in, bucket 1 moves as one commit.</li>
<li>2026-09-20: Blocks stored as <code>prelude</code>,
<code>cores</code>, <code>coda</code> fields. A flat tuple would keep
<code>paper_rep</code>'s checkpoint layout at K=1, but every loop-aware
code path would then slice it; a test helper maps old checkpoints by
index instead.</li>
<li>2026-09-20: K lives only in the static config. The two train steps
and the eval sweep all use
<code>dataclasses.replace(config, num_loops=K)</code>; no per-call
override.</li>
<li>2026-09-20: Growth lives in the train config. The model config
describes stored parameters and the default K; when K changes is a
schedule.</li>
<li>2026-09-20: First core pass receives the raw prelude output,
following the released code; the paper's Algorithm 1 implies
<code>alpha * e</code>. The two agree at alpha=1.</li>
<li>2026-09-20: Every gate is an iso-FLOP or FLOP-corrected comparison
inside this variant. The paper's growth runs train on more tokens than
their fixed-K parents, and Loop-2 spends 25 percent more FLOPs than
Operator-1 at equal steps.</li>
<li>2026-09-20: No mechanism outside the paper's family in the first
pass. PR #6387 tested FiLM, sampled K, and a consistency loss and all
lost; the paper's own B.4 found sampled K slightly worse than fixed
K.</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p>None yet.</p>
</div>
<footer class="provenance">
<p><em>Generated: 2026-09-20 18:55 UTC &middot; Viewing: <span id="viewing-time"></span></em></p>
<blockquote>
<p><em>Data: marinmirror — 217761 chunks, built 2026-09-20 · summaries
through 2026-09-07_2026-09-13 (refreshed this run). Also read: the
paper's released code at <a
href="https://github.com/qlabs-eng/scaling-exponents">github.com/qlabs-eng/scaling-exponents</a>
(main), PR <a
href="https://github.com/marin-community/marin/pull/9293">#9293</a>'s
<code>paper-replication</code> branch, PR <a
href="https://github.com/marin-community/marin/pull/6387">#6387</a>'s
<code>weaver/re-entrant-model-testing</code> branch at 8f1e197, and PR
<a href="https://github.com/marin-community/marin/pull/9281">#9281</a>'s
<code>boundary-operator-phase-0</code> branch.</em></p>
<p><em>Query: "carefully study implementations of looped transformers in
other repos ... propose a plan to add a grug-style looped transformer to
Levanter ... review with an adversarial eye ... share as a proposed
issue for the Marin repo ... describe how to replicate <a
href="https://arxiv.org/abs/2609.19107">https://arxiv.org/abs/2609.19107</a>"</em></p>
<p><em>Sub-queries: "Parcae injection.py recurrence vs backprop depth" ·
"LoopFormer time embeddings, alignment loss, always-true elif" · "Huginn
iterate_forward, KV cache, stopping criteria" · "Mixture-of-Recursions
expert-choice router" · "Looped-GPT recurrent_refinement, loop_steps
baseline" · "grug base/moe/hero model stacking, ArrayStacked, remat" ·
"Marin prior looping work: re-entrant #6387, boundary operator
#9280/#9281, paper_rep #9292/#9293, cross-layer tying #8032" · "grug
conventions: change-grug, grugformer.md, variant contracts, archive" ·
"FLOP/MFU/parameter accounting under weight tying" · "arXiv 2609.19107
architecture, recipes, ladders, growth, data-constrained regime, CORE" ·
"SMELT 2609.01343" · "Discord #architecture thread on the paper" ·
"Levanter epoch cycling and Muon zero-gradient safety"</em></p>
</blockquote>
</footer>
</div>
</body>
</html>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment