Last active
September 23, 2026 13:24
-
-
Save hammer/881fb086dc2d326c348e7eb16df76793 to your computer and use it in GitHub Desktop.
Expected MFU for the 535B-A23B hero on Megatron instead of Levanter
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>Expected MFU for the 535B-A23B hero on a Megatron stack instead of Levanter</title> | |
| <style> | |
| :root { | |
| --ink: #1a1a2e; --ink-secondary: #555770; --ink-faint: #8a8a9a; | |
| --accent: #8b2500; --accent-light: #c4530a; | |
| --surface: #faf9f6; --surface-raised: #f0eeea; --surface-code: #f5f3ef; | |
| --rule: #d4d0c8; --link: #8b2500; --link-hover: #c4530a; --ref-bg: #f7f5f1; | |
| --content-width: 650px; --sidenote-width: 230px; --sidenote-gap: 30px; | |
| } | |
| @media (prefers-color-scheme: dark) { | |
| :root:not([data-theme="light"]) { | |
| --ink: #d8d5cf; --ink-secondary: #9e9bab; --ink-faint: #6e6b7b; | |
| --accent: #d4764e; --accent-light: #e8956e; | |
| --surface: #1a1a24; --surface-raised: #242430; --surface-code: #20202c; | |
| --rule: #33333f; --link: #d4764e; --link-hover: #e8956e; --ref-bg: #1e1e2a; | |
| } | |
| } | |
| :root[data-theme="dark"] { | |
| --ink: #d8d5cf; --ink-secondary: #9e9bab; --ink-faint: #6e6b7b; | |
| --accent: #d4764e; --accent-light: #e8956e; | |
| --surface: #1a1a24; --surface-raised: #242430; --surface-code: #20202c; | |
| --rule: #33333f; --link: #d4764e; --link-hover: #e8956e; --ref-bg: #1e1e2a; | |
| } | |
| * { margin: 0; padding: 0; box-sizing: border-box; } | |
| body { | |
| background: var(--surface); color: var(--ink); | |
| font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; | |
| font-size: 16px; line-height: 1.7; | |
| -webkit-font-smoothing: antialiased; | |
| } | |
| .page { | |
| max-width: calc(var(--content-width) + var(--sidenote-width) + var(--sidenote-gap) + 80px); | |
| margin: 0 auto; padding: 3rem 40px 4rem; position: relative; | |
| } | |
| @media (max-width: 1060px) { | |
| .page { max-width: 100%; padding: 2rem 1.5rem 3rem; } | |
| } | |
| .content { max-width: var(--content-width); } | |
| .paper-header { max-width: var(--content-width); margin-bottom: 2.5rem; padding-bottom: 2rem; } | |
| .paper-header h1 { | |
| font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif; | |
| font-size: 2rem; font-weight: 600; line-height: 1.25; color: var(--ink); | |
| text-wrap: balance; margin-bottom: 0.6rem; letter-spacing: -0.01em; | |
| } | |
| .paper-meta { font-size: 0.875rem; color: var(--ink-secondary); line-height: 1.5; } | |
| .paper-meta .author { font-weight: 500; } | |
| .paper-meta .mumwelt-link { color: var(--ink-secondary); text-decoration: none; border-bottom: 1px dotted var(--ink-faint); } | |
| .paper-meta .mumwelt-link:hover { color: var(--accent); border-bottom-color: var(--accent); } | |
| .prompt-box { | |
| max-width: var(--content-width); margin-bottom: 2.5rem; position: relative; | |
| } | |
| .prompt-label { | |
| position: absolute; left: -0.6em; top: 50%; transform: translateY(-50%); | |
| font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif; | |
| font-size: 5rem; font-weight: 700; color: var(--ink); opacity: 0.08; | |
| line-height: 1; pointer-events: none; user-select: none; | |
| } | |
| .prompt-text { | |
| font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif; | |
| font-style: italic; font-size: 1.15rem; line-height: 1.55; color: var(--ink); | |
| position: relative; | |
| } | |
| .abstract { margin-bottom: 2.5rem; max-width: var(--content-width); } | |
| .abstract-label { | |
| font-size: 0.7rem; font-weight: 600; text-transform: uppercase; | |
| letter-spacing: 0.1em; color: var(--ink-secondary); margin-bottom: 0.5rem; | |
| } | |
| .abstract p { font-size: 0.92rem; line-height: 1.75; color: var(--ink); } | |
| h1, h2, h3 { | |
| font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif; | |
| font-weight: 600; color: var(--ink); text-wrap: balance; | |
| } | |
| h2 { font-size: 1.4rem; margin-top: 2.5rem; margin-bottom: 0.75rem; letter-spacing: -0.005em; } | |
| h3 { font-size: 1.1rem; margin-top: 1.75rem; margin-bottom: 0.5rem; } | |
| p { margin-bottom: 1rem; max-width: var(--content-width); } | |
| a { color: var(--link); text-decoration: none; border-bottom: 1px solid transparent; | |
| transition: border-color 0.15s, color 0.15s; } | |
| a:hover { color: var(--link-hover); border-bottom-color: var(--link-hover); } | |
| .date-label { cursor: default; border-bottom: 1px dotted var(--ink-faint); } | |
| strong { font-weight: 600; } | |
| .sidenote-checkbox { display: none; } | |
| .sidenote-toggle { display: none; } | |
| .sidenote { | |
| float: right; clear: right; width: var(--sidenote-width); | |
| margin-right: calc(-1 * (var(--sidenote-width) + var(--sidenote-gap))); | |
| margin-top: 0.2rem; margin-bottom: 1rem; | |
| font-size: 0.8rem; line-height: 1.5; color: var(--ink-secondary); | |
| } | |
| .sidenote-number { font-size: 0.7rem; font-weight: 600; color: var(--accent); margin-right: 0.3em; } | |
| @media (max-width: 1060px) { | |
| .sidenote-toggle { | |
| display: inline; cursor: pointer; color: var(--accent); | |
| font-size: 0.78rem; font-weight: 600; user-select: none; | |
| } | |
| .sidenote { | |
| float: none; display: none; width: 100%; margin: 0.4rem 0 0.75rem 0; | |
| font-size: 0.84rem; padding: 0.6rem 0.9rem; background: var(--surface-raised); | |
| border-radius: 4px; border-left: 2px solid var(--accent); | |
| } | |
| .sidenote-checkbox:checked + .sidenote { display: block; } | |
| .sidenote-number { display: none; } | |
| } | |
| blockquote { border-left: 2px solid var(--rule); padding-left: 1.25rem; | |
| margin: 1.25rem 0; color: var(--ink-secondary); font-style: italic; } | |
| code { font-family: 'SF Mono', Menlo, Consolas, monospace; font-size: 0.85em; | |
| background: var(--surface-code); padding: 0.15em 0.35em; border-radius: 3px; } | |
| pre { background: var(--surface-code); border: 1px solid var(--rule); border-radius: 4px; | |
| padding: 1rem 1.25rem; overflow-x: auto; margin: 1.25rem 0; max-width: var(--content-width); } | |
| pre code { background: none; padding: 0; font-size: 0.82rem; line-height: 1.6; } | |
| ul, ol { margin-bottom: 1rem; padding-left: 1.5rem; max-width: var(--content-width); } | |
| li { margin-bottom: 0.35rem; } | |
| li::marker { color: var(--ink-faint); } | |
| table { max-width: var(--content-width); border-collapse: collapse; width: 100%; | |
| margin: 1.25rem 0; font-variant-numeric: tabular-nums; font-size: 0.9rem; } | |
| thead { border-top: 2px solid var(--ink); border-bottom: 1px solid var(--ink); } | |
| th { font-weight: 600; padding: 0.3rem 0.75rem 0.35rem; text-align: left; | |
| line-height: 1.2; white-space: nowrap; font-size: 0.82rem; vertical-align: bottom; } | |
| td { padding: 0.35rem 0.75rem; border: none; vertical-align: top; } | |
| tbody { border-bottom: 1.5px solid var(--ink); } | |
| th:first-child, td:first-child { padding-left: 0; } | |
| th:last-child, td:last-child { padding-right: 0; } | |
| figure { margin: 2rem 0; max-width: var(--content-width); } | |
| figure img { width: 100%; border-radius: 3px; border: 1px solid var(--rule); } | |
| figcaption { font-size: 0.8rem; color: var(--ink-secondary); margin-top: 0.5rem; | |
| line-height: 1.5; font-style: italic; } | |
| footer.provenance { | |
| margin-top: 3rem; padding-top: 1rem; max-width: var(--content-width); | |
| font-size: 0.78rem; line-height: 1.6; color: var(--ink-faint); | |
| } | |
| footer.provenance p { margin-bottom: 0.3rem; } | |
| footer.provenance a { color: var(--ink-faint); } | |
| footer.provenance blockquote { margin: 0; padding: 0; border: none; color: inherit; } | |
| a[data-hover-title] { position: relative; } | |
| .hover-card { | |
| position: absolute; bottom: 100%; left: 50%; transform: translateX(-50%); | |
| width: 320px; max-width: 90vw; padding: 0.65rem 0.8rem; | |
| background: var(--surface-raised); border: 1px solid var(--rule); | |
| border-radius: 6px; box-shadow: 0 4px 12px rgba(0,0,0,0.1); | |
| font-size: 0.78rem; line-height: 1.45; color: var(--ink); | |
| pointer-events: none; z-index: 100; margin-bottom: 6px; | |
| opacity: 0; transition: opacity 0.12s; | |
| } | |
| a[data-hover-title]:hover .hover-card, | |
| a[data-hover-title]:focus .hover-card, | |
| a.cite:hover .hover-card { opacity: 1; } | |
| .hover-card .hc-title { font-weight: 600; margin-bottom: 0.2rem; } | |
| .hover-card .hc-meta { font-size: 0.72rem; color: var(--ink-faint); margin-bottom: 0.25rem; } | |
| .hover-card .hc-status { | |
| display: inline-block; font-size: 0.65rem; font-weight: 600; | |
| text-transform: uppercase; letter-spacing: 0.04em; | |
| padding: 0.1em 0.4em; border-radius: 3px; margin-right: 0.4em; | |
| } | |
| .hc-status-open { background: #fff3e0; color: #e65100; } | |
| .hc-status-closed, .hc-status-merged { background: #e8f5e9; color: #2e7d32; } | |
| @media (prefers-color-scheme: dark) { | |
| :root:not([data-theme="light"]) .hover-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); } | |
| :root:not([data-theme="light"]) .hc-status-open { background: #3a2a10; color: #ffb74d; } | |
| :root:not([data-theme="light"]) .hc-status-closed, | |
| :root:not([data-theme="light"]) .hc-status-merged { background: #1b3a1e; color: #66bb6a; } | |
| } | |
| :root[data-theme="dark"] .hover-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); } | |
| :root[data-theme="dark"] .hc-status-open { background: #3a2a10; color: #ffb74d; } | |
| :root[data-theme="dark"] .hc-status-closed, | |
| :root[data-theme="dark"] .hc-status-merged { background: #1b3a1e; color: #66bb6a; } | |
| .hover-card .hc-desc { color: var(--ink-secondary); } | |
| .cite { | |
| font-size: 0.72rem; vertical-align: super; line-height: 0; | |
| color: var(--accent); font-weight: 600; text-decoration: none; | |
| border-bottom: none !important; position: relative; | |
| } | |
| .cite:hover { color: var(--link-hover); } | |
| .references { margin-top: 3rem; max-width: var(--content-width); } | |
| .references h2 { font-size: 1.15rem; margin-bottom: 1rem; } | |
| .ref-list { list-style: none; padding: 0; counter-reset: ref; } | |
| .ref-list li { | |
| counter-increment: ref; display: flex; align-items: baseline; | |
| gap: 0.5em; font-size: 0.82rem; line-height: 1.55; | |
| margin-bottom: 0.4rem; color: var(--ink-secondary); | |
| } | |
| .ref-list li::before { | |
| content: "[" counter(ref) "]"; flex-shrink: 0; | |
| font-variant-numeric: tabular-nums; color: var(--ink-faint); | |
| font-size: 0.78rem; min-width: 2.2em; | |
| } | |
| .ref-list .ref-body { flex: 1; min-width: 0; } | |
| .ref-list .ref-title { font-weight: 500; color: var(--ink); } | |
| .ref-list .ref-url { | |
| font-family: 'SF Mono', Menlo, Consolas, monospace; font-size: 0.75rem; | |
| color: var(--ink-faint); word-break: break-all; margin-left: 0.4em; | |
| } | |
| .ref-list .ref-url a { color: var(--ink-faint); border-bottom: none; } | |
| .ref-list .ref-url a:hover { color: var(--link-hover); } | |
| .ref-back { | |
| color: var(--accent); text-decoration: none; border-bottom: none !important; | |
| margin-left: 0.3em; font-size: 0.78rem; | |
| } | |
| .ref-back:hover { color: var(--link-hover); } | |
| .status { | |
| display: inline-block; font-size: 0.65rem; font-weight: 600; | |
| text-transform: uppercase; letter-spacing: 0.04em; | |
| padding: 0.15em 0.5em; border-radius: 3px; vertical-align: middle; | |
| } | |
| .status-done { background: #e8f5e9; color: #2e7d32; } | |
| .status-open { background: #fff3e0; color: #e65100; } | |
| .status-blocked { background: #fce4ec; color: #c62828; } | |
| @media (prefers-color-scheme: dark) { | |
| :root:not([data-theme="light"]) .status-done { background: #1b3a1e; color: #66bb6a; } | |
| :root:not([data-theme="light"]) .status-open { background: #3a2a10; color: #ffb74d; } | |
| :root:not([data-theme="light"]) .status-blocked { background: #3a1520; color: #ef9a9a; } | |
| } | |
| :root[data-theme="dark"] .status-done { background: #1b3a1e; color: #66bb6a; } | |
| :root[data-theme="dark"] .status-open { background: #3a2a10; color: #ffb74d; } | |
| :root[data-theme="dark"] .status-blocked { background: #3a1520; color: #ef9a9a; } | |
| .cite-card { | |
| position: absolute; bottom: 100%; left: 50%; transform: translateX(-50%); | |
| width: 360px; max-width: 90vw; padding: 0.65rem 0.8rem; | |
| background: var(--surface-raised); border: 1px solid var(--rule); | |
| border-radius: 6px; box-shadow: 0 4px 12px rgba(0,0,0,0.1); | |
| font-size: 0.78rem; line-height: 1.45; color: var(--ink); | |
| pointer-events: none; z-index: 100; margin-bottom: 6px; | |
| opacity: 0; transition: opacity 0.12s; | |
| font-weight: 400; vertical-align: baseline; text-align: left; | |
| } | |
| a.cite:hover .cite-card { opacity: 1; } | |
| @media (prefers-color-scheme: dark) { | |
| :root:not([data-theme="light"]) .cite-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); } | |
| } | |
| :root[data-theme="dark"] .cite-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); } | |
| @media print { | |
| body { font-size: 11pt; } | |
| .page { max-width: 100%; padding: 0; } | |
| .sidenote { float: right; width: 180px; margin-right: -210px; } | |
| .sidenote-toggle { display: none; } | |
| a { color: inherit; border-bottom: none; } | |
| .hover-card { display: none; } | |
| .cite-card { display: none; } | |
| } | |
| .katex{font:normal 1.21em KaTeX_Main,Times New Roman,serif;line-height:1.2;position:relative;text-indent:0;text-rendering:auto}.katex *{-ms-high-contrast-adjust:none!important;border-color:currentColor}.katex .katex-version:after{content:"0.18.7"}.katex .katex-mathml{border:0;-webkit-clip-path:inset(50%);clip-path:inset(50%);height:1px;overflow:hidden;padding:0;position:absolute;width:1px}.katex .katex-html>.katex-newline{display:block}.katex .katex-base{position:relative;white-space:nowrap;width:-webkit-min-content;width:-moz-min-content;width:min-content}.katex .katex-base,.katex .katex-strut{display:inline-block}.katex .textbf{font-weight:700}.katex .textit{font-style:italic}.katex .textrm{font-family:KaTeX_Main}.katex .textsf{font-family:KaTeX_SansSerif}.katex .texttt{font-family:KaTeX_Typewriter}.katex .mathnormal{font-family:KaTeX_Math;font-style:italic}.katex .mathit{font-family:KaTeX_Main;font-style:italic}.katex .mathrm{font-style:normal}.katex .mathbf{font-family:KaTeX_Main;font-weight:700}.katex .boldsymbol{font-family:KaTeX_Math;font-style:italic;font-weight:700}.katex .amsrm,.katex .mathbb,.katex .textbb{font-family:KaTeX_AMS}.katex .mathcal{font-family:KaTeX_Caligraphic}.katex .mathfrak,.katex .textfrak{font-family:KaTeX_Fraktur}.katex .mathboldfrak,.katex .textboldfrak{font-family:KaTeX_Fraktur;font-weight:700}.katex .mathtt{font-family:KaTeX_Typewriter}.katex .mathscr,.katex .textscr{font-family:KaTeX_Script}.katex .mathsf,.katex .textsf{font-family:KaTeX_SansSerif}.katex .mathboldsf,.katex .textboldsf{font-family:KaTeX_SansSerif;font-weight:700}.katex .mathitsf,.katex .mathsfit,.katex .textitsf{font-family:KaTeX_SansSerif;font-style:italic}.katex .mainrm{font-family:KaTeX_Main;font-style:normal}.katex .vlist-t{border-collapse:collapse;display:inline-table;table-layout:fixed}.katex .vlist-r{display:table-row}.katex .vlist{display:table-cell;position:relative;vertical-align:bottom}.katex .vlist>span{display:block;height:0;position:relative}.katex .vlist>span>span{display:inline-block}.katex .vlist>span>.pstrut{overflow:hidden;width:0}.katex .vlist-t2{margin-right:-2px}.katex .vlist-s{display:table-cell;font-size:1px;min-width:2px;vertical-align:bottom;width:2px}.katex .katex-vbox{align-items:baseline;display:inline-flex;flex-direction:column}.katex .katex-thinbox{display:inline-flex;flex-direction:row;max-width:0;width:0}.katex .msupsub{text-align:left}.katex .mfrac>span>span{text-align:center}.katex .mfrac .frac-line{border-bottom-style:solid;display:inline-block;width:100%}.katex .katex-hdashline,.katex .katex-hline,.katex .katex-overline .overline-line,.katex .katex-rule,.katex .katex-underline .underline-line,.katex .mfrac .frac-line{min-height:1px}.katex .mspace{display:inline-block}.katex .katex-smash{display:inline;line-height:0}.katex .clap,.katex .llap,.katex .rlap{position:relative;width:0}.katex .clap>.katex-inner,.katex .llap>.katex-inner,.katex .rlap>.katex-inner{position:absolute}.katex .clap>.katex-fix,.katex .llap>.katex-fix,.katex .rlap>.katex-fix{display:inline-block}.katex .llap>.katex-inner{right:0}.katex .clap>.katex-inner,.katex .rlap>.katex-inner{left:0}.katex .clap>.katex-inner>span{margin-left:-50%;margin-right:50%}.katex .katex-rule{border:0 solid;display:inline-block;position:relative}.katex .katex-hline,.katex .katex-overline .overline-line,.katex .katex-underline .underline-line{border-bottom-style:solid;display:inline-block;width:100%}.katex .katex-hdashline{border-bottom-style:dashed;display:inline-block;width:100%}.katex .sqrt>.katex-root{margin-left:.2777777778em;margin-right:-.5555555556em}.katex .fontsize-ensurer.reset-size1.size1,.katex .katex-sizing.reset-size1.size1{font-size:1em}.katex .fontsize-ensurer.reset-size1.size2,.katex .katex-sizing.reset-size1.size2{font-size:1.2em}.katex .fontsize-ensurer.reset-size1.size3,.katex .katex-sizing.reset-size1.size3{font-size:1.4em}.katex .fontsize-ensurer.reset-size1.size4,.katex .katex-sizing.reset-size1.size4{font-size:1.6em}.katex .fontsize-ensurer.reset-size1.size5,.katex .katex-sizing.reset-size1.size5{font-size:1.8em}.katex .fontsize-ensurer.reset-size1.size6,.katex .katex-sizing.reset-size1.size6{font-size:2em}.katex .fontsize-ensurer.reset-size1.size7,.katex .katex-sizing.reset-size1.size7{font-size:2.4em}.katex .fontsize-ensurer.reset-size1.size8,.katex .katex-sizing.reset-size1.size8{font-size:2.88em}.katex .fontsize-ensurer.reset-size1.size9,.katex .katex-sizing.reset-size1.size9{font-size:3.456em}.katex .fontsize-ensurer.reset-size1.size10,.katex .katex-sizing.reset-size1.size10{font-size:4.148em}.katex .fontsize-ensurer.reset-size1.size11,.katex .katex-sizing.reset-size1.size11{font-size:4.976em}.katex .fontsize-ensurer.reset-size2.size1,.katex .katex-sizing.reset-size2.size1{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size2.size2,.katex .katex-sizing.reset-size2.size2{font-size:1em}.katex .fontsize-ensurer.reset-size2.size3,.katex .katex-sizing.reset-size2.size3{font-size:1.1666666667em}.katex .fontsize-ensurer.reset-size2.size4,.katex .katex-sizing.reset-size2.size4{font-size:1.3333333333em}.katex .fontsize-ensurer.reset-size2.size5,.katex .katex-sizing.reset-size2.size5{font-size:1.5em}.katex .fontsize-ensurer.reset-size2.size6,.katex .katex-sizing.reset-size2.size6{font-size:1.6666666667em}.katex .fontsize-ensurer.reset-size2.size7,.katex .katex-sizing.reset-size2.size7{font-size:2em}.katex .fontsize-ensurer.reset-size2.size8,.katex .katex-sizing.reset-size2.size8{font-size:2.4em}.katex .fontsize-ensurer.reset-size2.size9,.katex .katex-sizing.reset-size2.size9{font-size:2.88em}.katex .fontsize-ensurer.reset-size2.size10,.katex .katex-sizing.reset-size2.size10{font-size:3.4566666667em}.katex .fontsize-ensurer.reset-size2.size11,.katex .katex-sizing.reset-size2.size11{font-size:4.1466666667em}.katex .fontsize-ensurer.reset-size3.size1,.katex .katex-sizing.reset-size3.size1{font-size:.7142857143em}.katex .fontsize-ensurer.reset-size3.size2,.katex .katex-sizing.reset-size3.size2{font-size:.8571428571em}.katex .fontsize-ensurer.reset-size3.size3,.katex .katex-sizing.reset-size3.size3{font-size:1em}.katex .fontsize-ensurer.reset-size3.size4,.katex .katex-sizing.reset-size3.size4{font-size:1.1428571429em}.katex .fontsize-ensurer.reset-size3.size5,.katex .katex-sizing.reset-size3.size5{font-size:1.2857142857em}.katex .fontsize-ensurer.reset-size3.size6,.katex .katex-sizing.reset-size3.size6{font-size:1.4285714286em}.katex .fontsize-ensurer.reset-size3.size7,.katex .katex-sizing.reset-size3.size7{font-size:1.7142857143em}.katex .fontsize-ensurer.reset-size3.size8,.katex .katex-sizing.reset-size3.size8{font-size:2.0571428571em}.katex .fontsize-ensurer.reset-size3.size9,.katex .katex-sizing.reset-size3.size9{font-size:2.4685714286em}.katex .fontsize-ensurer.reset-size3.size10,.katex .katex-sizing.reset-size3.size10{font-size:2.9628571429em}.katex .fontsize-ensurer.reset-size3.size11,.katex .katex-sizing.reset-size3.size11{font-size:3.5542857143em}.katex .fontsize-ensurer.reset-size4.size1,.katex .katex-sizing.reset-size4.size1{font-size:.625em}.katex .fontsize-ensurer.reset-size4.size2,.katex .katex-sizing.reset-size4.size2{font-size:.75em}.katex .fontsize-ensurer.reset-size4.size3,.katex .katex-sizing.reset-size4.size3{font-size:.875em}.katex .fontsize-ensurer.reset-size4.size4,.katex .katex-sizing.reset-size4.size4{font-size:1em}.katex .fontsize-ensurer.reset-size4.size5,.katex .katex-sizing.reset-size4.size5{font-size:1.125em}.katex .fontsize-ensurer.reset-size4.size6,.katex .katex-sizing.reset-size4.size6{font-size:1.25em}.katex .fontsize-ensurer.reset-size4.size7,.katex .katex-sizing.reset-size4.size7{font-size:1.5em}.katex .fontsize-ensurer.reset-size4.size8,.katex .katex-sizing.reset-size4.size8{font-size:1.8em}.katex .fontsize-ensurer.reset-size4.size9,.katex .katex-sizing.reset-size4.size9{font-size:2.16em}.katex .fontsize-ensurer.reset-size4.size10,.katex .katex-sizing.reset-size4.size10{font-size:2.5925em}.katex .fontsize-ensurer.reset-size4.size11,.katex .katex-sizing.reset-size4.size11{font-size:3.11em}.katex .fontsize-ensurer.reset-size5.size1,.katex .katex-sizing.reset-size5.size1{font-size:.5555555556em}.katex .fontsize-ensurer.reset-size5.size2,.katex .katex-sizing.reset-size5.size2{font-size:.6666666667em}.katex .fontsize-ensurer.reset-size5.size3,.katex .katex-sizing.reset-size5.size3{font-size:.7777777778em}.katex .fontsize-ensurer.reset-size5.size4,.katex .katex-sizing.reset-size5.size4{font-size:.8888888889em}.katex .fontsize-ensurer.reset-size5.size5,.katex .katex-sizing.reset-size5.size5{font-size:1em}.katex .fontsize-ensurer.reset-size5.size6,.katex .katex-sizing.reset-size5.size6{font-size:1.1111111111em}.katex .fontsize-ensurer.reset-size5.size7,.katex .katex-sizing.reset-size5.size7{font-size:1.3333333333em}.katex .fontsize-ensurer.reset-size5.size8,.katex .katex-sizing.reset-size5.size8{font-size:1.6em}.katex .fontsize-ensurer.reset-size5.size9,.katex .katex-sizing.reset-size5.size9{font-size:1.92em}.katex .fontsize-ensurer.reset-size5.size10,.katex .katex-sizing.reset-size5.size10{font-size:2.3044444444em}.katex .fontsize-ensurer.reset-size5.size11,.katex .katex-sizing.reset-size5.size11{font-size:2.7644444444em}.katex .fontsize-ensurer.reset-size6.size1,.katex .katex-sizing.reset-size6.size1{font-size:.5em}.katex .fontsize-ensurer.reset-size6.size2,.katex .katex-sizing.reset-size6.size2{font-size:.6em}.katex .fontsize-ensurer.reset-size6.size3,.katex .katex-sizing.reset-size6.size3{font-size:.7em}.katex .fontsize-ensurer.reset-size6.size4,.katex .katex-sizing.reset-size6.size4{font-size:.8em}.katex .fontsize-ensurer.reset-size6.size5,.katex .katex-sizing.reset-size6.size5{font-size:.9em}.katex .fontsize-ensurer.reset-size6.size6,.katex .katex-sizing.reset-size6.size6{font-size:1em}.katex .fontsize-ensurer.reset-size6.size7,.katex .katex-sizing.reset-size6.size7{font-size:1.2em}.katex .fontsize-ensurer.reset-size6.size8,.katex .katex-sizing.reset-size6.size8{font-size:1.44em}.katex .fontsize-ensurer.reset-size6.size9,.katex .katex-sizing.reset-size6.size9{font-size:1.728em}.katex .fontsize-ensurer.reset-size6.size10,.katex .katex-sizing.reset-size6.size10{font-size:2.074em}.katex .fontsize-ensurer.reset-size6.size11,.katex .katex-sizing.reset-size6.size11{font-size:2.488em}.katex .fontsize-ensurer.reset-size7.size1,.katex .katex-sizing.reset-size7.size1{font-size:.4166666667em}.katex .fontsize-ensurer.reset-size7.size2,.katex .katex-sizing.reset-size7.size2{font-size:.5em}.katex .fontsize-ensurer.reset-size7.size3,.katex .katex-sizing.reset-size7.size3{font-size:.5833333333em}.katex .fontsize-ensurer.reset-size7.size4,.katex .katex-sizing.reset-size7.size4{font-size:.6666666667em}.katex .fontsize-ensurer.reset-size7.size5,.katex .katex-sizing.reset-size7.size5{font-size:.75em}.katex .fontsize-ensurer.reset-size7.size6,.katex .katex-sizing.reset-size7.size6{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size7.size7,.katex .katex-sizing.reset-size7.size7{font-size:1em}.katex .fontsize-ensurer.reset-size7.size8,.katex .katex-sizing.reset-size7.size8{font-size:1.2em}.katex .fontsize-ensurer.reset-size7.size9,.katex .katex-sizing.reset-size7.size9{font-size:1.44em}.katex .fontsize-ensurer.reset-size7.size10,.katex .katex-sizing.reset-size7.size10{font-size:1.7283333333em}.katex .fontsize-ensurer.reset-size7.size11,.katex .katex-sizing.reset-size7.size11{font-size:2.0733333333em}.katex .fontsize-ensurer.reset-size8.size1,.katex .katex-sizing.reset-size8.size1{font-size:.3472222222em}.katex .fontsize-ensurer.reset-size8.size2,.katex .katex-sizing.reset-size8.size2{font-size:.4166666667em}.katex .fontsize-ensurer.reset-size8.size3,.katex .katex-sizing.reset-size8.size3{font-size:.4861111111em}.katex .fontsize-ensurer.reset-size8.size4,.katex .katex-sizing.reset-size8.size4{font-size:.5555555556em}.katex .fontsize-ensurer.reset-size8.size5,.katex .katex-sizing.reset-size8.size5{font-size:.625em}.katex .fontsize-ensurer.reset-size8.size6,.katex .katex-sizing.reset-size8.size6{font-size:.6944444444em}.katex .fontsize-ensurer.reset-size8.size7,.katex .katex-sizing.reset-size8.size7{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size8.size8,.katex .katex-sizing.reset-size8.size8{font-size:1em}.katex .fontsize-ensurer.reset-size8.size9,.katex .katex-sizing.reset-size8.size9{font-size:1.2em}.katex .fontsize-ensurer.reset-size8.size10,.katex .katex-sizing.reset-size8.size10{font-size:1.4402777778em}.katex .fontsize-ensurer.reset-size8.size11,.katex .katex-sizing.reset-size8.size11{font-size:1.7277777778em}.katex .fontsize-ensurer.reset-size9.size1,.katex .katex-sizing.reset-size9.size1{font-size:.2893518519em}.katex .fontsize-ensurer.reset-size9.size2,.katex .katex-sizing.reset-size9.size2{font-size:.3472222222em}.katex .fontsize-ensurer.reset-size9.size3,.katex .katex-sizing.reset-size9.size3{font-size:.4050925926em}.katex .fontsize-ensurer.reset-size9.size4,.katex .katex-sizing.reset-size9.size4{font-size:.462962963em}.katex .fontsize-ensurer.reset-size9.size5,.katex .katex-sizing.reset-size9.size5{font-size:.5208333333em}.katex .fontsize-ensurer.reset-size9.size6,.katex .katex-sizing.reset-size9.size6{font-size:.5787037037em}.katex .fontsize-ensurer.reset-size9.size7,.katex .katex-sizing.reset-size9.size7{font-size:.6944444444em}.katex .fontsize-ensurer.reset-size9.size8,.katex .katex-sizing.reset-size9.size8{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size9.size9,.katex .katex-sizing.reset-size9.size9{font-size:1em}.katex .fontsize-ensurer.reset-size9.size10,.katex .katex-sizing.reset-size9.size10{font-size:1.2002314815em}.katex .fontsize-ensurer.reset-size9.size11,.katex .katex-sizing.reset-size9.size11{font-size:1.4398148148em}.katex .fontsize-ensurer.reset-size10.size1,.katex .katex-sizing.reset-size10.size1{font-size:.2410800386em}.katex .fontsize-ensurer.reset-size10.size2,.katex .katex-sizing.reset-size10.size2{font-size:.2892960463em}.katex .fontsize-ensurer.reset-size10.size3,.katex .katex-sizing.reset-size10.size3{font-size:.337512054em}.katex .fontsize-ensurer.reset-size10.size4,.katex .katex-sizing.reset-size10.size4{font-size:.3857280617em}.katex .fontsize-ensurer.reset-size10.size5,.katex .katex-sizing.reset-size10.size5{font-size:.4339440694em}.katex .fontsize-ensurer.reset-size10.size6,.katex .katex-sizing.reset-size10.size6{font-size:.4821600771em}.katex .fontsize-ensurer.reset-size10.size7,.katex .katex-sizing.reset-size10.size7{font-size:.5785920926em}.katex .fontsize-ensurer.reset-size10.size8,.katex .katex-sizing.reset-size10.size8{font-size:.6943105111em}.katex .fontsize-ensurer.reset-size10.size9,.katex .katex-sizing.reset-size10.size9{font-size:.8331726133em}.katex .fontsize-ensurer.reset-size10.size10,.katex .katex-sizing.reset-size10.size10{font-size:1em}.katex .fontsize-ensurer.reset-size10.size11,.katex .katex-sizing.reset-size10.size11{font-size:1.1996142719em}.katex .fontsize-ensurer.reset-size11.size1,.katex .katex-sizing.reset-size11.size1{font-size:.2009646302em}.katex .fontsize-ensurer.reset-size11.size2,.katex .katex-sizing.reset-size11.size2{font-size:.2411575563em}.katex .fontsize-ensurer.reset-size11.size3,.katex .katex-sizing.reset-size11.size3{font-size:.2813504823em}.katex .fontsize-ensurer.reset-size11.size4,.katex .katex-sizing.reset-size11.size4{font-size:.3215434084em}.katex .fontsize-ensurer.reset-size11.size5,.katex .katex-sizing.reset-size11.size5{font-size:.3617363344em}.katex .fontsize-ensurer.reset-size11.size6,.katex .katex-sizing.reset-size11.size6{font-size:.4019292605em}.katex .fontsize-ensurer.reset-size11.size7,.katex .katex-sizing.reset-size11.size7{font-size:.4823151125em}.katex .fontsize-ensurer.reset-size11.size8,.katex .katex-sizing.reset-size11.size8{font-size:.578778135em}.katex .fontsize-ensurer.reset-size11.size9,.katex .katex-sizing.reset-size11.size9{font-size:.6945337621em}.katex .fontsize-ensurer.reset-size11.size10,.katex .katex-sizing.reset-size11.size10{font-size:.8336012862em}.katex .fontsize-ensurer.reset-size11.size11,.katex .katex-sizing.reset-size11.size11{font-size:1em}.katex .delimsizing.size1{font-family:KaTeX_Size1}.katex .delimsizing.size2{font-family:KaTeX_Size2}.katex .delimsizing.size3{font-family:KaTeX_Size3}.katex .delimsizing.size4{font-family:KaTeX_Size4}.katex .delimsizing.mult .delim-size1>span{font-family:KaTeX_Size1}.katex .delimsizing.mult .delim-size4>span{font-family:KaTeX_Size4}.katex .nulldelimiter{display:inline-block;width:.12em}.katex .delimcenter,.katex .op-symbol{position:relative}.katex .op-symbol.small-op{font-family:KaTeX_Size1}.katex .op-symbol.large-op{font-family:KaTeX_Size2}.katex .katex-accent>.vlist-t,.katex .op-limits>.vlist-t{text-align:center}.katex .katex-accent .accent-body{position:relative}.katex .katex-accent .accent-body:not(.accent-full){width:0}.katex .katex-overlay{display:block}.katex .mtable .vertical-separator{display:inline-block;min-width:1px}.katex .mtable .arraycolsep{display:inline-block}.katex .mtable .col-align-c>.vlist-t{text-align:center}.katex .mtable .col-align-l>.vlist-t{text-align:left}.katex .mtable .col-align-r>.vlist-t{text-align:right}.katex .svg-align{text-align:left}.katex svg{fill:currentColor;stroke:currentColor;display:block;height:inherit;position:absolute;width:100%}.katex svg path{stroke:none}.katex svg{fill-rule:nonzero;fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1}.katex img{border-style:none;max-height:none;max-width:none;min-height:0;min-width:0}.katex .katex-stretchy{display:block;overflow:hidden;position:relative;width:100%}.katex .katex-stretchy:after,.katex .katex-stretchy:before{content:""}.katex .hide-tail{overflow:hidden;position:relative;width:100%}.katex .halfarrow-left{left:0;overflow:hidden;position:absolute;width:50.2%}.katex .halfarrow-right{overflow:hidden;position:absolute;right:0;width:50.2%}.katex .brace-left{left:0;overflow:hidden;position:absolute;width:25.1%}.katex .brace-center{left:25%;overflow:hidden;position:absolute;width:50%}.katex .brace-right{overflow:hidden;position:absolute;right:0;width:25.1%}.katex .x-arrow-pad{padding:0 .5em}.katex .cd-arrow-pad{padding:0 .55556em 0 .27778em}.katex .mover,.katex .munder,.katex .x-arrow{text-align:center}.katex .boxpad{padding:0 .3em}.katex .fbox,.katex .fcolorbox{border:.04em solid;box-sizing:border-box}.katex .cancel-pad{padding:0 .2em}.katex .cancel-lap{margin-left:-.2em;margin-right:-.2em}.katex .katex-sout{border-bottom-style:solid;border-bottom-width:.08em}.katex .angl{border-right:.049em solid;border-top:.049em solid;box-sizing:border-box;margin-right:.03889em}.katex .anglpad{padding:0 .03889em}.katex .reflectbox{display:inline-block;transform:scaleX(-1)}.katex .eqn-num:before{content:"(" counter(katexEqnNo) ")";counter-increment:katexEqnNo}.katex .mml-eqn-num:before{content:"(" counter(mmlEqnNo) ")";counter-increment:mmlEqnNo}.katex .mtr-glue{width:50%}.katex .cd-vert-arrow{display:inline-block;position:relative}.katex .cd-label-left{display:inline-block;position:absolute;right:calc(50% + .3em);text-align:left}.katex .cd-label-right{display:inline-block;left:calc(50% + .3em);position:absolute;text-align:right}.katex-display{display:block;margin:1em 0;text-align:center}.katex-display>.katex{display:block;text-align:center;white-space:nowrap}.katex-display>.katex>.katex-html{display:block;position:relative}.katex-display>.katex>.katex-html>.katex-tag{position:absolute;right:0}.katex-display.leqno>.katex>.katex-html>.katex-tag{left:0;right:auto}.katex-display.fleqn>.katex{padding-left:2em;text-align:left}body{counter-reset:katexEqnNo mmlEqnNo} | |
| </style> | |
| <script>(function() { | |
| function __mumInit() { | |
| // Restore external links that htmlpreview.github.io's loader rewrote to | |
| // in-page anchors (it treats any href containing '#' as a local anchor). | |
| // Runs now (htmlpreview re-executes this script after its rewrite), again | |
| // after delays, and on click as a last line of defense. | |
| function restoreOrigHrefs() { | |
| document.querySelectorAll('a[data-orig-href]').forEach(function(a) { | |
| var orig = a.getAttribute('data-orig-href'); | |
| if (orig && a.getAttribute('href') !== orig) a.setAttribute('href', orig); | |
| }); | |
| } | |
| restoreOrigHrefs(); | |
| setTimeout(restoreOrigHrefs, 100); | |
| setTimeout(restoreOrigHrefs, 1000); | |
| document.addEventListener('click', function(e) { | |
| var t = e.target && e.target.closest ? e.target.closest('a[data-orig-href]') : null; | |
| if (t) { var o = t.getAttribute('data-orig-href'); if (o && t.getAttribute('href') !== o) t.setAttribute('href', o); } | |
| }, true); | |
| var vt = document.getElementById('viewing-time'); | |
| if (vt) { | |
| function pad(n) { return n < 10 ? '0' + n : n; } | |
| function updateTime() { | |
| var d = new Date(); | |
| vt.textContent = d.getUTCFullYear() + '-' + pad(d.getUTCMonth()+1) + '-' + pad(d.getUTCDate()) + ' ' + pad(d.getUTCHours()) + ':' + pad(d.getUTCMinutes()) + ' UTC'; | |
| } | |
| updateTime(); | |
| setInterval(updateTime, 60000); | |
| } | |
| document.querySelectorAll('a[data-hover-title]').forEach(function(a) { | |
| var card = document.createElement('span'); | |
| card.className = 'hover-card'; | |
| var title = a.getAttribute('data-hover-title') || ''; | |
| var status = a.getAttribute('data-hover-status') || ''; | |
| var owner = a.getAttribute('data-hover-owner') || ''; | |
| var desc = a.getAttribute('data-hover-desc') || ''; | |
| var statusCls = 'hc-status hc-status-' + status.toLowerCase().replace(/[^a-z]/g, ''); | |
| var html = '<div class="hc-title">' + title + '</div>'; | |
| var meta = []; | |
| if (status) meta.push('<span class="' + statusCls + '">' + status + '</span>'); | |
| if (owner) meta.push(owner); | |
| if (meta.length) html += '<div class="hc-meta">' + meta.join(' ') + '</div>'; | |
| if (desc) html += '<div class="hc-desc">' + desc + '</div>'; | |
| card.innerHTML = html; | |
| a.appendChild(card); | |
| }); | |
| document.querySelectorAll('a.cite').forEach(function(a) { | |
| var href = a.getAttribute('href') || ''; | |
| if (!href.startsWith('#ref-')) return; | |
| var li = document.getElementById(href.slice(1)); | |
| if (!li) return; | |
| var body = li.querySelector('.ref-body'); | |
| if (!body) return; | |
| var refUrl = body.querySelector('.ref-url a'); | |
| var refHref = refUrl ? refUrl.getAttribute('href') : ''; | |
| var hoverSource = refHref ? document.querySelector('a[data-hover-title][href="' + refHref + '"]') : null; | |
| if (hoverSource) { | |
| var card = document.createElement('span'); | |
| card.className = 'hover-card'; | |
| var t = hoverSource.getAttribute('data-hover-title') || ''; | |
| var s = hoverSource.getAttribute('data-hover-status') || ''; | |
| var o = hoverSource.getAttribute('data-hover-owner') || ''; | |
| var d = hoverSource.getAttribute('data-hover-desc') || ''; | |
| var sc = 'hc-status hc-status-' + s.toLowerCase().replace(/[^a-z]/g, ''); | |
| var h = '<div class="hc-title">' + t + '</div>'; | |
| var m = []; | |
| if (s) m.push('<span class="' + sc + '">' + s + '</span>'); | |
| if (o) m.push(o); | |
| if (m.length) h += '<div class="hc-meta">' + m.join(' ') + '</div>'; | |
| if (d) h += '<div class="hc-desc">' + d + '</div>'; | |
| card.innerHTML = h; | |
| a.appendChild(card); | |
| } else { | |
| var title = body.querySelector('.ref-title'); | |
| if (!title) return; | |
| var card = document.createElement('span'); | |
| card.className = 'cite-card'; | |
| card.textContent = title.textContent; | |
| a.appendChild(card); | |
| } | |
| }); | |
| } | |
| if (document.readyState !== 'loading') { __mumInit(); } | |
| else { document.addEventListener('DOMContentLoaded', __mumInit); } | |
| })();</script> | |
| </head> | |
| <body> | |
| <div class="page"> | |
| <header class="paper-header"> | |
| <h1>Expected MFU for the 535B-A23B hero on a Megatron stack instead of Levanter</h1> | |
| <div class="paper-meta">Posed by <span class="author">hammer</span>, answered by <a class="mumwelt-link" href="https://github.com/marin-community/mumwelt">mumwelt</a> · <span class="date-label" title="Generated 2026-09-23 13:24 UTC">Published 2026-09-23</span></div> | |
| </header> | |
| <div class="prompt-box"> | |
| <div class="prompt-label">?</div> | |
| <div class="prompt-text">Given the 535B hero-run architecture we are training on CoreWeave, what MFU should we expect from a PyTorch/Megatron stack in place of Levanter, informed by external results on similar hardware?</div> | |
| </div> | |
| <div class="content"> | |
| <h2 id="short-answer">Short answer</h2> | |
| <p>A Megatron-Core stack running the hero's actual recipe (learned QB | |
| router with capacity and spill, MuonH, the full Grug operator set) is | |
| expected to train <strong>slower</strong> than Levanter does today: | |
| median about 20% MFU against Levanter's 24.0% in production, with | |
| roughly a 1-in-20 chance of beating it. Replacing MuonH with Adam makes | |
| it a coin flip, median about 23.5%. The 31 to 32% figures that NVIDIA | |
| and Marin's own calibration report for Megatron are systems ceilings | |
| measured with a force-balanced, dropless router, and no published BF16 | |
| number above 30% on GB200 was obtained any other way.</p> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th>Scenario for a Megatron port of the hero (BF16, 704 GB200, seq | |
| 4096)</th> | |
| <th>p10</th> | |
| <th>median</th> | |
| <th>p90</th> | |
| <th>P(beats 24%)</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td>Systems ceiling: force-balanced router, Adam</td> | |
| <td>29.8%</td> | |
| <td>31.6%</td> | |
| <td>33.5%</td> | |
| <td>1.00</td> | |
| </tr> | |
| <tr> | |
| <td>Production semantics with Adam</td> | |
| <td>20.0%</td> | |
| <td>23.5%</td> | |
| <td>26.8%</td> | |
| <td>0.43</td> | |
| </tr> | |
| <tr> | |
| <td>Production semantics with MuonH (the actual recipe)</td> | |
| <td>16.6%</td> | |
| <td>19.8%</td> | |
| <td>23.2%</td> | |
| <td>0.05</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <p>At the hero's 149 GFLOP per token on 704 GPUs, those levels mean 18T | |
| tokens in about 84 days at 20%, 74 days at 24%, and 57 days at 31%. The | |
| estimate rests on NVIDIA's published Megatron-Core numbers for the two | |
| closest open MoE shapes, on Marin's own July calibration of Megatron | |
| against JAX on one GB200 rack, and on DeepSeek-V3's production | |
| GPU-hours; the method and its uncertainty are laid out below.</p> | |
| <h2 id="what-levanter-delivers-today">What Levanter delivers today</h2> | |
| <p>The hero is a 535.3B-total, 22.76B-active Grug MoE: 48 layers, 384 | |
| routed experts of width 3072 with top-8 routing plus two shared experts, | |
| LatentMoE dispatch, sequence length 4096, trained with MuonH on 11 | |
| CoreWeave GB200 NVL72 racks (704 GPUs) with expert parallelism 64 inside | |
| each rack and data parallelism across racks (<a | |
| data-orig-href="https://github.com/marin-community/marin/issues/8435#issuecomment-5335872267" href="https://github.com/marin-community/marin/issues/8435#issuecomment-5335872267">job | |
| summary on #8435</a>). Its reported MFU divides a window-aware analytic | |
| model-FLOP count (<a | |
| data-orig-href="https://github.com/marin-community/marin/blob/cf3f4ffd1b8821a2216eac2082409306c2a29cb2/lib/levanter/src/levanter/utils/flop_utils.py#L31-L61" href="https://github.com/marin-community/marin/blob/cf3f4ffd1b8821a2216eac2082409306c2a29cb2/lib/levanter/src/levanter/utils/flop_utils.py#L31-L61">flop_utils.py</a>) | |
| by 2.5 PFLOP/s per GPU, the GB200 NVL72 dense BF16 peak (<a | |
| href="https://github.com/marin-community/marin/pull/7334">#7334</a>). | |
| Production went from about 21% at launch to 23.3% after the ragged | |
| expert-parallel relaunch and 24.0% after the PDL-off kernels (<a | |
| href="https://mws.oa.dev/summaries/summary-2026-09-07_2026-09-13.html">week | |
| of Sep 7</a>, <a | |
| href="https://mws.oa.dev/summaries/summary-2026-09-14_2026-09-20.html">week | |
| of Sep 14</a>); a one-rack diagnostic branch reads 27.64% with symmetric | |
| buffers and mask removal (<a | |
| href="https://github.com/marin-community/marin/pull/9315">#9315</a>). | |
| August profiles put GEMMs at 57.7% of the compute stream and the exposed | |
| expert-parallel collective residue at 2.4 to 3.2 seconds of a 17-second | |
| step (<a | |
| data-orig-href="https://github.com/marin-community/marin/issues/8317#issuecomment-5305649465" href="https://github.com/marin-community/marin/issues/8317#issuecomment-5305649465">#8317</a>, | |
| <a | |
| data-orig-href="https://github.com/marin-community/marin/issues/8311#issuecomment-5305698574" href="https://github.com/marin-community/marin/issues/8311#issuecomment-5305698574">#8311</a>).</p> | |
| <h2 | |
| id="marins-own-calibration-megatron-against-jax-on-the-same-rack">Marin's | |
| own calibration: Megatron against JAX on the same rack</h2> | |
| <p>In late July Rafal Wojdyla's agent built a Megatron-Core plus | |
| Transformer Engine image and ran a d5120, 48-layer, 256-expert top-8 MoE | |
| (about 250B total, 16B active) at sequence 4096 and global batch 1024 on | |
| one 64-GPU GB200 rack, the same 65,536 tokens per GPU per step the hero | |
| uses, scoring it on the 2.5 PFLOP/s denominator (<a | |
| href="https://github.com/marin-community/marin/issues/7668">#7668</a>). | |
| The ladder:</p> | |
| <ul> | |
| <li>Stock NCCL all-to-all dispatcher: 4.423% MFU, with all-to-all | |
| consuming 77.8% of the step; HybridEP replaced it.</li> | |
| <li>Force-balanced router, Adam, no selective recompute: 31.1 to 31.3% | |
| on the 8-of-256 shape and 32.4% on a promoted 4-of-128 shape, which the | |
| completion audit calls "32.407% Levanter-compatible common MFU" (<a | |
| data-orig-href="https://github.com/marin-community/marin/issues/7668#issuecomment-5104907558" href="https://github.com/marin-community/marin/issues/7668#issuecomment-5104907558">completion | |
| audit</a>).</li> | |
| <li>The full Grug operator set under the balanced router: 26.6%.</li> | |
| <li>Stock auxiliary-loss-free learned router: 14.7%, from | |
| expert-parallel rank overload. Tensor parallel 2 as a memory escape: | |
| 12.1%.</li> | |
| <li>A faithful port of Grug's QB routing with capacity 1.0625 and three | |
| spill attempts, 350 steps: "3.667 MFU percentage points below the | |
| corrected" JAX point (<a | |
| data-orig-href="https://github.com/marin-community/marin/issues/7668#issuecomment-5100527250" href="https://github.com/marin-community/marin/issues/7668#issuecomment-5100527250">#7668</a>), | |
| that point being Matt Wittmann's 20.708% at 1.44% drops on the same rack | |
| two days earlier (<a | |
| data-orig-href="https://github.com/marin-community/marin/issues/7279#issuecomment-5084892846" href="https://github.com/marin-community/marin/issues/7279#issuecomment-5084892846">#7279</a>); | |
| so about 17.0%.</li> | |
| <li>MuonH through Megatron's layer-wise distributed optimizer "does not | |
| recover EP64 throughput," and the close-out lists "a usable | |
| MuonH/HyperBall optimizer path" among the remaining framework gaps (<a | |
| data-orig-href="https://github.com/marin-community/marin/issues/7668#issuecomment-5100527250" href="https://github.com/marin-community/marin/issues/7668#issuecomment-5100527250">#7668</a>). | |
| Every Megatron figure above used Adam.</li> | |
| </ul> | |
| <p>The exercise was a calibration, not a candidate stack. Jeff's June | |
| synthesis on the derisking epic set the plan to "stay on the JAX/Grug | |
| (Levanter) stack and close the gap with native kernels, not migrate to | |
| Megatron/TE" (<a | |
| data-orig-href="https://github.com/marin-community/marin/issues/4302#issuecomment-4727421242" href="https://github.com/marin-community/marin/issues/4302#issuecomment-4727421242">#4302</a>), | |
| against David Hall's bar of "within ~10% of the best Megatron/TE BF16 | |
| B200 result on the same relevant shape" (<a | |
| data-orig-href="https://github.com/marin-community/marin/issues/4311#issuecomment-4464443239" href="https://github.com/marin-community/marin/issues/4311#issuecomment-4464443239">#4311</a>). | |
| Earlier microbenchmarks predate the kernel work: Megatron was 28% faster | |
| on one GH200 in May (<a | |
| data-orig-href="https://github.com/marin-community/marin/issues/4311#issuecomment-4414534370" href="https://github.com/marin-community/marin/issues/4311#issuecomment-4414534370">#4311</a>) | |
| and 3.8 times faster on an 8×B200 slice (<a | |
| href="https://github.com/marin-community/marin/issues/5815">#5815</a>), | |
| after which Grug closed to within the 10% band by June 4 (<a | |
| data-orig-href="https://github.com/marin-community/marin/issues/6139#issuecomment-4619246391" href="https://github.com/marin-community/marin/issues/6139#issuecomment-4619246391">#6139</a>).</p> | |
| <h2 id="external-evidence-converted-to-the-heros-denominator">External | |
| evidence, converted to the hero's denominator</h2> | |
| <p>NVIDIA reports "TFLOPS/GPU" as delivered model FLOPs per second, so | |
| dividing by 2.5 PFLOP/s gives MFU on Levanter's definition. Two | |
| conversions of my own are marked.</p> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th>Source</th> | |
| <th>Model, stack, hardware</th> | |
| <th>Precision</th> | |
| <th>Routing</th> | |
| <th>MFU</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td><a href="https://arxiv.org/abs/2603.07685">Megatron-Core MoE report, | |
| Table 11</a></td> | |
| <td>Qwen3-235B-A22B, Megatron-Core, 256 GB200, seq 4096, 5,100 | |
| tok/s/GPU</td> | |
| <td>BF16</td> | |
| <td>force-balanced, dropless</td> | |
| <td>30.0%</td> | |
| </tr> | |
| <tr> | |
| <td>same</td> | |
| <td>DeepSeek-V3 (37B active), 256 GB200, 3,298 tok/s/GPU</td> | |
| <td>BF16</td> | |
| <td>force-balanced</td> | |
| <td>34.3%</td> | |
| </tr> | |
| <tr> | |
| <td>same</td> | |
| <td>DeepSeek-V3, 256 GB200</td> | |
| <td>MXFP8</td> | |
| <td>force-balanced</td> | |
| <td>41.9%</td> | |
| </tr> | |
| <tr> | |
| <td>same</td> | |
| <td>Qwen3-235B-A22B, 256 H100</td> | |
| <td>BF16</td> | |
| <td>force-balanced</td> | |
| <td>32.4% of H100 peak</td> | |
| </tr> | |
| <tr> | |
| <td><a | |
| href="https://docs.nvidia.com/nemo/megatron-bridge/0.6.0/performance-summary-archive.html">Megatron | |
| Bridge 26.06 tables</a></td> | |
| <td>DeepSeek-V3, 256 GB200, TP1/PP4</td> | |
| <td>MXFP8</td> | |
| <td>force-balanced, dropless</td> | |
| <td>51.7%</td> | |
| </tr> | |
| <tr> | |
| <td>same</td> | |
| <td>Qwen3-235B, 256 GB200, PP8/EP32</td> | |
| <td>MXFP8</td> | |
| <td>force-balanced, dropless</td> | |
| <td>43.7%</td> | |
| </tr> | |
| <tr> | |
| <td><a | |
| href="https://developer.nvidia.com/blog/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72/">NVIDIA | |
| GB300 record post</a></td> | |
| <td>DeepSeek-V3, "early software version" GB200 NVL72, 256 GPUs, 606 | |
| TFLOPS/GPU</td> | |
| <td>unstated</td> | |
| <td>unstated</td> | |
| <td>24.2%</td> | |
| </tr> | |
| <tr> | |
| <td><a | |
| href="https://pytorch.org/blog/enabling-up-to-41-faster-pre-training-mxfp8-and-deepep-for-deepseek-v3-on-b200-with-torchtitan/">TorchTitan | |
| on Nebius</a></td> | |
| <td>DeepSeek-V3, PyTorch-native, 256 HGX B200, EP32 over InfiniBand, seq | |
| 8192, 859 tok/s/GPU</td> | |
| <td>BF16 + DeepEP</td> | |
| <td>learned</td> | |
| <td>11.8% (my conversion, with attention FLOPs)</td> | |
| </tr> | |
| <tr> | |
| <td><a href="https://arxiv.org/abs/2412.19437">DeepSeek-V3 | |
| report</a></td> | |
| <td>production run, DeepSeek's own stack, 2,048 H800, 2.664M GPU-hours | |
| for 14.8T tokens</td> | |
| <td>FP8</td> | |
| <td>learned, aux-loss-free</td> | |
| <td>34.6% of H800 BF16 peak on 6N alone, about 38% with attention (my | |
| derivation)</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <p>Four readings. First, NVIDIA's MoE benchmarks are force-balanced and | |
| dropless by their own statement on the Bridge performance page, so they | |
| are ceilings of the same kind as Marin's 31.3%, and Marin's two-day port | |
| landed within two points of NVIDIA's number for a comparable shape. | |
| Second, the closest published shape to the hero, Qwen3-235B-A22B at 22B | |
| active in BF16 on GB200, reads 30.0%. Third, DeepSeek-V3 is the only | |
| production number with a learned router at scale, and it lands in the | |
| same band as NVIDIA's balanced H100 benchmarks, so a mature stack can | |
| recover most of the routing discount. Fourth, a PyTorch-native stack | |
| without NVIDIA's dispatcher and kernels sits near 12%, which is what | |
| "PyTorch" means if it does not also mean Megatron-Core. The | |
| Megatron-Core report's DeepSeek-V3 recipe on GB200 is TP1/PP4/EP64 with | |
| HybridEP and CUDA Graphs (Table 17), and the report notes that on NVL72 | |
| <a href="https://arxiv.org/abs/2603.07685">"EP64 stays entirely within | |
| the NVLink domain"</a>, which is the hero's layout too. The Nemotron 3 | |
| reports state their GB200 parallelism (CP64, TP2, EP64 for Super) but no | |
| throughput (<a href="https://arxiv.org/abs/2604.12374">arXiv | |
| 2604.12374</a>); the SemiAnalysis GB200 NVL72 measurements are | |
| paywalled, and its free section shows dense Llama 405B near 54% BF16 MFU | |
| on H100 with NeMo (<a | |
| href="https://newsletter.semianalysis.com/p/h100-vs-gb200-nvl72-training-benchmarks">SemiAnalysis</a>), | |
| a reminder that MoE stacks run well below dense ones.</p> | |
| <h2 id="extrapolation">Extrapolation</h2> | |
| <p>The starting point is the BF16 anchors for Qwen3-235B and | |
| DeepSeek-V3, cross-checked against Marin's own 31.3% ceiling, drawn as a | |
| triangular distribution from 29% to 34.5% with mode 31%. Four | |
| adjustments follow, each a triangular distribution over the range the | |
| evidence supports:</p> | |
| <ul> | |
| <li><strong>Shape</strong>, 0.97 to 1.10 with mode 1.03. The hero holds | |
| 6 experts per GPU at width 3072 and dispatches in a 3072-dimensional | |
| latent, which favors GEMM efficiency and halves all-to-all volume | |
| relative to Qwen3's thinner experts; DeepSeek-V3's larger active size | |
| reading 4 points above Qwen3 sets the upper end.</li> | |
| <li><strong>Scale</strong>, 0.95 to 0.99. NVIDIA reports 98.5% per-GPU | |
| retention from 256 to 1,024 GPUs on GB300; Marin sees about a point from | |
| one rack to eleven.</li> | |
| <li><strong>Routing</strong>, 0.60 to 0.97 with mode 0.85. This is the | |
| move from a force-balanced benchmark router to a learned router with | |
| capacity 1.15 and spill. Marin's untuned port paid 0.54; DeepSeek's | |
| mature production stack pays close to nothing; Levanter's own ragged | |
| transport now drops 0.01% of assignments at negligible cost. A Marin | |
| Megatron deployment would sit between, and where it sits is the largest | |
| single uncertainty.</li> | |
| <li><strong>Operator fidelity</strong>, 0.85 to 0.99 with mode 0.92. The | |
| full Grug operator set cost 0.855 in July before any kernel work; NVIDIA | |
| now ships LatentMoE natively for Nemotron 3.</li> | |
| <li><strong>MuonH</strong>, 0.70 to 0.97 with mode 0.87, applied only in | |
| the recipe-faithful scenario.</li> | |
| </ul> | |
| <p>Two hundred thousand draws give the table in the short answer. The | |
| ceiling scenario multiplies base, shape and scale; the Adam scenario | |
| adds routing and operator fidelity; the MuonH scenario adds the | |
| optimizer factor.</p> | |
| <h2 id="what-would-move-the-estimate">What would move the estimate</h2> | |
| <ul> | |
| <li>The routing discount is engineering time, not hardware. Getting a | |
| Megatron learned-router path to DeepSeek-class balance would lift the | |
| Adam scenario toward its 27% upper tail. There is no internal precedent | |
| for that effort, and the JAX stack keeps moving in the meantime: the | |
| one-rack branch is already at 27.6%, with about 3% from coordinated | |
| garbage collection (<a | |
| href="https://github.com/marin-community/marin/pull/9266">#9266</a>) and | |
| 4% from the SM100 attention backward (<a | |
| href="https://github.com/marin-community/marin/pull/9228">#9228</a>) | |
| queued.</li> | |
| <li>The dispatcher is Megatron's one structural advantage. HybridEP cuts | |
| exclusive communication to about 6% of the step on NVL72, against 2.4 to | |
| 3.2 exposed seconds per 17-second step in Levanter, worth about 5 | |
| points. That argues for porting the kernel, not the framework.</li> | |
| <li>Precision changes the picture more than the framework does. NVIDIA's | |
| MXFP8 numbers run 1.2 to 1.5 times its BF16 numbers on the same | |
| hardware; the hero is BF16 throughout by choice.</li> | |
| </ul> | |
| <h2 id="caveats">Caveats</h2> | |
| <ul> | |
| <li>No external run exists at the hero's expert count, at its width, or | |
| at 704 GB200s in BF16; the shape adjustment is my inference from GEMM | |
| sizes.</li> | |
| <li>NVIDIA's 24.2% GB200 point, labelled an <a | |
| href="https://developer.nvidia.com/blog/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72/">"early | |
| software version"</a>, has no stated precision or routing.</li> | |
| <li>The Marin trainable-lane number is a two-month-old snapshot after | |
| two days of tuning; a rerun would gain, but by an unmeasured | |
| amount.</li> | |
| <li>Long context is a separate question. The only 65K-sequence hero | |
| numbers are Levanter pipeline-parallel variants on 192 H100s at 15.5% | |
| (<a | |
| href="https://github.com/marin-community/marin/issues/9277">#9277</a>), | |
| with no Megatron counterpart anywhere.</li> | |
| </ul> | |
| <p>Treat the medians as good to about plus or minus three points and the | |
| ordering of the three scenarios as robust.</p> | |
| </div> | |
| <footer class="provenance"> | |
| <p><em>Corpus: 2026-09-22 05:29 UTC · Generated: 2026-09-23 13:24 UTC · Viewing: <span id="viewing-time"></span></em></p> | |
| <blockquote> | |
| <p><em>Data: marinmirror — 221146 chunks, built 2026-09-22 05:29 UTC · | |
| summaries through 2026-09-14_2026-09-20 (not refreshed); web sources | |
| fetched 2026-09-23: arXiv 2603.07685 (PDF, Tables 11 and 17), NVIDIA | |
| Megatron Bridge performance summary and archive, NVIDIA GB300 record | |
| post, PyTorch TorchTitan B200 post, DeepSeek-V3 report (PDF), Nemotron 3 | |
| Ultra and Super reports (PDF), SemiAnalysis free section.</em></p> | |
| <p><em>Query: "given the hero run 535B model architecture that we are | |
| currently training on CoreWeave, what is the expected MFU we'd get if we | |
| used PyTorch and Megatron in place of Levanter, informed by external | |
| results on similar hardware?"</em></p> | |
| <p><em>Sub-queries (corpus): "535B-A23B hero architecture and config" · | |
| "hero MFU definition and history" · "Marin's own Megatron usage" · | |
| "framework decision record" · "external MFU references in the corpus" · | |
| "kernel stack and step-time breakdown" · "pipeline parallelism results" | |
| · "PyTorch trials in Marin" · "CoreWeave GB200 hardware" · "MoE dispatch | |
| and transport" · "MuonH overhead" · code lane. Sub-queries (web): | |
| Megatron-Core MoE report tables · Megatron Bridge performance pages · | |
| GB200/GB300 MoE record post · TorchTitan DeepSeek-V3 on B200 · | |
| DeepSeek-V3 GPU-hours · Nemotron 3 training throughput · MLPerf GB200 | |
| NVL72 · third-party GB200 MoE benchmarks.</em></p> | |
| </blockquote> | |
| </footer> | |
| </div> | |
| </body> | |
| </html> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment