Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Select an option

  • Save hammer/881fb086dc2d326c348e7eb16df76793 to your computer and use it in GitHub Desktop.

Select an option

Save hammer/881fb086dc2d326c348e7eb16df76793 to your computer and use it in GitHub Desktop.
Expected MFU for the 535B-A23B hero on Megatron instead of Levanter
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Expected MFU for the 535B-A23B hero on a Megatron stack instead of Levanter</title>
<style>
:root {
--ink: #1a1a2e; --ink-secondary: #555770; --ink-faint: #8a8a9a;
--accent: #8b2500; --accent-light: #c4530a;
--surface: #faf9f6; --surface-raised: #f0eeea; --surface-code: #f5f3ef;
--rule: #d4d0c8; --link: #8b2500; --link-hover: #c4530a; --ref-bg: #f7f5f1;
--content-width: 650px; --sidenote-width: 230px; --sidenote-gap: 30px;
}
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) {
--ink: #d8d5cf; --ink-secondary: #9e9bab; --ink-faint: #6e6b7b;
--accent: #d4764e; --accent-light: #e8956e;
--surface: #1a1a24; --surface-raised: #242430; --surface-code: #20202c;
--rule: #33333f; --link: #d4764e; --link-hover: #e8956e; --ref-bg: #1e1e2a;
}
}
:root[data-theme="dark"] {
--ink: #d8d5cf; --ink-secondary: #9e9bab; --ink-faint: #6e6b7b;
--accent: #d4764e; --accent-light: #e8956e;
--surface: #1a1a24; --surface-raised: #242430; --surface-code: #20202c;
--rule: #33333f; --link: #d4764e; --link-hover: #e8956e; --ref-bg: #1e1e2a;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: var(--surface); color: var(--ink);
font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif;
font-size: 16px; line-height: 1.7;
-webkit-font-smoothing: antialiased;
}
.page {
max-width: calc(var(--content-width) + var(--sidenote-width) + var(--sidenote-gap) + 80px);
margin: 0 auto; padding: 3rem 40px 4rem; position: relative;
}
@media (max-width: 1060px) {
.page { max-width: 100%; padding: 2rem 1.5rem 3rem; }
}
.content { max-width: var(--content-width); }
.paper-header { max-width: var(--content-width); margin-bottom: 2.5rem; padding-bottom: 2rem; }
.paper-header h1 {
font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif;
font-size: 2rem; font-weight: 600; line-height: 1.25; color: var(--ink);
text-wrap: balance; margin-bottom: 0.6rem; letter-spacing: -0.01em;
}
.paper-meta { font-size: 0.875rem; color: var(--ink-secondary); line-height: 1.5; }
.paper-meta .author { font-weight: 500; }
.paper-meta .mumwelt-link { color: var(--ink-secondary); text-decoration: none; border-bottom: 1px dotted var(--ink-faint); }
.paper-meta .mumwelt-link:hover { color: var(--accent); border-bottom-color: var(--accent); }
.prompt-box {
max-width: var(--content-width); margin-bottom: 2.5rem; position: relative;
}
.prompt-label {
position: absolute; left: -0.6em; top: 50%; transform: translateY(-50%);
font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif;
font-size: 5rem; font-weight: 700; color: var(--ink); opacity: 0.08;
line-height: 1; pointer-events: none; user-select: none;
}
.prompt-text {
font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif;
font-style: italic; font-size: 1.15rem; line-height: 1.55; color: var(--ink);
position: relative;
}
.abstract { margin-bottom: 2.5rem; max-width: var(--content-width); }
.abstract-label {
font-size: 0.7rem; font-weight: 600; text-transform: uppercase;
letter-spacing: 0.1em; color: var(--ink-secondary); margin-bottom: 0.5rem;
}
.abstract p { font-size: 0.92rem; line-height: 1.75; color: var(--ink); }
h1, h2, h3 {
font-family: 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif;
font-weight: 600; color: var(--ink); text-wrap: balance;
}
h2 { font-size: 1.4rem; margin-top: 2.5rem; margin-bottom: 0.75rem; letter-spacing: -0.005em; }
h3 { font-size: 1.1rem; margin-top: 1.75rem; margin-bottom: 0.5rem; }
p { margin-bottom: 1rem; max-width: var(--content-width); }
a { color: var(--link); text-decoration: none; border-bottom: 1px solid transparent;
transition: border-color 0.15s, color 0.15s; }
a:hover { color: var(--link-hover); border-bottom-color: var(--link-hover); }
.date-label { cursor: default; border-bottom: 1px dotted var(--ink-faint); }
strong { font-weight: 600; }
.sidenote-checkbox { display: none; }
.sidenote-toggle { display: none; }
.sidenote {
float: right; clear: right; width: var(--sidenote-width);
margin-right: calc(-1 * (var(--sidenote-width) + var(--sidenote-gap)));
margin-top: 0.2rem; margin-bottom: 1rem;
font-size: 0.8rem; line-height: 1.5; color: var(--ink-secondary);
}
.sidenote-number { font-size: 0.7rem; font-weight: 600; color: var(--accent); margin-right: 0.3em; }
@media (max-width: 1060px) {
.sidenote-toggle {
display: inline; cursor: pointer; color: var(--accent);
font-size: 0.78rem; font-weight: 600; user-select: none;
}
.sidenote {
float: none; display: none; width: 100%; margin: 0.4rem 0 0.75rem 0;
font-size: 0.84rem; padding: 0.6rem 0.9rem; background: var(--surface-raised);
border-radius: 4px; border-left: 2px solid var(--accent);
}
.sidenote-checkbox:checked + .sidenote { display: block; }
.sidenote-number { display: none; }
}
blockquote { border-left: 2px solid var(--rule); padding-left: 1.25rem;
margin: 1.25rem 0; color: var(--ink-secondary); font-style: italic; }
code { font-family: 'SF Mono', Menlo, Consolas, monospace; font-size: 0.85em;
background: var(--surface-code); padding: 0.15em 0.35em; border-radius: 3px; }
pre { background: var(--surface-code); border: 1px solid var(--rule); border-radius: 4px;
padding: 1rem 1.25rem; overflow-x: auto; margin: 1.25rem 0; max-width: var(--content-width); }
pre code { background: none; padding: 0; font-size: 0.82rem; line-height: 1.6; }
ul, ol { margin-bottom: 1rem; padding-left: 1.5rem; max-width: var(--content-width); }
li { margin-bottom: 0.35rem; }
li::marker { color: var(--ink-faint); }
table { max-width: var(--content-width); border-collapse: collapse; width: 100%;
margin: 1.25rem 0; font-variant-numeric: tabular-nums; font-size: 0.9rem; }
thead { border-top: 2px solid var(--ink); border-bottom: 1px solid var(--ink); }
th { font-weight: 600; padding: 0.3rem 0.75rem 0.35rem; text-align: left;
line-height: 1.2; white-space: nowrap; font-size: 0.82rem; vertical-align: bottom; }
td { padding: 0.35rem 0.75rem; border: none; vertical-align: top; }
tbody { border-bottom: 1.5px solid var(--ink); }
th:first-child, td:first-child { padding-left: 0; }
th:last-child, td:last-child { padding-right: 0; }
figure { margin: 2rem 0; max-width: var(--content-width); }
figure img { width: 100%; border-radius: 3px; border: 1px solid var(--rule); }
figcaption { font-size: 0.8rem; color: var(--ink-secondary); margin-top: 0.5rem;
line-height: 1.5; font-style: italic; }
footer.provenance {
margin-top: 3rem; padding-top: 1rem; max-width: var(--content-width);
font-size: 0.78rem; line-height: 1.6; color: var(--ink-faint);
}
footer.provenance p { margin-bottom: 0.3rem; }
footer.provenance a { color: var(--ink-faint); }
footer.provenance blockquote { margin: 0; padding: 0; border: none; color: inherit; }
a[data-hover-title] { position: relative; }
.hover-card {
position: absolute; bottom: 100%; left: 50%; transform: translateX(-50%);
width: 320px; max-width: 90vw; padding: 0.65rem 0.8rem;
background: var(--surface-raised); border: 1px solid var(--rule);
border-radius: 6px; box-shadow: 0 4px 12px rgba(0,0,0,0.1);
font-size: 0.78rem; line-height: 1.45; color: var(--ink);
pointer-events: none; z-index: 100; margin-bottom: 6px;
opacity: 0; transition: opacity 0.12s;
}
a[data-hover-title]:hover .hover-card,
a[data-hover-title]:focus .hover-card,
a.cite:hover .hover-card { opacity: 1; }
.hover-card .hc-title { font-weight: 600; margin-bottom: 0.2rem; }
.hover-card .hc-meta { font-size: 0.72rem; color: var(--ink-faint); margin-bottom: 0.25rem; }
.hover-card .hc-status {
display: inline-block; font-size: 0.65rem; font-weight: 600;
text-transform: uppercase; letter-spacing: 0.04em;
padding: 0.1em 0.4em; border-radius: 3px; margin-right: 0.4em;
}
.hc-status-open { background: #fff3e0; color: #e65100; }
.hc-status-closed, .hc-status-merged { background: #e8f5e9; color: #2e7d32; }
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) .hover-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); }
:root:not([data-theme="light"]) .hc-status-open { background: #3a2a10; color: #ffb74d; }
:root:not([data-theme="light"]) .hc-status-closed,
:root:not([data-theme="light"]) .hc-status-merged { background: #1b3a1e; color: #66bb6a; }
}
:root[data-theme="dark"] .hover-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); }
:root[data-theme="dark"] .hc-status-open { background: #3a2a10; color: #ffb74d; }
:root[data-theme="dark"] .hc-status-closed,
:root[data-theme="dark"] .hc-status-merged { background: #1b3a1e; color: #66bb6a; }
.hover-card .hc-desc { color: var(--ink-secondary); }
.cite {
font-size: 0.72rem; vertical-align: super; line-height: 0;
color: var(--accent); font-weight: 600; text-decoration: none;
border-bottom: none !important; position: relative;
}
.cite:hover { color: var(--link-hover); }
.references { margin-top: 3rem; max-width: var(--content-width); }
.references h2 { font-size: 1.15rem; margin-bottom: 1rem; }
.ref-list { list-style: none; padding: 0; counter-reset: ref; }
.ref-list li {
counter-increment: ref; display: flex; align-items: baseline;
gap: 0.5em; font-size: 0.82rem; line-height: 1.55;
margin-bottom: 0.4rem; color: var(--ink-secondary);
}
.ref-list li::before {
content: "[" counter(ref) "]"; flex-shrink: 0;
font-variant-numeric: tabular-nums; color: var(--ink-faint);
font-size: 0.78rem; min-width: 2.2em;
}
.ref-list .ref-body { flex: 1; min-width: 0; }
.ref-list .ref-title { font-weight: 500; color: var(--ink); }
.ref-list .ref-url {
font-family: 'SF Mono', Menlo, Consolas, monospace; font-size: 0.75rem;
color: var(--ink-faint); word-break: break-all; margin-left: 0.4em;
}
.ref-list .ref-url a { color: var(--ink-faint); border-bottom: none; }
.ref-list .ref-url a:hover { color: var(--link-hover); }
.ref-back {
color: var(--accent); text-decoration: none; border-bottom: none !important;
margin-left: 0.3em; font-size: 0.78rem;
}
.ref-back:hover { color: var(--link-hover); }
.status {
display: inline-block; font-size: 0.65rem; font-weight: 600;
text-transform: uppercase; letter-spacing: 0.04em;
padding: 0.15em 0.5em; border-radius: 3px; vertical-align: middle;
}
.status-done { background: #e8f5e9; color: #2e7d32; }
.status-open { background: #fff3e0; color: #e65100; }
.status-blocked { background: #fce4ec; color: #c62828; }
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) .status-done { background: #1b3a1e; color: #66bb6a; }
:root:not([data-theme="light"]) .status-open { background: #3a2a10; color: #ffb74d; }
:root:not([data-theme="light"]) .status-blocked { background: #3a1520; color: #ef9a9a; }
}
:root[data-theme="dark"] .status-done { background: #1b3a1e; color: #66bb6a; }
:root[data-theme="dark"] .status-open { background: #3a2a10; color: #ffb74d; }
:root[data-theme="dark"] .status-blocked { background: #3a1520; color: #ef9a9a; }
.cite-card {
position: absolute; bottom: 100%; left: 50%; transform: translateX(-50%);
width: 360px; max-width: 90vw; padding: 0.65rem 0.8rem;
background: var(--surface-raised); border: 1px solid var(--rule);
border-radius: 6px; box-shadow: 0 4px 12px rgba(0,0,0,0.1);
font-size: 0.78rem; line-height: 1.45; color: var(--ink);
pointer-events: none; z-index: 100; margin-bottom: 6px;
opacity: 0; transition: opacity 0.12s;
font-weight: 400; vertical-align: baseline; text-align: left;
}
a.cite:hover .cite-card { opacity: 1; }
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) .cite-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); }
}
:root[data-theme="dark"] .cite-card { box-shadow: 0 4px 12px rgba(0,0,0,0.3); }
@media print {
body { font-size: 11pt; }
.page { max-width: 100%; padding: 0; }
.sidenote { float: right; width: 180px; margin-right: -210px; }
.sidenote-toggle { display: none; }
a { color: inherit; border-bottom: none; }
.hover-card { display: none; }
.cite-card { display: none; }
}
.katex{font:normal 1.21em KaTeX_Main,Times New Roman,serif;line-height:1.2;position:relative;text-indent:0;text-rendering:auto}.katex *{-ms-high-contrast-adjust:none!important;border-color:currentColor}.katex .katex-version:after{content:"0.18.7"}.katex .katex-mathml{border:0;-webkit-clip-path:inset(50%);clip-path:inset(50%);height:1px;overflow:hidden;padding:0;position:absolute;width:1px}.katex .katex-html>.katex-newline{display:block}.katex .katex-base{position:relative;white-space:nowrap;width:-webkit-min-content;width:-moz-min-content;width:min-content}.katex .katex-base,.katex .katex-strut{display:inline-block}.katex .textbf{font-weight:700}.katex .textit{font-style:italic}.katex .textrm{font-family:KaTeX_Main}.katex .textsf{font-family:KaTeX_SansSerif}.katex .texttt{font-family:KaTeX_Typewriter}.katex .mathnormal{font-family:KaTeX_Math;font-style:italic}.katex .mathit{font-family:KaTeX_Main;font-style:italic}.katex .mathrm{font-style:normal}.katex .mathbf{font-family:KaTeX_Main;font-weight:700}.katex .boldsymbol{font-family:KaTeX_Math;font-style:italic;font-weight:700}.katex .amsrm,.katex .mathbb,.katex .textbb{font-family:KaTeX_AMS}.katex .mathcal{font-family:KaTeX_Caligraphic}.katex .mathfrak,.katex .textfrak{font-family:KaTeX_Fraktur}.katex .mathboldfrak,.katex .textboldfrak{font-family:KaTeX_Fraktur;font-weight:700}.katex .mathtt{font-family:KaTeX_Typewriter}.katex .mathscr,.katex .textscr{font-family:KaTeX_Script}.katex .mathsf,.katex .textsf{font-family:KaTeX_SansSerif}.katex .mathboldsf,.katex .textboldsf{font-family:KaTeX_SansSerif;font-weight:700}.katex .mathitsf,.katex .mathsfit,.katex .textitsf{font-family:KaTeX_SansSerif;font-style:italic}.katex .mainrm{font-family:KaTeX_Main;font-style:normal}.katex .vlist-t{border-collapse:collapse;display:inline-table;table-layout:fixed}.katex .vlist-r{display:table-row}.katex .vlist{display:table-cell;position:relative;vertical-align:bottom}.katex .vlist>span{display:block;height:0;position:relative}.katex .vlist>span>span{display:inline-block}.katex .vlist>span>.pstrut{overflow:hidden;width:0}.katex .vlist-t2{margin-right:-2px}.katex .vlist-s{display:table-cell;font-size:1px;min-width:2px;vertical-align:bottom;width:2px}.katex .katex-vbox{align-items:baseline;display:inline-flex;flex-direction:column}.katex .katex-thinbox{display:inline-flex;flex-direction:row;max-width:0;width:0}.katex .msupsub{text-align:left}.katex .mfrac>span>span{text-align:center}.katex .mfrac .frac-line{border-bottom-style:solid;display:inline-block;width:100%}.katex .katex-hdashline,.katex .katex-hline,.katex .katex-overline .overline-line,.katex .katex-rule,.katex .katex-underline .underline-line,.katex .mfrac .frac-line{min-height:1px}.katex .mspace{display:inline-block}.katex .katex-smash{display:inline;line-height:0}.katex .clap,.katex .llap,.katex .rlap{position:relative;width:0}.katex .clap>.katex-inner,.katex .llap>.katex-inner,.katex .rlap>.katex-inner{position:absolute}.katex .clap>.katex-fix,.katex .llap>.katex-fix,.katex .rlap>.katex-fix{display:inline-block}.katex .llap>.katex-inner{right:0}.katex .clap>.katex-inner,.katex .rlap>.katex-inner{left:0}.katex .clap>.katex-inner>span{margin-left:-50%;margin-right:50%}.katex .katex-rule{border:0 solid;display:inline-block;position:relative}.katex .katex-hline,.katex .katex-overline .overline-line,.katex .katex-underline .underline-line{border-bottom-style:solid;display:inline-block;width:100%}.katex .katex-hdashline{border-bottom-style:dashed;display:inline-block;width:100%}.katex .sqrt>.katex-root{margin-left:.2777777778em;margin-right:-.5555555556em}.katex .fontsize-ensurer.reset-size1.size1,.katex .katex-sizing.reset-size1.size1{font-size:1em}.katex .fontsize-ensurer.reset-size1.size2,.katex .katex-sizing.reset-size1.size2{font-size:1.2em}.katex .fontsize-ensurer.reset-size1.size3,.katex .katex-sizing.reset-size1.size3{font-size:1.4em}.katex .fontsize-ensurer.reset-size1.size4,.katex .katex-sizing.reset-size1.size4{font-size:1.6em}.katex .fontsize-ensurer.reset-size1.size5,.katex .katex-sizing.reset-size1.size5{font-size:1.8em}.katex .fontsize-ensurer.reset-size1.size6,.katex .katex-sizing.reset-size1.size6{font-size:2em}.katex .fontsize-ensurer.reset-size1.size7,.katex .katex-sizing.reset-size1.size7{font-size:2.4em}.katex .fontsize-ensurer.reset-size1.size8,.katex .katex-sizing.reset-size1.size8{font-size:2.88em}.katex .fontsize-ensurer.reset-size1.size9,.katex .katex-sizing.reset-size1.size9{font-size:3.456em}.katex .fontsize-ensurer.reset-size1.size10,.katex .katex-sizing.reset-size1.size10{font-size:4.148em}.katex .fontsize-ensurer.reset-size1.size11,.katex .katex-sizing.reset-size1.size11{font-size:4.976em}.katex .fontsize-ensurer.reset-size2.size1,.katex .katex-sizing.reset-size2.size1{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size2.size2,.katex .katex-sizing.reset-size2.size2{font-size:1em}.katex .fontsize-ensurer.reset-size2.size3,.katex .katex-sizing.reset-size2.size3{font-size:1.1666666667em}.katex .fontsize-ensurer.reset-size2.size4,.katex .katex-sizing.reset-size2.size4{font-size:1.3333333333em}.katex .fontsize-ensurer.reset-size2.size5,.katex .katex-sizing.reset-size2.size5{font-size:1.5em}.katex .fontsize-ensurer.reset-size2.size6,.katex .katex-sizing.reset-size2.size6{font-size:1.6666666667em}.katex .fontsize-ensurer.reset-size2.size7,.katex .katex-sizing.reset-size2.size7{font-size:2em}.katex .fontsize-ensurer.reset-size2.size8,.katex .katex-sizing.reset-size2.size8{font-size:2.4em}.katex .fontsize-ensurer.reset-size2.size9,.katex .katex-sizing.reset-size2.size9{font-size:2.88em}.katex .fontsize-ensurer.reset-size2.size10,.katex .katex-sizing.reset-size2.size10{font-size:3.4566666667em}.katex .fontsize-ensurer.reset-size2.size11,.katex .katex-sizing.reset-size2.size11{font-size:4.1466666667em}.katex .fontsize-ensurer.reset-size3.size1,.katex .katex-sizing.reset-size3.size1{font-size:.7142857143em}.katex .fontsize-ensurer.reset-size3.size2,.katex .katex-sizing.reset-size3.size2{font-size:.8571428571em}.katex .fontsize-ensurer.reset-size3.size3,.katex .katex-sizing.reset-size3.size3{font-size:1em}.katex .fontsize-ensurer.reset-size3.size4,.katex .katex-sizing.reset-size3.size4{font-size:1.1428571429em}.katex .fontsize-ensurer.reset-size3.size5,.katex .katex-sizing.reset-size3.size5{font-size:1.2857142857em}.katex .fontsize-ensurer.reset-size3.size6,.katex .katex-sizing.reset-size3.size6{font-size:1.4285714286em}.katex .fontsize-ensurer.reset-size3.size7,.katex .katex-sizing.reset-size3.size7{font-size:1.7142857143em}.katex .fontsize-ensurer.reset-size3.size8,.katex .katex-sizing.reset-size3.size8{font-size:2.0571428571em}.katex .fontsize-ensurer.reset-size3.size9,.katex .katex-sizing.reset-size3.size9{font-size:2.4685714286em}.katex .fontsize-ensurer.reset-size3.size10,.katex .katex-sizing.reset-size3.size10{font-size:2.9628571429em}.katex .fontsize-ensurer.reset-size3.size11,.katex .katex-sizing.reset-size3.size11{font-size:3.5542857143em}.katex .fontsize-ensurer.reset-size4.size1,.katex .katex-sizing.reset-size4.size1{font-size:.625em}.katex .fontsize-ensurer.reset-size4.size2,.katex .katex-sizing.reset-size4.size2{font-size:.75em}.katex .fontsize-ensurer.reset-size4.size3,.katex .katex-sizing.reset-size4.size3{font-size:.875em}.katex .fontsize-ensurer.reset-size4.size4,.katex .katex-sizing.reset-size4.size4{font-size:1em}.katex .fontsize-ensurer.reset-size4.size5,.katex .katex-sizing.reset-size4.size5{font-size:1.125em}.katex .fontsize-ensurer.reset-size4.size6,.katex .katex-sizing.reset-size4.size6{font-size:1.25em}.katex .fontsize-ensurer.reset-size4.size7,.katex .katex-sizing.reset-size4.size7{font-size:1.5em}.katex .fontsize-ensurer.reset-size4.size8,.katex .katex-sizing.reset-size4.size8{font-size:1.8em}.katex .fontsize-ensurer.reset-size4.size9,.katex .katex-sizing.reset-size4.size9{font-size:2.16em}.katex .fontsize-ensurer.reset-size4.size10,.katex .katex-sizing.reset-size4.size10{font-size:2.5925em}.katex .fontsize-ensurer.reset-size4.size11,.katex .katex-sizing.reset-size4.size11{font-size:3.11em}.katex .fontsize-ensurer.reset-size5.size1,.katex .katex-sizing.reset-size5.size1{font-size:.5555555556em}.katex .fontsize-ensurer.reset-size5.size2,.katex .katex-sizing.reset-size5.size2{font-size:.6666666667em}.katex .fontsize-ensurer.reset-size5.size3,.katex .katex-sizing.reset-size5.size3{font-size:.7777777778em}.katex .fontsize-ensurer.reset-size5.size4,.katex .katex-sizing.reset-size5.size4{font-size:.8888888889em}.katex .fontsize-ensurer.reset-size5.size5,.katex .katex-sizing.reset-size5.size5{font-size:1em}.katex .fontsize-ensurer.reset-size5.size6,.katex .katex-sizing.reset-size5.size6{font-size:1.1111111111em}.katex .fontsize-ensurer.reset-size5.size7,.katex .katex-sizing.reset-size5.size7{font-size:1.3333333333em}.katex .fontsize-ensurer.reset-size5.size8,.katex .katex-sizing.reset-size5.size8{font-size:1.6em}.katex .fontsize-ensurer.reset-size5.size9,.katex .katex-sizing.reset-size5.size9{font-size:1.92em}.katex .fontsize-ensurer.reset-size5.size10,.katex .katex-sizing.reset-size5.size10{font-size:2.3044444444em}.katex .fontsize-ensurer.reset-size5.size11,.katex .katex-sizing.reset-size5.size11{font-size:2.7644444444em}.katex .fontsize-ensurer.reset-size6.size1,.katex .katex-sizing.reset-size6.size1{font-size:.5em}.katex .fontsize-ensurer.reset-size6.size2,.katex .katex-sizing.reset-size6.size2{font-size:.6em}.katex .fontsize-ensurer.reset-size6.size3,.katex .katex-sizing.reset-size6.size3{font-size:.7em}.katex .fontsize-ensurer.reset-size6.size4,.katex .katex-sizing.reset-size6.size4{font-size:.8em}.katex .fontsize-ensurer.reset-size6.size5,.katex .katex-sizing.reset-size6.size5{font-size:.9em}.katex .fontsize-ensurer.reset-size6.size6,.katex .katex-sizing.reset-size6.size6{font-size:1em}.katex .fontsize-ensurer.reset-size6.size7,.katex .katex-sizing.reset-size6.size7{font-size:1.2em}.katex .fontsize-ensurer.reset-size6.size8,.katex .katex-sizing.reset-size6.size8{font-size:1.44em}.katex .fontsize-ensurer.reset-size6.size9,.katex .katex-sizing.reset-size6.size9{font-size:1.728em}.katex .fontsize-ensurer.reset-size6.size10,.katex .katex-sizing.reset-size6.size10{font-size:2.074em}.katex .fontsize-ensurer.reset-size6.size11,.katex .katex-sizing.reset-size6.size11{font-size:2.488em}.katex .fontsize-ensurer.reset-size7.size1,.katex .katex-sizing.reset-size7.size1{font-size:.4166666667em}.katex .fontsize-ensurer.reset-size7.size2,.katex .katex-sizing.reset-size7.size2{font-size:.5em}.katex .fontsize-ensurer.reset-size7.size3,.katex .katex-sizing.reset-size7.size3{font-size:.5833333333em}.katex .fontsize-ensurer.reset-size7.size4,.katex .katex-sizing.reset-size7.size4{font-size:.6666666667em}.katex .fontsize-ensurer.reset-size7.size5,.katex .katex-sizing.reset-size7.size5{font-size:.75em}.katex .fontsize-ensurer.reset-size7.size6,.katex .katex-sizing.reset-size7.size6{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size7.size7,.katex .katex-sizing.reset-size7.size7{font-size:1em}.katex .fontsize-ensurer.reset-size7.size8,.katex .katex-sizing.reset-size7.size8{font-size:1.2em}.katex .fontsize-ensurer.reset-size7.size9,.katex .katex-sizing.reset-size7.size9{font-size:1.44em}.katex .fontsize-ensurer.reset-size7.size10,.katex .katex-sizing.reset-size7.size10{font-size:1.7283333333em}.katex .fontsize-ensurer.reset-size7.size11,.katex .katex-sizing.reset-size7.size11{font-size:2.0733333333em}.katex .fontsize-ensurer.reset-size8.size1,.katex .katex-sizing.reset-size8.size1{font-size:.3472222222em}.katex .fontsize-ensurer.reset-size8.size2,.katex .katex-sizing.reset-size8.size2{font-size:.4166666667em}.katex .fontsize-ensurer.reset-size8.size3,.katex .katex-sizing.reset-size8.size3{font-size:.4861111111em}.katex .fontsize-ensurer.reset-size8.size4,.katex .katex-sizing.reset-size8.size4{font-size:.5555555556em}.katex .fontsize-ensurer.reset-size8.size5,.katex .katex-sizing.reset-size8.size5{font-size:.625em}.katex .fontsize-ensurer.reset-size8.size6,.katex .katex-sizing.reset-size8.size6{font-size:.6944444444em}.katex .fontsize-ensurer.reset-size8.size7,.katex .katex-sizing.reset-size8.size7{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size8.size8,.katex .katex-sizing.reset-size8.size8{font-size:1em}.katex .fontsize-ensurer.reset-size8.size9,.katex .katex-sizing.reset-size8.size9{font-size:1.2em}.katex .fontsize-ensurer.reset-size8.size10,.katex .katex-sizing.reset-size8.size10{font-size:1.4402777778em}.katex .fontsize-ensurer.reset-size8.size11,.katex .katex-sizing.reset-size8.size11{font-size:1.7277777778em}.katex .fontsize-ensurer.reset-size9.size1,.katex .katex-sizing.reset-size9.size1{font-size:.2893518519em}.katex .fontsize-ensurer.reset-size9.size2,.katex .katex-sizing.reset-size9.size2{font-size:.3472222222em}.katex .fontsize-ensurer.reset-size9.size3,.katex .katex-sizing.reset-size9.size3{font-size:.4050925926em}.katex .fontsize-ensurer.reset-size9.size4,.katex .katex-sizing.reset-size9.size4{font-size:.462962963em}.katex .fontsize-ensurer.reset-size9.size5,.katex .katex-sizing.reset-size9.size5{font-size:.5208333333em}.katex .fontsize-ensurer.reset-size9.size6,.katex .katex-sizing.reset-size9.size6{font-size:.5787037037em}.katex .fontsize-ensurer.reset-size9.size7,.katex .katex-sizing.reset-size9.size7{font-size:.6944444444em}.katex .fontsize-ensurer.reset-size9.size8,.katex .katex-sizing.reset-size9.size8{font-size:.8333333333em}.katex .fontsize-ensurer.reset-size9.size9,.katex .katex-sizing.reset-size9.size9{font-size:1em}.katex .fontsize-ensurer.reset-size9.size10,.katex .katex-sizing.reset-size9.size10{font-size:1.2002314815em}.katex .fontsize-ensurer.reset-size9.size11,.katex .katex-sizing.reset-size9.size11{font-size:1.4398148148em}.katex .fontsize-ensurer.reset-size10.size1,.katex .katex-sizing.reset-size10.size1{font-size:.2410800386em}.katex .fontsize-ensurer.reset-size10.size2,.katex .katex-sizing.reset-size10.size2{font-size:.2892960463em}.katex .fontsize-ensurer.reset-size10.size3,.katex .katex-sizing.reset-size10.size3{font-size:.337512054em}.katex .fontsize-ensurer.reset-size10.size4,.katex .katex-sizing.reset-size10.size4{font-size:.3857280617em}.katex .fontsize-ensurer.reset-size10.size5,.katex .katex-sizing.reset-size10.size5{font-size:.4339440694em}.katex .fontsize-ensurer.reset-size10.size6,.katex .katex-sizing.reset-size10.size6{font-size:.4821600771em}.katex .fontsize-ensurer.reset-size10.size7,.katex .katex-sizing.reset-size10.size7{font-size:.5785920926em}.katex .fontsize-ensurer.reset-size10.size8,.katex .katex-sizing.reset-size10.size8{font-size:.6943105111em}.katex .fontsize-ensurer.reset-size10.size9,.katex .katex-sizing.reset-size10.size9{font-size:.8331726133em}.katex .fontsize-ensurer.reset-size10.size10,.katex .katex-sizing.reset-size10.size10{font-size:1em}.katex .fontsize-ensurer.reset-size10.size11,.katex .katex-sizing.reset-size10.size11{font-size:1.1996142719em}.katex .fontsize-ensurer.reset-size11.size1,.katex .katex-sizing.reset-size11.size1{font-size:.2009646302em}.katex .fontsize-ensurer.reset-size11.size2,.katex .katex-sizing.reset-size11.size2{font-size:.2411575563em}.katex .fontsize-ensurer.reset-size11.size3,.katex .katex-sizing.reset-size11.size3{font-size:.2813504823em}.katex .fontsize-ensurer.reset-size11.size4,.katex .katex-sizing.reset-size11.size4{font-size:.3215434084em}.katex .fontsize-ensurer.reset-size11.size5,.katex .katex-sizing.reset-size11.size5{font-size:.3617363344em}.katex .fontsize-ensurer.reset-size11.size6,.katex .katex-sizing.reset-size11.size6{font-size:.4019292605em}.katex .fontsize-ensurer.reset-size11.size7,.katex .katex-sizing.reset-size11.size7{font-size:.4823151125em}.katex .fontsize-ensurer.reset-size11.size8,.katex .katex-sizing.reset-size11.size8{font-size:.578778135em}.katex .fontsize-ensurer.reset-size11.size9,.katex .katex-sizing.reset-size11.size9{font-size:.6945337621em}.katex .fontsize-ensurer.reset-size11.size10,.katex .katex-sizing.reset-size11.size10{font-size:.8336012862em}.katex .fontsize-ensurer.reset-size11.size11,.katex .katex-sizing.reset-size11.size11{font-size:1em}.katex .delimsizing.size1{font-family:KaTeX_Size1}.katex .delimsizing.size2{font-family:KaTeX_Size2}.katex .delimsizing.size3{font-family:KaTeX_Size3}.katex .delimsizing.size4{font-family:KaTeX_Size4}.katex .delimsizing.mult .delim-size1>span{font-family:KaTeX_Size1}.katex .delimsizing.mult .delim-size4>span{font-family:KaTeX_Size4}.katex .nulldelimiter{display:inline-block;width:.12em}.katex .delimcenter,.katex .op-symbol{position:relative}.katex .op-symbol.small-op{font-family:KaTeX_Size1}.katex .op-symbol.large-op{font-family:KaTeX_Size2}.katex .katex-accent>.vlist-t,.katex .op-limits>.vlist-t{text-align:center}.katex .katex-accent .accent-body{position:relative}.katex .katex-accent .accent-body:not(.accent-full){width:0}.katex .katex-overlay{display:block}.katex .mtable .vertical-separator{display:inline-block;min-width:1px}.katex .mtable .arraycolsep{display:inline-block}.katex .mtable .col-align-c>.vlist-t{text-align:center}.katex .mtable .col-align-l>.vlist-t{text-align:left}.katex .mtable .col-align-r>.vlist-t{text-align:right}.katex .svg-align{text-align:left}.katex svg{fill:currentColor;stroke:currentColor;display:block;height:inherit;position:absolute;width:100%}.katex svg path{stroke:none}.katex svg{fill-rule:nonzero;fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1}.katex img{border-style:none;max-height:none;max-width:none;min-height:0;min-width:0}.katex .katex-stretchy{display:block;overflow:hidden;position:relative;width:100%}.katex .katex-stretchy:after,.katex .katex-stretchy:before{content:""}.katex .hide-tail{overflow:hidden;position:relative;width:100%}.katex .halfarrow-left{left:0;overflow:hidden;position:absolute;width:50.2%}.katex .halfarrow-right{overflow:hidden;position:absolute;right:0;width:50.2%}.katex .brace-left{left:0;overflow:hidden;position:absolute;width:25.1%}.katex .brace-center{left:25%;overflow:hidden;position:absolute;width:50%}.katex .brace-right{overflow:hidden;position:absolute;right:0;width:25.1%}.katex .x-arrow-pad{padding:0 .5em}.katex .cd-arrow-pad{padding:0 .55556em 0 .27778em}.katex .mover,.katex .munder,.katex .x-arrow{text-align:center}.katex .boxpad{padding:0 .3em}.katex .fbox,.katex .fcolorbox{border:.04em solid;box-sizing:border-box}.katex .cancel-pad{padding:0 .2em}.katex .cancel-lap{margin-left:-.2em;margin-right:-.2em}.katex .katex-sout{border-bottom-style:solid;border-bottom-width:.08em}.katex .angl{border-right:.049em solid;border-top:.049em solid;box-sizing:border-box;margin-right:.03889em}.katex .anglpad{padding:0 .03889em}.katex .reflectbox{display:inline-block;transform:scaleX(-1)}.katex .eqn-num:before{content:"(" counter(katexEqnNo) ")";counter-increment:katexEqnNo}.katex .mml-eqn-num:before{content:"(" counter(mmlEqnNo) ")";counter-increment:mmlEqnNo}.katex .mtr-glue{width:50%}.katex .cd-vert-arrow{display:inline-block;position:relative}.katex .cd-label-left{display:inline-block;position:absolute;right:calc(50% + .3em);text-align:left}.katex .cd-label-right{display:inline-block;left:calc(50% + .3em);position:absolute;text-align:right}.katex-display{display:block;margin:1em 0;text-align:center}.katex-display>.katex{display:block;text-align:center;white-space:nowrap}.katex-display>.katex>.katex-html{display:block;position:relative}.katex-display>.katex>.katex-html>.katex-tag{position:absolute;right:0}.katex-display.leqno>.katex>.katex-html>.katex-tag{left:0;right:auto}.katex-display.fleqn>.katex{padding-left:2em;text-align:left}body{counter-reset:katexEqnNo mmlEqnNo}
</style>
<script>(function() {
function __mumInit() {
// Restore external links that htmlpreview.github.io's loader rewrote to
// in-page anchors (it treats any href containing '#' as a local anchor).
// Runs now (htmlpreview re-executes this script after its rewrite), again
// after delays, and on click as a last line of defense.
function restoreOrigHrefs() {
document.querySelectorAll('a[data-orig-href]').forEach(function(a) {
var orig = a.getAttribute('data-orig-href');
if (orig && a.getAttribute('href') !== orig) a.setAttribute('href', orig);
});
}
restoreOrigHrefs();
setTimeout(restoreOrigHrefs, 100);
setTimeout(restoreOrigHrefs, 1000);
document.addEventListener('click', function(e) {
var t = e.target && e.target.closest ? e.target.closest('a[data-orig-href]') : null;
if (t) { var o = t.getAttribute('data-orig-href'); if (o && t.getAttribute('href') !== o) t.setAttribute('href', o); }
}, true);
var vt = document.getElementById('viewing-time');
if (vt) {
function pad(n) { return n < 10 ? '0' + n : n; }
function updateTime() {
var d = new Date();
vt.textContent = d.getUTCFullYear() + '-' + pad(d.getUTCMonth()+1) + '-' + pad(d.getUTCDate()) + ' ' + pad(d.getUTCHours()) + ':' + pad(d.getUTCMinutes()) + ' UTC';
}
updateTime();
setInterval(updateTime, 60000);
}
document.querySelectorAll('a[data-hover-title]').forEach(function(a) {
var card = document.createElement('span');
card.className = 'hover-card';
var title = a.getAttribute('data-hover-title') || '';
var status = a.getAttribute('data-hover-status') || '';
var owner = a.getAttribute('data-hover-owner') || '';
var desc = a.getAttribute('data-hover-desc') || '';
var statusCls = 'hc-status hc-status-' + status.toLowerCase().replace(/[^a-z]/g, '');
var html = '<div class="hc-title">' + title + '</div>';
var meta = [];
if (status) meta.push('<span class="' + statusCls + '">' + status + '</span>');
if (owner) meta.push(owner);
if (meta.length) html += '<div class="hc-meta">' + meta.join(' ') + '</div>';
if (desc) html += '<div class="hc-desc">' + desc + '</div>';
card.innerHTML = html;
a.appendChild(card);
});
document.querySelectorAll('a.cite').forEach(function(a) {
var href = a.getAttribute('href') || '';
if (!href.startsWith('#ref-')) return;
var li = document.getElementById(href.slice(1));
if (!li) return;
var body = li.querySelector('.ref-body');
if (!body) return;
var refUrl = body.querySelector('.ref-url a');
var refHref = refUrl ? refUrl.getAttribute('href') : '';
var hoverSource = refHref ? document.querySelector('a[data-hover-title][href="' + refHref + '"]') : null;
if (hoverSource) {
var card = document.createElement('span');
card.className = 'hover-card';
var t = hoverSource.getAttribute('data-hover-title') || '';
var s = hoverSource.getAttribute('data-hover-status') || '';
var o = hoverSource.getAttribute('data-hover-owner') || '';
var d = hoverSource.getAttribute('data-hover-desc') || '';
var sc = 'hc-status hc-status-' + s.toLowerCase().replace(/[^a-z]/g, '');
var h = '<div class="hc-title">' + t + '</div>';
var m = [];
if (s) m.push('<span class="' + sc + '">' + s + '</span>');
if (o) m.push(o);
if (m.length) h += '<div class="hc-meta">' + m.join(' ') + '</div>';
if (d) h += '<div class="hc-desc">' + d + '</div>';
card.innerHTML = h;
a.appendChild(card);
} else {
var title = body.querySelector('.ref-title');
if (!title) return;
var card = document.createElement('span');
card.className = 'cite-card';
card.textContent = title.textContent;
a.appendChild(card);
}
});
}
if (document.readyState !== 'loading') { __mumInit(); }
else { document.addEventListener('DOMContentLoaded', __mumInit); }
})();</script>
</head>
<body>
<div class="page">
<header class="paper-header">
<h1>Expected MFU for the 535B-A23B hero on a Megatron stack instead of Levanter</h1>
<div class="paper-meta">Posed by <span class="author">hammer</span>, answered by <a class="mumwelt-link" href="https://github.com/marin-community/mumwelt">mumwelt</a> &middot; <span class="date-label" title="Generated 2026-09-23 13:24 UTC">Published 2026-09-23</span></div>
</header>
<div class="prompt-box">
<div class="prompt-label">?</div>
<div class="prompt-text">Given the 535B hero-run architecture we are training on CoreWeave, what MFU should we expect from a PyTorch/Megatron stack in place of Levanter, informed by external results on similar hardware?</div>
</div>
<div class="content">
<h2 id="short-answer">Short answer</h2>
<p>A Megatron-Core stack running the hero's actual recipe (learned QB
router with capacity and spill, MuonH, the full Grug operator set) is
expected to train <strong>slower</strong> than Levanter does today:
median about 20% MFU against Levanter's 24.0% in production, with
roughly a 1-in-20 chance of beating it. Replacing MuonH with Adam makes
it a coin flip, median about 23.5%. The 31 to 32% figures that NVIDIA
and Marin's own calibration report for Megatron are systems ceilings
measured with a force-balanced, dropless router, and no published BF16
number above 30% on GB200 was obtained any other way.</p>
<table>
<thead>
<tr>
<th>Scenario for a Megatron port of the hero (BF16, 704 GB200, seq
4096)</th>
<th>p10</th>
<th>median</th>
<th>p90</th>
<th>P(beats 24%)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Systems ceiling: force-balanced router, Adam</td>
<td>29.8%</td>
<td>31.6%</td>
<td>33.5%</td>
<td>1.00</td>
</tr>
<tr>
<td>Production semantics with Adam</td>
<td>20.0%</td>
<td>23.5%</td>
<td>26.8%</td>
<td>0.43</td>
</tr>
<tr>
<td>Production semantics with MuonH (the actual recipe)</td>
<td>16.6%</td>
<td>19.8%</td>
<td>23.2%</td>
<td>0.05</td>
</tr>
</tbody>
</table>
<p>At the hero's 149 GFLOP per token on 704 GPUs, those levels mean 18T
tokens in about 84 days at 20%, 74 days at 24%, and 57 days at 31%. The
estimate rests on NVIDIA's published Megatron-Core numbers for the two
closest open MoE shapes, on Marin's own July calibration of Megatron
against JAX on one GB200 rack, and on DeepSeek-V3's production
GPU-hours; the method and its uncertainty are laid out below.</p>
<h2 id="what-levanter-delivers-today">What Levanter delivers today</h2>
<p>The hero is a 535.3B-total, 22.76B-active Grug MoE: 48 layers, 384
routed experts of width 3072 with top-8 routing plus two shared experts,
LatentMoE dispatch, sequence length 4096, trained with MuonH on 11
CoreWeave GB200 NVL72 racks (704 GPUs) with expert parallelism 64 inside
each rack and data parallelism across racks (<a
data-orig-href="https://github.com/marin-community/marin/issues/8435#issuecomment-5335872267" href="https://github.com/marin-community/marin/issues/8435#issuecomment-5335872267">job
summary on #8435</a>). Its reported MFU divides a window-aware analytic
model-FLOP count (<a
data-orig-href="https://github.com/marin-community/marin/blob/cf3f4ffd1b8821a2216eac2082409306c2a29cb2/lib/levanter/src/levanter/utils/flop_utils.py#L31-L61" href="https://github.com/marin-community/marin/blob/cf3f4ffd1b8821a2216eac2082409306c2a29cb2/lib/levanter/src/levanter/utils/flop_utils.py#L31-L61">flop_utils.py</a>)
by 2.5 PFLOP/s per GPU, the GB200 NVL72 dense BF16 peak (<a
href="https://github.com/marin-community/marin/pull/7334">#7334</a>).
Production went from about 21% at launch to 23.3% after the ragged
expert-parallel relaunch and 24.0% after the PDL-off kernels (<a
href="https://mws.oa.dev/summaries/summary-2026-09-07_2026-09-13.html">week
of Sep 7</a>, <a
href="https://mws.oa.dev/summaries/summary-2026-09-14_2026-09-20.html">week
of Sep 14</a>); a one-rack diagnostic branch reads 27.64% with symmetric
buffers and mask removal (<a
href="https://github.com/marin-community/marin/pull/9315">#9315</a>).
August profiles put GEMMs at 57.7% of the compute stream and the exposed
expert-parallel collective residue at 2.4 to 3.2 seconds of a 17-second
step (<a
data-orig-href="https://github.com/marin-community/marin/issues/8317#issuecomment-5305649465" href="https://github.com/marin-community/marin/issues/8317#issuecomment-5305649465">#8317</a>,
<a
data-orig-href="https://github.com/marin-community/marin/issues/8311#issuecomment-5305698574" href="https://github.com/marin-community/marin/issues/8311#issuecomment-5305698574">#8311</a>).</p>
<h2
id="marins-own-calibration-megatron-against-jax-on-the-same-rack">Marin's
own calibration: Megatron against JAX on the same rack</h2>
<p>In late July Rafal Wojdyla's agent built a Megatron-Core plus
Transformer Engine image and ran a d5120, 48-layer, 256-expert top-8 MoE
(about 250B total, 16B active) at sequence 4096 and global batch 1024 on
one 64-GPU GB200 rack, the same 65,536 tokens per GPU per step the hero
uses, scoring it on the 2.5 PFLOP/s denominator (<a
href="https://github.com/marin-community/marin/issues/7668">#7668</a>).
The ladder:</p>
<ul>
<li>Stock NCCL all-to-all dispatcher: 4.423% MFU, with all-to-all
consuming 77.8% of the step; HybridEP replaced it.</li>
<li>Force-balanced router, Adam, no selective recompute: 31.1 to 31.3%
on the 8-of-256 shape and 32.4% on a promoted 4-of-128 shape, which the
completion audit calls "32.407% Levanter-compatible common MFU" (<a
data-orig-href="https://github.com/marin-community/marin/issues/7668#issuecomment-5104907558" href="https://github.com/marin-community/marin/issues/7668#issuecomment-5104907558">completion
audit</a>).</li>
<li>The full Grug operator set under the balanced router: 26.6%.</li>
<li>Stock auxiliary-loss-free learned router: 14.7%, from
expert-parallel rank overload. Tensor parallel 2 as a memory escape:
12.1%.</li>
<li>A faithful port of Grug's QB routing with capacity 1.0625 and three
spill attempts, 350 steps: "3.667 MFU percentage points below the
corrected" JAX point (<a
data-orig-href="https://github.com/marin-community/marin/issues/7668#issuecomment-5100527250" href="https://github.com/marin-community/marin/issues/7668#issuecomment-5100527250">#7668</a>),
that point being Matt Wittmann's 20.708% at 1.44% drops on the same rack
two days earlier (<a
data-orig-href="https://github.com/marin-community/marin/issues/7279#issuecomment-5084892846" href="https://github.com/marin-community/marin/issues/7279#issuecomment-5084892846">#7279</a>);
so about 17.0%.</li>
<li>MuonH through Megatron's layer-wise distributed optimizer "does not
recover EP64 throughput," and the close-out lists "a usable
MuonH/HyperBall optimizer path" among the remaining framework gaps (<a
data-orig-href="https://github.com/marin-community/marin/issues/7668#issuecomment-5100527250" href="https://github.com/marin-community/marin/issues/7668#issuecomment-5100527250">#7668</a>).
Every Megatron figure above used Adam.</li>
</ul>
<p>The exercise was a calibration, not a candidate stack. Jeff's June
synthesis on the derisking epic set the plan to "stay on the JAX/Grug
(Levanter) stack and close the gap with native kernels, not migrate to
Megatron/TE" (<a
data-orig-href="https://github.com/marin-community/marin/issues/4302#issuecomment-4727421242" href="https://github.com/marin-community/marin/issues/4302#issuecomment-4727421242">#4302</a>),
against David Hall's bar of "within ~10% of the best Megatron/TE BF16
B200 result on the same relevant shape" (<a
data-orig-href="https://github.com/marin-community/marin/issues/4311#issuecomment-4464443239" href="https://github.com/marin-community/marin/issues/4311#issuecomment-4464443239">#4311</a>).
Earlier microbenchmarks predate the kernel work: Megatron was 28% faster
on one GH200 in May (<a
data-orig-href="https://github.com/marin-community/marin/issues/4311#issuecomment-4414534370" href="https://github.com/marin-community/marin/issues/4311#issuecomment-4414534370">#4311</a>)
and 3.8 times faster on an 8×B200 slice (<a
href="https://github.com/marin-community/marin/issues/5815">#5815</a>),
after which Grug closed to within the 10% band by June 4 (<a
data-orig-href="https://github.com/marin-community/marin/issues/6139#issuecomment-4619246391" href="https://github.com/marin-community/marin/issues/6139#issuecomment-4619246391">#6139</a>).</p>
<h2 id="external-evidence-converted-to-the-heros-denominator">External
evidence, converted to the hero's denominator</h2>
<p>NVIDIA reports "TFLOPS/GPU" as delivered model FLOPs per second, so
dividing by 2.5 PFLOP/s gives MFU on Levanter's definition. Two
conversions of my own are marked.</p>
<table>
<thead>
<tr>
<th>Source</th>
<th>Model, stack, hardware</th>
<th>Precision</th>
<th>Routing</th>
<th>MFU</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="https://arxiv.org/abs/2603.07685">Megatron-Core MoE report,
Table 11</a></td>
<td>Qwen3-235B-A22B, Megatron-Core, 256 GB200, seq 4096, 5,100
tok/s/GPU</td>
<td>BF16</td>
<td>force-balanced, dropless</td>
<td>30.0%</td>
</tr>
<tr>
<td>same</td>
<td>DeepSeek-V3 (37B active), 256 GB200, 3,298 tok/s/GPU</td>
<td>BF16</td>
<td>force-balanced</td>
<td>34.3%</td>
</tr>
<tr>
<td>same</td>
<td>DeepSeek-V3, 256 GB200</td>
<td>MXFP8</td>
<td>force-balanced</td>
<td>41.9%</td>
</tr>
<tr>
<td>same</td>
<td>Qwen3-235B-A22B, 256 H100</td>
<td>BF16</td>
<td>force-balanced</td>
<td>32.4% of H100 peak</td>
</tr>
<tr>
<td><a
href="https://docs.nvidia.com/nemo/megatron-bridge/0.6.0/performance-summary-archive.html">Megatron
Bridge 26.06 tables</a></td>
<td>DeepSeek-V3, 256 GB200, TP1/PP4</td>
<td>MXFP8</td>
<td>force-balanced, dropless</td>
<td>51.7%</td>
</tr>
<tr>
<td>same</td>
<td>Qwen3-235B, 256 GB200, PP8/EP32</td>
<td>MXFP8</td>
<td>force-balanced, dropless</td>
<td>43.7%</td>
</tr>
<tr>
<td><a
href="https://developer.nvidia.com/blog/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72/">NVIDIA
GB300 record post</a></td>
<td>DeepSeek-V3, "early software version" GB200 NVL72, 256 GPUs, 606
TFLOPS/GPU</td>
<td>unstated</td>
<td>unstated</td>
<td>24.2%</td>
</tr>
<tr>
<td><a
href="https://pytorch.org/blog/enabling-up-to-41-faster-pre-training-mxfp8-and-deepep-for-deepseek-v3-on-b200-with-torchtitan/">TorchTitan
on Nebius</a></td>
<td>DeepSeek-V3, PyTorch-native, 256 HGX B200, EP32 over InfiniBand, seq
8192, 859 tok/s/GPU</td>
<td>BF16 + DeepEP</td>
<td>learned</td>
<td>11.8% (my conversion, with attention FLOPs)</td>
</tr>
<tr>
<td><a href="https://arxiv.org/abs/2412.19437">DeepSeek-V3
report</a></td>
<td>production run, DeepSeek's own stack, 2,048 H800, 2.664M GPU-hours
for 14.8T tokens</td>
<td>FP8</td>
<td>learned, aux-loss-free</td>
<td>34.6% of H800 BF16 peak on 6N alone, about 38% with attention (my
derivation)</td>
</tr>
</tbody>
</table>
<p>Four readings. First, NVIDIA's MoE benchmarks are force-balanced and
dropless by their own statement on the Bridge performance page, so they
are ceilings of the same kind as Marin's 31.3%, and Marin's two-day port
landed within two points of NVIDIA's number for a comparable shape.
Second, the closest published shape to the hero, Qwen3-235B-A22B at 22B
active in BF16 on GB200, reads 30.0%. Third, DeepSeek-V3 is the only
production number with a learned router at scale, and it lands in the
same band as NVIDIA's balanced H100 benchmarks, so a mature stack can
recover most of the routing discount. Fourth, a PyTorch-native stack
without NVIDIA's dispatcher and kernels sits near 12%, which is what
"PyTorch" means if it does not also mean Megatron-Core. The
Megatron-Core report's DeepSeek-V3 recipe on GB200 is TP1/PP4/EP64 with
HybridEP and CUDA Graphs (Table 17), and the report notes that on NVL72
<a href="https://arxiv.org/abs/2603.07685">"EP64 stays entirely within
the NVLink domain"</a>, which is the hero's layout too. The Nemotron 3
reports state their GB200 parallelism (CP64, TP2, EP64 for Super) but no
throughput (<a href="https://arxiv.org/abs/2604.12374">arXiv
2604.12374</a>); the SemiAnalysis GB200 NVL72 measurements are
paywalled, and its free section shows dense Llama 405B near 54% BF16 MFU
on H100 with NeMo (<a
href="https://newsletter.semianalysis.com/p/h100-vs-gb200-nvl72-training-benchmarks">SemiAnalysis</a>),
a reminder that MoE stacks run well below dense ones.</p>
<h2 id="extrapolation">Extrapolation</h2>
<p>The starting point is the BF16 anchors for Qwen3-235B and
DeepSeek-V3, cross-checked against Marin's own 31.3% ceiling, drawn as a
triangular distribution from 29% to 34.5% with mode 31%. Four
adjustments follow, each a triangular distribution over the range the
evidence supports:</p>
<ul>
<li><strong>Shape</strong>, 0.97 to 1.10 with mode 1.03. The hero holds
6 experts per GPU at width 3072 and dispatches in a 3072-dimensional
latent, which favors GEMM efficiency and halves all-to-all volume
relative to Qwen3's thinner experts; DeepSeek-V3's larger active size
reading 4 points above Qwen3 sets the upper end.</li>
<li><strong>Scale</strong>, 0.95 to 0.99. NVIDIA reports 98.5% per-GPU
retention from 256 to 1,024 GPUs on GB300; Marin sees about a point from
one rack to eleven.</li>
<li><strong>Routing</strong>, 0.60 to 0.97 with mode 0.85. This is the
move from a force-balanced benchmark router to a learned router with
capacity 1.15 and spill. Marin's untuned port paid 0.54; DeepSeek's
mature production stack pays close to nothing; Levanter's own ragged
transport now drops 0.01% of assignments at negligible cost. A Marin
Megatron deployment would sit between, and where it sits is the largest
single uncertainty.</li>
<li><strong>Operator fidelity</strong>, 0.85 to 0.99 with mode 0.92. The
full Grug operator set cost 0.855 in July before any kernel work; NVIDIA
now ships LatentMoE natively for Nemotron 3.</li>
<li><strong>MuonH</strong>, 0.70 to 0.97 with mode 0.87, applied only in
the recipe-faithful scenario.</li>
</ul>
<p>Two hundred thousand draws give the table in the short answer. The
ceiling scenario multiplies base, shape and scale; the Adam scenario
adds routing and operator fidelity; the MuonH scenario adds the
optimizer factor.</p>
<h2 id="what-would-move-the-estimate">What would move the estimate</h2>
<ul>
<li>The routing discount is engineering time, not hardware. Getting a
Megatron learned-router path to DeepSeek-class balance would lift the
Adam scenario toward its 27% upper tail. There is no internal precedent
for that effort, and the JAX stack keeps moving in the meantime: the
one-rack branch is already at 27.6%, with about 3% from coordinated
garbage collection (<a
href="https://github.com/marin-community/marin/pull/9266">#9266</a>) and
4% from the SM100 attention backward (<a
href="https://github.com/marin-community/marin/pull/9228">#9228</a>)
queued.</li>
<li>The dispatcher is Megatron's one structural advantage. HybridEP cuts
exclusive communication to about 6% of the step on NVL72, against 2.4 to
3.2 exposed seconds per 17-second step in Levanter, worth about 5
points. That argues for porting the kernel, not the framework.</li>
<li>Precision changes the picture more than the framework does. NVIDIA's
MXFP8 numbers run 1.2 to 1.5 times its BF16 numbers on the same
hardware; the hero is BF16 throughout by choice.</li>
</ul>
<h2 id="caveats">Caveats</h2>
<ul>
<li>No external run exists at the hero's expert count, at its width, or
at 704 GB200s in BF16; the shape adjustment is my inference from GEMM
sizes.</li>
<li>NVIDIA's 24.2% GB200 point, labelled an <a
href="https://developer.nvidia.com/blog/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72/">"early
software version"</a>, has no stated precision or routing.</li>
<li>The Marin trainable-lane number is a two-month-old snapshot after
two days of tuning; a rerun would gain, but by an unmeasured
amount.</li>
<li>Long context is a separate question. The only 65K-sequence hero
numbers are Levanter pipeline-parallel variants on 192 H100s at 15.5%
(<a
href="https://github.com/marin-community/marin/issues/9277">#9277</a>),
with no Megatron counterpart anywhere.</li>
</ul>
<p>Treat the medians as good to about plus or minus three points and the
ordering of the three scenarios as robust.</p>
</div>
<footer class="provenance">
<p><em>Corpus: 2026-09-22 05:29 UTC &middot; Generated: 2026-09-23 13:24 UTC &middot; Viewing: <span id="viewing-time"></span></em></p>
<blockquote>
<p><em>Data: marinmirror — 221146 chunks, built 2026-09-22 05:29 UTC ·
summaries through 2026-09-14_2026-09-20 (not refreshed); web sources
fetched 2026-09-23: arXiv 2603.07685 (PDF, Tables 11 and 17), NVIDIA
Megatron Bridge performance summary and archive, NVIDIA GB300 record
post, PyTorch TorchTitan B200 post, DeepSeek-V3 report (PDF), Nemotron 3
Ultra and Super reports (PDF), SemiAnalysis free section.</em></p>
<p><em>Query: "given the hero run 535B model architecture that we are
currently training on CoreWeave, what is the expected MFU we'd get if we
used PyTorch and Megatron in place of Levanter, informed by external
results on similar hardware?"</em></p>
<p><em>Sub-queries (corpus): "535B-A23B hero architecture and config" ·
"hero MFU definition and history" · "Marin's own Megatron usage" ·
"framework decision record" · "external MFU references in the corpus" ·
"kernel stack and step-time breakdown" · "pipeline parallelism results"
· "PyTorch trials in Marin" · "CoreWeave GB200 hardware" · "MoE dispatch
and transport" · "MuonH overhead" · code lane. Sub-queries (web):
Megatron-Core MoE report tables · Megatron Bridge performance pages ·
GB200/GB300 MoE record post · TorchTitan DeepSeek-V3 on B200 ·
DeepSeek-V3 GPU-hours · Nemotron 3 training throughput · MLPerf GB200
NVL72 · third-party GB200 MoE benchmarks.</em></p>
</blockquote>
</footer>
</div>
</body>
</html>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment