Skip to content

Instantly share code, notes, and snippets.

@belisarius222

belisarius222/imposter-results.html

Last active April 27, 2026 13:03
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save belisarius222/50822f4909ef1db97e72c8dd98d4b9a5 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save belisarius222/50822f4909ef1db97e72c8dd98d4b9a5 to your computer and use it in GitHub Desktop.
Download ZIP
IMPOSTER training results - Qwen chat-template from-scratch run (Apr 25, 2026); supersedes Apr 9 results. #pagedrop
Raw
imposter-results.html
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>IMPOSTER: chat-template run results &amp; span rubric</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Fraunces:opsz,wght@9..144,600;9..144,800&family=IBM+Plex+Mono:wght@400;600&family=Source+Serif+4:opsz,wght@8..60,400;8..60,600&display=swap" rel="stylesheet">
<style>
:root {
--ink: #17120d;
--muted: #6b5e50;
--paper: #f6efe2;
--panel: #fff9ed;
--line: #d8c7ad;
--red: #a8442e;
--green: #2f7550;
--blue: #285f7f;
--gold: #b78933;
--shadow: 0 24px 70px rgba(55, 38, 18, 0.18);
}
* { box-sizing: border-box; }
body {
margin: 0;
color: var(--ink);
background:
radial-gradient(circle at top left, rgba(183, 137, 51, 0.22), transparent 30rem),
linear-gradient(120deg, rgba(168, 68, 46, 0.08), transparent 24rem),
var(--paper);
font-family: "Source Serif 4", Georgia, serif;
line-height: 1.45;
}
main {
width: min(1120px, calc(100vw - 32px));
margin: 0 auto;
padding: 48px 0 72px;
}
.hero {
display: grid;
grid-template-columns: 1.2fr 0.8fr;
gap: 28px;
align-items: stretch;
margin-bottom: 28px;
}
.hero-card,
.card,
.prompt {
background: rgba(255, 249, 237, 0.86);
border: 1px solid var(--line);
border-radius: 24px;
box-shadow: var(--shadow);
}
.hero-card {
padding: 34px;
position: relative;
overflow: hidden;
}
.hero-card::after {
content: "";
position: absolute;
right: -70px;
top: -70px;
width: 220px;
height: 220px;
border: 1px solid rgba(183, 137, 51, 0.45);
border-radius: 50%;
}
.eyebrow {
font-family: "IBM Plex Mono", monospace;
font-size: 0.78rem;
letter-spacing: 0.12em;
text-transform: uppercase;
color: var(--blue);
font-weight: 600;
margin-bottom: 12px;
}
h1, h2, h3 {
font-family: "Fraunces", Georgia, serif;
line-height: 0.98;
margin: 0;
}
h1 {
font-size: clamp(2.4rem, 6.4vw, 5.2rem);
max-width: 780px;
letter-spacing: -0.055em;
}
h2 {
font-size: clamp(1.6rem, 3vw, 2.4rem);
letter-spacing: -0.035em;
margin-bottom: 14px;
}
h3 {
font-size: 1.25rem;
letter-spacing: -0.02em;
margin: 0 0 10px;
}
.lead {
max-width: 760px;
margin: 22px 0 0;
color: var(--muted);
font-size: 1.18rem;
}
.decision {
padding: 28px;
display: flex;
flex-direction: column;
justify-content: space-between;
gap: 18px;
}
.decision strong {
display: block;
font-family: "Fraunces", Georgia, serif;
font-size: 1.85rem;
line-height: 1.02;
letter-spacing: -0.04em;
}
.decision p {
margin: 0;
color: var(--muted);
}
.pill-row {
display: flex;
flex-wrap: wrap;
gap: 8px;
margin-top: 18px;
}
.pill {
border: 1px solid var(--line);
border-radius: 999px;
padding: 7px 10px;
background: rgba(255, 255, 255, 0.45);
font-family: "IBM Plex Mono", monospace;
font-size: 0.75rem;
color: #4b4035;
}
.grid {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 20px;
margin: 20px 0;
}
.grid-3 {
display: grid;
grid-template-columns: repeat(3, minmax(0, 1fr));
gap: 20px;
margin: 20px 0;
}
.card {
padding: 26px;
}
ul {
margin: 14px 0 0;
padding-left: 1.2rem;
}
li {
margin: 8px 0;
}
.good h2 { color: var(--green); }
.bad h2 { color: var(--red); }
.score {
margin-top: 20px;
overflow: hidden;
}
table {
width: 100%;
border-collapse: collapse;
font-size: 0.95rem;
}
th, td {
border-bottom: 1px solid var(--line);
padding: 12px 10px;
text-align: left;
vertical-align: top;
}
td.num, th.num { text-align: right; font-variant-numeric: tabular-nums; }
th {
font-family: "IBM Plex Mono", monospace;
font-size: 0.72rem;
text-transform: uppercase;
letter-spacing: 0.08em;
color: var(--blue);
background: rgba(40, 95, 127, 0.07);
}
tr:last-child td { border-bottom: 0; }
tr.best td { background: rgba(47, 117, 80, 0.08); }
tr.worst td { background: rgba(168, 68, 46, 0.06); }
.rule {
display: grid;
grid-template-columns: 180px 1fr;
gap: 18px;
align-items: start;
margin-top: 20px;
}
.threshold {
background: #172018;
color: #f6efe2;
border-radius: 22px;
padding: 22px;
box-shadow: var(--shadow);
}
.threshold code {
color: #d7f3d5;
font-family: "IBM Plex Mono", monospace;
font-size: 0.9rem;
}
.prompt {
margin-top: 20px;
padding: 0;
overflow: hidden;
}
pre {
margin: 0;
padding: 22px;
overflow-x: auto;
color: #efe8d8;
background: #181512;
font-family: "IBM Plex Mono", monospace;
font-size: 0.86rem;
line-height: 1.5;
}
.footer-note {
margin-top: 26px;
color: var(--muted);
font-size: 0.95rem;
text-align: center;
}
.section-title {
margin: 36px 0 4px;
}
.section-sub {
color: var(--muted);
margin: 0 0 8px;
max-width: 820px;
}
.stat {
font-family: "IBM Plex Mono", monospace;
font-size: 0.78rem;
color: var(--muted);
}
.verdict {
border-left: 4px solid var(--gold);
padding: 10px 16px;
margin-top: 14px;
background: rgba(183, 137, 51, 0.08);
border-radius: 8px;
color: var(--ink);
font-size: 0.98rem;
}
.recco {
background: #172018;
color: #f6efe2;
border-radius: 22px;
padding: 28px;
box-shadow: var(--shadow);
margin-top: 26px;
}
.recco h2 { color: #f6d984; }
.recco li { color: #efe8d8; }
.recco .eyebrow { color: #e1c778; }
@media (max-width: 820px) {
main { padding-top: 24px; }
.hero, .grid, .grid-3, .rule { grid-template-columns: 1fr; }
.hero-card, .card, .decision { padding: 22px; border-radius: 18px; }
table { font-size: 0.86rem; }
th, td { padding: 10px 8px; }
}
</style>
</head>
<body>
<main>
<section class="hero">
<article class="hero-card">
<div class="eyebrow">IMPOSTER &middot; chat-template run &middot; 2026-04-25</div>
<h1>The discriminator was fooled. The judges weren&rsquo;t.</h1>
<p class="lead">
IMPOSTER is a legal-text infill generator: given a passage from a legal opinion
with one paragraph removed, it tries to write a paragraph that fits between the
before-context and the after-context. The generator is Qwen2.5-72B-Instruct
plus a LoRA adapter, trained with a GRPO-style RL loop whose reward comes from a
small text classifier (the &ldquo;discriminator&rdquo;) that tries to tell the
model&rsquo;s paragraph apart from the original human paragraph.
</p>
<p class="lead">
This page reports the final-checkpoint evaluation of run
<span class="stat">grpo-lora-qwen-chat-template-from-scratch-20260424</span>: a
controlled rerun from scratch using Qwen&rsquo;s correct chat template at every
stage, with the discriminator architecture, co-training schedule, training data,
and held-out eval set held constant. Headline: the final LoRA fools the
discriminator more than earlier checkpoints, but GPT-4.1 and GPT-5.4 prefer
earlier-checkpoint and even base-Qwen text. That is a textbook
reward-model-overfitting / proxy-drift signature, not a real quality gain.
</p>
<div class="pill-row" aria-label="Run summary">
<span class="pill">Qwen2.5-72B-Instruct + LoRA</span>
<span class="pill">14400 / 14400 steps</span>
<span class="pill">5.6 s/step</span>
<span class="pill">chat template</span>
<span class="pill">co-trained discriminator</span>
</div>
<p class="stat" style="margin-top: 18px;">
Supersedes prior PageDrop (Apr 9, pre-chat-template fix):
<a href="https://pagedrop.ai/g/belisarius222/50822f4909ef1db97e72c8dd98d4b9a5">pagedrop.ai/g/belisarius222/50822f4909ef1db97e72c8dd98d4b9a5</a>.
</p>
</article>
<aside class="decision card">
<div>
<div class="eyebrow">Decision</div>
<strong>Freeze this run. Step 6500 stays the best LoRA. Build a better eval before more RL.</strong>
</div>
<p>
Final improved every discriminator-style metric and lost on every LLM-judge metric.
That&rsquo;s reward-model overfitting / style drift, not better legal infill.
</p>
</aside>
</section>
<h2 class="section-title">Executive summary</h2>
<p class="section-sub">
One screen, plain language. Everything below this section unpacks these five points
in detail.
</p>
<article class="card" style="margin-top: 14px;">
<ol style="margin: 0 0 0 1.2rem; padding: 0;">
<li style="margin-bottom: 10px;">
<strong>What IMPOSTER is.</strong>
A Qwen2.5-72B-Instruct legal-text infill generator. It receives a chunk of a legal
opinion with one paragraph elided, plus the surrounding before / after context,
and is asked to write a paragraph that fits in the missing slot. The model is
trained with LoRA adapters using a GRPO-style RL loop. The reward signal is the
score of a small DeBERTa-style classifier (the &ldquo;span discriminator&rdquo;)
that tries to distinguish model paragraphs from human paragraphs.
</li>
<li style="margin-bottom: 10px;">
<strong>Why this run exists.</strong>
A previous audit found the original training/eval prompt path was malformed: it
used a raw causal-completion string with truncation that could drop part of the
gap or the after-context, which made base Qwen look weaker than it really is. The
fix is to use Qwen&rsquo;s built-in chat template at every stage. This run is a
controlled retrain from scratch with the chat template applied everywhere and the
rest of the stack (discriminator, co-training, LR, data, eval set, seed) held
constant. The only intentional change is the prompt format.
</li>
<li style="margin-bottom: 10px;">
<strong>What happened in training.</strong>
14400/14400 outer steps over ~57k seconds (5.6 s/step), 10248 of which the
trainer counted as &ldquo;successful&rdquo; (the rest were sampled prompts whose
generations failed and were dropped). Periodic LoRA merges fed back into vLLM 29
times; discriminator co-training fired 5 of 7 scheduled times (two late intervals
failed on vLLM-side errors but training continued). The run completed cleanly.
</li>
<li style="margin-bottom: 10px;">
<strong>What the evals say.</strong>
Three lenses on 100 held-out prompts, plus pairwise LLM judging. Frozen
discriminator: <em>final is the best.</em> Fresh-from-scratch classifier head:
<em>final is the hardest to separate from human, but only marginally better than
step 6500.</em> GPT-4.1 pairwise (30 prompts): <em>final loses to step 6500 9-19
and slightly loses to base 13-16.</em> GPT-5.4 xhigh spot check (10 prompts):
<em>final loses 0-10 to the human reference</em> (step 6500 had also lost 0-10
on the prior version of this spot check).
</li>
<li>
<strong>What follows.</strong>
The contradiction between &ldquo;discriminator is happier&rdquo; and
&ldquo;judges are unhappier&rdquo; is itself the result. Continuing to train
against this discriminator-only reward is now optimizing past the point where
the proxy tracks quality. The best LoRA checkpoint for any user-visible follow-up
is <strong>step 6500</strong>, not final. The next investment is a better eval
and reward stack &mdash; tighter span selection, a human-anchored LLM judge,
adversarial data diversity, and held-out-judge early-stopping &mdash; not more
training on the same reward.
</li>
</ol>
</article>
<h2 class="section-title">Primer: the task and the actors</h2>
<p class="section-sub">
What problem the model is solving and who the players are. Skip if you already know.
</p>
<section class="grid">
<article class="card">
<h3>The task: legal-text infill</h3>
<p>
Each example is a passage from a publicly-available legal opinion in which one
paragraph (the &ldquo;span&rdquo;) has been removed. The model receives the text
before the missing span and the text after, and must write a paragraph that
plausibly fills the gap. We score each candidate paragraph against the
<em>original</em> paragraph that was removed (the &ldquo;human reference&rdquo;).
</p>
<p>
A good span is constrained by context: facts named in the before-text need to be
carried forward, parties and dates need to stay consistent, the procedural posture
matters, and the legal reasoning needs to bridge what comes before and what comes
after. A weak span (a heading, a signature block, a citation list) can be
completed by surface pattern-matching and tells us little about quality &mdash;
which is exactly the reason this page ends with a span-selection rubric.
</p>
</article>
<article class="card">
<h3>The actors compared</h3>
<ul>
<li><strong>Human reference</strong> &mdash; the actual paragraph that was removed
from the opinion. The gold standard each candidate is compared against.</li>
<li><strong>Base Qwen (chat)</strong> &mdash; Qwen2.5-72B-Instruct with no
IMPOSTER training, prompted via Qwen&rsquo;s chat template. This is the
honest baseline; its older raw-prompt counterpart was the malformed run.</li>
<li><strong>LoRA step 6500</strong> &mdash; the IMPOSTER LoRA adapter at training
step 6500 of this run, applied on top of base Qwen. The best LoRA so far on
user-visible quality.</li>
<li><strong>LoRA final</strong> &mdash; the LoRA adapter at the end of training
(step 14400). Trained the longest against the discriminator reward.</li>
<li><strong>Span discriminator</strong> &mdash; a small DeBERTa-style classifier
that, given before-context + a candidate paragraph + after-context, outputs
a logit estimating &ldquo;is this paragraph human?&rdquo;. The training
reward for the LoRA is roughly &ldquo;raise that logit.&rdquo;</li>
<li><strong>LLM judges</strong> &mdash; capable language models (GPT-4.1 and
GPT-5.4) presented with two candidate paragraphs (anonymized A/B) and asked
which one is the better continuation of the legal context. They are a
<em>quality</em> proxy, not a detection proxy.</li>
</ul>
</article>
</section>
<h2 class="section-title">Glossary</h2>
<p class="section-sub">
Just enough to read the rest of the page. Skim the ones you don&rsquo;t know.
</p>
<section class="card score">
<table>
<thead>
<tr>
<th style="width: 28%;">Term</th>
<th>Meaning in this report</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Qwen2.5-72B-Instruct</strong></td>
<td>The 72-billion-parameter open-weights chat model from Alibaba used as the frozen base. &ldquo;Instruct&rdquo; means it was fine-tuned to follow chat-formatted messages.</td>
</tr>
<tr>
<td><strong>Qwen chat template</strong></td>
<td>Qwen&rsquo;s expected message-rendering format (system / user / assistant turns wrapped with specific tokens). Using a raw concatenated string instead of this template degrades the model&rsquo;s output. The original IMPOSTER prompt path did exactly that; the chat-template fix uses the official template at every stage.</td>
</tr>
<tr>
<td><strong>LoRA</strong></td>
<td>Low-Rank Adaptation. A small set of trainable matrices added on top of a frozen base model so we can adapt behavior without touching the 72B weights. Cheap to train, cheap to swap, easy to merge into vLLM for inference.</td>
</tr>
<tr>
<td><strong>GRPO-style loop</strong></td>
<td>The RL loop used here: sample several candidate paragraphs per prompt, score them with the discriminator (and any auxiliary signals), and update the LoRA so that higher-scored samples get more probability. &ldquo;-style&rdquo; because the implementation is the team&rsquo;s in-house variant, not a literal call to a published GRPO library.</td>
</tr>
<tr>
<td><strong>Span discriminator</strong></td>
<td>The small DeBERTa-style binary classifier that says &ldquo;human or machine&rdquo; for a candidate paragraph in context. Its score (or a transformed version) is the RL reward.</td>
</tr>
<tr>
<td><strong>Co-training</strong></td>
<td>Periodically retraining the discriminator on fresh model outputs so it doesn&rsquo;t go stale as the LoRA improves. Scheduled every 2000 steps in this run.</td>
</tr>
<tr>
<td><strong>step 6500 vs final</strong></td>
<td>Two LoRA snapshots of <em>this same run</em>. Step 6500 is a mid-training checkpoint; final is the last checkpoint at step 14400. Comparing them shows what continued training after step 6500 actually changed.</td>
</tr>
<tr>
<td><strong>Frozen discriminator</strong></td>
<td>Take the in-loop discriminator from training, freeze its weights, and re-score held-out prompts with it. Measures &ldquo;how well did the LoRA optimize the proxy that was driving its training.&rdquo;</td>
</tr>
<tr>
<td><strong>Fresh head</strong></td>
<td>A new classifier head trained from scratch on a small split (70 records train / 30 records eval) for each candidate model. Less biased toward the in-loop discriminator&rsquo;s exact state, so it&rsquo;s a partially independent &ldquo;is this still detectable&rdquo; check.</td>
</tr>
<tr>
<td><strong>Confused rate</strong></td>
<td>Fraction of held-out examples where the discriminator&rsquo;s confidence in &ldquo;machine&rdquo; was effectively a coin-flip (rather than confidently classifying it as machine). Higher = better for the candidate.</td>
</tr>
<tr>
<td><strong>Logit gap</strong></td>
<td>(Discriminator logit on the human paragraph) minus (logit on the candidate paragraph). Larger gap = the discriminator is more sure the human is real and the candidate is fake. Smaller gap = the candidate is harder to tell apart.</td>
</tr>
<tr>
<td><strong>AUC</strong></td>
<td>Area Under the ROC Curve for the fresh-head classifier. 0.5 = chance, 1.0 = perfect separation. Lower AUC = the candidate fools small classifiers more often.</td>
</tr>
<tr>
<td><strong>Beats human</strong></td>
<td>Fraction of held-out examples where the discriminator scored the candidate as <em>more</em> human-like than the actual human paragraph. Mostly a curiosity; small numbers here are noisy.</td>
</tr>
<tr>
<td><strong>GPT-4.1 / GPT-5.4 judge</strong></td>
<td>Capable LLMs prompted to compare two anonymized paragraphs (A/B) given the same legal context, and pick the better continuation, returning a confidence 1&ndash;5. GPT-5.4 here uses xhigh reasoning effort (one prompt fell back to high after empty responses on retry).</td>
</tr>
<tr>
<td><strong>vLLM</strong></td>
<td>The high-throughput open-source inference server used to generate samples during RL training and during eval. Periodically reloads merged LoRA weights so the policy used for sampling tracks the policy being trained.</td>
</tr>
<tr>
<td><strong>Volta2</strong></td>
<td>The team&rsquo;s artifact / metric-logging stack. Used here to upload checkpoints and eval bundles to GCS. <em>Note:</em> the in-process metric-logging client failed during this run due to a missing OpenSSL on the training host, but the separate Volta2 uploader daemon worked, so artifacts still landed.</td>
</tr>
</tbody>
</table>
</section>
<h2 class="section-title">Why we reran from scratch</h2>
<p class="section-sub">
The story in three steps. Each one reframes what we thought we knew about IMPOSTER.
</p>
<article class="card" style="margin-top: 14px;">
<h3>1. The original headline relied on a malformed prompt</h3>
<p>
The very first IMPOSTER training and eval pipeline didn&rsquo;t use Qwen&rsquo;s
chat template. Instead, it concatenated the before-context, a placeholder for the
missing span, and the after-context into a single raw string and asked Qwen to
continue from there. That path also had a truncation bug: in long examples, the
truncation could chop out part of the gap or the after-context entirely. The
practical effect was that base Qwen sometimes saw a corrupted prompt while the
LoRA, having been trained against the same corrupt prompt, had effectively
memorized how to behave under it. The classic outcome: an inflated apparent
difference between LoRA and base on the in-loop discriminator.
</p>
</article>
<article class="card" style="margin-top: 14px;">
<h3>2. The corrected chat-template eval shrank that gap</h3>
<p>
Re-prompting <em>only the eval</em> with Qwen&rsquo;s official chat template (so
base Qwen was given a fair shot) closed most of the original LoRA-vs-base
frozen-discriminator gap. After correction, base Qwen confused the discriminator
43% of the time and step 6500 confused it 48% &mdash; a marginal lead, not the
large win the malformed-prompt eval had reported. GPT-4.1 pairwise comparison
between corrected base and corrected step 6500 was similarly tighter than before.
</p>
<p>
That left an open question: <em>was the chat-template fix only changing
evaluation, or would training itself benefit from being run end-to-end on the
corrected prompt path?</em>
</p>
</article>
<article class="card" style="margin-top: 14px;">
<h3>3. So we reran training from scratch &mdash; with everything else held constant</h3>
<p>
The April 25 run is the controlled answer. The Qwen chat template is now applied
consistently in training prompts, in the vLLM generation server that produces
rollouts, and in evaluation. Everything else &mdash; data, eval set, seed, LoRA
config, discriminator architecture, co-training schedule, and reward shape &mdash;
is held constant so that the chat-template fix is the only intentional change.
That keeps the comparison clean: any improvement <em>or</em> regression is
attributable to training-time prompt format, not to swapping rewards or models.
</p>
</article>
<h2 class="section-title">What was held constant, and why each thing matters</h2>
<p class="section-sub">
Each held-constant choice is here for a reason. Changing any of them at the same
time as the chat-template fix would have made the result uninterpretable.
</p>
<section class="card score">
<table>
<thead>
<tr>
<th style="width: 32%;">Held constant</th>
<th>Why it matters</th>
</tr>
</thead>
<tbody>
<tr>
<td>Generator base = Qwen2.5-72B-Instruct</td>
<td>The point of the experiment is to evaluate the chat-template fix on the same model family the prior run used. Swapping the base model would confound the prompt-format change with a base-quality change.</td>
</tr>
<tr>
<td>Same GRPO-style training code path</td>
<td>If we changed the RL algorithm, we couldn&rsquo;t separate the effect of the prompt fix from the effect of the new optimizer behavior.</td>
</tr>
<tr>
<td>LoRA rank 16; max completion length 64</td>
<td>The capacity of the adapter and the length budget at sampling time both affect what the policy can learn. Holding them constant lets us read changes as policy behavior, not capacity.</td>
</tr>
<tr>
<td>Same learning rate &amp; optimizer config</td>
<td>Inherited from the previous productive IMPOSTER run. Sweeping LR alongside a prompt fix would have been a two-variable experiment.</td>
</tr>
<tr>
<td>Same span discriminator (small DeBERTa-style)</td>
<td>The discriminator <em>is</em> the reward. Strengthening it would make the run a different experiment about reward-model power, not about prompt format.</td>
</tr>
<tr>
<td>Same co-training cadence (every 2000 steps)</td>
<td>Co-training keeps the discriminator from going stale. Changing the cadence simultaneously would mix &ldquo;fresher reward&rdquo; into the chat-template result.</td>
</tr>
<tr>
<td>Same training corpus and same 100-prompt held-out eval set, same seed</td>
<td>Base, step 6500, and final are evaluated on the <em>same</em> examples in the same order, so all three columns of every results table are directly comparable.</td>
</tr>
<tr>
<td>Same eval protocol as the corrected step-6500 chat-template eval</td>
<td>This run is meant to slot into the existing comparison. The corrected step-6500 numbers in the tables below are unchanged from the prior eval; only the &ldquo;final&rdquo; column is new.</td>
</tr>
</tbody>
</table>
</section>
<h2 class="section-title">What changed, and why each change was needed</h2>
<p class="section-sub">
The intentional change is the prompt path. The other changes are operational hygiene
that protect the experiment from being polluted by old state.
</p>
<section class="card score">
<table>
<thead>
<tr>
<th style="width: 32%;">Change</th>
<th>Why it was needed</th>
</tr>
</thead>
<tbody>
<tr>
<td>Qwen chat template applied at every stage</td>
<td>The single intentional change. Training prompts, vLLM generation, and eval prompts all now use the official Qwen chat-formatting (system/user/assistant tokens) instead of a raw concatenated string. This is what the experiment is testing.</td>
</tr>
<tr>
<td><span class="stat">IMPOSTER_DISABLE_AUTO_RESUME=1</span></td>
<td>The training framework normally auto-resumes from the latest local checkpoint. We had pre-fix checkpoints sitting around; auto-resume would have silently mixed them into a &ldquo;from-scratch&rdquo; run. This env var disables that path so the run truly starts at step 0.</td>
</tr>
<tr>
<td>Disabled stale IMPOSTER cron jobs on the orchestrator</td>
<td>The orchestrator host had cron entries from earlier IMPOSTER work that could have relaunched stale runs or uploaded outdated artifacts mid-experiment. Disabling them removes that source of contamination from the artifact stream.</td>
</tr>
<tr>
<td>Volta2 retention naming fix (commit <span class="stat">bcf4029</span>)</td>
<td>Earlier checkpoint artifacts had the timestamp at the start of the filename, so the retention policy treated each one as a separate slot family and kept too many. Moving the timestamp to the end (<span class="stat">checkpoint_lora_*_TIMESTAMP.tgz</span>) lets retention recognize all LoRA checkpoints as one family. The training run was stopped right after step 4000, the fix was deployed, and the run resumed explicitly from <span class="stat">lora_step_4000</span> &mdash; so retention behavior is correct for steps 4500 onward without changing the modeling outcome.</td>
</tr>
</tbody>
</table>
</section>
<h2 class="section-title">Run timeline &amp; operational notes</h2>
<p class="section-sub">
How the training actually executed. None of the operational issues below changed the
run&rsquo;s scientific premise &mdash; the comparison between base, step 6500, and
final remains valid &mdash; but they affect how to interpret &ldquo;successful
steps&rdquo; and how often the co-training schedule actually fired.
</p>
<article class="card" style="margin-top: 14px;">
<h3>Topology &amp; launch</h3>
<p>
Training is a two-host setup: one host runs the LoRA policy update and the
discriminator, the other host runs <em>vLLM</em> serving the current policy for
sampling rollouts. Splitting them keeps the policy update step from contending
with high-throughput inference for GPU memory. vLLM initially serves the
un-adapted base Qwen; whenever the trainer hits a checkpoint+merge interval, the
latest LoRA is merged into the base weights, vLLM is restarted with the merged
weights, and rollouts thereafter come from the new policy. That merge+restart
cycle ran 29 times across this run.
</p>
<p>
The first launch attempt used <em>torchrun / DDP</em> (multi-process distributed
training): one rank crashed with an OOM because the discriminator had to coexist
with a policy shard on the same GPU. A CPU-discriminator variant of the torchrun
path was tried as a fallback, but it ran orders of magnitude too slow because the
discriminator forward pass became the throughput bottleneck. The launch that
actually completed reverted to the same single-process
<span class="stat">device_map=auto</span> setup used by the previous productive
IMPOSTER run &mdash; this fits the model + adapter + discriminator into memory and
matches the comparator run&rsquo;s training shape, which is what we want for an
apples-to-apples comparison.
</p>
</article>
<article class="card" style="margin-top: 14px;">
<h3>Cadence &amp; artifacts</h3>
<p>
The trainer saves a LoRA checkpoint every 500 steps and runs a discriminator
co-training pass every 2000 steps. Co-training is itself a small training run for
the discriminator: pull a fresh batch of model rollouts, label them as machine,
pair them with human paragraphs, and update the classifier so it stays useful as
the policy improves.
</p>
<ul>
<li>Final LoRA saved <span class="stat">2026-04-25 19:51:41 UTC</span>; the
corresponding merge into vLLM completed <span class="stat">19:59:32 UTC</span>.</li>
<li>Training-complete log line:
<em>&ldquo;Training complete in 57187s (10248 successful steps, 5.6 s/step).&rdquo;</em>
The outer loop reached the configured 14400/14400 steps; the lower
&ldquo;successful steps&rdquo; count reflects iterations where one or more
sampled rollouts came back invalid (vLLM error, length-zero, format-broken)
and were dropped by the trainer instead of being retried. <strong>Read this
as:</strong> the run completed cleanly, but ~29% of attempted iterations
contributed nothing to the gradient.</li>
<li>Run-level log counts: 1424 GRPO log lines, 29 successful merge completions,
<strong>5 of 7 scheduled co-training passes succeeded</strong>, 2 co-training
attempts failed on vLLM-side errors (at the step 10000 and step 14000
intervals), 1 routine vLLM &ldquo;failed&rdquo; warning of the kind that
normally appears under load.</li>
<li>Volta2 artifact uploads worked throughout. Side note: the in-process
<span class="stat">volta.init</span> metric-logging client failed during the
run because the training host was missing an OpenSSL library, but the separate
Volta2 uploader daemon was unaffected, so checkpoints and the eval bundle
still landed in GCS.</li>
<li>Final LoRA checkpoint path:
<span class="stat">/scratch/imposter/runs/grpo-lora-qwen-chat-template-from-scratch-20260424/checkpoints/generator/lora_final</span>.</li>
<li>Final artifact name: <span class="stat">checkpoint_lora_final_1777146701505.tgz</span>.</li>
<li>Eval bundle uploaded to GCS:
<span class="stat">gs://volta-489906-artifacts/imposter/grpo-lora-qwen-chat-template-from-scratch-20260424/keep_eval_lora_final_chat/1777154851289_keep_eval_lora_final_chat_20260425_180722.tgz</span>.</li>
<li>Local combined eval summary:
<span class="stat">/Users/ted/v/imposter/evals/chat_final_20260425/summary_all.json</span>.</li>
</ul>
</article>
<article class="card" style="margin-top: 14px;">
<h3>Caveats worth flagging up front</h3>
<ul>
<li><strong>Co-training did not fire on schedule late in the run.</strong> Two of
the seven scheduled co-training passes failed (steps 10000 and 14000) due to
vLLM-side generation errors. Training continued, so the LoRA kept updating
against an in-loop discriminator that was slightly more stale by the end of
the run than by mid-run. If you read the late-run discriminator-confusion
climb and want to argue &ldquo;some of that gain is the discriminator getting
stale rather than the policy getting better,&rdquo; this is the lever.</li>
<li><strong>Scope of the result.</strong> The numbers below are evidence about
<em>this specific reward stack</em> (this discriminator + this co-training
schedule + this prompt format). They do not bound how good Qwen + LoRA could
be on legal infill under a different reward design.</li>
<li><strong>No humans scored these completions.</strong> All quality claims rest
on LLM judges. The recommendation below treats GPT-4.1 (30 prompts, blind
pairwise) as the load-bearing user-visible-quality signal and the GPT-5.4
xhigh 10-prompt run as a coarse ceiling check.</li>
<li><strong>Restart from step 4000 was operational, not scientific.</strong> The
run was paused right after step 4000 to deploy the Volta retention-naming fix,
then resumed explicitly from <span class="stat">lora_step_4000</span>. The
modeling trajectory is the same; only the artifact retention behavior changed.</li>
</ul>
</article>
<h2 class="section-title">How to read the evals</h2>
<p class="section-sub">
Three lenses, each measuring something different. The headline of this report is
that they <em>disagree</em>. Below, for each lens, what it measures, what it can
tell us, and what it can&rsquo;t.
</p>
<div class="grid-3">
<article class="card">
<h3>Frozen discriminator (n = 100)</h3>
<p>
Take the in-loop discriminator at the end of training, freeze it, and re-score
held-out prompts. Each row is a candidate paragraph (from base, step 6500, or
final) plus the corresponding human paragraph; we record both logits.
</p>
<p style="margin-top: 8px;"><strong>What it can tell us:</strong> how well training
moved the policy in the direction the reward function was pointing. If the LoRA
improved on this metric, the RL step actually had effect.</p>
<p style="margin-top: 8px;"><strong>What it can&rsquo;t tell us:</strong> whether
the policy improved on anything <em>other than</em> what this discriminator
measures. Since the LoRA was trained against a near-identical signal, this metric
is the most likely to be overfit. Don&rsquo;t treat it as a quality benchmark.</p>
</article>
<article class="card">
<h3>Fresh discriminator head (70 train / 30 eval)</h3>
<p>
For each candidate model, hold out 30 prompts. Use the remaining 70 to train a
brand-new classifier head from scratch on (human paragraph, candidate paragraph)
pairs. Then evaluate that fresh head on the 30 held-out prompts. Reported
numbers: accuracy and AUC of that fresh head.
</p>
<p style="margin-top: 8px;"><strong>What it can tell us:</strong> whether
<em>any</em> small classifier &mdash; not just the one IMPOSTER was trained
against &mdash; can still tell the candidate apart from human. Insensitive to the
in-loop discriminator&rsquo;s exact state.</p>
<p style="margin-top: 8px;"><strong>What it can&rsquo;t tell us:</strong> it&rsquo;s
still a small classifier in the same model family / vocabulary as the reward
model. A persuasive paragraph that fools the fresh head can still read as wrong
to a strong language-model judge or to a lawyer. Treat AUC as a
&ldquo;detectability&rdquo; metric, not a quality metric.</p>
</article>
<article class="card">
<h3>LLM judges (GPT-4.1, GPT-5.4)</h3>
<p>
Anonymized pairwise judging. The judge is shown the before-context, the
after-context, and two candidate paragraphs labeled A and B (one is the human
reference or another candidate, one is the model under test) and is asked which
paragraph is the better continuation, with a 1&ndash;5 confidence rating.
</p>
<p style="margin-top: 8px;"><strong>What it can tell us:</strong> the closest
available proxy for user-visible quality. The judge is reasoning about legal
style, factual continuity, internal consistency, and whether the paragraph
actually fits between the before- and after-text. GPT-4.1 over 30 prompts is the
load-bearing user-visible signal here. GPT-5.4 xhigh over 10 prompts is a
stricter spot check.</p>
<p style="margin-top: 8px;"><strong>What it can&rsquo;t tell us:</strong> this
is an LLM, not a lawyer. It can be biased by stylistic cues, length, and
formatting. Treat 30-prompt and 10-prompt judge runs as directional, not
statistically tight.</p>
</article>
</div>
<h2 class="section-title">Frozen discriminator (n = 100)</h2>
<p class="section-sub">
Same 100 held-out prompts, same seed, same protocol as the prior corrected
chat-template eval &mdash; only the &ldquo;final&rdquo; row is new. Each candidate
paragraph is scored by the same in-loop discriminator the LoRA was trained against.
Higher <em>confused</em> and lower <em>logit gap</em> mean the candidate is harder
for that discriminator to tell apart from the human paragraph. Remember:
<em>this is the metric the run was optimizing.</em>
</p>
<article class="card" style="margin: 14px 0;">
<h3>Column glossary</h3>
<ul>
<li><strong>Cand. logit</strong> &mdash; the discriminator&rsquo;s raw score on the candidate paragraph. Higher = the model thinks it looks more human.</li>
<li><strong>Logit gap</strong> &mdash; (logit on the human paragraph) &minus; (logit on the candidate). Smaller = the discriminator is less sure which one is real.</li>
<li><strong>Cand. p</strong> &mdash; same logit, run through a sigmoid; the discriminator&rsquo;s probability that the candidate is human.</li>
<li><strong>Confused</strong> &mdash; fraction of examples where the discriminator was effectively a coin-flip on the candidate. Higher = better for the candidate.</li>
<li><strong>Gap &lt; 5 / Gap &lt; 10</strong> &mdash; fraction of examples where the gap between the human and candidate logit is below 5 / below 10. Both rise as the candidate gets harder to detect.</li>
<li><strong>Beats human</strong> &mdash; fraction of examples where the discriminator scored the candidate as <em>more</em> human-like than the actual human paragraph. A noisy curiosity at this sample size; don&rsquo;t over-read.</li>
</ul>
</article>
<section class="card score">
<table>
<thead>
<tr>
<th>Model</th>
<th class="num">Cand. logit</th>
<th class="num">Logit gap</th>
<th class="num">Cand. p</th>
<th class="num">Confused</th>
<th class="num">Gap &lt; 5</th>
<th class="num">Gap &lt; 10</th>
<th class="num">Beats human</th>
</tr>
</thead>
<tbody>
<tr>
<td>Base Qwen (chat)</td>
<td class="num">1.41</td>
<td class="num">8.37</td>
<td class="num">0.439</td>
<td class="num">43%</td>
<td class="num">31%</td>
<td class="num">54%</td>
<td class="num">18%</td>
</tr>
<tr>
<td>LoRA step 6500</td>
<td class="num">1.50</td>
<td class="num">8.27</td>
<td class="num">0.513</td>
<td class="num">48%</td>
<td class="num">37%</td>
<td class="num">59%</td>
<td class="num">19%</td>
</tr>
<tr class="best">
<td>LoRA final</td>
<td class="num">2.54</td>
<td class="num">7.23</td>
<td class="num">0.596</td>
<td class="num">61%</td>
<td class="num">49%</td>
<td class="num">65%</td>
<td class="num">21%</td>
</tr>
</tbody>
</table>
<p class="verdict">
<strong>Read this as:</strong> training did exactly what RL training is supposed
to do &mdash; it moved the policy in the direction the reward function was
pointing. Confused jumps 43% &rarr; 48% &rarr; 61%, logit gap shrinks 8.37 &rarr;
7.23, and candidate-p climbs from 0.44 to 0.60. Final wins this metric cleanly.
The interesting question is whether that gain corresponds to anything outside the
reward model. The next two sections answer no.
</p>
</section>
<h2 class="section-title">Fresh discriminator head (train 70 / eval 30)</h2>
<p class="section-sub">
For each candidate model, a brand-new classifier head is trained from scratch on 70
prompts (140 paired items, 10 epochs), then evaluated on 30 held-out prompts (60
items). This is partially independent of the in-loop discriminator: the head is
reset, but the architecture and feature space are similar. It tests whether
&ldquo;any small classifier&rdquo; can still tell the candidate apart from human,
not just the specific reward model that was driving training.
</p>
<article class="card" style="margin: 14px 0;">
<h3>Column glossary</h3>
<ul>
<li><strong>Eval acc.</strong> &mdash; accuracy of the fresh head on the 60-item eval set. Lower is better for the candidate (means the classifier is wrong more often).</li>
<li><strong>AUC</strong> &mdash; area under the ROC curve over the eval set. 0.5 = chance, 1.0 = perfect separation. Lower = candidate is harder to detect.</li>
<li><strong>Confused</strong> &mdash; fraction of held-out candidate items the fresh head couldn&rsquo;t classify confidently as machine.</li>
<li><strong>Beats human</strong> &mdash; fraction of held-out candidate items scored more human than their paired human paragraph. Same noisy-curiosity caveat as in the previous table.</li>
<li><strong>Logit gap</strong> &mdash; same definition as the frozen-discriminator table, but using the fresh head&rsquo;s scores.</li>
</ul>
</article>
<section class="card score">
<table>
<thead>
<tr>
<th>Model</th>
<th class="num">Eval acc.</th>
<th class="num">AUC</th>
<th class="num">Confused</th>
<th class="num">Beats human</th>
<th class="num">Logit gap</th>
</tr>
</thead>
<tbody>
<tr>
<td>Base Qwen (chat)</td>
<td class="num">76.7%</td>
<td class="num">0.826</td>
<td class="num">23.3%</td>
<td class="num">13.3%</td>
<td class="num">1.244</td>
</tr>
<tr>
<td>LoRA step 6500</td>
<td class="num">73.3%</td>
<td class="num">0.757</td>
<td class="num">30.0%</td>
<td class="num">23.3%</td>
<td class="num">0.973</td>
</tr>
<tr class="best">
<td>LoRA final</td>
<td class="num">70.0%</td>
<td class="num">0.756</td>
<td class="num">40.0%</td>
<td class="num">23.3%</td>
<td class="num">0.837</td>
</tr>
</tbody>
</table>
<p class="verdict">
<strong>Read this as:</strong> a fresh classifier confirms that final is harder to
separate from human than base or step 6500 (eval accuracy drops 76.7% &rarr; 73.3%
&rarr; 70.0%, confused rate climbs 23.3% &rarr; 30.0% &rarr; 40.0%) &mdash; but
AUC is essentially flat between step 6500 and final (0.7567 vs 0.7556). In other
words, most of the gain a fresh head can detect was already there at step 6500;
the additional 7900 training steps mostly paid out on the in-loop discriminator&rsquo;s
idiosyncratic surface, not on a deeper detectability change. That is consistent
with proxy drift.
</p>
</section>
<h2 class="section-title">LLM judges</h2>
<p class="section-sub">
Pairwise judging by capable language models. The judge is shown the surrounding
legal context and two anonymized candidate paragraphs (A and B), and is asked which
paragraph is the better continuation, with a 1&ndash;5 confidence score. This is
where we test &ldquo;is the text actually better,&rdquo; not &ldquo;is it harder to
detect.&rdquo; <strong>This is the metric that disagrees with the discriminator
story.</strong>
</p>
<div class="grid">
<article class="card">
<h3>GPT-4.1 pairwise (30 prompts)</h3>
<table>
<thead>
<tr>
<th>Pairing</th>
<th class="num">A</th>
<th class="num">B</th>
<th class="num">Tie</th>
<th class="num">Win rate</th>
</tr>
</thead>
<tbody>
<tr>
<td>Final (A) vs base (B)</td>
<td class="num">13</td>
<td class="num">16</td>
<td class="num">1</td>
<td class="num">44.8%</td>
</tr>
<tr class="worst">
<td>Final (A) vs step 6500 (B)</td>
<td class="num">9</td>
<td class="num">19</td>
<td class="num">2</td>
<td class="num">32.1%</td>
</tr>
<tr>
<td>Human (A) vs final (B)</td>
<td class="num">20</td>
<td class="num">9</td>
<td class="num">1</td>
<td class="num">31.0%<sup>*</sup></td>
</tr>
<tr>
<td>Human vs step 6500 (prior)</td>
<td class="num">18</td>
<td class="num">12</td>
<td class="num">0</td>
<td class="num">40.0%<sup>*</sup></td>
</tr>
<tr>
<td>Human vs base (prior)</td>
<td class="num">23</td>
<td class="num">6</td>
<td class="num">1</td>
<td class="num">20.7%<sup>*</sup></td>
</tr>
</tbody>
</table>
<p class="stat" style="margin-top:8px"><sup>*</sup> Model win rate vs human, ties excluded. Mean confidence 4.57&ndash;4.70.</p>
</article>
<article class="card">
<h3>GPT-5.4 spot check (10 prompts, mostly xhigh)</h3>
<table>
<thead>
<tr>
<th>Pairing</th>
<th class="num">Human</th>
<th class="num">Model</th>
<th class="num">Tie</th>
</tr>
</thead>
<tbody>
<tr><td>Human vs LoRA final</td><td class="num">10</td><td class="num">0</td><td class="num">0</td></tr>
<tr><td>Human vs LoRA step 6500 (prior)</td><td class="num">10</td><td class="num">0</td><td class="num">0</td></tr>
</tbody>
</table>
<p class="verdict">
<strong>Read this as:</strong> on the strictest available judge, both LoRA
checkpoints are shut out by the human reference. GPT-5.4 xhigh, given a 10-prompt
slice, never preferred either LoRA over the human paragraph. This run cannot
distinguish step 6500 from final using GPT-5.4 (both went 0/10), so don&rsquo;t
treat this as a comparison between checkpoints; treat it as a ceiling check
saying neither LoRA is anywhere near human-quality at this sample size.
</p>
</article>
</div>
<article class="card" style="margin-top: 18px;">
<h3>How to read the GPT-4.1 numbers</h3>
<ul>
<li><strong>Final vs base, 13&ndash;16&ndash;1.</strong> 30 prompts. Final loses to
corrected base by three. Win rate excluding ties is 44.8%. <em>Read this as:</em>
a near-draw with a slight lean toward base. Don&rsquo;t over-state &mdash; the
sample is small and the absolute gap is three judgments &mdash; but base is
certainly not <em>worse</em> than final on this judge.</li>
<li><strong>Final vs step 6500, 9&ndash;19&ndash;2.</strong> 30 prompts. Final
loses to step 6500 by ten judgments. Win rate excluding ties is 32.1%. This
<em>is</em> a meaningful regression at this sample size; it is the headline
finding.</li>
<li><strong>Human vs final, 20&ndash;9&ndash;1.</strong> 30 prompts. The human
reference is preferred about 7&ndash;3 over final excluding ties (model win
rate 31.0%). For comparison, the prior corrected eval had human vs step 6500
at 18&ndash;12 (model win rate 40.0%) and human vs base at 23&ndash;6 with one
tie (base model win rate 20.7%).</li>
<li><strong>What this means in plain language:</strong> compared to base, final
beats the human a little more often (model win rate 31.0% vs 20.7%). Compared
to step 6500, final beats the human <em>less</em> often (31.0% vs 40.0%).
Whatever final got better at, it isn&rsquo;t making the paragraph more
human-preferred than step 6500 was.</li>
<li><strong>Confidence numbers</strong> sit in the 4.57&ndash;4.70 range out of
5, meaning the judge was generally decisive on the comparisons it made &mdash;
this isn&rsquo;t a pile of low-confidence coin-flips.</li>
</ul>
</article>
<h2 class="section-title">Text-shape stats (same 100 generations)</h2>
<p class="section-sub">
Not quality metrics on their own. They describe surface properties of the
candidates and exist to suggest <em>mechanism</em> for the discriminator-vs-judge
contradiction.
</p>
<section class="card score">
<table>
<thead>
<tr>
<th>Model</th>
<th class="num">Mean chars</th>
<th class="num">Median chars</th>
<th class="num">Mean words</th>
<th class="num">Empty</th>
<th class="num">&lt;FILL&gt;</th>
<th class="num">Label leak</th>
<th class="num">Too short (&lt;20)</th>
<th class="num">Newline-heavy (&ge;3)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Base Qwen</td>
<td class="num">232.9</td><td class="num">265</td><td class="num">36.9</td>
<td class="num">0</td><td class="num">0</td><td class="num">0</td>
<td class="num">4</td><td class="num">20</td>
</tr>
<tr>
<td>LoRA step 6500</td>
<td class="num">227.9</td><td class="num">264</td><td class="num">36.1</td>
<td class="num">0</td><td class="num">0</td><td class="num">0</td>
<td class="num">2</td><td class="num">37</td>
</tr>
<tr class="worst">
<td>LoRA final</td>
<td class="num">217.1</td><td class="num">244.5</td><td class="num">34.3</td>
<td class="num">0</td><td class="num">0</td><td class="num">0</td>
<td class="num">4</td><td class="num">65</td>
</tr>
</tbody>
</table>
<p class="verdict">
<strong>Read this as:</strong> mean and median character counts drop slightly from
step 6500 to final (227.9 &rarr; 217.1 mean; 264 &rarr; 244.5 median). None of the
candidates produces empty outputs, leaks the &lt;FILL&gt; placeholder, or leaks
labels &mdash; the prompt format is clean. The notable shift is the
<em>newline-heavy</em> count: examples with 3+ newlines almost double from step
6500 (37) to final (65). That means a much larger fraction of final&rsquo;s output
is broken into short lines, lists, or bulleted fragments rather than continuous
paragraph prose.
</p>
<p class="verdict" style="border-left-color: var(--red); background: rgba(168, 68, 46, 0.07);">
<strong>Why this is the suspected mechanism:</strong> small span discriminators
learn from token-level surface cues; once they are confused by short-line layouts
in the training distribution, the policy can lower its detectability by drifting
toward that shape. LLM judges, on the other hand, read for legal continuity and
penalize fragmented or list-shaped paragraphs in continuous opinion text. So the
same drift simultaneously raises &ldquo;confused rate&rdquo; and lowers
&ldquo;judge prefers it.&rdquo; This is plausible-but-not-proven; we have not
run an ablation that holds output formatting fixed.
</p>
</section>
<h2 class="section-title">The contradiction is the result</h2>
<p class="section-sub">
If you only looked at one lens, you&rsquo;d either declare success or declare
failure. Looking at both is what makes this run informative.
</p>
<article class="card" style="margin-top: 14px;">
<h3>The two stories side by side</h3>
<table>
<thead>
<tr>
<th style="width: 20%;">Lens</th>
<th>Step 6500 &rarr; final</th>
<th>What it implies if you only saw this lens</th>
</tr>
</thead>
<tbody>
<tr>
<td>Frozen discriminator</td>
<td>Confused 48% &rarr; 61%; logit gap 8.27 &rarr; 7.23.</td>
<td>&ldquo;Training continued to make the LoRA harder for the in-loop discriminator. The reward signal was successfully optimized.&rdquo;</td>
</tr>
<tr>
<td>Fresh head</td>
<td>Eval acc 73.3% &rarr; 70.0%; AUC 0.7567 &rarr; 0.7556 (essentially flat).</td>
<td>&ldquo;A from-scratch classifier still finds final slightly harder, but most of the detectability gain was already captured at step 6500.&rdquo;</td>
</tr>
<tr>
<td>GPT-4.1 pairwise (30)</td>
<td>Final loses to step 6500 9&ndash;19&ndash;2 (32.1% win rate); slightly loses to base 13&ndash;16&ndash;1 (44.8%).</td>
<td>&ldquo;Whatever final improved at, it&rsquo;s not user-visible legal-continuation quality. Step 6500 reads better.&rdquo;</td>
</tr>
<tr>
<td>GPT-5.4 spot check (10)</td>
<td>Final 0&ndash;10 vs human; step 6500 had also gone 0&ndash;10 vs human in the prior version.</td>
<td>&ldquo;Neither LoRA is anywhere near human-level on the strictest available judge. This run can&rsquo;t separate them on this signal.&rdquo;</td>
</tr>
<tr>
<td>Text shape (newline-heavy)</td>
<td>20 (base) &rarr; 37 (step 6500) &rarr; 65 (final).</td>
<td>&ldquo;A lot of what the policy changed is layout, not legal substance.&rdquo;</td>
</tr>
</tbody>
</table>
</article>
<article class="card" style="margin-top: 14px;">
<h3>What that mismatch means</h3>
<p>
The reward signal in this stack is &ldquo;raise the discriminator&rsquo;s
probability that this paragraph is human.&rdquo; The most direct way to do that
is to write a paragraph that&rsquo;s actually a better legal continuation. A less
direct, often easier way is to drift the surface form toward what the
discriminator finds hard to score &mdash; e.g., shorter lines, lists, fragmented
layout. From step 6500 onward, the policy increasingly took the easier path: the
in-loop discriminator was happier, but a partially independent classifier
plateaued and an LLM judge actively preferred the older checkpoint and (slightly)
the base model.
</p>
<p>
The label for this pattern is <strong>reward-model overfitting / proxy
drift</strong>: the policy is still optimizing the reward, but the reward has
decoupled from quality. Newline-heavy text shape is a plausible mechanism for
that decoupling here, but the decoupling itself is the load-bearing finding.
</p>
</article>
<h2 class="section-title">What this proves &mdash; and what it does not</h2>
<section class="grid">
<article class="card good">
<h2>What it proves</h2>
<ul>
<li>The chat-template-from-scratch run <em>succeeded</em> at the narrow objective
it was optimizing: the in-loop and a fresh-from-scratch discriminator both
find LoRA final harder to separate from human than base or step 6500.</li>
<li>Discriminator confusion is no longer a useful single objective in this setup.
From step 6500 to final, frozen-discriminator confused jumped 48% &rarr; 61%
while GPT-4.1 quality regressed; that is a clean demonstration of
<strong>reward-model overfitting / proxy drift</strong> in IMPOSTER&rsquo;s
current discriminator-only co-training loop.</li>
<li>Style is the most plausible mechanism: text-shape stats show newline-heavy
completions almost double from step 6500 to final. Lists and short paragraphs
fool span discriminators; LLM judges read them as worse continuations of legal
prose.</li>
<li>Step 6500 is the best LoRA checkpoint by GPT-4.1 quality and is the right
source for any follow-up samples or downstream demos.</li>
</ul>
</article>
<article class="card bad">
<h2>What it does not prove</h2>
<ul>
<li>It does not prove the LoRA final checkpoint is useless. It is useful evidence
of where the reward signal breaks down and is the right artifact for
diagnosing discriminator failure modes.</li>
<li>It does not prove base Qwen is definitively better than LoRA final overall.
GPT-4.1 had base 16, final 13, ties 1 over 30 examples &mdash; closer to a
draw with a slight lean toward base, not a sweep. Treat this as &ldquo;final
did not beat base in this judge run,&rdquo; not as &ldquo;LoRA hurts.&rdquo;</li>
<li>The GPT-5.4 result (10 prompts, mostly xhigh, one fallback to high after
repeated empty responses) is a coarse spot check, not a benchmark. It is
consistent with both LoRAs being well below frontier-quality but should not
be cited as a quantitative comparison.</li>
<li>No human reviewers scored these outputs in this run. Until that is added, all
user-visible quality claims rest on LLM judges.</li>
</ul>
</article>
</section>
<h2 class="section-title">Why span selection matters more now</h2>
<p class="section-sub">
The rubric below was designed before this run; the run gives it new urgency. The
reasoning is short.
</p>
<article class="card" style="margin-top: 14px;">
<h3>From the run back to the rubric</h3>
<ul>
<li>Final&rsquo;s drift was largely <em>formatting</em>: short lines, more
newlines, list-like fragments. That kind of drift is most rewarded on spans
where formatting itself carries information &mdash; signature blocks, headings,
counsel lists, citation strings, captions, short procedural bridges.</li>
<li>If the eval set or the reward set leans on those weak spans, formatting drift
looks like quality progress. The discriminator gets fooled, and the policy
gets pushed further down a non-quality slope.</li>
<li>The fix is <em>before</em> training: choose spans where the missing text has
to do real legal work &mdash; carry forward facts and parties, apply a
standard of review, frame an issue, transition between argument steps. On
spans like that, formatting tricks don&rsquo;t buy detectability and the
reward stays correlated with quality.</li>
<li>Useful diagnostic: keep a small labeled slice of bad/adversarial spans on
purpose &mdash; they&rsquo;re great for studying discriminator failure
modes &mdash; but exclude them from product-quality eval averages.</li>
</ul>
</article>
<h2 class="section-title">What this implies for span selection</h2>
<p class="section-sub">
The rubric proper. Carried forward from the prior PageDrop unchanged in substance,
with one bullet added in the &ldquo;bad spans&rdquo; column reflecting the
formatting-drift mechanism observed in this run.
</p>
<section class="grid">
<article class="card good">
<h2>Good spans</h2>
<h3>Keep when the missing text is meaningfully constrained by context.</h3>
<ul>
<li>Substantive legal text: reasoning, factual application, issue framing, standard of review, procedural history, or statutory interpretation.</li>
<li>Long enough to test quality: roughly 80-500 words by default.</li>
<li>Naturally bounded at paragraph or multi-paragraph boundaries.</li>
<li>Before and after context make factual or logical errors visible.</li>
<li>Requires consistency with parties, dates, claims, statutes, charges, remedies, and prior paragraph logic.</li>
<li>The human answer is coherent, cleanly extracted, and not mostly quotation or citation filler.</li>
</ul>
</article>
<article class="card bad">
<h2>Bad spans</h2>
<h3>Reject spans where generic or mechanical completion can look good.</h3>
<ul>
<li>Signature blocks, judge names, captions, counsel lists, citations, dates, headers, footers, and costs boilerplate.</li>
<li>Very short completions: initials, paragraph numbers, isolated citations, or one-line dispositions.</li>
<li>Chopped spans that start or end mid-word, mid-sentence, or mid-citation.</li>
<li>Long statutory quotations or copied block quotes where generation becomes transcription.</li>
<li>OCR damage, table artifacts, page numbers, extraction junk, or language mismatch.</li>
<li>Any span containing chat markers, prompt text, explanation prose, or model-role leakage.</li>
<li>Spans dominated by lists, headings, or whitespace that reward style drift over substance.</li>
</ul>
</article>
</section>
<section class="card score">
<h2>LLM triage scores</h2>
<table>
<thead>
<tr>
<th>Dimension</th>
<th>What to score 1-5</th>
<th>Keep target</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>substance</strong></td>
<td>Does the span contain meaningful legal or factual reasoning?</td>
<td>4+</td>
</tr>
<tr>
<td><strong>context_dependence</strong></td>
<td>Does a good answer require using the surrounding case context?</td>
<td>4+</td>
</tr>
<tr>
<td><strong>bridge_quality</strong></td>
<td>Can before/after context reveal whether the continuation fits?</td>
<td>4+</td>
</tr>
<tr>
<td><strong>extraction_cleanliness</strong></td>
<td>Is the target clean, readable, and naturally bounded?</td>
<td>4+</td>
</tr>
<tr>
<td><strong>non_boilerplate</strong></td>
<td>Is it not metadata, signature, caption, citation filler, or generic disposition?</td>
<td>4+</td>
</tr>
<tr>
<td><strong>evaluation_value</strong></td>
<td>Would human-vs-model comparison here teach us about generator quality?</td>
<td>4+</td>
</tr>
</tbody>
</table>
</section>
<section class="rule">
<div class="threshold">
<div class="eyebrow" style="color:#e1c778">Keep threshold</div>
<code>
all scores &gt;= 4<br>
word_count: 80-500<br>
no reject flags
</code>
</div>
<article class="card">
<h2>Hard reject flags</h2>
<p>
Reject immediately if any flag is true:
<strong>is_signature_block</strong>, <strong>is_caption_or_counsel_block</strong>,
<strong>is_mostly_citation</strong>, <strong>is_mostly_quote</strong>,
<strong>is_too_short</strong>, <strong>is_extraction_artifact</strong>,
<strong>has_prompt_or_chat_markers</strong>, <strong>starts_or_ends_mid_word</strong>,
or <strong>language_mismatch</strong>.
</p>
</article>
</section>
<section class="prompt">
<pre>You are selecting legal-text infill samples for evaluating a generator.
Given BEFORE, MISSING_SPAN, and AFTER, decide whether this is a high-quality
evaluation span.
A good span requires substantive legal reasoning or factual/procedural
continuation, is naturally bounded, cleanly extracted, context-dependent,
and not mere boilerplate.
Return JSON:
{
"keep": true/false,
"scores": {
"substance": 1-5,
"context_dependence": 1-5,
"bridge_quality": 1-5,
"extraction_cleanliness": 1-5,
"non_boilerplate": 1-5,
"evaluation_value": 1-5
},
"reject_flags": [...],
"span_type": "reasoning|facts|procedure|disposition|caption|signature|citation|quote|other",
"rationale": "one concise sentence"
}</pre>
</section>
<section class="recco">
<div class="eyebrow">Recommendation</div>
<h2>Don&rsquo;t train longer. Rebuild the eval/reward stack first.</h2>
<ul>
<li><strong>Freeze the run.</strong> Treat
<span class="stat">grpo-lora-qwen-chat-template-from-scratch-20260424</span> as
evidence of proxy drift and stop training against the same discriminator-only /
co-training loop. Continuing only buys more drift.</li>
<li><strong>Use step 6500 as the working LoRA</strong> for follow-up samples,
qualitative demos, or any downstream task that wants a non-base IMPOSTER output.</li>
<li><strong>Tighten span selection</strong> using the rubric below. Filter out
heading-, list-, signature-, and citation-heavy spans; favor legally substantive
paragraph spans where formatting does not carry the win.</li>
<li><strong>Add a human-anchored LLM judge</strong> as the standing eval (and
potentially as a reward component, distilled or used as a critic). GPT-4.1
pairwise vs human reference is a reasonable starting point.</li>
<li><strong>Diversify the adversarial discriminator&rsquo;s training data:</strong>
multiple base models, multiple temperatures and seeds, varied decoding styles,
and explicit formatting variants. The current discriminator was fooled mostly by
style; a more diverse training set raises that bar.</li>
<li><strong>Early-stop and select on held-out signals.</strong> In-loop discriminator
reward should not be the criterion; held-out GPT/judge metrics and a fresh-head
classifier are. Step 6500 would have been the selection if we&rsquo;d been
applying that policy.</li>
<li><strong>Consider richer feedback.</strong> Scalar P(human) is a thin signal.
Reward-interpreter / per-span critique / structured rubric scores are worth
prototyping before another long GRPO run.</li>
<li><strong>Acknowledge the ceiling.</strong> Even step 6500 lost 10/10 to the human
under the GPT-5.4 xhigh spot check. Neither LoRA is product-ready; closing that
gap is a different project than another GRPO run.</li>
</ul>
</section>
<h2 class="section-title">Sources &amp; provenance</h2>
<p class="section-sub">
Where every number on this page came from, and what each artifact contains.
</p>
<article class="card" style="margin-top: 14px;">
<h3>Eval artifacts</h3>
<ul>
<li><strong>Combined eval summary (single source of truth for tables on this page):</strong>
<span class="stat">/Users/ted/v/imposter/evals/chat_final_20260425/summary_all.json</span>.
Keys: <span class="stat">frozen_{base,step6500,final}</span>,
<span class="stat">fresh_{base,step6500,final}</span>,
<span class="stat">gpt41_final_vs_base</span>,
<span class="stat">gpt41_final_vs_step6500</span>,
<span class="stat">gpt41_human_vs_final</span>,
<span class="stat">gpt54_human_vs_final_10</span>.</li>
<li><strong>Local generations and judge outputs:</strong>
<span class="stat">/Users/ted/v/imposter/evals/chat_final_20260425/</span>
contains per-prompt JSONL files: <span class="stat">base_chat_generations.jsonl</span>,
<span class="stat">lora_step6500_chat_generations.jsonl</span>,
<span class="stat">lora_final_generations.jsonl</span>, plus the GPT-4.1 and
GPT-5.4 judge directories with prompt-by-prompt verdicts and confidences.</li>
<li><strong>Remote eval bundle (uploaded via Volta2):</strong>
<span class="stat">gs://volta-489906-artifacts/imposter/grpo-lora-qwen-chat-template-from-scratch-20260424/keep_eval_lora_final_chat/1777154851289_keep_eval_lora_final_chat_20260425_180722.tgz</span>.
The packed bundle of all the above, suitable for sharing.</li>
<li><strong>Final LoRA checkpoint:</strong>
<span class="stat">/scratch/imposter/runs/grpo-lora-qwen-chat-template-from-scratch-20260424/checkpoints/generator/lora_final</span>.
Final artifact name: <span class="stat">checkpoint_lora_final_1777146701505.tgz</span>.</li>
</ul>
</article>
<article class="card" style="margin-top: 14px;">
<h3>Companion reports in this repo</h3>
<ul>
<li><span class="stat">report/imposter_eval_report_chat_template_20260424.tex</span> &mdash; longer LaTeX writeup of the corrected chat-template eval (step 6500-era results).</li>
<li><span class="stat">report/imposter_eval_report_20260424.tex</span> &mdash; pre-correction LaTeX writeup, useful for reading what the malformed-prompt era reported.</li>
<li><span class="stat">report/imposter_report.tex</span> &mdash; original IMPOSTER report; section that mentions the prior PageDrop URL is at line 229.</li>
</ul>
</article>
<h2 class="section-title">More context (Codex CLI session data on disk)</h2>
<p class="section-sub">
This page is a summary. The decision-making, debugging steps, log spelunking, and
operational backstory behind it live in Codex CLI session rollouts on the operator&rsquo;s
machine. If you are a future Claude or Codex agent picking up this work and need
more than what fit on this page (failed launch attempts, raw log excerpts, side
experiments, intermediate checkpoint behavior, judge prompt details, etc.), read
these first.
</p>
<article class="card">
<h3>Where to look</h3>
<ul>
<li><strong>Main current Codex session</strong> (this audit + retrain + final eval):
<span class="stat">/Users/ted/.codex/sessions/2026/04/25/rollout-2026-04-25T02-54-49-019dc36b-6a1e-7da3-b01b-a8c3fed844d4.jsonl</span>.</li>
<li><strong>Prior related Codex sessions</strong> (chat-template fix, step 6500 corrected eval, retrain bring-up):
<ul>
<li><span class="stat">/Users/ted/.codex/sessions/2026/04/24/rollout-2026-04-24T21-33-31-019dc245-4164-7ed1-984f-7db3bba887ce.jsonl</span></li>
<li><span class="stat">/Users/ted/.codex/sessions/2026/04/24/rollout-2026-04-24T18-46-34-019dc1ac-67d4-7311-949d-84f22edb262b.jsonl</span></li>
<li><span class="stat">/Users/ted/.codex/sessions/2026/04/24/rollout-2026-04-24T15-19-44-019dc0ef-0a7e-74a1-b0a4-df21d5e23d30.jsonl</span></li>
</ul>
</li>
<li>The Codex sessions are JSONL transcripts. They are not required reading to
understand this page &mdash; everything load-bearing is in the tables and
sections above &mdash; but they are the right place to recover detail that
didn&rsquo;t make it into a section here.</li>
</ul>
</article>
<p class="footer-note">
Span rubric carried over from the prior PageDrop
(<a href="https://pagedrop.ai/g/belisarius222/50822f4909ef1db97e72c8dd98d4b9a5">pagedrop.ai/g/belisarius222/50822f4909ef1db97e72c8dd98d4b9a5</a>,
Apr 9, pre-chat-template fix). Eval tables, operational notes, and recommendation
above reflect the 2026-04-25 final-run results and supersede that page.
</p>
</main>
</body>
</html>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment