belisarius222 · May 28, 2026 18:00
diff --git a/2026-05-28-arda-poc-status-report.html b/2026-05-28-arda-poc-status-report.html
 <!doctype html>
 <html lang="en" data-theme="dark">
 <head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>ARDA POC Status Report</title>
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@picocss/pico@2/css/pico.min.css">
  <style>
    :root {
      color-scheme: dark;
      --bg: #0f172a;
      --panel: #1e2937;
      --ink: #e2e8f0;
      --muted: #94a3b8;
      --line: #334155;
      --accent: #14b8a6;
      --accent-soft: #134e4b;
      --warn: #fbbf24;
      --warn-soft: #451a03;
      --bad: #f87171;
      --bad-soft: #450a0a;
      --good: #4ade80;
      --good-soft: #052e16;
      --code: #1e2937;
    }
    * { box-sizing: border-box; }
    body {
      margin: 0;
      background: var(--bg);
      color: var(--ink);
      font: 15px/1.55 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
    }
    main {
      max-width: 1120px;
      margin: 0 auto;
      padding: 40px 24px 64px;
    }
    header {
      border-bottom: 1px solid var(--line);
      margin-bottom: 28px;
      padding-bottom: 24px;
    }
    h1 {
      margin: 0 0 10px;
      font-size: 34px;
      line-height: 1.15;
      letter-spacing: 0;
    }
    h2 {
      margin: 34px 0 12px;
      font-size: 22px;
      letter-spacing: 0;
    }
    h3 {
      margin: 24px 0 8px;
      font-size: 17px;
      letter-spacing: 0;
    }
    p { margin: 8px 0 12px; }
    ul { margin: 8px 0 16px 22px; padding: 0; }
    li { margin: 4px 0; }
    code, pre {
      font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
      font-size: 0.92em;
    }
    code {
      background: var(--code);
      border-radius: 4px;
      padding: 1px 4px;
    }
    pre {
      background: #0f172a;
      color: #e2e8f0;
      border-radius: 8px;
      overflow: auto;
      padding: 14px 16px;
      line-height: 1.45;
      border: 1px solid var(--line);
    }
    table {
      width: 100%;
      border-collapse: collapse;
      margin: 12px 0 20px;
      background: var(--panel);
    }
    th, td {
      border: 1px solid var(--line);
      padding: 9px 10px;
      text-align: left;
      vertical-align: top;
    }
    th {
      background: #1e2937;
      font-weight: 650;
    }
    .card {
      background: var(--panel);
      border: 1px solid var(--line);
      border-radius: 8px;
      padding: 18px 20px;
      margin: 14px 0;
    }
    .grid {
      display: grid;
      grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
      gap: 14px;
      margin: 14px 0;
    }
    .metric {
      background: var(--panel);
      border: 1px solid var(--line);
      border-radius: 8px;
      padding: 14px 16px;
    }
    .metric strong {
      display: block;
      font-size: 21px;
      margin-bottom: 3px;
    }
    .muted { color: var(--muted); }
    .tag {
      display: inline-block;
      border-radius: 999px;
      padding: 2px 9px;
      font-size: 12px;
      font-weight: 650;
      margin-right: 6px;
      white-space: nowrap;
    }
    .good { background: var(--good-soft); color: var(--good); }
    .warn { background: var(--warn-soft); color: var(--warn); }
    .bad { background: var(--bad-soft); color: var(--bad); }
    .info { background: var(--accent-soft); color: var(--accent); }
    .toc a {
      color: var(--accent);
      text-decoration: none;
    }
    .toc a:hover { text-decoration: underline; }
    /* Pico integration note: base typography from Pico, custom components override for report fidelity */
  </style>
 </head>
 <body>
 <main>
  <header>
    <h1>ARDA POC Status Report</h1>
    <p class="muted">Generated May 28, 2026 from the local workspace at <code>/Users/ted/v/arda</code>.</p>
    <p>
      This report summarizes the current state of the ARDA proof of concept: what has been built,
      what has been verified, what has not worked yet, and what is actively in progress.
    </p>
  </header>

  <section class="card">
    <h2 id="executive-summary">Executive Summary</h2>
    <p>
      The project has moved from a written plan to a runnable Python POC with an online causal ARDA tree,
      disk-backed node storage, per-head frontier search, Hugging Face/Qwen integration, long-context
      scenario generation, remote 8-GPU orchestration, and evaluation/reporting artifacts.
    </p>
    <p>
      The system is not yet a completed 100M-token proof. The strongest verified result so far is that
      capped smoke and timing runs execute through the 8-rank B200 harness, use distinct CUDA devices
      <code>cuda:0..cuda:7</code>, and report FlashAttention 4. Those runs are intentionally tiny, so
      they prove infrastructure and some ARDA mechanics, not final long-context quality.
    </p>
    <p>
      Current local test state: <strong>221 passed, 1 skipped</strong>. The latest in-progress code adds
      a dedicated answer-review packet for manual baseline-vs-ARDA qualitative review.
    </p>
  </section>

  <nav class="card toc">
    <h2>Contents</h2>
    <ul>
      <li><a href="#current-state">Current State</a></li>
      <li><a href="#components">Components Built</a></li>
      <li><a href="#tests">Tests and Verification</a></li>
      <li><a href="#remote">Remote B200 Runs</a></li>
      <li><a href="#worked">What Has Worked</a></li>
      <li><a href="#not-worked">What Has Not Worked Yet</a></li>
      <li><a href="#current-work">Current Work</a></li>
      <li><a href="#remaining">Remaining Work for Full POC</a></li>
    </ul>
  </nav>

  <section>
    <h2 id="current-state">Current State</h2>
    <div class="grid">
      <div class="metric">
        <strong>Latest pushed commit</strong>
        <code>1db7484</code><br>
        <span class="muted">Document POC completion gates</span>
      </div>
      <div class="metric">
        <strong>Local tests</strong>
        <code>221 passed, 1 skipped</code><br>
        <span class="muted">Includes the answer-review packet changes</span>
      </div>
      <div class="metric">
        <strong>Remote hardware path</strong>
        <code>8 ranks / cuda:0..7</code><br>
        <span class="muted">Verified on capped smoke runs</span>
      </div>
      <div class="metric">
        <strong>Full 100M POC</strong>
        <span class="tag bad">Not complete</span><br>
        <span class="muted">No uncapped 100M run has completed yet</span>
      </div>
    </div>

    <p>
      The branch is <code>main</code>. Before this report, the most recent pushed commits were:
    </p>
    <pre>1db7484 Document POC completion gates
 e7c1a6e Require model tokens for target completion
 10f5928 Support resumable comparison runs
 265001e Report full POC completion evidence
 fef05de Avoid duplicate frontier delta reads
 4cd458a Document GQA stat cache timing
 6b16ffb Reuse selected GQA node stats
 0007c15 Document shared GQA frontier setup</pre>

    <p>
      There are currently local, tested, not-yet-pushed changes in:
      <code>arda/evaluation_report.py</code>, <code>scripts/run_long_context_poc.py</code>, and
      <code>tests/test_evaluation_report.py</code>. They add and test the answer-review packet described
      later in this report.
    </p>
  </section>

  <section>
    <h2 id="components">Components Built</h2>
    <table>
      <thead>
        <tr>
          <th>Component</th>
          <th>Main files</th>
          <th>Status</th>
          <th>Notes</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td>Online causal tree</td>
          <td><code>arda/online_tree.py</code>, <code>arda/ingest.py</code></td>
          <td><span class="tag good">Built and tested</span></td>
          <td>Appends real model-produced leaves and seals aligned parent nodes. Enforces the fresh-tail invariant.</td>
        </tr>
        <tr>
          <td>Raw-file node storage</td>
          <td><code>arda/storage.py</code>, <code>arda/payload_codec.py</code></td>
          <td><span class="tag good">Built and tested</span></td>
          <td>Append-only binary files for nodes, child arrays, payloads, and per-level indexes. Chosen over SQLite for hot path locality and simple indexed access.</td>
        </tr>
        <tr>
          <td>Payload/statistics layer</td>
          <td><code>arda/payloads.py</code>, <code>arda/stats.py</code>, <code>arda/hf_cache.py</code></td>
          <td><span class="tag good">Built and tested</span></td>
          <td>Stores per-layer/head key and value summary stats, low-rank covariance pieces, boxes, and value correction data.</td>
        </tr>
        <tr>
          <td>RoPE-aware bounds</td>
          <td><code>arda/bounds.py</code></td>
          <td><span class="tag good">Built and tested</span></td>
          <td>Implements the span-aware box/angle primitive, including angle wrapping and full-period behavior.</td>
        </tr>
        <tr>
          <td>Frontier search</td>
          <td><code>arda/frontier.py</code>, <code>arda/baseline_frontiers.py</code></td>
          <td><span class="tag good">Built and tested</span></td>
          <td>Best-first refinement of a legal tree cut, independent per layer/head/token for the correctness baseline. Includes lazy re-score diagnostics and fixed/recency/all-leaf baselines.</td>
        </tr>
        <tr>
          <td>Qwen attention integration</td>
          <td><code>arda/qwen_attention.py</code>, <code>arda/decode_state.py</code>, <code>arda/gqa_routing.py</code></td>
          <td><span class="tag good">Built and tested</span></td>
          <td>Patches Qwen attention forward paths for ARDA readout. GQA routing shares query-independent setup and selected durable node stats while preserving independent per-head searches.</td>
        </tr>
        <tr>
          <td>HF projection and ingestion hooks</td>
          <td><code>arda/hf_ingest.py</code>, <code>arda/hf_projections.py</code>, <code>arda/hf_queries.py</code></td>
          <td><span class="tag good">Built and tested</span></td>
          <td>Captures pre-RoPE key/value projections during model forward passes so leaves come from the actual online model context.</td>
        </tr>
        <tr>
          <td>Long-context scenarios</td>
          <td><code>arda/long_context.py</code></td>
          <td><span class="tag good">Built and tested</span></td>
          <td>Three coherent synthetic-but-realistic conversation families: legal case review, codebase audit, and chemistry literature review. They stream to 100M target tokens without materializing the full text.</td>
        </tr>
        <tr>
          <td>Local/remote runners</td>
          <td><code>scripts/run_long_context_poc.py</code>, <code>scripts/run_comparison_long_context_poc.py</code>, <code>scripts/run_resumable_long_context_poc.py</code>, <code>arda/remote.py</code></td>
          <td><span class="tag good">Built and tested</span></td>
          <td>Supports runtime checks, model smoke, ingest smoke, route smoke, patched decode smoke, streaming ARDA eval, resumable part-boundary runs, and scenario/mode comparisons.</td>
        </tr>
        <tr>
          <td>Evaluation reports</td>
          <td><code>arda/evaluation_report.py</code>, <code>arda/comparison_report.py</code>, <code>scripts/summarize_long_context_poc.py</code></td>
          <td><span class="tag good">Built and tested</span></td>
          <td>Aggregates rank artifacts, stream progress, throughput projections, completion gates, hardware proof, and answer-review summaries.</td>
        </tr>
        <tr>
          <td>Manual answer-review packet</td>
          <td><code>arda/evaluation_report.py</code>, <code>scripts/run_long_context_poc.py</code></td>
          <td><span class="tag warn">In progress</span></td>
          <td>New local changes write <code>answer_review.md</code> with short oracle context, baseline answer, ARDA answer, and review checklist. Tests are passing but the change has not yet been committed at the time this report was generated.</td>
        </tr>
      </tbody>
    </table>
  </section>

  <section>
    <h2 id="tests">Tests and Verification</h2>
    <p>
      The current local suite covers unit, integration, runner, reporting, and synthetic long-context planning behavior.
      The latest full run completed with:
    </p>
    <pre>221 passed, 1 skipped in 2.79s</pre>

    <h3>Major test categories</h3>
    <ul>
      <li><strong>Tree and storage invariants:</strong> sealing schedule, fresh tail, reopen/continue behavior, file-backed indexed lookup.</li>
      <li><strong>Math and bound correctness:</strong> RoPE/angle box primitive, scale handling, value/stat merging, payload serialization.</li>
      <li><strong>Frontier search invariants:</strong> legal cut coverage, budget adherence, no partial synthetic nodes, independent per-head contexts, baseline frontiers.</li>
      <li><strong>HF/Qwen integration:</strong> projection capture, ingestion, patched decode, query extraction, all-leaf/manual attention agreement paths.</li>
      <li><strong>Long-context scenario generation:</strong> 100M-token plans, bounded streaming segments, coherent conversation turns, short-context oracle questions.</li>
      <li><strong>Remote orchestration:</strong> 8-GPU torchrun command generation, resumable part boundaries, comparison runner mode/scenario matrix.</li>
      <li><strong>Reporting:</strong> rank aggregation, progress-only reports, completion gates, throughput projection, POC hardware proof, comparison summaries, answer-review packet.</li>
    </ul>

    <h3>Completion gates now tested</h3>
    <p>
      A report can only mark the full target complete when every rank has completed progress and all of these reach
      the requested target: <code>document_target_tokens_ingested</code>, tokenizer-side <code>prompt_tokens</code>,
      and ARDA <code>tree_token_count</code>. A separate POC-complete gate also requires at least 8 GPUs,
      distinct <code>cuda:0..cuda:7</code> evaluation devices, FlashAttention availability, and
      <code>flash_attention_4</code> as the active attention implementation.
    </p>
  </section>

  <section>
    <h2 id="remote">Remote B200 Runs</h2>
    <p>
      Remote execution uses the 8x B200 machine at <code>198.13.252.92</code>, with the repo synced to
      <code>/scratch/arda/repo</code> and venv at <code>/scratch/arda/venv</code>.
    </p>

    <table>
      <thead>
        <tr>
          <th>Run</th>
          <th>What it proved</th>
          <th>Key results</th>
          <th>Limitations</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td><code>comparison_resumable_smoke_10f5928</code></td>
          <td>Real resumable comparison path over two torchrun invocations.</td>
          <td>8 ranks, distinct CUDA devices <code>[0..7]</code>, FlashAttention 4 used, progress status completed on all ranks.</td>
          <td>Capped at 16 document-target tokens. Not a quality or 100M completion proof.</td>
        </tr>
        <tr>
          <td><code>comparison_cap16_fef05de_timing_cold_budget48_lazy</code></td>
          <td>Small timing probe for frontier delta-read cleanup.</td>
          <td>8 rank artifacts, CUDA devices <code>[0..7]</code>, FlashAttention 4 used. The cleanup did not improve mean timing.</td>
          <td>Capped at 16 target tokens; no generated answer tokens.</td>
        </tr>
        <tr>
          <td><code>comparison_d7b99fa_answer_smoke_4tok_complete</code></td>
          <td>ARDA can generate tokens through the remote comparison harness.</td>
          <td>8 ranks and 4 generated tokens. Example behavior was close to the baseline on a tiny legal-case prompt.</td>
          <td>Old artifact predates the stricter POC hardware fields and is capped. Not enough for final quality judgment.</td>
        </tr>
      </tbody>
    </table>

    <h3>Timing observations</h3>
    <ul>
      <li>Shared initial frontier setup reduced small-run mean search time from about <code>24.189s</code> to <code>17.842s</code>.</li>
      <li>Selected GQA stat reuse moved the same small-run mean search time from about <code>17.842s</code> to <code>17.390s</code>.</li>
      <li>The duplicate frontier-delta-read cleanup was semantically good but did not show a timing improvement in the cap-16 probe.</li>
      <li>Per-head, per-layer, per-token cold-start routing is still expensive; the next performance gains likely require batching/vectorized priority scoring or lower-overhead routing loops.</li>
    </ul>
  </section>

  <section>
    <h2 id="worked">What Has Worked</h2>
    <ul>
      <li>The offline-tree design flaw was corrected. Leaves are now produced online by the model under the same approximate-attention regime used for retrieval/generation.</li>
      <li>The tree and frontier APIs enforce fresh-tail behavior structurally, which reduces the chance of policy drift in routers.</li>
      <li>The raw-file storage path exercises disk-backed topology and payload reads instead of assuming the whole tree fits in RAM.</li>
      <li>The Qwen integration can ingest tokens, patch attention, construct ARDA frontiers, and generate through the ARDA path on smoke runs.</li>
      <li>The remote harness can launch 8-process torchrun jobs and produce per-rank artifacts on B200 hardware.</li>
      <li>Completion reporting has become much stricter, making it harder to mistake a capped smoke run for a full POC.</li>
      <li>The synthetic scenarios are coherent enough to be useful: legal case analysis, codebase audit, and chemistry literature review each have consistent topics, repeated document-style packets, assistant acknowledgements, and short-context oracle questions.</li>
    </ul>
  </section>

  <section>
    <h2 id="not-worked">What Has Not Worked Yet</h2>
    <ul>
      <li><strong>No full 100M run has completed.</strong> Existing remote runs are capped smoke or timing runs.</li>
      <li><strong>Holistic answer quality is not yet established.</strong> The old expected-label mention check is too weak; the new answer-review packet is being added to support manual judgment.</li>
      <li><strong>Cold-start per-head routing is slow.</strong> It is the clean correctness baseline, but likely too expensive for a full-quality 100M adaptive run without more batching or routing optimization.</li>
      <li><strong>The duplicate delta-read optimization did not improve measured wall time.</strong> It still simplifies one path and removes redundant work, but the bottleneck is elsewhere.</li>
      <li><strong>Full adaptive-vs-fixed quality curves are not done.</strong> There are comparison runners and capped artifacts, but not a complete long-context quality study.</li>
      <li><strong>Needle/value-channel validation is not yet the main remote result.</strong> The mathematical primitives and tests exist, but the end-to-end long-context evidence still needs to be generated.</li>
      <li><strong>Storage tiering is still basic.</strong> The raw-file store exists, but detailed tensor blob layout, eviction, and hot/cold locality policies have not been optimized.</li>
    </ul>
  </section>

  <section>
    <h2 id="current-work">What I Am Working On Now</h2>
    <p>
      The current active slice is the answer-review packet. The goal is to make each evaluation directory contain a
      human-readable <code>answer_review.md</code> artifact with:
    </p>
    <ul>
      <li>the exact evaluation question,</li>
      <li>the expected evidence label,</li>
      <li>the short-context oracle excerpt used for the fully uncompressed baseline,</li>
      <li>the baseline answer,</li>
      <li>the ARDA answer,</li>
      <li>a short checklist for manual review of evidence match, fact preservation, and relative quality.</li>
    </ul>
    <p>
      This directly supports the user-requested qualitative comparison: read the answers yourself and compare long
      compressed-context behavior with the small fully uncompressed baseline. The tests for this slice currently pass,
      and the full suite passes with the changes included.
    </p>
  </section>

  <section>
    <h2 id="remaining">Remaining Work for Full POC</h2>
    <ol>
      <li><strong>Commit the answer-review packet support.</strong> It is implemented locally and tested, but should be committed and pushed.</li>
      <li><strong>Run an uncapped or meaningfully larger resumable remote job.</strong> Use the resumable comparison runner to start a real 100M scenario/mode directory and let it advance in bounded chunks.</li>
      <li><strong>Monitor throughput and storage growth.</strong> Use progress files and evaluation summaries to estimate whether the current routing path can finish in practical time.</li>
      <li><strong>Optimize routing if needed.</strong> The likely next targets are vectorized priority scoring, lower-overhead queue loops, and possibly Rust for CPU-heavy frontier logic if profiling justifies it.</li>
      <li><strong>Complete holistic answer review.</strong> Generate nontrivial answers, inspect <code>answer_review.md</code>, and compare ARDA vs. baseline answers manually.</li>
      <li><strong>Compare adaptive vs. fixed/recency baselines.</strong> Run the same scenario/question set across routing modes at comparable realized slot counts.</li>
      <li><strong>Only then mark the full goal complete.</strong> The completion audit should require current evidence for all explicit user requirements: strict TDD, realistic coherent 100M conversations, model execution on 8 B200 GPUs, FlashAttention 4, all 8 GPUs used, and manual answer-quality comparison against small exact-context baselines.</li>
    </ol>
  </section>

  <section class="card">
    <h2>Useful Commands</h2>
    <pre># Full local suite
 python -m pytest -q

 # Summarize one remote run directory
 ssh -o StrictHostKeyChecking=no -o ConnectTimeout=15 -i ~/.ssh/pi0-prime-intellect-b200 \
  ubuntu@198.13.252.92 \
  'cd /scratch/arda/repo && /scratch/arda/venv/bin/python scripts/summarize_long_context_poc.py \
   --output-dir /scratch/arda/runs/comparison_resumable_smoke_10f5928/recency'

 # Resumable comparison skeleton for a real long run
 /scratch/arda/venv/bin/python scripts/run_comparison_long_context_poc.py \
  --scenario legal_case_review \
  --output-dir /scratch/arda/runs/&lt;run-name&gt; \
  --modes cold_start_per_layer_head_token,fixed_uniform,recency \
  --parts-per-run 64 \
  --max-iterations 1</pre>
  </section>
 </main>
 <script src="https://pagedrop.ai/pagedrop.js"></script>
 </body>
 </html>
	<!doctype html>
	<html lang="en" data-theme="dark">
	<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>ARDA POC Status Report</title>
	<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@picocss/pico@2/css/pico.min.css">
	<style>
	:root {
	color-scheme: dark;
	--bg: #0f172a;
	--panel: #1e2937;
	--ink: #e2e8f0;
	--muted: #94a3b8;
	--line: #334155;
	--accent: #14b8a6;
	--accent-soft: #134e4b;
	--warn: #fbbf24;
	--warn-soft: #451a03;
	--bad: #f87171;
	--bad-soft: #450a0a;
	--good: #4ade80;
	--good-soft: #052e16;
	--code: #1e2937;
	}
	* { box-sizing: border-box; }
	body {
	margin: 0;
	background: var(--bg);
	color: var(--ink);
	font: 15px/1.55 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
	}
	main {
	max-width: 1120px;
	margin: 0 auto;
	padding: 40px 24px 64px;
	}
	header {
	border-bottom: 1px solid var(--line);
	margin-bottom: 28px;
	padding-bottom: 24px;
	}
	h1 {
	margin: 0 0 10px;
	font-size: 34px;
	line-height: 1.15;
	letter-spacing: 0;
	}
	h2 {
	margin: 34px 0 12px;
	font-size: 22px;
	letter-spacing: 0;
	}
	h3 {
	margin: 24px 0 8px;
	font-size: 17px;
	letter-spacing: 0;
	}
	p { margin: 8px 0 12px; }
	ul { margin: 8px 0 16px 22px; padding: 0; }
	li { margin: 4px 0; }
	code, pre {
	font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
	font-size: 0.92em;
	}
	code {
	background: var(--code);
	border-radius: 4px;
	padding: 1px 4px;
	}
	pre {
	background: #0f172a;
	color: #e2e8f0;
	border-radius: 8px;
	overflow: auto;
	padding: 14px 16px;
	line-height: 1.45;
	border: 1px solid var(--line);
	}
	table {
	width: 100%;
	border-collapse: collapse;
	margin: 12px 0 20px;
	background: var(--panel);
	}
	th, td {
	border: 1px solid var(--line);
	padding: 9px 10px;
	text-align: left;
	vertical-align: top;
	}
	th {
	background: #1e2937;
	font-weight: 650;
	}
	.card {
	background: var(--panel);
	border: 1px solid var(--line);
	border-radius: 8px;
	padding: 18px 20px;
	margin: 14px 0;
	}
	.grid {
	display: grid;
	grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
	gap: 14px;
	margin: 14px 0;
	}
	.metric {
	background: var(--panel);
	border: 1px solid var(--line);
	border-radius: 8px;
	padding: 14px 16px;
	}
	.metric strong {
	display: block;
	font-size: 21px;
	margin-bottom: 3px;
	}
	.muted { color: var(--muted); }
	.tag {
	display: inline-block;
	border-radius: 999px;
	padding: 2px 9px;
	font-size: 12px;
	font-weight: 650;
	margin-right: 6px;
	white-space: nowrap;
	}
	.good { background: var(--good-soft); color: var(--good); }
	.warn { background: var(--warn-soft); color: var(--warn); }
	.bad { background: var(--bad-soft); color: var(--bad); }
	.info { background: var(--accent-soft); color: var(--accent); }
	.toc a {
	color: var(--accent);
	text-decoration: none;
	}
	.toc a:hover { text-decoration: underline; }
	/* Pico integration note: base typography from Pico, custom components override for report fidelity */
	</style>
	</head>
	<body>
	<main>
	<header>
	<h1>ARDA POC Status Report</h1>
	<p class="muted">Generated May 28, 2026 from the local workspace at <code>/Users/ted/v/arda</code>.</p>
	<p>
	This report summarizes the current state of the ARDA proof of concept: what has been built,
	what has been verified, what has not worked yet, and what is actively in progress.
	</p>
	</header>

	<section class="card">
	<h2 id="executive-summary">Executive Summary</h2>
	<p>
	The project has moved from a written plan to a runnable Python POC with an online causal ARDA tree,
	disk-backed node storage, per-head frontier search, Hugging Face/Qwen integration, long-context
	scenario generation, remote 8-GPU orchestration, and evaluation/reporting artifacts.
	</p>
	<p>
	The system is not yet a completed 100M-token proof. The strongest verified result so far is that
	capped smoke and timing runs execute through the 8-rank B200 harness, use distinct CUDA devices
	<code>cuda:0..cuda:7</code>, and report FlashAttention 4. Those runs are intentionally tiny, so
	they prove infrastructure and some ARDA mechanics, not final long-context quality.
	</p>
	<p>
	Current local test state: <strong>221 passed, 1 skipped</strong>. The latest in-progress code adds
	a dedicated answer-review packet for manual baseline-vs-ARDA qualitative review.
	</p>
	</section>

	<nav class="card toc">
	<h2>Contents</h2>
	<ul>
	<li><a href="#current-state">Current State</a></li>
	<li><a href="#components">Components Built</a></li>
	<li><a href="#tests">Tests and Verification</a></li>
	<li><a href="#remote">Remote B200 Runs</a></li>
	<li><a href="#worked">What Has Worked</a></li>
	<li><a href="#not-worked">What Has Not Worked Yet</a></li>
	<li><a href="#current-work">Current Work</a></li>
	<li><a href="#remaining">Remaining Work for Full POC</a></li>
	</ul>
	</nav>

	<section>
	<h2 id="current-state">Current State</h2>
	<div class="grid">
	<div class="metric">
	<strong>Latest pushed commit</strong>
	<code>1db7484</code><br>
	<span class="muted">Document POC completion gates</span>
	</div>
	<div class="metric">
	<strong>Local tests</strong>
	<code>221 passed, 1 skipped</code><br>
	<span class="muted">Includes the answer-review packet changes</span>
	</div>
	<div class="metric">
	<strong>Remote hardware path</strong>
	<code>8 ranks / cuda:0..7</code><br>
	<span class="muted">Verified on capped smoke runs</span>
	</div>
	<div class="metric">
	<strong>Full 100M POC</strong>
	<span class="tag bad">Not complete</span><br>
	<span class="muted">No uncapped 100M run has completed yet</span>
	</div>
	</div>

	<p>
	The branch is <code>main</code>. Before this report, the most recent pushed commits were:
	</p>
	<pre>1db7484 Document POC completion gates
	e7c1a6e Require model tokens for target completion
	10f5928 Support resumable comparison runs
	265001e Report full POC completion evidence
	fef05de Avoid duplicate frontier delta reads
	4cd458a Document GQA stat cache timing
	6b16ffb Reuse selected GQA node stats
	0007c15 Document shared GQA frontier setup</pre>

	<p>
	There are currently local, tested, not-yet-pushed changes in:
	<code>arda/evaluation_report.py</code>, <code>scripts/run_long_context_poc.py</code>, and
	<code>tests/test_evaluation_report.py</code>. They add and test the answer-review packet described
	later in this report.
	</p>
	</section>

	<section>
	<h2 id="components">Components Built</h2>
	<table>
	<thead>
	<tr>
	<th>Component</th>
	<th>Main files</th>
	<th>Status</th>
	<th>Notes</th>
	</tr>
	</thead>
	<tbody>
	<tr>
	<td>Online causal tree</td>
	<td><code>arda/online_tree.py</code>, <code>arda/ingest.py</code></td>
	<td><span class="tag good">Built and tested</span></td>
	<td>Appends real model-produced leaves and seals aligned parent nodes. Enforces the fresh-tail invariant.</td>
	</tr>
	<tr>
	<td>Raw-file node storage</td>
	<td><code>arda/storage.py</code>, <code>arda/payload_codec.py</code></td>
	<td><span class="tag good">Built and tested</span></td>
	<td>Append-only binary files for nodes, child arrays, payloads, and per-level indexes. Chosen over SQLite for hot path locality and simple indexed access.</td>
	</tr>
	<tr>
	<td>Payload/statistics layer</td>
	<td><code>arda/payloads.py</code>, <code>arda/stats.py</code>, <code>arda/hf_cache.py</code></td>
	<td><span class="tag good">Built and tested</span></td>
	<td>Stores per-layer/head key and value summary stats, low-rank covariance pieces, boxes, and value correction data.</td>
	</tr>
	<tr>
	<td>RoPE-aware bounds</td>
	<td><code>arda/bounds.py</code></td>
	<td><span class="tag good">Built and tested</span></td>
	<td>Implements the span-aware box/angle primitive, including angle wrapping and full-period behavior.</td>
	</tr>
	<tr>
	<td>Frontier search</td>
	<td><code>arda/frontier.py</code>, <code>arda/baseline_frontiers.py</code></td>
	<td><span class="tag good">Built and tested</span></td>
	<td>Best-first refinement of a legal tree cut, independent per layer/head/token for the correctness baseline. Includes lazy re-score diagnostics and fixed/recency/all-leaf baselines.</td>
	</tr>
	<tr>
	<td>Qwen attention integration</td>
	<td><code>arda/qwen_attention.py</code>, <code>arda/decode_state.py</code>, <code>arda/gqa_routing.py</code></td>
	<td><span class="tag good">Built and tested</span></td>
	<td>Patches Qwen attention forward paths for ARDA readout. GQA routing shares query-independent setup and selected durable node stats while preserving independent per-head searches.</td>
	</tr>
	<tr>
	<td>HF projection and ingestion hooks</td>
	<td><code>arda/hf_ingest.py</code>, <code>arda/hf_projections.py</code>, <code>arda/hf_queries.py</code></td>
	<td><span class="tag good">Built and tested</span></td>
	<td>Captures pre-RoPE key/value projections during model forward passes so leaves come from the actual online model context.</td>
	</tr>
	<tr>
	<td>Long-context scenarios</td>
	<td><code>arda/long_context.py</code></td>
	<td><span class="tag good">Built and tested</span></td>
	<td>Three coherent synthetic-but-realistic conversation families: legal case review, codebase audit, and chemistry literature review. They stream to 100M target tokens without materializing the full text.</td>
	</tr>
	<tr>
	<td>Local/remote runners</td>
	<td><code>scripts/run_long_context_poc.py</code>, <code>scripts/run_comparison_long_context_poc.py</code>, <code>scripts/run_resumable_long_context_poc.py</code>, <code>arda/remote.py</code></td>
	<td><span class="tag good">Built and tested</span></td>
	<td>Supports runtime checks, model smoke, ingest smoke, route smoke, patched decode smoke, streaming ARDA eval, resumable part-boundary runs, and scenario/mode comparisons.</td>
	</tr>
	<tr>
	<td>Evaluation reports</td>
	<td><code>arda/evaluation_report.py</code>, <code>arda/comparison_report.py</code>, <code>scripts/summarize_long_context_poc.py</code></td>
	<td><span class="tag good">Built and tested</span></td>
	<td>Aggregates rank artifacts, stream progress, throughput projections, completion gates, hardware proof, and answer-review summaries.</td>
	</tr>
	<tr>
	<td>Manual answer-review packet</td>
	<td><code>arda/evaluation_report.py</code>, <code>scripts/run_long_context_poc.py</code></td>
	<td><span class="tag warn">In progress</span></td>
	<td>New local changes write <code>answer_review.md</code> with short oracle context, baseline answer, ARDA answer, and review checklist. Tests are passing but the change has not yet been committed at the time this report was generated.</td>
	</tr>
	</tbody>
	</table>
	</section>

	<section>
	<h2 id="tests">Tests and Verification</h2>
	<p>
	The current local suite covers unit, integration, runner, reporting, and synthetic long-context planning behavior.
	The latest full run completed with:
	</p>
	<pre>221 passed, 1 skipped in 2.79s</pre>

	<h3>Major test categories</h3>
	<ul>
	<li><strong>Tree and storage invariants:</strong> sealing schedule, fresh tail, reopen/continue behavior, file-backed indexed lookup.</li>
	<li><strong>Math and bound correctness:</strong> RoPE/angle box primitive, scale handling, value/stat merging, payload serialization.</li>
	<li><strong>Frontier search invariants:</strong> legal cut coverage, budget adherence, no partial synthetic nodes, independent per-head contexts, baseline frontiers.</li>
	<li><strong>HF/Qwen integration:</strong> projection capture, ingestion, patched decode, query extraction, all-leaf/manual attention agreement paths.</li>
	<li><strong>Long-context scenario generation:</strong> 100M-token plans, bounded streaming segments, coherent conversation turns, short-context oracle questions.</li>
	<li><strong>Remote orchestration:</strong> 8-GPU torchrun command generation, resumable part boundaries, comparison runner mode/scenario matrix.</li>
	<li><strong>Reporting:</strong> rank aggregation, progress-only reports, completion gates, throughput projection, POC hardware proof, comparison summaries, answer-review packet.</li>
	</ul>

	<h3>Completion gates now tested</h3>
	<p>
	A report can only mark the full target complete when every rank has completed progress and all of these reach
	the requested target: <code>document_target_tokens_ingested</code>, tokenizer-side <code>prompt_tokens</code>,
	and ARDA <code>tree_token_count</code>. A separate POC-complete gate also requires at least 8 GPUs,
	distinct <code>cuda:0..cuda:7</code> evaluation devices, FlashAttention availability, and
	<code>flash_attention_4</code> as the active attention implementation.
	</p>
	</section>

	<section>
	<h2 id="remote">Remote B200 Runs</h2>
	<p>
	Remote execution uses the 8x B200 machine at <code>198.13.252.92</code>, with the repo synced to
	<code>/scratch/arda/repo</code> and venv at <code>/scratch/arda/venv</code>.
	</p>

	<table>
	<thead>
	<tr>
	<th>Run</th>
	<th>What it proved</th>
	<th>Key results</th>
	<th>Limitations</th>
	</tr>
	</thead>
	<tbody>
	<tr>
	<td><code>comparison_resumable_smoke_10f5928</code></td>
	<td>Real resumable comparison path over two torchrun invocations.</td>
	<td>8 ranks, distinct CUDA devices <code>[0..7]</code>, FlashAttention 4 used, progress status completed on all ranks.</td>
	<td>Capped at 16 document-target tokens. Not a quality or 100M completion proof.</td>
	</tr>
	<tr>
	<td><code>comparison_cap16_fef05de_timing_cold_budget48_lazy</code></td>
	<td>Small timing probe for frontier delta-read cleanup.</td>
	<td>8 rank artifacts, CUDA devices <code>[0..7]</code>, FlashAttention 4 used. The cleanup did not improve mean timing.</td>
	<td>Capped at 16 target tokens; no generated answer tokens.</td>
	</tr>
	<tr>
	<td><code>comparison_d7b99fa_answer_smoke_4tok_complete</code></td>
	<td>ARDA can generate tokens through the remote comparison harness.</td>
	<td>8 ranks and 4 generated tokens. Example behavior was close to the baseline on a tiny legal-case prompt.</td>
	<td>Old artifact predates the stricter POC hardware fields and is capped. Not enough for final quality judgment.</td>
	</tr>
	</tbody>
	</table>

	<h3>Timing observations</h3>
	<ul>
	<li>Shared initial frontier setup reduced small-run mean search time from about <code>24.189s</code> to <code>17.842s</code>.</li>
	<li>Selected GQA stat reuse moved the same small-run mean search time from about <code>17.842s</code> to <code>17.390s</code>.</li>
	<li>The duplicate frontier-delta-read cleanup was semantically good but did not show a timing improvement in the cap-16 probe.</li>
	<li>Per-head, per-layer, per-token cold-start routing is still expensive; the next performance gains likely require batching/vectorized priority scoring or lower-overhead routing loops.</li>
	</ul>
	</section>

	<section>
	<h2 id="worked">What Has Worked</h2>
	<ul>
	<li>The offline-tree design flaw was corrected. Leaves are now produced online by the model under the same approximate-attention regime used for retrieval/generation.</li>
	<li>The tree and frontier APIs enforce fresh-tail behavior structurally, which reduces the chance of policy drift in routers.</li>
	<li>The raw-file storage path exercises disk-backed topology and payload reads instead of assuming the whole tree fits in RAM.</li>
	<li>The Qwen integration can ingest tokens, patch attention, construct ARDA frontiers, and generate through the ARDA path on smoke runs.</li>
	<li>The remote harness can launch 8-process torchrun jobs and produce per-rank artifacts on B200 hardware.</li>
	<li>Completion reporting has become much stricter, making it harder to mistake a capped smoke run for a full POC.</li>
	<li>The synthetic scenarios are coherent enough to be useful: legal case analysis, codebase audit, and chemistry literature review each have consistent topics, repeated document-style packets, assistant acknowledgements, and short-context oracle questions.</li>
	</ul>
	</section>

	<section>
	<h2 id="not-worked">What Has Not Worked Yet</h2>
	<ul>
	<li><strong>No full 100M run has completed.</strong> Existing remote runs are capped smoke or timing runs.</li>
	<li><strong>Holistic answer quality is not yet established.</strong> The old expected-label mention check is too weak; the new answer-review packet is being added to support manual judgment.</li>
	<li><strong>Cold-start per-head routing is slow.</strong> It is the clean correctness baseline, but likely too expensive for a full-quality 100M adaptive run without more batching or routing optimization.</li>
	<li><strong>The duplicate delta-read optimization did not improve measured wall time.</strong> It still simplifies one path and removes redundant work, but the bottleneck is elsewhere.</li>
	<li><strong>Full adaptive-vs-fixed quality curves are not done.</strong> There are comparison runners and capped artifacts, but not a complete long-context quality study.</li>
	<li><strong>Needle/value-channel validation is not yet the main remote result.</strong> The mathematical primitives and tests exist, but the end-to-end long-context evidence still needs to be generated.</li>
	<li><strong>Storage tiering is still basic.</strong> The raw-file store exists, but detailed tensor blob layout, eviction, and hot/cold locality policies have not been optimized.</li>
	</ul>
	</section>

	<section>
	<h2 id="current-work">What I Am Working On Now</h2>
	<p>
	The current active slice is the answer-review packet. The goal is to make each evaluation directory contain a
	human-readable <code>answer_review.md</code> artifact with:
	</p>
	<ul>
	<li>the exact evaluation question,</li>
	<li>the expected evidence label,</li>
	<li>the short-context oracle excerpt used for the fully uncompressed baseline,</li>
	<li>the baseline answer,</li>
	<li>the ARDA answer,</li>
	<li>a short checklist for manual review of evidence match, fact preservation, and relative quality.</li>
	</ul>
	<p>
	This directly supports the user-requested qualitative comparison: read the answers yourself and compare long
	compressed-context behavior with the small fully uncompressed baseline. The tests for this slice currently pass,
	and the full suite passes with the changes included.
	</p>
	</section>

	<section>
	<h2 id="remaining">Remaining Work for Full POC</h2>
	<ol>
	<li><strong>Commit the answer-review packet support.</strong> It is implemented locally and tested, but should be committed and pushed.</li>
	<li><strong>Run an uncapped or meaningfully larger resumable remote job.</strong> Use the resumable comparison runner to start a real 100M scenario/mode directory and let it advance in bounded chunks.</li>
	<li><strong>Monitor throughput and storage growth.</strong> Use progress files and evaluation summaries to estimate whether the current routing path can finish in practical time.</li>
	<li><strong>Optimize routing if needed.</strong> The likely next targets are vectorized priority scoring, lower-overhead queue loops, and possibly Rust for CPU-heavy frontier logic if profiling justifies it.</li>
	<li><strong>Complete holistic answer review.</strong> Generate nontrivial answers, inspect <code>answer_review.md</code>, and compare ARDA vs. baseline answers manually.</li>
	<li><strong>Compare adaptive vs. fixed/recency baselines.</strong> Run the same scenario/question set across routing modes at comparable realized slot counts.</li>
	<li><strong>Only then mark the full goal complete.</strong> The completion audit should require current evidence for all explicit user requirements: strict TDD, realistic coherent 100M conversations, model execution on 8 B200 GPUs, FlashAttention 4, all 8 GPUs used, and manual answer-quality comparison against small exact-context baselines.</li>
	</ol>
	</section>

	<section class="card">
	<h2>Useful Commands</h2>
	<pre># Full local suite
	python -m pytest -q

	# Summarize one remote run directory
	ssh -o StrictHostKeyChecking=no -o ConnectTimeout=15 -i ~/.ssh/pi0-prime-intellect-b200 \
	ubuntu@198.13.252.92 \
	'cd /scratch/arda/repo && /scratch/arda/venv/bin/python scripts/summarize_long_context_poc.py \
	--output-dir /scratch/arda/runs/comparison_resumable_smoke_10f5928/recency'

	# Resumable comparison skeleton for a real long run
	/scratch/arda/venv/bin/python scripts/run_comparison_long_context_poc.py \
	--scenario legal_case_review \
	--output-dir /scratch/arda/runs/<run-name> \
	--modes cold_start_per_layer_head_token,fixed_uniform,recency \
	--parts-per-run 64 \
	--max-iterations 1</pre>
	</section>
	</main>
	<script src="https://pagedrop.ai/pagedrop.js"></script>
	</body>
	</html>
No results found