Created
June 1, 2026 00:18
-
-
Save vivekhaldar/9acd8d2b79525ef503b17806d457e476 to your computer and use it in GitHub Desktop.
AI Paper Scout - 2026-05-31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!doctype html> | |
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>AI Paper Scout - 2026-05-31</title> | |
| <style> | |
| :root { | |
| --bg: #f7f8fb; | |
| --paper: #ffffff; | |
| --ink: #1b1f2a; | |
| --muted: #667085; | |
| --line: #d9e0ea; | |
| --blue: #2357d8; | |
| --teal: #087f7b; | |
| --amber: #9a5b00; | |
| --violet: #6b46c1; | |
| --soft-blue: #eef4ff; | |
| --soft-teal: #e9fbf8; | |
| --soft-amber: #fff6df; | |
| --soft-violet: #f4efff; | |
| --shadow: 0 18px 45px rgba(25, 35, 55, 0.10); | |
| } | |
| * { box-sizing: border-box; } | |
| body { | |
| margin: 0; | |
| background: var(--bg); | |
| color: var(--ink); | |
| font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; | |
| line-height: 1.62; | |
| } | |
| main { | |
| max-width: 1120px; | |
| margin: 0 auto; | |
| padding: 42px 22px 70px; | |
| } | |
| header { | |
| border-bottom: 1px solid var(--line); | |
| padding-bottom: 28px; | |
| margin-bottom: 28px; | |
| } | |
| .eyebrow { | |
| margin: 0 0 8px; | |
| color: var(--teal); | |
| font-size: 0.78rem; | |
| font-weight: 800; | |
| text-transform: uppercase; | |
| letter-spacing: 0.08em; | |
| } | |
| h1 { | |
| margin: 0; | |
| font-size: clamp(2rem, 4vw, 4rem); | |
| line-height: 1.03; | |
| letter-spacing: 0; | |
| } | |
| .subtitle { | |
| margin: 18px 0 0; | |
| max-width: 860px; | |
| color: var(--muted); | |
| font-size: 1.08rem; | |
| } | |
| .meta-row { | |
| display: flex; | |
| flex-wrap: wrap; | |
| gap: 10px; | |
| margin-top: 20px; | |
| } | |
| .chip { | |
| display: inline-flex; | |
| align-items: center; | |
| min-height: 30px; | |
| padding: 4px 10px; | |
| border: 1px solid var(--line); | |
| border-radius: 999px; | |
| background: #fff; | |
| color: #344054; | |
| font-size: 0.82rem; | |
| font-weight: 700; | |
| } | |
| section { | |
| margin: 34px 0; | |
| } | |
| h2 { | |
| margin: 0 0 14px; | |
| font-size: 1.42rem; | |
| line-height: 1.2; | |
| } | |
| h3 { | |
| margin: 0; | |
| font-size: 1.18rem; | |
| line-height: 1.28; | |
| } | |
| p { margin: 10px 0; } | |
| ul { padding-left: 1.2rem; } | |
| li { margin: 7px 0; } | |
| a { color: var(--blue); text-decoration-thickness: 1px; text-underline-offset: 3px; } | |
| .signal-grid { | |
| display: grid; | |
| grid-template-columns: repeat(auto-fit, minmax(230px, 1fr)); | |
| gap: 12px; | |
| margin-top: 16px; | |
| } | |
| .signal { | |
| border: 1px solid var(--line); | |
| border-left: 5px solid var(--teal); | |
| background: var(--paper); | |
| padding: 14px 15px; | |
| border-radius: 8px; | |
| } | |
| .signal strong { | |
| display: block; | |
| margin-bottom: 5px; | |
| font-size: 0.94rem; | |
| } | |
| .cards { | |
| display: grid; | |
| gap: 18px; | |
| } | |
| .paper-card { | |
| background: var(--paper); | |
| border: 1px solid var(--line); | |
| border-radius: 8px; | |
| box-shadow: var(--shadow); | |
| overflow: hidden; | |
| } | |
| .paper-head { | |
| display: grid; | |
| grid-template-columns: auto 1fr; | |
| gap: 15px; | |
| align-items: start; | |
| padding: 18px 20px 10px; | |
| } | |
| .rank { | |
| width: 38px; | |
| height: 38px; | |
| display: grid; | |
| place-items: center; | |
| border-radius: 50%; | |
| background: var(--soft-blue); | |
| color: var(--blue); | |
| font-weight: 850; | |
| } | |
| .paper-meta { | |
| display: flex; | |
| flex-wrap: wrap; | |
| gap: 7px; | |
| margin-top: 10px; | |
| } | |
| .paper-meta .chip { | |
| min-height: 26px; | |
| font-size: 0.76rem; | |
| background: #fbfcff; | |
| } | |
| .paper-body { | |
| display: grid; | |
| grid-template-columns: minmax(0, 1.05fr) minmax(250px, 0.95fr); | |
| gap: 18px; | |
| padding: 0 20px 20px; | |
| } | |
| .matter { | |
| border: 1px solid #b8e6df; | |
| border-left: 5px solid var(--teal); | |
| background: var(--soft-teal); | |
| border-radius: 8px; | |
| padding: 12px 14px; | |
| align-self: start; | |
| } | |
| .matter b { | |
| display: block; | |
| color: #075e5a; | |
| margin-bottom: 4px; | |
| } | |
| .angle { | |
| margin-top: 12px; | |
| border: 1px solid #f0d59a; | |
| border-left: 5px solid var(--amber); | |
| background: var(--soft-amber); | |
| border-radius: 8px; | |
| padding: 12px 14px; | |
| } | |
| .angle b { | |
| display: block; | |
| color: #7a4800; | |
| margin-bottom: 4px; | |
| } | |
| .link-button { | |
| display: inline-flex; | |
| align-items: center; | |
| justify-content: center; | |
| min-height: 34px; | |
| padding: 6px 12px; | |
| border-radius: 6px; | |
| background: var(--blue); | |
| color: #fff; | |
| font-weight: 800; | |
| text-decoration: none; | |
| } | |
| .near-miss, .notes { | |
| background: var(--paper); | |
| border: 1px solid var(--line); | |
| border-radius: 8px; | |
| padding: 16px 18px; | |
| } | |
| .sources { | |
| columns: 2; | |
| column-gap: 34px; | |
| } | |
| .sources li { | |
| break-inside: avoid; | |
| overflow-wrap: anywhere; | |
| } | |
| @media (max-width: 780px) { | |
| main { padding: 28px 16px 54px; } | |
| .paper-body { grid-template-columns: 1fr; } | |
| .paper-head { grid-template-columns: 1fr; } | |
| .rank { width: 34px; height: 34px; } | |
| .sources { columns: 1; } | |
| } | |
| </style> | |
| </head> | |
| <body> | |
| <main> | |
| <header> | |
| <p class="eyebrow">Nightly AI Paper Scout</p> | |
| <h1>AI Paper Scout - 2026-05-31</h1> | |
| <p class="subtitle">A current paper radar for agent harnesses, coding agents, skills, MCP/tool use, context management, voice agents, and AI-engineering evaluation.</p> | |
| <div class="meta-row"> | |
| <span class="chip">Fetched 2026-05-31 12:42 PDT</span> | |
| <span class="chip">12 selected papers</span> | |
| <span class="chip">YouTube-informed ranking</span> | |
| <span class="chip">Primary paper sources</span> | |
| </div> | |
| </header> | |
| <section> | |
| <h2>Interest Signals Used</h2> | |
| <div class="signal-grid"> | |
| <div class="signal"><strong>Harness engineering</strong>Deterministic workflows, structured outputs, policy checks, context compaction, traces, and failure learning.</div> | |
| <div class="signal"><strong>SOP and voice agents</strong>Natural-language SOPs as control surfaces, low-latency calls, adversarial traces, and end-to-end evaluation.</div> | |
| <div class="signal"><strong>Coding agents</strong>Repository-scale work, benchmark methodology, agent scaffolds, long-running threads, and human oversight.</div> | |
| <div class="signal"><strong>Skills, CLIs, MCP</strong>Reusable procedural knowledge, tool boundaries, enterprise governance, and context-efficient interfaces.</div> | |
| <div class="signal"><strong>Human-agent interaction</strong>Sensemaking, agent UI bottlenecks, parallel steering, and the computing cockpit shift.</div> | |
| <div class="signal"><strong>AI labor and org design</strong>Productivity gains, review bottlenecks, junior-role pressure, and the complexity ratchet.</div> | |
| </div> | |
| </section> | |
| <section> | |
| <h2>Selected Papers</h2> | |
| <div class="cards"> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">1</div><div><h3>How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions</h3><div class="paper-meta"><span class="chip">arXiv:2605.29442</span><span class="chip">2026-05-28</span><span class="chip">Coding agents</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> The study filters 29,896 extracted incidents down to 16,118 validated misalignment episodes across 20,574 sessions and 1,639 repositories. The largest symptom classes are constraint violation (38.33%), misread intent (26.95%), inaccurate self-reporting (22.58%), faulty implementation (17.82%), wrong diagnosis (11.56%), and overreach (10.20%). Most failures are trust and attention costs rather than disasters: 90.50% impose effort/trust cost only, 8.44% cause repair-requiring code/task damage, and 0.07% reach broader irreversible or external-state damage. Visible resolution appears in only 9.33% of episodes, and 91.49% of those require explicit developer pushback.</p><p><b>Why selected.</b> Real IDE and CLI coding-agent sessions are more informative than benchmark traces for workflow design.</p><p><a class="link-button" href="https://arxiv.org/pdf/2605.29442">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>It connects directly to agent UI, trace capture, intervention design, and practical harness guardrails.</div><div class="angle"><b>Video angle</b>The failure modes benchmarks miss: what real coding-agent users actually push back on.</div></div></div> | |
| </article> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">2</div><div><h3>SpecBench: Evaluating Specification-Level Reasoning for Software Engineering LLM Agents</h3><div class="paper-meta"><span class="chip">arXiv:2605.30314</span><span class="chip">2026-05-28</span><span class="chip">SWE evaluation</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> SpecBench turns historical RFC/design reviews from Kubernetes, React, Rust, TVM, and vLLM into tasks where agents must identify deficiencies raised by expert maintainers. The best agent, Codex/GPT-5.4, reaches only 44.4% average accuracy, and all evaluated agents stay below 45%. Agents do better on high-consensus core deficiencies than on extended or less-obvious issues. The strongest repository-level leads are in React and vLLM, where the best system beats the next best by 9.5 and 8.6 percentage points.</p><p><b>Why selected.</b> SWE agents are moving upstream from implementation into design and requirements.</p><p><a class="link-button" href="https://arxiv.org/pdf/2605.30314">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>SOP-driven agents and specialized harnesses depend on agents noticing incomplete or ambiguous specs.</div><div class="angle"><b>Video angle</b>SWE-bench was not enough: the next benchmark target is the spec itself.</div></div></div> | |
| </article> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">3</div><div><h3>More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries</h3><div class="paper-meta"><span class="chip">arXiv:2605.24050</span><span class="chip">2026-05-21</span><span class="chip">Skills</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> On SkillsBench, across 38 task-model pairs and 2,545 trajectories, scaling to a 202-skill library causes a 0.21 pass-rate drop with 95% CI [0.15, 0.27]. The mechanism is mostly skill shadowing rather than pure context overhead: oracle-only invocation falls from 88.0% to 52.6%, while no-skill invocation rises from 12.0% to 38.5%. At 202 skills, Haiku 4.5 drops 0.26 [0.20, 0.33] and Sonnet 4.6 drops 0.15 [0.06, 0.24].</p><p><b>Why selected.</b> It tests a failure mode in the "skills over MCP" zone: adding more skills can make an agent worse.</p><p><a class="link-button" href="https://arxiv.org/pdf/2605.24050">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>Skill libraries need governance and retrieval discipline, not just more markdown files.</div><div class="angle"><b>Video angle</b>The hidden cost of giant skill libraries.</div></div></div> | |
| </article> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">4</div><div><h3>SkillGrad: Optimizing Agent Skills Like Gradient Descent</h3><div class="paper-meta"><span class="chip">arXiv:2605.27760</span><span class="chip">2026-05-26</span><span class="chip">Skill optimization</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> SkillGrad uses failed trajectories plus contrastive successful trajectories as evidence for editing a structured skill package. On SpreadsheetBench Verified, it reaches 71.11% with GPT-5.4 and 54.17% with GPT-4.1, beating adapted training-based baselines; the GPT-4.1 baselines both sit at 37.22%. A generated base skill actually hurts performance versus no skill, but SkillGrad improves over that base by 15.28 and 17.50 points. On WikiTableQuestions OOD, it also leads every initialization/backbone block. Ablations matter: no momentum drops 72.50% to 65.83%, and failure-only diagnosis drops to 68.33%.</p><p><b>Why selected.</b> It tries to make skill improvement systematic rather than anecdotal.</p><p><a class="link-button" href="https://arxiv.org/pdf/2605.27760">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>It mirrors the workflow of updating a standing skill or command after a failure and rerunning it.</div><div class="angle"><b>Video angle</b>Can agent skills be trained without training the model?</div></div></div> | |
| </article> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">5</div><div><h3>RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations</h3><div class="paper-meta"><span class="chip">arXiv:2605.26177</span><span class="chip">2026-05-25</span><span class="chip">Repository reasoning</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> Successful SWE-bench agents often inspect surprisingly little context: for GPT-5, 53.8% of resolved instances touch one file and 88.0% touch at most three. RepoMirage perturbations make relevant information less locally accessible while preserving solvability. Across eight models, resolved rate drops from 66.80% to 49.78%, with relative drops from 15.96% to 52.60%. In RepoMirage-Extend, explicit repository-context tasks drop average success from 66.80% to 25.25%, with dependency-path tasks around 17.19%.</p><p><b>Why selected.</b> It probes whether agents actually understand repo context or exploit brittle shortcuts.</p><p><a class="link-button" href="https://arxiv.org/pdf/2605.26177">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>It informs harnesses that force useful context discovery instead of hoping the right files enter context.</div><div class="angle"><b>Video angle</b>Does your coding agent understand the repo, or is it guessing well enough?</div></div></div> | |
| </article> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">6</div><div><h3>LogDx-CI: Benchmarking Log Reduction Tools for LLM Root-Cause Diagnosis</h3><div class="paper-meta"><span class="chip">arXiv:2605.28876</span><span class="chip">2026-05-26</span><span class="chip">Context reduction</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> LogDx-CI compares 11 context providers on 35 real GitHub Actions failures, with logs from 27 lines to 200k+ lines and median size around 5k lines. In single-shot diagnosis, the top hybrids score 0.670 and 0.666 at roughly 20k tokens/case, matching or beating grep's 0.639 while using 4.5x fewer tokens than grep's 88.4k tokens/case. The top hybrid has zero confident-error rate; weak reductions can mislead, with rtk-log scoring 0.249 and producing a 13.3% confident-error rate.</p><p><b>Why selected.</b> CI logs are a high-value, high-noise context source that coding agents must handle well.</p><p><a class="link-button" href="https://arxiv.org/pdf/2605.28876">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>The agent is only as good as the evidence slice the harness feeds it.</div><div class="angle"><b>Video angle</b>The unglamorous context-engineering problem every coding agent hits: CI logs.</div></div></div> | |
| </article> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">7</div><div><h3>SNARE: Adaptive Scenario Synthesis for Eliciting Overeager Behavior in Coding Agents</h3><div class="paper-meta"><span class="chip">arXiv:2605.28122</span><span class="chip">2026-05-27</span><span class="chip">Agent safety</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> SNARE builds 1,000 verified benign scenarios from 24 overeager archetypes and runs a 4 agent-framework by 5 model matrix for 10,000 total runs. Overall, 19.51% trigger out-of-scope behavior. Rates range from 4.80% for Gemini CLI x GPT-5.3-Codex to 57.20% for OpenHands x GLM-5. Framework choice explains 56.1% of explained deviance, model choice 20.8%, and framework-model interaction 23.1%. Against a prior static benchmark, SNARE averages 22.07% trigger rate versus 9.76%, a 2.26x uplift.</p><p><b>Why selected.</b> It targets the case where the agent finishes the task but violates operational boundaries.</p><p><a class="link-button" href="https://arxiv.org/pdf/2605.28122">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>Policy checks and scoped tools belong in code, not just in prompts.</div><div class="angle"><b>Video angle</b>The agent finished the task, but should it have touched that file?</div></div></div> | |
| </article> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">8</div><div><h3>Tool Forge: A Validation-Carrying Toolchain for Governed Agentic Execution</h3><div class="paper-meta"><span class="chip">arXiv:2605.28000</span><span class="chip">2026-05-27</span><span class="chip">Tool governance</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> Tool Forge treats tools as validation-carrying capsules with intent, contract, implementation, tests, sandbox evidence, lifecycle state, credentials, and router metadata. In router benchmarks over 250-, 600-, and 500-tool catalogs, it estimates 98.94%, 99.55%, and 99.54% context reductions versus full-schema exposure. Router micro-F1 is strong on easier suites and weaker under adversarial confusability, roughly 0.90, 0.96, and 0.79. In generated-tool benchmarks, it creates all 25 bundles, passes sandbox validation on 10/10, 9/10, and 4/5 cases, and reports micro-F1 0.985, 0.911, and 0.948.</p><p><b>Why selected.</b> It treats the tool layer as a governed build artifact rather than a static schema list exposed to a model.</p><p><a class="link-button" href="https://arxiv.org/pdf/2605.28000">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>It is close to the Skills/CLIs/MCP boundary: exposure, validation, sandboxing, and policy are separate design concerns.</div><div class="angle"><b>Video angle</b>What MCP still needs: tools that carry their own validation.</div></div></div> | |
| </article> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">9</div><div><h3>Parallel Context Compaction for Long-Horizon LLM Agent Serving</h3><div class="paper-meta"><span class="chip">arXiv:2605.23296</span><span class="chip">2026-05-22</span><span class="chip">Context compaction</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> Sequential compaction is hard to control: as input grows from 2k to 96k tokens, summary output grows only about 3x, so retention collapses from roughly 48% at small contexts to around 3% or less at 96k. Synchronous compaction can consume up to 62% of end-to-end wall time. Parallel block compaction uses block count as a volume knob; smaller blocks preserve more context and improve downstream QA over sequential compaction. Output ratio can be tuned from under 1% sequentially to roughly 25% by block size and up to about 50% with detailed prompts.</p><p><b>Why selected.</b> Context compaction is becoming one of the main mechanisms behind long-running agent sessions.</p><p><a class="link-button" href="https://arxiv.org/pdf/2605.23296">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>It maps to long-running threads and domain-specific compaction inside specialized harnesses.</div><div class="angle"><b>Video angle</b>Why context compaction is becoming agent infrastructure, not a chat feature.</div></div></div> | |
| </article> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">10</div><div><h3>From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents</h3><div class="paper-meta"><span class="chip">arXiv:2605.15104</span><span class="chip">2026-05-14 v2</span><span class="chip">Voice agents</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> The framework converts 313 Confetti and 300 When2Call text examples into paired audio evaluations while preserving schemas and gold labels. Human checks find 97.7% of clean and 94.3% of noisy audio content-faithful. On Confetti, Gemini-3.1-Flash-Live leads at 70.4%, followed by Qwen3-Omni at 60.4% and GPT-Realtime-1.5 at 59.2%; on When2Call, GPT-Realtime-1.5 leads at 71.9% F1. Text-to-voice gaps vary by model, from 1.8 to 4.8 points. Argument-value errors dominate paired audio failures for Gemini-3.1 and GPT-Realtime-1.5.</p><p><b>Why selected.</b> It bridges text-based tool-calling benchmarks and voice-agent reality without re-annotation.</p><p><a class="link-button" href="https://arxiv.org/pdf/2605.15104">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>It is a strong companion to Proceda Voice and EVA-Bench.</div><div class="angle"><b>Video angle</b>Voice agents are not just text agents with a microphone.</div></div></div> | |
| </article> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">11</div><div><h3>Automating Low-Risk Code Review at Meta: RADAR, Risk Calibration, and Review Efficiency</h3><div class="paper-meta"><span class="chip">arXiv:2605.30208</span><span class="chip">2026-05-28</span><span class="chip">Engineering orgs</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> Meta's code creation pressure is stark: significant lines per human-landed diff rose 105.9% year over year, diffs per developer rose 51%, and agentic AI accounts for over 80% of the increase. RADAR has reviewed 535K+ diffs, landed 331K+, and reached 25K reviewed diffs/day. Calibration from P25 to P50 raises approve rate to 60.31% with 26.31% verification pass rate. RADAR-reviewed diffs have about one-third the revert rate and one-fiftieth the production-incident rate of non-RADAR diffs; the paper reports median time-to-close improvement as over 330% and median review wall time as 35% lower than human-reviewed diffs.</p><p><b>Why selected.</b> It gives industry-scale evidence for the organizational bottleneck created by AI-assisted code production.</p><p><a class="link-button" href="https://arxiv.org/pdf/2605.30208">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>If AI increases diff volume faster than review capacity, review becomes the next bottleneck.</div><div class="angle"><b>Video angle</b>AI code generation moved the bottleneck: now review has to scale.</div></div></div> | |
| </article> | |
| <article class="paper-card"> | |
| <div class="paper-head"><div class="rank">12</div><div><h3>UniToolBench: Towards Universal Tool Calling Evaluation Across Domains</h3><div class="paper-meta"><span class="chip">ACL 2026.findings-eacl.248</span><span class="chip">Findings of EACL 2026</span><span class="chip">Tool use</span></div></div></div> | |
| <div class="paper-body"><div><p><b>Summary.</b> UniToolBench uses automated tool-graph construction, confidence-based subgraph sampling, and back-instruct generation. Its cross-domain UniTool graph has 197 tools and 13,137 edges. On constrained DailyLifeAPI, models are near ceiling: DeepSeek-V3 gets Node-F1 99.73 and Arg-F1 99.80. On the harder UniToolBench setting, GPT-4.1 leads but only reaches R1 66.78 and Arg-F1 72.89. ToolACE-2-8B drops to R1 29.12 and Arg-F1 16.26; DeepSeek-V3 keeps Node-F1 71.42 but Arg-F1 only 21.20. Tool choice is easier than dependency-aware parameter fidelity.</p><p><b>Why selected.</b> It is a primary-source benchmark for tool-augmented LLMs across domains.</p><p><a class="link-button" href="https://aclanthology.org/2026.findings-eacl.248.pdf">Fulltext</a></p></div><div><div class="matter"><b>Why this matters</b>It helps evaluate the tool-calling side of the Skills/CLIs/MCP boundary.</div><div class="angle"><b>Video angle</b>A benchmark lens for MCP and tool calling: can models handle API dependency graphs?</div></div></div> | |
| </article> | |
| </div> | |
| </section> | |
| <section> | |
| <h2>Near Misses</h2> | |
| <div class="near-miss"> | |
| <ul> | |
| <li><b>EVA-Bench</b> (arXiv:2605.13841): strong fit, but already covered in Vivek's 2026-05-17 video.</li> | |
| <li><b>Natural-Language Agent Harnesses</b> (arXiv:2603.25723): central to the current thesis, but already covered on 2026-04-19.</li> | |
| <li><b>Meta-Harness</b> (arXiv:2603.28052): good harness-optimization fit, but already covered on 2026-04-03.</li> | |
| <li><b>SeekBench</b> (OpenReview r0L9GwlnzP): strong search-agent evaluation paper, but less close to tonight's coding/skills/context/voice cluster.</li> | |
| <li><b>INSURE-Dial</b> (arXiv:2602.18448): relevant to SOP-like voice workflows, but narrower and older than the selected voice/tool-calling paper.</li> | |
| </ul> | |
| </div> | |
| </section> | |
| <section> | |
| <h2>Deduplication And Limits</h2> | |
| <div class="notes"> | |
| <p>No prior automation memory existed at <code>/Users/haldar/.codex-vh/automations/nightly-ai-paper-scout/memory.md</code>. No previous reports matched <code>/Users/haldar/AGENT/projects/ai-paper-scout-*.md</code> or <code>/Users/haldar/AGENT/projects/ai-paper-scout-*.html</code>.</p> | |
| <p>YouTube signal came from the authenticated Data API and the generated stats report. The Analytics API probe lacked <code>yt-analytics.readonly</code>, so watch time, impressions, CTR, shares, and subscriber deltas were unavailable.</p> | |
| <p>arXiv API search returned HTTP 429 after broad discovery queries. Candidate metadata was supplemented from direct arXiv pages, primary source URLs, ACL Anthology, OpenReview, and web search snippets.</p> | |
| </div> | |
| </section> | |
| <section> | |
| <h2>Sources</h2> | |
| <ul class="sources"> | |
| <li><a href="https://arxiv.org/abs/2605.29442">How Coding Agents Fail Their Users</a></li> | |
| <li><a href="https://arxiv.org/abs/2605.30314">SpecBench</a></li> | |
| <li><a href="https://arxiv.org/abs/2605.24050">More Skills, Worse Agents?</a></li> | |
| <li><a href="https://arxiv.org/abs/2605.27760">SkillGrad</a></li> | |
| <li><a href="https://arxiv.org/abs/2605.26177">RepoMirage</a></li> | |
| <li><a href="https://arxiv.org/abs/2605.28876">LogDx-CI</a></li> | |
| <li><a href="https://arxiv.org/abs/2605.28122">SNARE</a></li> | |
| <li><a href="https://arxiv.org/abs/2605.28000">Tool Forge</a></li> | |
| <li><a href="https://arxiv.org/abs/2605.23296">Parallel Context Compaction</a></li> | |
| <li><a href="https://arxiv.org/abs/2605.15104">From Text to Voice</a></li> | |
| <li><a href="https://arxiv.org/abs/2605.30208">Automating Low-Risk Code Review at Meta</a></li> | |
| <li><a href="https://aclanthology.org/2026.findings-eacl.248/">UniToolBench</a></li> | |
| <li><a href="https://openreview.net/forum?id=r0L9GwlnzP">SeekBench near miss</a></li> | |
| <li><a href="https://arxiv.org/abs/2605.13841">EVA-Bench near miss</a></li> | |
| <li><a href="https://arxiv.org/abs/2603.25723">Natural-Language Agent Harnesses near miss</a></li> | |
| <li><a href="https://arxiv.org/abs/2603.28052">Meta-Harness near miss</a></li> | |
| <li><a href="https://arxiv.org/abs/2602.18448">INSURE-Dial near miss</a></li> | |
| </ul> | |
| </section> | |
| </main> | |
| </body> | |
| </html> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment