Task Eval Analysis Guide

Deep dive reference for analyzing Horizon evaluation results.

Quick Assessment

# View summary
cat eval_results.json | python3 -m json.tool

# Key metrics
jq '.tasks[].pass_rate' eval_results.json
jq '.tasks[].runs[].score' eval_results.json

Decision Matrix

Thresholds below use mean score across all runs for biggie-nebula (8 runs required for acceptance). Thresholds differ by backend: ≤0.85 on teapot (hosted/firecracker), <0.50 on docker (local). See Review Guide: Backend-Specific Thresholds for how to identify the backend.

Mean Score (`biggie-nebula`)	Failure Pattern	Verdict
Below threshold	Multiple modes	PASS — Task is appropriately difficult
Below threshold	All same check	NEEDS WORK — Fix grader timing
Above threshold	N/A	NEEDS WORK — Task too easy
0.0	Agent stuck early	NEEDS WORK — Task unclear or blocked
0.0	Agent progresses, grader fails	NEEDS WORK — Grader issue

Failure Mode Analysis

Step 1: Collect Failure Data

For API-downloaded transcripts (from scripts/download-transcripts.py):

# Download docker (biggie-nebula) transcripts — default
python scripts/download-transcripts.py <uuid> --output-dir eval_transcripts

# Download teapot (nighthawk) transcripts — version auto-resolved via fingerprinting
python scripts/download-transcripts.py <uuid> --model nighthawk --output-dir eval_transcripts

Note: Nighthawk (teapot) rollouts have null version numbers in the API. The script resolves them automatically by fingerprint-matching prompt content against versioned rollouts. Check resolved_version and resolution_method fields in the output JSON to verify.

# List all run scores
for f in eval_transcripts/<uuid>-run*-batch*.json; do
  score=$(python3 -c "import json; print(json.load(open('$f'))['extracted_score'])")
  echo "$f: $score"
done

# Extract per-check subscores
python3 -c "
import json, glob
for f in sorted(glob.glob('eval_transcripts/<uuid>-run*-batch*.json')):
    d = json.load(open(f))
    gr = d['grade_result'] if isinstance(d['grade_result'], dict) else json.loads(d['grade_result'])
    checks = ' '.join(f\"{k}={'PASS' if v else 'FAIL'}\" for k,v in gr['subscores'].items())
    print(f\"Run {d['run_number']}: score={d['extracted_score']} | {checks}\")
"

Step 2: Categorize Failures

Category A: Genuine Difficulty

Symptom: Multiple different failure modes across runs
Agent tries reasonable approaches and makes progress
Failures at various stages; some runs get further than others
Verdict: Task is appropriately challenging (good)

Category B: Task Clarity / Underspecification

Symptom: Agent attempts wrong approach or reasonable-but-ungraded approach
Agent doesn't understand objective, or interprets it differently than grader expects
Multiple runs try fundamentally different (incorrect) strategies
Fix: Clarify task.yaml prompt

Category C: Environment / Timing

Symptom: Early failures before agent starts real work, or correct work not detected
Connection refused, service unavailable, resource not ready when grader checks
Bootstrap or setup.sh problems
Fix: Debug setup.sh, add wait loops in grader.py

Category D: Grader Bug

Symptom: Agent clearly completes work but grader reports failure
Grader checks wrong resource, wrong namespace, or has inverted logic
Non-deterministic check produces different results for same state across runs
Grader crashes or produces malformed output
Fix: Debug and fix the check function

Category E: Grader Path Restriction

Symptom: Agent achieves correct outcome via different method, grader rejects
Grader checks specific implementation artifacts (file paths, command side-effects) rather than outcomes
task.yaml doesn't mandate the specific approach the grader expects
Fix: Make grader check outcomes not mechanisms, or constrain task.yaml to specify the approach

Category F: Redundant / Composite Check

Symptom: A check always fails when another specific check fails (perfectly correlated)
Check is a superset of others — can only pass if prerequisites pass too
Creates double-penalty for a single conceptual miss
Fix: Remove redundant check or replace with an independent verification

Step 3: Per-Check Analysis

# Analyze which checks fail most often (API-downloaded transcripts)
from collections import Counter
import json, glob

failures = Counter()
for f in glob.glob("eval_transcripts/*-run*.json"):
    with open(f) as fp:
        data = json.load(fp)
        gr = json.loads(data['grade_result'])
        for check, passed in gr.get('subscores', {}).items():
            if passed < 1.0:
                failures[check] += 1

for check, count in failures.most_common():
    print(f"{check}: {count} failures")

Step 4: Zero-Pass Check Investigation

Any check with 0 passes across all runs is a red flag. Do not assume genuine difficulty — investigate:

Read the grader function — Does the check test what it claims? Is the logic correct?
Read transcripts — Did any agent attempt to address this? Did any come close?
Compare grader vs task.yaml — Is the requirement discoverable from the prompt alone?
Check for path restriction — Could the agent have achieved the goal via a different approach the grader doesn't accept?
Check for dependency — Is this check a composite that depends on other failing checks?

Possible classifications:

Genuine extreme difficulty — Agents understand but this aspect is very hard
Hidden requirement — Not discoverable from task.yaml alone
Grader defect — Bug, path restriction, or non-determinism
Cascading failure — Depends on another failing check

Transcript Analysis

Reading API-Downloaded Transcripts

# Extract agent actions from API transcript
import json
data = json.load(open('eval_transcripts/<uuid>-run1.json'))
for msg in data['messages']:
    if msg['role'] == 'assistant':
        print(f"[{msg['sequence_number']}] {msg['content'][:200]}")

What to Look For

Early turns: Does agent understand the task?
Middle turns: Is agent making progress toward graded objectives?
Late turns: Did agent complete or get stuck? Did it do unscored work?
Final state: What did agent accomplish vs. what grader expected?
Correct-but-rejected work: Did agent achieve the outcome via a path the grader doesn't check?

Red Flags

Pattern	Indicates
Agent asks clarifying questions	Task unclear
Agent tries multiple unrelated approaches	Task ambiguous
Agent completes then does unrelated work	Task scope unclear
Agent stuck in loop	Blocker or unclear success criteria
Agent finishes early, grader fails	Grader timing or alignment issue
Agent achieves goal, grader says FAIL	Grader path restriction or bug
Same agent behavior, different grader result across runs	Non-deterministic grader
Grader feedback within a run shows physically-impossible check pairs (e.g., `probes: N/N OK` alongside `unreachable`)	Grader raced a rollout / reconcile; check `transcript-summary.py <uuid> race-check`
Agent never attempts a check's requirement	Hidden/undiscoverable requirement

Grader Debugging

Test Grader Independently

docker exec -it nebula-fast-boot-test bash

# After running solution manually:
python3 << 'EOF'
import sys
sys.path.insert(0, '/path/to/task')
from grader import grade
result = grade('')
print(f'Score: {result.score}')
print(f'Subscores: {result.subscores}')
print(f'Feedback: {result.feedback}')
EOF

Grader Defect Patterns

When reviewing grader.py during eval analysis, look for these patterns:

Race conditions / timing:

# Bad: Check immediately
def check_deployment():
    code, out, _ = run("kubectl get deployment foo -n bar")
    return code == 0

# Good: Wait for readiness
def check_deployment():
    for _ in range(30):
        code, out, _ = run("kubectl rollout status deployment/foo -n bar --timeout=10s")
        if code == 0:
            return True
        time.sleep(10)
    return False

Wrong resource / namespace:

# Bad: Assume namespace
run("kubectl get secret tls-cert")

# Good: Explicit namespace
run("kubectl get secret tls-cert -n ingress-nginx")

Brittle string matching:

# Bad: Exact string match
return output == "Running"

# Good: Flexible matching
return "Running" in output or output.strip() == "Running"

Path restriction (checks mechanism, not outcome):

# Bad: Checks specific file exists (agent might use a different approach)
def check_tls():
    return os.path.exists("/etc/nginx/ssl/cert.pem")

# Good: Checks actual outcome (TLS is working)
def check_tls():
    code, out, _ = run("curl -sk https://app.local")
    return code == 0

Non-deterministic ordering:

# Bad: Depends on kubectl output order
def check_labels():
    out = run("kubectl get pods -l app=web -o name")
    return out.split('\n')[0] == "pod/web-abc123"

# Good: Checks set membership
def check_labels():
    out = run("kubectl get pods -l app=web -o name")
    return len(out.strip().split('\n')) >= 1

Redundant / composite check:

# Problematic: This can only pass if checks 1-3 also pass
# Creates double-penalty for any single miss
def cluster_fully_resolved():
    return (check_a() and check_b() and check_c())

# Better: Each check tests an independent milestone

Reporting Format

When reporting eval analysis:

## Eval Analysis: <task-name>

**Summary:**
- Model: biggie-nebula
- Runs: 8
- Pass rate: X%
- Verdict: PASS / NEEDS WORK

**Failure Breakdown:**
| Check | Failures | Category |
|-------|----------|----------|
| check_foo | 3 | Timing |
| check_bar | 7 | Genuine |

**Recommendations:**
1. [Specific fix for timing issues]
2. [Or: Task is appropriately difficult, approve]

**Sample Transcript Notes:**
- Run 3: Agent completed but grader missed due to timing
- Run 7: Agent took wrong approach, unclear from task.yaml

arubis/TASK_EVAL_ANALYSIS.md

Select an option

No results found