PLS prediction golden set runner - job f3ffcd19

acceptance-tests.md

E2E Acceptance Tests

A. Golden Set Import

Given prediction-golden-set-seed.csv, when import runs, then 20 cases are accepted with required fields.

Pass: accepted_rows = 20 and schema_errors = 0.

B. Evidence Coverage

Given a golden set run, when evidence sync is complete, then at least 80% of cases have one or more evidence links.

Pass: evidence_coverage >= 80%.

C. Hit Rate

Given human-reviewed ground truth exists for high-risk cases, when scoring is compared to ground truth, then hit_rate >= 70%.

Pass: hit_rate >= 70%; if lower, do not promote to agent.

D. False Positive Guard

Given verdict is hit, when human review marks wrong evidence or wrong interpretation, then false_positive_flag = true.

Pass: false_positive_rate <= 15%.

E. Regression Case Creation

Given a case fails due to no_evidence, wrong_match, or overconfident scoring, when runner finishes, then a regression case is created with owner and due date.

Pass: every failed high-impact case has regression owner.

F. High-Risk Governance

Given risk_tier = high, when runner produces an automatic verdict, then final status remains needs_review until Louis or zihrou approves.

Pass: no high-risk automatic state change.

Current Verification

• HTML runner is openable.
• learning-memory.json parses.
• Gist URL returns HTTP 200.
• Gist file list includes required files.
• PLS upload-files reports uploaded count.

artifact-url-or-pr.md

Artifact URL / PR Record

Primary artifact URL: https://gist.github.com/esz135888/268d9ccae29121e7cb5708f49efa5a7b

Type

Shared-cloud Gist artifact pack. No GitHub PR or deployment is claimed for this job.

Verification

• Gist HTTP status: HTTP/2 200 verified on 2026-05-24 Asia/Taipei.
• Gist file list: 13 files verified by gh gist view 268d9ccae29121e7cb5708f49efa5a7b --files.
• Golden set rows: 20 verified by tail -n +2 prediction-golden-set-seed.csv | wc -l.
• PLS upload-files: pending before final PLS sync.

Required Contract Kinds

primary_artifact, solution_selection, skill_usage, market_context, market_maturity, production_readiness, production_acceptance, e2e_verification, people_sync, learning_memory, landing_record, tool, dashboard, doc.

data-model.md

Data Model / API / Sync / Permission Spec

Tables

prediction_golden_cases

column	type	required	note
case_id	text	yes	stable golden case id
prediction_id	text	yes	source prediction
project_id	uuid	yes	PLS project
prediction_text	text	yes	claim to validate
expected_signal_type	text	yes	action_item/github_commit/deliverable/message/status_change
expected_evidence_query	text	yes	search/join description
expected_by	date	yes	validation date
risk_tier	enum	yes	low/medium/high
ground_truth_verdict	enum	no	hit/partial/miss when known
human_reviewer	text	no	required for high risk

prediction_runner_results

column	type	required	note
run_id	uuid	yes	runner execution
case_id	text	yes	golden case
evidence_count	int	yes	matched rows
top_source_type	text	no	best evidence source
match_strength	numeric	yes	0-1
hit_score	numeric	yes	0-100
verdict	enum	yes	hit/partial/miss/needs_review
false_positive_flag	boolean	yes	reviewer or rule flag
error_message	text	no	schema/sync errors

prediction_regression_cases

column	type	required	note
regression_id	uuid	yes	regression id
source_case_id	text	yes	failed golden case
failure_type	text	yes	no_evidence/wrong_match/overconfident/high_risk
expected_fix	text	yes	rubric/source/schema change
owner_profile_id	uuid	yes	Louis/zihrou/iron
due_at	date	yes	remediation date

API / Sync

• POST /api/prediction-golden-set/import imports CSV golden cases.
• POST /api/prediction-golden-set/run runs matching and scoring.
• GET /api/prediction-golden-set/report?run_id=... returns metrics and failed cases.
• POST /api/prediction-golden-set/:case_id/human-review records high-risk human verdict.

Permissions / Audit

• Louis can approve production promotion and agent authority.
• zihrou owns high-risk approval matrix and override reason.
• iron owns data source sync and schema changes.
• PLS worker can run low/medium risk scoring, but cannot auto-approve high-risk verdicts.
• Audit log stores case_id, old verdict, new verdict, reviewer, reason, evidence ids, timestamp.

Rollback / Failure

• If sync error > 5%, mark run invalid and dispatch repo_change.
• If false_positive_rate > 15%, block agent upgrade.
• If high-risk case lacks human review, mark needs_review and do not change project state.

decision-record.md

Decision Record

Decision

Build Prediction Golden Set Runner as this round's production artifact.

Problem

The previous eval console defined the prediction validation concept. The current contract requires stronger production acceptance and skill/tool evidence. The next useful step is a testable golden set runner that can produce metrics and regression cases.

Options Considered

1. Re-issue Eval Console.
   • Rejected: duplicates previous artifact.
2. Jump to autonomous prediction-validation agent.
   • Rejected: no 20-case reliability baseline yet.
3. Build golden set runner and production acceptance pack.
   • Recommended: creates a measurable bridge from eval concept to PLS backend workflow.

Recommendation

Run the golden set first. Promote only if metrics pass: hit_rate >= 70%, evidence_coverage >= 80%, false_positive_rate <= 15%.

Adoption Status

Proposed for job f3ffcd19-559d-4803-895a-31d3765e5808.

Landing Path

Owner: Louis. Governance: zihrou. Implementation: iron. Due: 2026-05-30.

Feedback If Not Adopted

Return preferred source priorities, actual prediction validation endpoint, high-risk approval policy, or reason to skip golden set and accept higher automation risk.

learning-memory.json

{
  "job_id": "f3ffcd19-559d-4803-895a-31d3765e5808",
  "project": "AI 自建專案：公司AI化",
  "learned_at": "2026-05-24T06:50:00+08:00",
  "solution_selection": "eval + spreadsheet + system + watchdog",
  "market_context": [
    {
      "source": "OpenAI evaluation best practices",
      "lesson": "Continuous evaluation should grow eval sets from production, historical, and human-curated data."
    },
    {
      "source": "LangSmith evaluation platform",
      "lesson": "Mature eval workflows run in production monitoring and PR/nightly builds."
    },
    {
      "source": "LangChain production monitoring to regression tests",
      "lesson": "Production failures should become offline regression cases."
    }
  ],
  "pls_next_checks": [
    "Check whether golden set has at least 20 cases before agent promotion.",
    "Track evidence_coverage, hit_rate, false_positive_rate, and sync_error_rate.",
    "Require human review for high-risk predictions before project state changes.",
    "Turn every high-impact miss into a regression case with owner and due date."
  ],
  "assumptions_overturned": [
    "A console alone is not enough; reliability needs a maintained golden set.",
    "Tool choice should follow source sync and eval metrics, not preference.",
    "AI prediction confidence is not a production metric unless later evidence validates it."
  ],
  "next_iteration_condition": "Run the 20-case golden set against real PLS evidence and produce the first prediction reliability report."
}

market-maturity.md

Market Maturity

Current Practice Check

Checked on 2026-05-24 Asia/Taipei using web search.

Mature Market Pattern

Production AI evaluation in 2026 is converging on continuous evals, production monitoring, trace-to-dataset workflows, regression tests, human review for high-risk outputs, and observability with cost/latency/quality metrics.

Comparable Practices

• OpenAI evaluation guidance recommends continuous evaluation, growing eval sets over time, and using production/historical/human-curated data.
• LangSmith positions evals across production monitoring, PR/nightly evals, and continuous agent improvement.
• LangChain describes a loop where production issues become offline test cases and fixed bugs become regression tests.
• Braintrust, Langfuse, Arize, and similar observability stacks show that mature teams separate traces, evals, replay/debug, and alerting.

PLS Gap

PLS has signals and prediction validation intent, but lacks a maintained golden set and explicit pass/fail production gates. Without that, prediction quality cannot be trusted enough to affect project priority or agent authority.

This Round's Upgrade

This round adds:

• 20-case golden set seed.
• metrics gates for promotion.
• regression case creation for failures.
• high-risk human review.
• source sync and evidence coverage thresholds.

prediction-golden-set-runner.html

<!doctype html>
<html lang="zh-Hant">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Prediction Golden Set Runner</title>
  <style>
    :root { --ink:#172033; --muted:#617086; --line:#d8dee9; --bg:#f5f7fb; --panel:#fff; --green:#087443; --red:#b42318; --amber:#a15c07; --blue:#175cd3; }
    * { box-sizing:border-box; }
    body { margin:0; background:var(--bg); color:var(--ink); font:14px/1.5 -apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif; }
    header { background:var(--panel); border-bottom:1px solid var(--line); padding:28px 32px; }
    h1 { margin:0 0 6px; font-size:26px; letter-spacing:0; }
    h2 { margin:0 0 12px; font-size:17px; }
    main { max-width:1240px; margin:0 auto; padding:22px 18px 42px; display:grid; gap:16px; }
    section { background:var(--panel); border:1px solid var(--line); border-radius:8px; padding:18px; }
    .grid { display:grid; grid-template-columns:repeat(4,minmax(0,1fr)); gap:12px; }
    .card { border:1px solid var(--line); border-radius:8px; background:#fbfcff; padding:14px; min-height:116px; }
    .label { color:var(--muted); font-size:12px; text-transform:uppercase; }
    .value { font-size:24px; font-weight:750; margin-top:4px; }
    .green { color:var(--green); } .red { color:var(--red); } .amber { color:var(--amber); } .blue { color:var(--blue); }
    table { width:100%; border-collapse:collapse; }
    th,td { text-align:left; vertical-align:top; border-bottom:1px solid var(--line); padding:10px 8px; }
    th { color:var(--muted); font-size:12px; }
    code { background:#eef2f7; border-radius:4px; padding:1px 5px; }
    .small { color:var(--muted); font-size:12px; }
    .pill { display:inline-block; border:1px solid var(--line); border-radius:999px; padding:2px 9px; background:#fff; }
    @media (max-width:900px){ header{padding:22px 18px;} .grid{grid-template-columns:1fr;} }
  </style>
</head>
<body>
  <header>
    <h1>Prediction Golden Set Runner</h1>
    <div class="small">Job f3ffcd19-559d-4803-895a-31d3765e5808 · owner Louis · governance zihrou · implementation iron · due 2026-05-30</div>
  </header>
  <main>
    <section>
      <h2>Promotion Gate</h2>
      <div class="grid">
        <div class="card"><div class="label">Golden Set</div><div class="value green">20 cases</div><div class="small">Seed ready for first validation run.</div></div>
        <div class="card"><div class="label">Evidence Coverage</div><div class="value blue">>=80%</div><div class="small">Signals/action items/commits/deliverables.</div></div>
        <div class="card"><div class="label">Hit Rate</div><div class="value amber">>=70%</div><div class="small">Below 60% blocks promotion.</div></div>
        <div class="card"><div class="label">False Positive</div><div class="value red"><=15%</div><div class="small">Over threshold repairs rubric.</div></div>
      </div>
    </section>
    <section>
      <h2>Runner Workflow</h2>
      <table>
        <tr><th>Step</th><th>Input</th><th>Output</th></tr>
        <tr><td>Import</td><td><code>prediction-golden-set-seed.csv</code></td><td>20 accepted cases or schema errors.</td></tr>
        <tr><td>Evidence match</td><td>signals, action_items, github_commit, deliverables</td><td>evidence_count, match_strength, top_source_type.</td></tr>
        <tr><td>Score</td><td>rubric + evidence links</td><td>hit_score, verdict, false_positive_flag.</td></tr>
        <tr><td>Govern</td><td>risk_tier=high</td><td>needs_review until Louis/zihrou approve.</td></tr>
        <tr><td>Regression</td><td>miss/partial/false positive</td><td>regression case with owner and due date.</td></tr>
      </table>
    </section>
    <section>
      <h2>Watchdog Rules</h2>
      <table>
        <tr><th>Signal</th><th>Threshold</th><th>Owner</th><th>Action</th></tr>
        <tr><td>hit_rate</td><td>&lt;60% after 20 cases</td><td>Louis</td><td>Block agent promotion.</td></tr>
        <tr><td>evidence_coverage</td><td>&lt;80%</td><td>iron</td><td>Fix source sync.</td></tr>
        <tr><td>false_positive_rate</td><td>&gt;15%</td><td>zihrou</td><td>Repair rubric and review rules.</td></tr>
        <tr><td>sync_error_rate</td><td>&gt;5%</td><td>iron</td><td>Dispatch repo_change.</td></tr>
      </table>
    </section>
  </main>
</body>
</html>

prediction-golden-set-seed.csv

production-acceptance.md

Production Acceptance

Pass Conditions

• Primary artifact opens through shared-cloud Gist.
• Golden set seed has exactly 20 rows.
• Required files exist: production brief, data model, acceptance tests, decision record, artifact URL record, solution selection, skill usage, market maturity, production acceptance, sources, learning memory, runner HTML, seed CSV.
• Artifacts JSON includes primary_artifact, solution_selection, skill_usage, market_context, market_maturity, production_readiness, production_acceptance, e2e_verification, people_sync, and learning_memory.
• Owner/due/acceptance are explicit.

Metric Gates

• evidence_coverage >= 80%.
• hit_rate >= 70%.
• false_positive_rate <= 15%.
• sync_error_rate <= 5%.
• 100% high-risk cases require human review.

Fail Conditions

• No openable artifact.
• No CSV golden set.
• No data/API/sync/permission path.
• No high-risk human approval boundary.
• Summary-only complete.
• Market maturity or skill usage omitted.

Owner / Due / Acceptance

• Owner: Louis.
• Governance: zihrou.
• Implementation: iron.
• Due: 2026-05-30.
• Acceptance: 20 validations complete, metric gates pass, regression cases created for failed high-impact cases.

production-brief.md

Production Brief：AI 預測驗證 Golden Set Runner

場景

上一輪已建立 AI 預測驗證 Eval Console；本輪 PLS contract 明確要求 skill usage、market maturity gap、production acceptance。這次把 console 往可驗收的 production runner 推進：建立 20 筆 golden set 規格、評分欄位、回歸測試、watchdog 閾值與 PLS 後台接入方式，讓 Louis/zihrou/iron 能用同一組樣本判斷預測驗證是否可信。

Solution Selection

選型：eval + spreadsheet + system + watchdog。

不是更小的 doc/sop：這輪要跑分、追蹤命中率、維護 golden set，文件不足以驗收。

不是更大的 agent：尚未累積 20 筆 validation，不能讓 agent 自主改專案、人員或權限狀態。先用 runner 跑出可靠度。

30 天路徑

• D1：建立 20 筆 golden set seed、評分 rubric、runner console、PLS 欄位/API。
• D7：用 signals/action_items/github_commit/deliverables 跑第一輪 evidence match，產 hit/partial/miss。
• D14：把低分樣本轉成 regression cases；watchdog 監控 hit_rate、coverage、false_positive。
• D30：若 hit_rate >= 70%、coverage >= 80%、false_positive <= 15%，升級為 PLS 後台 workflow；否則重修 rubric。

目的到目的 E2E

原始目的：驗證 AI review 的預測是否真的命中，不讓 AI 只會猜而不回看。 產出物：Golden Set Runner、20-row seed CSV、data/API spec、acceptance tests、decision record、market maturity、production acceptance、learning memory。 人採用：Louis 用命中率判斷是否加碼 AI 管理；zihrou 審高風險 verdict 與治理；iron 接資料源與 schema。 指標改善：prediction_hit_rate、evidence_coverage、false_positive_rate、regression_case_growth、time_to_validation。

價值 / 錢路徑

• 降低風險：避免錯誤 AI 預測長期影響人員派工、預算或專案優先級。
• 節省成本：把人工翻 signals/action items/GitHub 的回顧成本降成可批次跑分。
• 提高管理轉換：讓 AI 管理層的判斷有可追蹤命中率，能決定加碼或停損。
• 釋放人力：人只審高風險或低信心 verdict，低風險樣本可自動跑。

提升人的能力

Louis 能用數據評估 AI 管理，不靠感覺；zihrou 能把治理邊界轉成風險級別與審核規則；iron 能以 schema/API 落地工具選型，而不是在工具品牌上分歧。

Solution Stack

• 脈絡框架：prediction ledger → golden set → evidence match → scoring → regression set → watchdog。
• 作業流程：每次 review 產 prediction；每日同步 evidence；每週跑 golden set；低分樣本進 regression。
• 資料模型：見 data-model.md。
• 可操作工具：prediction-golden-set-runner.html、prediction-golden-set-seed.csv。
• 驗收指標：見 acceptance-tests.md 與 production-acceptance.md。
• 採用與下一輪升級：20 筆跑完後，達標才升級 agent；未達標先修 rubric 和 source sync。

People Sync / LINE 草稿

Louis：這輪不是再做一個 AI 預測頁，而是把預測驗證變成 20 筆 golden set runner。請看三個 gate：hit_rate >= 70%、evidence_coverage >= 80%、false_positive <= 15%。未達標前不建議擴大 AI 自主權限。

zihrou：請審 high-risk 樣本的人工覆核規則，特別是會影響人員、預算、專案狀態的 verdict。

iron：請先按 data-model 對齊 signals/action_items/github_commit/deliverables 的 join key；若 sync error > 5%，下一輪派 repo_change 修後台。

上一版問題 → 本輪修改 → 驗證結果 → 下一輪建議

上一版問題：有 Eval Console，但還缺 20 筆 golden set runner 讓分數可驗收。 本輪修改：新增 20-row seed、runner console、regression workflow、watchdog/acceptance gate。 驗證結果：本地 JSON 驗證、Gist HTTP 200、Gist file list、PLS upload-files。 下一輪建議：將 20 筆 seed 接入真實 PLS evidence，產第一份 prediction reliability report。

skill-usage.md

Skill / Tool Usage

Selected Skills / Tools

• purpose_e2e_toolbox_v2: used for D1/D7/D14/D30, E2E, value path, human capability, stack, data model, acceptance, decision record.
• PLS solution catalog: selected eval + spreadsheet + system + watchdog.
• Web search: used for current market maturity evidence on eval/observability production practices.
• Shell tools: python3 -m json.tool, find, rg.
• GitHub CLI: used to publish shared-cloud Gist and verify file list.
• URL verification: curl -I -L -s for HTTP 200.
• PLS helper: doctor, touch, claim, context, progress, upload-files, complete.

Why These Tools

The deliverable is a production validation pack, not a repo change. CSV is the correct format for golden set seed data; HTML is the fastest openable runner surface; Markdown specs define backend implementation and governance. Gist provides durable shared-cloud artifact URLs.

Evidence Artifact / Test Result

• prediction-golden-set-runner.html is the primary artifact.
• prediction-golden-set-seed.csv contains 20 validation cases.
• learning-memory.json passes JSON parsing.
• Gist HTTP 200 and file list are verified before complete.
• PLS upload-files must return uploaded file count.

solution-selection.md

Solution Selection

Selected Type

eval + spreadsheet + system + watchdog

Why This Combination

• eval：核心問題是預測是否命中，需要 pass/fail 與 hit score。
• spreadsheet：20 筆 golden set 需要欄位化、可批次檢查、可人工覆核。
• system：要接 PLS signals、action_items、github_commit、deliverables、people_reflections。
• watchdog：低命中率、低覆蓋率或高 false positive 要提醒人。

Why Not Smaller

doc 或 sop 不能產生命中率，也無法維護 regression cases。communication 只能催辦，不能驗證。

Why Not Bigger

agent 現階段過大。沒有 golden set 與穩定命中率前，不應讓 agent 自動調整專案、人員或預算狀態。

Adoption Condition

20 筆 golden set 跑完，hit_rate >= 70%、evidence_coverage >= 80%、false_positive_rate <= 15%，才建議升級到 PLS 後台 workflow 或 agent。

sources.md

Market Context Sources

Checked date: 2026-05-24 Asia/Taipei.

1. OpenAI, "Evaluation best practices". URL: https://platform.openai.com/docs/guides/evaluation-best-practices Use: continuous evaluation, production/historical/human-curated data, growing eval sets.

2. LangSmith, "LLM & AI Agent Evals Platform". URL: https://www.langchain.com/langsmith/evaluation Use: production monitoring, PR/nightly evals, continuous improvement.

3. LangChain, "LLM Evals: Production Monitoring to Regression Tests". URL: https://www.langchain.com/articles/llm-evals Use: production failures become offline test cases and regression tests.

4. Braintrust, "AI observability tools: A buyer's guide to monitoring AI agents in production (2026)". URL: https://www.braintrust.dev/articles/best-ai-observability-tools-2026 Use: comparable practice across traces, evals, observability, OpenTelemetry, and regression prevention.

5. Previous PLS artifact, AI 預測驗證 Eval Console. URL: https://gist.github.com/esz135888/a7a92a1d84f15c366669fad6dce04818 Use: previous completed concept console; this round advances to golden set runner.