PLS job d76edb7c Tokenmaxxing value metrics guardrail production pack

acceptance-tests.md

Acceptance Tests

1. Primary artifact URL returns HTTP 200.
2. Required appendix files exist.
3. D1 / D7 / D14 / D30 path exists.
4. Purpose-to-purpose E2E connects Tokenmaxxing risk to value metric governance and money path.
5. Token consumption, prompt count, and AI usage leaderboard are blocked for performance evaluation.
6. Token cost can exist only as diagnostic-only metric.
7. Allowed metrics require formula, data source, baseline, quality gate, and anti-gaming check.
8. Data model includes schema, API, sync, permissions, and audit.
9. Market maturity includes at least two external sources.
10. People sync and learning memory exist.

E2E Scenario

Given a manager proposes token_count_per_employee for AI performance ranking, when the governance gate runs, then the metric is blocked, rationale cites Tokenmaxxing risk, and the system recommends task completion rate, actual time saved, quality score, and customer satisfaction as alternatives.

artifact-url-or-pr.md

Artifact URL or PR

Primary artifact URL: https://gist.github.com/esz135888/8648380fb94850cd560b750a87949ee9#file-tokenmaxxing-value-metrics-guardrail-html

Artifact type: public Gist production pack.

data-model.md

Data Model

ai_value_metrics

• id: uuid primary key.
• metric_key: text unique.
• metric_name: text.
• metric_type: enum task_completion, time_saved, quality, customer_satisfaction, cost_diagnostic, blocked_activity.
• status: enum proposed, approved, blocked, diagnostic_only, deprecated.
• formula: text.
• data_source: text.
• owner_id: uuid.
• baseline_required: boolean.
• quality_gate: text.
• anti_gaming_check: text.

metric_policy_rules

• id: uuid primary key.
• metric_id: uuid.
• policy: enum allow, block, diagnostic_only, requires_exception.
• rationale: text.
• approved_by: uuid.
• effective_at: timestamptz.

ai_workflow_outcomes

• id: uuid primary key.
• workflow_id: uuid.
• metric_id: uuid.
• baseline_value: numeric nullable.
• observed_value: numeric.
• evidence_ref: text.
• measured_at: timestamptz.

token_cost_diagnostics

• id: uuid primary key.
• workflow_id: uuid.
• token_count: numeric.
• model: text.
• cost: numeric.
• period_start: timestamptz.
• period_end: timestamptz.
• diagnostic_note: text.

gaming_alerts

• id: uuid primary key.
• workflow_id: uuid.
• alert_type: enum token_spike_no_outcome, prompt_spike_no_quality, usage_rank_chasing, quality_drop, self_report_mismatch.
• severity: enum low, medium, high.
• evidence_json: jsonb.
• created_at: timestamptz.

metric_exceptions

• id: uuid primary key.
• metric_id: uuid.
• requested_by: uuid.
• reason: text.
• expires_at: timestamptz.
• approved_by: uuid nullable.
• status: enum pending, approved, rejected, expired.

API / Sync

• POST /ai-value/metrics
• POST /ai-value/outcomes
• GET /ai-value/scorecard
• GET /ai-value/cost-diagnostics
• POST /ai-value/exceptions

Permissions / Audit

Team owners can propose value metrics. Governance owners approve or block. Finance can view token cost diagnostics. Performance admins cannot use blocked metrics for ranking or compensation. All exceptions require reason, expiry, and audit log.

decision-record.md

Decision Record

Decision

Use governance / eval / scorecard.

Why

Tokenmaxxing is an incentive design failure. The production answer is not a memo but a governance mechanism that prevents bad metrics from entering scorecards and redirects measurement to value outcomes.

Options Considered

• Communication: useful for warning people, but does not stop bad metrics.
• Doc: preserves the lesson but cannot enforce it.
• Governance / eval / scorecard: best fit because metric policy, evaluation, exception handling, and scorecard are all required.

Adoption Status

Recommended. D7 should implement the first value metric registry and block token-based performance metrics.

Feedback Needed If Rejected

Clarify whether token consumption is needed for finance/cost diagnostics only and who can approve temporary exceptions.

e2e-verification.md

E2E Verification

Status: pass.

Verified target:

• Primary artifact URL: https://gist.github.com/esz135888/8648380fb94850cd560b750a87949ee9#file-tokenmaxxing-value-metrics-guardrail-html
• Gist pack URL: https://gist.github.com/esz135888/8648380fb94850cd560b750a87949ee9

Checks:

• Primary artifact URL returns HTTP 200.
• Gist includes primary HTML and appendices.
• Upload-files succeeds for the deliverable.
• Complete uses stable shared cloud URLs.

learning-memory.json

{
  "project": "AI 自建專案：亞馬遜 Tokenmaxxing 案例提醒",
  "job_id": "d76edb7c-71f9-4bb9-8399-19776c74c1c6",
  "selected_solution": "governance/eval/scorecard",
  "learned_signal": "Using token consumption to measure AI benefit causes meaningless metric gaming; value metrics must focus on actual outcomes.",
  "market_learning": "Mature productivity measurement avoids simple activity metrics and balances outcome, quality, satisfaction, flow, stability, and cost diagnostics.",
  "next_run_bias": "Block token-based performance metrics by default; allow token count only as diagnostic cost data.",
  "must_check_next": [
    "Are token metrics used only for finance/cost diagnostics?",
    "Are prompt count and usage rankings excluded from performance decisions?",
    "Does each value metric have baseline and evidence?",
    "Are gaming alerts connected to workflow outcomes?"
  ]
}

market-maturity.md

Market Maturity

Comparable Practices

• McKinsey developer productivity measurement warns against misuse of simple activity metrics: https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/yes-you-can-measure-software-developer-productivity?cid=other-eml-mtg-mip-mck
• SPACE framework balances multiple dimensions to avoid over-optimizing a single visible activity: https://space-framework.com/
• DORA metrics emphasize delivery performance, stability, and recovery rather than raw activity volume: https://dora.dev/guides/dora-metrics/

PLS Gap

PLS has the Tokenmaxxing warning, but needs production guardrails so token-based metrics cannot be used for scorecards, ranking, or compensation.

This Round Upgrade

This pack adds value metric taxonomy, blocked metric policy, cost diagnostic boundary, gaming signals, data model, APIs, permissions, acceptance tests, and D30 console path.

people-sync.md

People Sync

LINE Draft

Louis，這輪我把 Tokenmaxxing 案例做成「AI 價值指標防護台」。規則很直接：Token 消耗量、prompt count、AI 使用排行只能作診斷或直接 blocked，不能進績效或排名；真正可以用的是任務完成率、實際節省時間、產出品質評分、客戶滿意度，而且每個都要有 baseline、公式、資料源、品質門檻和 anti-gaming check。

Ask

請確認：Token 消耗量是否只保留作成本診斷，不允許進任何績效評估或個人排行？

If No Reply

先把所有 token/prompt/usage ranking 指標標成 blocked 或 diagnostic_only；任何例外都要 expiry 和 decision record。

production-brief.md

Production Brief

場景

專案：AI 自建專案：亞馬遜 Tokenmaxxing 案例提醒。

核心 insight：用 Token 消耗量衡量 AI 使用效益會導致員工刷無意義指標，指標設計應聚焦實際產出價值。

本輪產出

建立「Tokenmaxxing 價值指標防護作戰台」，把案例提醒變成 production governance pack：禁止指標、替代指標、資料模型、權限、gaming signal、驗收、people sync、learning memory。

D1 / D7 / D14 / D30

• D1: 定義禁止指標、替代價值指標、資料欄位、反作弊訊號與 owner。
• D7: 完成 AI value scorecard v1，套用到 3 個真實 AI workflow。
• D14: 導入 gaming detection。
• D30: 接進 Operating Console 或 PLS 後台，支援預算與考核決策。

Owner / Due / Acceptance

• Owner: AI governance owner.
• Due: D7 AI value scorecard v1.
• Acceptance: Token 消耗量、prompt count、AI usage leaderboard 不得用於績效；每個 allowed metric 都有公式、資料源、baseline、品質門檻與 anti-gaming check。

production-readiness.md

Production Readiness

Ready Now

• Openable value metric guardrail console.
• Required production appendix pack.
• Metric policy and data model.
• D1/D7/D14/D30 path.
• People sync and learning memory.

Integration Required

• Add metric policy gate in Operating Console or PLS backend.
• Tag token count as diagnostic-only.
• Add gaming signal detection.
• Add exception approval with expiry.

Failure / Rollback

If token-based metrics already influence performance decisions, freeze the decision, relabel token metrics as diagnostic-only, and re-run scorecards with outcome metrics before continuing.

skill-usage.md

Skill / Tool Usage

Tools Used

• PLS helper: doctor, touch, claim, context/progress attempted, upload-files, complete.
• Web search: checked McKinsey productivity measurement, SPACE framework, and DORA metrics.
• GitHub CLI: publishes production pack as public Gist.
• curl: verifies primary artifact URL returns HTTP 200.

Evidence

Claim succeeded. Initial context/progress hit transient fetch failures, so claim payload was used as source and context is retried before upload. External maturity references were checked. Final artifact is verified before completion.

solution-selection.md

Solution Selection

Selected route: governance / eval / scorecard.

The problem is behavior induced by bad metrics. The deliverable must govern metric selection, evaluate whether metrics create value or gaming, and show an operating scorecard. This is larger than a communication note and smaller than a full platform build.

Production stack:

• Framework: metric proposal -> policy gate -> diagnostic/blocked/approved -> observation -> gaming alert -> decision.
• Workflow: D1 rules, D7 scorecard, D14 gaming detection, D30 console integration.
• Data model: value metrics, policy rules, workflow outcomes, token diagnostics, gaming alerts, exceptions.
• Tool: openable HTML guardrail console.
• Acceptance: token-based performance metric is blocked.
• Upgrade: Operating Console UI blocker and cost diagnostic dashboard.

tokenmaxxing-value-metrics-guardrail.html

<!doctype html>
<html lang="zh-Hant">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Tokenmaxxing 價值指標防護作戰台</title>
  <style>
    :root{--ink:#17202a;--muted:#617080;--line:#d9e1e8;--paper:#f6f8fb;--card:#fff;--blue:#1d4ed8;--green:#0f7f5c;--amber:#a16207;--red:#b3361d;--teal:#0f766e}
    *{box-sizing:border-box}body{margin:0;background:var(--paper);color:var(--ink);font-family:Inter,ui-sans-serif,system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;line-height:1.5}
    header{background:#fff;border-bottom:1px solid var(--line);padding:28px clamp(20px,4vw,56px)}main{padding:24px clamp(20px,4vw,56px) 48px}
    h1{margin:0 0 12px;font-size:clamp(30px,4vw,52px);line-height:1.05;max-width:1100px}h2{margin:0 0 12px;font-size:22px}h3{margin:0 0 6px;font-size:16px}p{margin-top:0}code{background:#eef3f8;padding:1px 5px;border-radius:4px}
    .sub{max-width:1080px;color:var(--muted);font-size:17px}.grid{display:grid;gap:16px}.kpis{grid-template-columns:repeat(4,minmax(0,1fr));margin-top:22px}.two{grid-template-columns:1.08fr .92fr}.three{grid-template-columns:repeat(3,minmax(0,1fr))}.timeline{grid-template-columns:repeat(4,minmax(0,1fr))}.flow{grid-template-columns:repeat(5,minmax(0,1fr))}
    .card{background:var(--card);border:1px solid var(--line);border-radius:8px;padding:18px;box-shadow:0 1px 2px rgba(23,32,42,.04)}.metric{font-size:34px;font-weight:780}.label{color:var(--muted);font-size:13px}
    .pill{display:inline-flex;border:1px solid var(--line);border-radius:999px;padding:4px 10px;font-size:12px;background:#fff;margin:0 6px 8px 0;white-space:nowrap}.ok{color:var(--green)}.warn{color:var(--amber)}.bad{color:var(--red)}.info{color:var(--blue)}
    table{width:100%;border-collapse:collapse;font-size:14px}th,td{text-align:left;padding:10px;border-bottom:1px solid var(--line);vertical-align:top}th{color:var(--muted);font-size:12px;text-transform:uppercase}.badcell{color:var(--red);font-weight:700}.goodcell{color:var(--green);font-weight:700}
    .day{border-left:4px solid var(--teal)}.step{border:1px solid var(--line);border-radius:8px;padding:12px;min-height:126px;background:#fbfdff}.step strong{display:block;color:var(--teal);margin-bottom:6px}.source a{color:var(--blue);word-break:break-word}
    @media(max-width:920px){.kpis,.two,.three,.timeline,.flow{grid-template-columns:1fr}h1{font-size:34px}}
  </style>
</head>
<body>
  <header>
    <span class="pill info">PLS production delivery pack</span><span class="pill ok">Solution: governance / eval / scorecard</span>
    <h1>Tokenmaxxing 價值指標防護作戰台</h1>
    <p class="sub">把「亞馬遜 Tokenmaxxing 案例提醒」變成可執行的 AI 指標防護系統：禁止用 Token 消耗量代表 AI 效益，將評估重心轉向實際產出價值、任務完成、節省時間、品質與客戶滿意，並建立 gaming signal 與治理例外流程。</p>
    <section class="grid kpis">
      <div class="card"><div class="metric bad">0</div><div class="label">Token 用量不得進績效排名</div></div>
      <div class="card"><div class="metric ok">4</div><div class="label">核心價值指標：完成、時間、品質、滿意</div></div>
      <div class="card"><div class="metric">D7</div><div class="label">完成 AI value scorecard v1</div></div>
      <div class="card"><div class="metric">D30</div><div class="label">導入 governance gate 與異常偵測</div></div>
    </section>
  </header>
  <main class="grid">
    <section class="grid two">
      <div class="card">
        <h2>本輪問題</h2>
        <p>Tokenmaxxing 的風險是把成本或活動量誤當價值。當公司用 Token 消耗量衡量 AI 使用效益，員工會被誘導去刷 prompt、開工具、製造看似繁忙的 AI 用量，最後增加成本、降低信任，卻沒有提升客戶或任務結果。</p>
        <span class="pill">Owner: AI governance owner</span><span class="pill">Due: D7 value scorecard</span><span class="pill">Acceptance: token metric blocked</span>
      </div>
      <div class="card">
        <h2>解法選型</h2>
        <p><strong>governance / eval / scorecard</strong>。這不是單純教育宣導，而是管理制度設計；必須有指標准入、反作弊訊號、例外審批、資料模型與驗收規則。</p>
      </div>
    </section>

    <section class="card">
      <h2>D1 / D7 / D14 / D30 路徑</h2>
      <div class="grid timeline">
        <div class="card day"><h3>D1</h3><p>定義禁止指標、替代價值指標、資料欄位、反作弊訊號與 owner。</p></div>
        <div class="card day"><h3>D7</h3><p>完成 AI value scorecard v1，套用到 3 個實際 AI 工作流，建立 baseline。</p></div>
        <div class="card day"><h3>D14</h3><p>導入 gaming detection：Token 暴增但任務/品質不升、工具啟動暴增但客戶滿意不升。</p></div>
        <div class="card day"><h3>D30</h3><p>把 value scorecard 接進 Operating Console 或 PLS 後台，支援預算、考核、加碼/停止決策。</p></div>
      </div>
    </section>

    <section class="card">
      <h2>Purpose-to-Purpose E2E</h2>
      <div class="grid flow">
        <div class="step"><strong>原始目的</strong>避免 AI 使用效益被 Token 消耗量綁架。</div>
        <div class="step"><strong>產出物</strong>價值指標防護台、metric registry、gaming signal、驗收規則。</div>
        <div class="step"><strong>人採用</strong>主管用 outcome metrics 評估，員工專注完成有價值任務。</div>
        <div class="step"><strong>指標改善</strong>成本浪費下降、完成率上升、品質提升、客戶滿意改善。</div>
        <div class="step"><strong>錢路徑</strong>AI 預算流向有效 workflow，停掉只燒 Token 不創值的使用場景。</div>
      </div>
    </section>

    <section class="grid two">
      <div class="card">
        <h2>指標防護規則</h2>
        <table>
          <thead><tr><th>Metric</th><th>Policy</th><th>Production Rule</th></tr></thead>
          <tbody>
            <tr><td>Token consumption</td><td class="badcell">Blocked for performance</td><td>只能作成本診斷，不得用於績效、排行或獎金。</td></tr>
            <tr><td>Prompt count</td><td class="badcell">Blocked</td><td>高 prompt count 不代表成果；只可作 debug signal。</td></tr>
            <tr><td>AI usage leaderboard</td><td class="badcell">Blocked</td><td>禁止個人排行；改看 team outcome。</td></tr>
            <tr><td>Task completion rate</td><td class="goodcell">Allowed</td><td>需有完成定義、驗收證據與品質門檻。</td></tr>
            <tr><td>Actual time saved</td><td class="goodcell">Allowed</td><td>需 baseline 與抽樣驗證，不能自報即算。</td></tr>
            <tr><td>Output quality score</td><td class="goodcell">Allowed</td><td>需 rubric、reviewer 與樣本數。</td></tr>
            <tr><td>Customer satisfaction</td><td class="goodcell">Allowed</td><td>需連到具體 workflow 和客戶回饋來源。</td></tr>
          </tbody>
        </table>
      </div>
      <div class="card">
        <h2>資料 / API / 權限</h2>
        <p><strong>Tables:</strong> <code>ai_value_metrics</code>, <code>metric_policy_rules</code>, <code>ai_workflow_outcomes</code>, <code>token_cost_diagnostics</code>, <code>gaming_alerts</code>, <code>metric_exceptions</code>.</p>
        <p><strong>APIs:</strong> <code>POST /ai-value/metrics</code>, <code>POST /ai-value/outcomes</code>, <code>GET /ai-value/scorecard</code>, <code>POST /ai-value/exceptions</code>.</p>
        <p><strong>Permissions:</strong> team owner can propose value metric; governance owner approves; finance can view token cost diagnostics; performance admins cannot use blocked metrics.</p>
      </div>
    </section>

    <section class="grid three">
      <div class="card"><h2>價值 / 錢路徑</h2><p>把 AI 成本從「燒多少 Token」改成「每筆任務創造多少成果」，能降低浪費、提升預算配置品質，並避免錯誤考核造成行為扭曲。</p></div>
      <div class="card"><h2>人的能力提升</h2><p>主管學會設計不可被輕易操弄的價值指標；員工知道 AI 使用目標是完成更好的工作，而不是刷活動量。</p></div>
      <div class="card"><h2>下一輪升級</h2><p>接 Operating Console 指標設定 UI，新增 Token metric blocker、cost diagnostic dashboard 與 gaming alert。</p></div>
    </section>

    <section class="card source">
      <h2>Market Maturity Inputs</h2>
      <p>McKinsey warns that productivity data can damage organizations if overly simple activity metrics are misused: <a href="https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/yes-you-can-measure-software-developer-productivity?cid=other-eml-mtg-mip-mck">McKinsey productivity measurement</a>.</p>
      <p>The SPACE framework balances satisfaction, performance, activity, communication, and efficiency to prevent over-optimization of one metric: <a href="https://space-framework.com/">SPACE framework</a>.</p>
      <p>DORA metrics focus on delivery performance and reliability rather than raw activity volume: <a href="https://dora.dev/guides/dora-metrics/">DORA metrics guide</a>.</p>
    </section>
  </main>
</body>
</html>