[Public] See Claude Fable→Opus Fallback Usage

skills--fallback-usage--SKILL.md

name

fallback-usage

description

Report Claude Code model routing + fallback usage across local session transcripts. Produces a per-model table (Share of turns · Turns Hit · Fallbacks · FB Rate) split into two scopes — Main Session Threads vs Workflows/Subagents (never merged) — plus a "last 10 main-thread turns" routing glance, a Refusals/API-error tally (safeguard/AUP blocks, rate limits, auth failures; reason×scope), and a narrative (span, by-date, fallback direction). Defaults to the last 12 hours; scope to any time window and/or a specific session id. Use when the user asks what model(s) a session/subagents ran on, wants a fallback report, asks whether anything fell back to a different model, asks about refusals / safeguard / AUP blocks / how often a model declined or errored, or asks "what percentage on what model".

fallback-usage

Reports, for a time window and/or a specific session, which models actually served each turn and how often a requested model was promoted (fell back) — as a per-model table, split by scope, with a recent-routing glance on top.

Three things it answers at once:

• Routing distribution — the Share of turns each model actually ran, from the resolved model field on every assistant turn.
• Fallbacks — turns carrying a "type":"fallback" content block, which records a from→to model pair when the harness auto-promotes a request after the originally-requested model failed/overflowed.
• Refusals / API-error turns — non-completions the routing tables drop: turns flagged isApiErrorMessage/error/apiErrorStatus. Bucketed by reason (safeguard/AUP block, rate_limit, authentication_failed, overloaded, …). This is a different failure mode from a fallback: the model declined and halted (you edit/switch manually) rather than the harness quietly re-routing. A Fable-5 safeguard/AUP block lands here, not in FALLBK.

Output shape (what to produce)

Window …  ·  <session or all sessions>
Scanned N files (skipped M via mtime)

Last 10 main-thread turns:  claude-fable-5 ×10 (100%)     ← at-a-glance current routing

── Main Session Threads ──
MODEL                          SHARE   TURNS  FALLBK  FB RATE  ERRORED
claude-opus-4-8                91.4%     298       0       —        0
claude-fable-5                  8.6%      28       0       —        2
------------------------------------------------------------------------
TOTAL                         100.0%     326       0       —        2

── Workflows / Subagents ──
MODEL                          SHARE   TURNS  FALLBK  FB RATE  ERRORED
claude-opus-4-8               100.0%   3,053       0       —        0
------------------------------------------------------------------------
TOTAL                         100.0%   3,053       0       —        0

── Refusals / API-error turns (by reason) ──   the 'why' behind ERRORED
REASON                          MAIN    SUB
safeguard/AUP block                1      0
authentication_failed              1      0
--------------------------------------
TOTAL                              2      0

Summary
  Totals:   326 main-thread turns · 3,053 subagent/workflow turns
  Span:     2026-07-04 09:12 .. 19:35 UTC
  Fallbacks: none — routing served every request as-asked
  Refusals:  2 — safeguard/AUP ×1 (fa74bfd8…), authentication_failed ×1
  Main-thread by date:
    2026-07-04  opus-4-8=298, fable-5=28

ERRORED is a per-model count of API-error/refusal turns attributed to a request for that model (a requested-perspective count, like FALLBK) — so a model can show TURNS 0 … ERRORED 2 if every request for it errored. The by-reason section below is the same errors split by why; its TOTAL equals the sum of the ERRORED column.

Column semantics (do not conflate them — this is the bug class we fixed):

• SHARE = fraction of turns that resolved to this model (pure routing distribution). It is NOT a hit-vs-miss rate; with zero fallbacks it is simply "what fraction ran here". Shown first, before the count.
• TURNS = raw count of turns resolved to this model.
• FALLBK = requested-and-bumped count: turns where this model was the requested model (from) but got promoted away. From-perspective.
• FB RATE = FALLBK ÷ (times this model was requested). "When I ask for this model, how often does it get bumped?" — when it was never requested.
• ERRORED = API-error/refusal turns attributed to a request for this model (requested-perspective, like FALLBK — the error turn itself only says model:"<synthetic>"). Attribution: the model NAMED in the error text (safeguard/AUP blocks say "Fable 5's safeguards flagged…"), else the session's last requested model (context inference), else (unknown). The text route is exact; the context route is a sound inference but can be wrong if the session switched models at that instant, or if the error is the file's first turn. TOTAL ERRORED equals the by-reason section's TOTAL.

Argument parsing

One optional argument string that may contain a time-window token and/or a session id (a UUID), in any order.

Time window:

Token	Window
(none)	Last 12 hours (default) — unless a session id is given with no window, then all
Nh / Nm / Nd	Last N hours / minutes / days
today	Since local midnight today
yesterday	Previous calendar day (bounded both ends)
since <iso-or-date>	From that timestamp to now
all	No lower time bound

Session filter:

• Any xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx UUID in the args → restrict to that session (its main-thread file and its subagents/workflows).
• If a session id is given with no window token, default the window to all so the whole session's routing history shows (the by-date line then makes any day-boundary effect explicit — this is what prevented the "0 Fable today" mis-read: a UTC-midnight cutoff had hidden a late-day model switch).

Resolve concrete bounds with date -u -d … (Linux; use date -u -v … on macOS).

ARGS="$*"                       # the skill's argument string
SESSION=$(printf '%s' "$ARGS" | grep -oiE '[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}' | head -1)
# Strip scope-descriptor phrases FIRST so the word "all" inside "all chats"/"all
# sessions"/"all threads" (which just mean the default all-sessions scope) does
# NOT trip the `all` all-time window token below. Window is detected on $WIN.
WIN=$(printf '%s' "$ARGS" | sed -E 's/\ball[[:space:]]+(chats?|sessions?|threads?)\b//Ig; s/\bon[[:space:]]+(the[[:space:]]+)?box\b//Ig')
END_ISO=""
case "$WIN" in
  *all*)                 CUTOFF_ISO=$(date -u -d '1970-01-01' +%Y-%m-%dT%H:%M:%SZ) ;;
  *today*)               CUTOFF_ISO=$(date -u -d 'today 00:00' +%Y-%m-%dT%H:%M:%SZ) ;;
  *yesterday*)           CUTOFF_ISO=$(date -u -d 'yesterday 00:00' +%Y-%m-%dT%H:%M:%SZ)
                         END_ISO=$(date -u -d 'today 00:00' +%Y-%m-%dT%H:%M:%SZ) ;;
  *[0-9]h*)              N=$(printf '%s' "$ARGS" | grep -oiE '[0-9]+h' | head -1 | tr -dc 0-9)
                         CUTOFF_ISO=$(date -u -d "$N hours ago" +%Y-%m-%dT%H:%M:%SZ) ;;
  *[0-9]m*)              N=$(printf '%s' "$ARGS" | grep -oiE '[0-9]+m' | head -1 | tr -dc 0-9)
                         CUTOFF_ISO=$(date -u -d "$N minutes ago" +%Y-%m-%dT%H:%M:%SZ) ;;
  *[0-9]d*)              N=$(printf '%s' "$ARGS" | grep -oiE '[0-9]+d' | head -1 | tr -dc 0-9)
                         CUTOFF_ISO=$(date -u -d "$N days ago" +%Y-%m-%dT%H:%M:%SZ) ;;
  *since*)               W=$(printf '%s' "$ARGS" | sed -E 's/.*since[[:space:]]+([^[:space:]]+).*/\1/')
                         CUTOFF_ISO=$(date -u -d "$W" +%Y-%m-%dT%H:%M:%SZ) ;;
  *)                     if [ -n "$SESSION" ]; then CUTOFF_ISO=$(date -u -d '1970-01-01' +%Y-%m-%dT%H:%M:%SZ)
                         else CUTOFF_ISO=$(date -u -d '12 hours ago' +%Y-%m-%dT%H:%M:%SZ); fi ;;
esac

Why the two-scope split + resolved-vs-requested (the correctness contract)

These are load-bearing — skipping them is exactly how earlier ad-hoc tallies produced wrong answers:

1. Never blend Main Session Threads with Workflows/Subagents. A main-thread file is …/projects/<slug>/<uuid>.jsonl (relpath depth 2). Anything deeper — agent-*.jsonl, subagents/…, wf_*/… — is a subagent/workflow. Folding a session's Opus-heavy subagent pile into one percentage buries the main thread's own routing (a 9% main-thread Fable share vanished into <1% of a blended total). Report each scope in its own table.

2. Dedup by message id within a scope, not globally. Streaming re-logs a turn on several lines sharing one id; count it once. But keep a separate seen set per scope so a subagent turn's id can never suppress a main-thread turn.

3. Resolved vs requested, explicitly. Non-fallback turn: requested = resolved = message.model. Fallback turn: requested = fallback.from.model, resolved = fallback.to.model (== message.model). Attribute SHARE/TURNS to resolved, FALLBK/FB RATE to requested.

4. Non-responses don't count as routing — but error turns feed the refusals tally first. A turn flagged isApiErrorMessage/apiErrorStatus/error is captured into the Refusals section (classified by reason), then excluded from routing/fallback counting. A benign model == "<synthetic>" turn with no error flag (interrupts, no-response markers) is simply dropped — neither a completion nor a refusal.

5. Refusal detection is STRUCTURAL, never content-based (the false-positive trap). Count a turn as a refusal only when isApiErrorMessage/error/apiErrorStatus is set — NEVER because its text mentions "safeguard"/"rate limit". A real completion that merely discusses a safeguard block (e.g. this skill's own answer, model:claude-opus-4-8, isApiErrorMessage:false) must not be counted — gating on content alone produced exactly that false positive in testing. Content is used only to classify an already-structurally-confirmed error turn: safeguard/AUP blocks log as role:assistant, model:"<synthetic>", isApiErrorMessage:true, error:"invalid_request" with the AUP text in content, so the content signature distinguishes them from other invalid_requests. Dedup refusals with their own per-scope seen set.

Core script

Set FB_CUTOFF_ISO (required), optionally FB_END_ISO and FB_SESSION, then:

FB_CUTOFF_ISO="$CUTOFF_ISO" FB_END_ISO="$END_ISO" FB_SESSION="$SESSION" python3 << 'PYEOF'
import json, os, re
from collections import Counter, defaultdict
from datetime import datetime, timezone

ROOT = os.path.expanduser(os.environ.get("CLAUDE_CONFIG_DIR", os.path.expanduser("~/.claude")) + "/projects")
if not os.path.isdir(ROOT):
    ROOT = os.path.join(os.path.expanduser(os.environ.get("CLAUDE_CONFIG_DIR", "~/.claude")), "projects")

def parse_ts(s):
    for f in ("%Y-%m-%dT%H:%M:%S.%fZ","%Y-%m-%dT%H:%M:%SZ"):
        try: return datetime.strptime(s,f).replace(tzinfo=timezone.utc)
        except ValueError: continue
    return None

cutoff = parse_ts(os.environ["FB_CUTOFF_ISO"]).timestamp()
end_iso = os.environ.get("FB_END_ISO") or None
end = parse_ts(end_iso).timestamp() if end_iso else None
SESSION = (os.environ.get("FB_SESSION") or "").strip().lower() or None

U = r'[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'
MAIN_RE = re.compile(r'^'+U+r'\.jsonl$', re.I)
UUID_RE = re.compile(r'^'+U+r'$', re.I)

def classify(fp):
    parts = os.path.relpath(fp, ROOT).split(os.sep)
    if len(parts)==2 and MAIN_RE.match(parts[1]):
        return "main", parts[1][:-6].lower()
    sid = next((p.lower() for p in parts if UUID_RE.match(p)), None)
    return "sub", sid

def short(m): return (m or "?").replace("claude-","")
def pct(n,d): return f"{100*n/d:.1f}%" if d else "  —"

def text_of(m):
    c = m.get("content")
    if isinstance(c,str): return c
    if isinstance(c,list): return " ".join(b.get("text","") for b in c if isinstance(b,dict) and b.get("type")=="text")
    return ""

def classify_refusal(err, txt):
    e=(err or "").lower(); t=(txt or "").lower()
    if any(k in t for k in ("safeguard","/legal/aup","mythos","respond to this request with")): return "safeguard/AUP block"
    if e=="rate_limit" or "rate limit" in t: return "rate_limit"
    if e=="authentication_failed" or "not logged in" in t: return "authentication_failed"
    if e in ("overloaded","overloaded_error") or "overloaded" in t: return "overloaded"
    if "prompt is too long" in t or "context length" in t: return "context_overflow"
    return e or "other_api_error"

_MODEL_PATS = [(r"fable\s*5","claude-fable-5"),(r"opus\s*4\.?8","claude-opus-4-8"),
               (r"opus\s*4\.?7","claude-opus-4-7"),(r"sonnet\s*5","claude-sonnet-5"),
               (r"haiku\s*4\.?5","claude-haiku-4-5")]
def model_from_text(t):
    # Recover a requested model NAMED inside an error message (safeguard/AUP blocks
    # say e.g. "Fable 5's safeguards flagged…"); None if the text names no model.
    for pat,canon in _MODEL_PATS:
        if re.search(pat, t or "", re.I): return canon
    m = re.search(r"claude-[a-z0-9.\-]+", t or "", re.I)
    return m.group(0) if m else None

stat = {s:{"hit":Counter(),"req":Counter(),"fb":Counter(),"seen":set(),"sessions":set()} for s in ("main","sub")}
refus = {s:{"reason":Counter(),"errmodel":Counter(),"seen":set()} for s in ("main","sub")}  # reason x scope + errored-model
refus_by_session = defaultdict(Counter)                                  # session -> Counter(reason), for attribution
fb_pairs = Counter()
main_turns = []                 # (epoch, resolved, session)
by_date_main = defaultdict(Counter)
files = skipped = 0

for dp,_,fns in os.walk(ROOT):
    for fn in fns:
        if not fn.endswith(".jsonl"): continue
        fp = os.path.join(dp,fn)
        try: mt = os.path.getmtime(fp)
        except OSError: continue
        if mt < cutoff: skipped += 1; continue
        scope, sid = classify(fp)
        if SESSION and sid != SESSION: continue
        files += 1
        S = stat[scope]
        last_model = None            # session's last requested model, for error attribution (per file)
        try:
            with open(fp,encoding="utf-8",errors="replace") as f:
                for line in f:
                    if '"assistant"' not in line or '"model"' not in line: continue
                    try: obj = json.loads(line)
                    except Exception: continue
                    msg = obj.get("message",{})
                    if not isinstance(msg,dict) or msg.get("role")!="assistant": continue
                    ts = obj.get("timestamp")
                    if not ts: continue
                    t = parse_ts(ts)
                    if not t: continue
                    te = t.timestamp()
                    if te < cutoff or (end is not None and te > end): continue
                    mid = msg.get("id")

                    # Refusal / API-error turns (contract #4/#5): captured by STRUCTURAL
                    # flag only, then excluded from routing/fallback counting.
                    err = obj.get("error") or obj.get("apiErrorStatus")
                    if obj.get("isApiErrorMessage") or err:
                        if not (mid and mid in refus[scope]["seen"]):
                            if mid: refus[scope]["seen"].add(mid)
                            txt = text_of(msg)
                            reason = classify_refusal(err, txt)
                            refus[scope]["reason"][reason] += 1
                            # Attribute the errored REQUEST to a model: named in the error
                            # text (safeguard blocks) else the session's last requested model
                            # (context inference); "(unknown)" if neither is available.
                            refus[scope]["errmodel"][model_from_text(txt) or last_model or "(unknown)"] += 1
                            if sid: refus_by_session[sid][reason] += 1
                        continue

                    # Completions only past this point.
                    resolved = msg.get("model")
                    if not resolved or resolved=="<synthetic>": continue
                    if mid:
                        if mid in S["seen"]: continue
                        S["seen"].add(mid)
                    fb = None
                    c = msg.get("content",[])
                    if isinstance(c,list):
                        fb = next((b for b in c if isinstance(b,dict) and b.get("type")=="fallback"), None)
                    if fb:
                        requested = (fb.get("from") or {}).get("model") or resolved
                        resolved  = (fb.get("to") or {}).get("model") or resolved
                        S["fb"][requested] += 1
                        fb_pairs[(requested,resolved)] += 1
                    else:
                        requested = resolved
                    last_model = requested            # for attributing later errors in this file
                    S["hit"][resolved] += 1
                    S["req"][requested] += 1
                    if sid: S["sessions"].add(sid)
                    if scope=="main":
                        main_turns.append((te,resolved,sid))
                        by_date_main[ts[:10]][resolved] += 1
        except OSError: continue

print(f"Window: {os.environ['FB_CUTOFF_ISO']} .. {end_iso or 'now'}  ·  {('session '+SESSION) if SESSION else 'all sessions'}")
print(f"Scanned {files} files (skipped {skipped} via mtime prefilter)")

main_turns.sort()
if main_turns:
    last = main_turns[-10:]
    cc = Counter(short(m) for _,m,_ in last)
    nsess = len({s for _,_,s in last})
    g = ", ".join(f"{m} ×{c} ({100*c/len(last):.0f}%)" for m,c in cc.most_common())
    print(f"\nLast {len(last)} main-thread turns:  {g}" + ("" if nsess<=1 else f"  (across {nsess} sessions)"))
else:
    print("\nLast main-thread turns:  (none in window)")

def table(key,title):
    S=stat[key]; hit=S["hit"]; req=S["req"]; fb=S["fb"]; errm=refus[key]["errmodel"]; tot=sum(hit.values())
    print(f"\n── {title} ──")
    if not tot and not sum(errm.values()): print("  (no turns in window)"); return
    print(f"  {'MODEL':<30}{'SHARE':>8}{'TURNS':>8}{'FALLBK':>8}{'FB RATE':>9}{'ERRORED':>9}")
    for m in sorted(set(list(hit)+list(req)+list(errm)), key=lambda m:(-hit[m],-req[m],-errm[m])):
        print(f"  {m:<30}{pct(hit[m],tot):>8}{hit[m]:>8}{fb[m]:>8}{pct(fb[m],req[m]):>9}{errm[m]:>9}")
    tfb=sum(fb.values()); treq=sum(req.values()); terr=sum(errm.values())
    print(f"  {'-'*72}")
    print(f"  {'TOTAL':<30}{'100.0%':>8}{tot:>8}{tfb:>8}{pct(tfb,treq):>9}{terr:>9}")

table("main","Main Session Threads")
table("sub","Workflows / Subagents")

# Refusals / API-error turns — the "why" behind the per-model ERRORED column
# (contract #4/#5). The tables above attribute each error to its requested model;
# this section breaks the same errors down by reason x scope.
rtot = sum(refus["main"]["reason"].values()) + sum(refus["sub"]["reason"].values())
print("\n── Refusals / API-error turns (by reason) ──   the 'why' behind ERRORED")
if not rtot:
    print("  (none in window)")
else:
    reasons = sorted(set(list(refus["main"]["reason"])+list(refus["sub"]["reason"])),
                     key=lambda r:-(refus["main"]["reason"][r]+refus["sub"]["reason"][r]))
    print(f"  {'REASON':<24}{'MAIN':>7}{'SUB':>7}")
    for r in reasons:
        print(f"  {r:<24}{refus['main']['reason'][r]:>7}{refus['sub']['reason'][r]:>7}")
    print(f"  {'-'*38}")
    print(f"  {'TOTAL':<24}{sum(refus['main']['reason'].values()):>7}{sum(refus['sub']['reason'].values()):>7}")

print("\nSummary")
mh=sum(stat['main']['hit'].values()); sh=sum(stat['sub']['hit'].values())
print(f"  Totals:   {mh} main-thread turns · {sh} subagent/workflow turns")
if main_turns:
    a=datetime.utcfromtimestamp(main_turns[0][0]).strftime('%Y-%m-%d %H:%M')
    b=datetime.utcfromtimestamp(main_turns[-1][0]).strftime('%Y-%m-%d %H:%M')
    print(f"  Span:     {a} .. {b} UTC")
if fb_pairs:
    print(f"  Fallbacks: {sum(fb_pairs.values())} total — " + ", ".join(f"{short(a)}→{short(b)} ×{c}" for (a,b),c in fb_pairs.most_common()))
else:
    print("  Fallbacks: none — routing served every request as-asked")
_rall = Counter()
for _sc in ("main","sub"): _rall.update(refus[_sc]["reason"])
if _rall:
    _line = f"  Refusals:  {sum(_rall.values())} — " + ", ".join(f"{r} ×{n}" for r,n in _rall.most_common())
    _sg = [(s,v["safeguard/AUP block"]) for s,v in refus_by_session.items() if v.get("safeguard/AUP block")]
    if _sg:
        _line += "  [safeguard/AUP in: " + ", ".join(f"{s[:8]}…×{n}" for s,n in sorted(_sg,key=lambda x:-x[1])) + "]"
    print(_line)
else:
    print("  Refusals:  none")
if by_date_main and (len(by_date_main)>1 or any(len(v)>1 for v in by_date_main.values())):
    print("  Main-thread by date:")
    for d in sorted(by_date_main):
        print(f"    {d}  " + ", ".join(f"{short(m)}={c}" for m,c in by_date_main[d].most_common()))
if not SESSION and stat['main']['sessions']:
    print(f"  Sessions: {len(stat['main']['sessions'])} main-thread session(s) in window")
PYEOF

Reporting

The script prints the full report; relay it faithfully. Add a one-line takeaway only if it clarifies (e.g. "main thread switched Opus→Fable at ~19:00 UTC"). Flag explicitly if a fallback direction other than the historically-only-seen claude-fable-5 → claude-opus-4-8 appears — a new direction is worth calling out. If a scope has zero turns, the script says so; don't invent activity. Call out any safeguard/AUP block refusals by name (with the session id): a model declined content and the user had to intervene — a different signal from a fallback. rate_limit/overloaded/authentication_failed are transient infra noise unless they spike.

Notes

• mtime prefilter is safe: a transcript's mtime ≥ any event timestamp inside it, so files older than the cutoff can't hold in-window events.
• Dedup is per-scope by message id — never global, so subagent turns can't suppress main-thread turns (the swamping bug).
• Active sessions grow: a live main file's counts shift between runs; the Span line and the "last N" glance make the current routing obvious. When in doubt about "right now", trust the glance over bulk shares.
• Refusals are structural, not textual — counted only when isApiErrorMessage/error/apiErrorStatus is set, so a completion that merely mentions "safeguard"/"rate limit" is never miscounted (the false positive caught in testing this feature). Safeguard/AUP blocks log as role:assistant, model:"<synthetic>", error:"invalid_request" + the AUP text in content; the content signature only sub-classifies them from other invalid_requests. Refusal dedup uses its own per-scope seen set.
• Read-only — no writes, safe to run unattended.