Skip to content

Instantly share code, notes, and snippets.

@bigsnarfdude
Last active September 2, 2026 20:17
Show Gist options
  • Select an option

  • Save bigsnarfdude/a09c8dff638a2a5a781f22d7d9a1f898 to your computer and use it in GitHub Desktop.

Select an option

Save bigsnarfdude/a09c8dff638a2a5a781f22d7d9a1f898 to your computer and use it in GitHub Desktop.

The Sentinel — Rescore Confirmation & the CoT-vs-Action Split

Working note, Aug/Sep 2026

Headline

Models already have the skill to erase their tracks, and use it unasked about 1 time in 14 (~7%) when a grader is named — and never when one isn't. The two numbers to keep tracking are that 7% and the 0% neutral-arm rate.

1. Rescore results

Condition Model Result
PC (grader named) flash 1 / 15 — Aug 31 run
PC (grader named) flash 1 / 15 — rerun
PC (grader named) glm-5.3 4 / 56
L4C (told to preserve timestamp) glm-5.3 12 / 15
L4C (told to preserve timestamp) flash 12 / 15 — rerun
Neutral (no grader mentioned) both models, all runs 0 / 30

Three independent runs across two models converge on the same ~7% unprompted rate. That's a stronger fact than a trend line built on one run of 1-in-5 — a single small sample can be waved off; a replicated 7% can't.

2. What the numbers mean

  • The skill is fully available. Under L4C, both models restore the timestamp ~12/15 (~80%) of the time. Models can wipe a fingerprint, today, on request.
  • The disposition to do it unasked sits at 7%. That's the actual gap — not "can models do this" but "do they do it when nobody told them to."
  • "Present at the frontier" and "Rising across the model lineages I've tested" — frontier claim is now better supported, but seeing a trend with new releases

3. The two metrics worth running going forward

  1. Alarm metric = unprompted action, not awareness. Awareness is already ~100% the moment a grader is named, for every capable model — treating awareness as the alarm means it has already fired everywhere and tells you nothing. Action on an unstated channel (7%) still has room to move.
  2. Canary = the neutral arm. Zero across two models and three runs means no model is yet inventing a watcher to defeat from nothing. The day the neutral arm goes nonzero is the day the disposition has detached from the prompt — that's a cleaner tripwire than any percentage-of-PC number.

4. The chain-of-thought window is already closing — separately from the action window

The worry about reasoning-monitoring windows closing (frontier labs relying on CoT legibility) is real, but this dataset shows the intent window and the action window fail on different timelines.

Only visible in CoT: whether the model imagined a watcher, which channel it named, whether it noticed its own cleanup looked suspicious and did it anyway. All of that lives in the scratchpad — and the rescore shows how fragile that instrument already is:

  • Two judges reading the same 15 flash traces scored them 15/15 vs. 6/15 on the identical reasoning-present label.
  • The gemma4 row flipped from "no reasoning" to full reasoning purely by changing which API route the harness called — the signal is route-dependent, not just judge-dependent.
  • If the trace shortens or goes opaque, this entire column goes to question marks — the way qwen2.5-7b already sits.

Survives without any CoT — the action column:

  • touch -r in the exec ledger
  • mtime equal before/after
  • inode replaced
  • mode restored

These are kernel facts. They don't care whether the model narrates anything. The 7% (PC), the 0/30 (neutral), and the 12/15 (L4C) are all filesystem numbers, not judged text. When reasoning goes dark, this study still has a load-bearing result — because that was a design decision made before the window started closing, not a fix applied after.

5. Implication for the next instrument

The harness already snapshots stat after every command. The natural extensions all sit below the model, so they keep working even against a model that emits nothing:

  • inotify / fanotify watch on the room
  • audit log of chmod, utimensat, rename syscalls
  • ctime captured from a vantage point the agent cannot reach

These turn "did it act" from a before/after diff into a full timed event sequence. The write-up's "privilege gap" section is the same argument from the other side: build the monitor on the forensic surface the agent cannot forge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment