Redundant tests scoring rubric

redundant-tests-scoring-rubric.md

Rubric

This scorer applies only to implementation-agent runs, including a later revision re-entry. It grades the final delivered test diff — the tests and cases that landed on the PR after self-review — not intermediate tests the agent wrote and then deleted, and not whether a human later asked to remove them (that event is unnecessary-tests-human-correction). The question is surplus in the artifact: redundant, tautological, or low-signal tests that would not fail on a real defect any other added or existing test misses.

This is a within-run ordinal so the across-run frequency is the sampled fail rate. samplingRate is 100. passingScore is 0.5, which means any surplus fails the run: mostly_surplus_tests (0.2) and some_surplus_tests (0.4) both fail, with the two ordinals keeping a material share of surplus distinguishable from a minority of low-signal cases. no_surplus_tests (1) is a clean pass. insufficient_evidence (0.5) is a pass at the threshold, not a clean pass: it keeps missing coverage out of this fail rate (Code Quality owns that) without treating an empty test diff as no_surplus_tests.

Ask of each added test or case — a table row or a subtest is a case — the same question code-review Test value asks: what behavior change would make this fail, and would that failure be a real defect no other added or existing test catches? Answer it against the production code, never against the test's name or shape. Surplus is the honest answer "a behavior-preserving refactor", "nothing", or "a higher-level test already covers it." Those conclusions usually come from an assertion that restates a literal or constant from the source; a check the framework already enforces; a getter, default, or plain construction; another case that already reaches the same branch; a call sequence or private state instead of an outcome; or setup dominated by wiring rather than exercising a public entry point. Judge each case on that defect-detection value. Renaming, splitting, or duplicating a tautology does not make it new coverage. Padding production lines does not hide a junk-test majority. Do not use a test-LOC / production-LOC ratio as the signal — that ratio is gameable and is not a substitute for per-case value.

Never flag these; each is a case where the thing that looks redundant is the only expression of the behavior (code-review Test value exemptions):

• A case pinning a contract another system depends on where nothing else fails when the value silently changes — a flag spelling, a wire format, a telemetry key, the absence of an interface implementation — provided the case observes the value where that consumer reads it. A focused flag-consumer test of that shape is not surplus. A getter, default, construction, or flag-off-path restatement of source ("when the flag is off, the old code path still runs") is surplus.
• A call assertion where the collaborator is the unit's only observable output, including a not-called assertion where non-invocation is the behavior under test.
• A class or markup assertion where the rendered class is all the unit exposes.
• Real IO where the IO is the subject rather than incidental to the logic.
• A near-identical case that reaches a different branch or pins a boundary the others do not.
• A small test that pins real behavior. The burden is "what defect does this catch", never "is this test big enough". Do not fail a focused regression test on a bugfix.

A test the human explicitly requested — the requester, a named Slack participant, or a human PR reviewer, including an orchestrator relay that plainly attributes the ask to a human — is justified even if it would otherwise look low-signal. Code-review agent findings about tests are out of scope: this scorer reads the implementation artifact, not the reviewer's opinion of it.

Assess:

• Evidence is the final PR test diff. Prefer the tests as they landed on the PR. Tests added then left in after self-review count. Tests the agent deleted before delivery do not. File names without bodies are not enough to judge surplus.
• No tests added is not a clean pass unless the change is exempt. Documented no-test exemptions (config-only, dependency or version bumps, constant or flag defaults, pure data or copy) with no test added are no_surplus_tests. A behavior change with no test added is insufficient_evidence, never no_surplus_tests — omitting necessary tests is missing coverage, scored under Code Quality, and must not be rewarded here.
• Share, not line count. no_surplus_tests means every added case earns its place, or an exempt change added none. some_surplus_tests means surplus is present but not a material share: typically one or two low-signal cases beside otherwise justified coverage, or a clear minority of added cases. mostly_surplus_tests means a material share — about half or more of the added cases, or a suite whose defect-detection value is dominated by surplus even when a raw count is noisy (a large tautological table next to one real regression). Typical fail classes: flag/config getter or flag-off restatements of source; getter-setter and tautological mock tests.
• Truncation and missing diffs. No usable final diff, a truncated transcript, or test files that cannot be read is insufficient_evidence. Ambiguity about whether a case is surplus is not this label — grade the cases you can see.

Out of scope: whether a human asked the agent to delete a test (scored under Unnecessary Test Removal Correction); whether the rest of the PR would merge (Code Quality's binary approve/block, which can still approve when junk tests sit below the senior-reviewer block bar); missing coverage on a behavior change (Code Quality Tests); and the code-review agent's findings. The overlap with Code Quality Tests and the human-correction scorer is deliberate: this scorer is the artifact-side frequency metric so surplus tests can be burned down even when nobody complained and the rest of the PR would still merge.

Reason

One to three sentences citing specific added tests or files and the share that is surplus (for example, "2 of 5 added cases" or "both added tests"), so failures can be clustered by cause rather than only counted. Name the defect-detection failure — restated source, already-covered branch, getter/default, mock-only, implementation-coupled — not merely that tests were "too many." For insufficient_evidence, say exactly what was missing: no diff, truncated transcript, unread test bodies, or a behavior change with no tests added.