Akash Das akash-coded

The Noise Band Ladder — is this delta real?

Six rungs. Climb until you can answer; stop at the first "no" — the rungs above cost more and cannot rescue a failure below.

#	Rung	The check	If it fails
1	Same eval set?	same slice, same `n`, same questions	nothing else means anything
2	Same config?	print the config beside the number, every time	a stale kernel is the most common "real" delta
3	Deterministic?	run it twice — identical?	your delta includes run-to-run variance
4	Outside the band?	paired bootstrap; does the 95% interval exclude zero?	inside the band ≠ regressed — it means not measurable at this n

	"""BM25's IDF from the probabilistic relevance model -- and where it goes negative.

	IDF(t) = log( (N - df + 0.5) / (df + 0.5) )

	Read the fraction: documents WITHOUT the term over documents WITH it. Above half
	the collection the ratio is below 1 and the log is negative: a ubiquitous term is
	evidence AGAINST relevance. That is the principled stop list -- no word list to
	maintain, adapts per corpus. The 0.5s keep df=0 and df=N finite. Mirrors lab L03.

	The bug that is not in the formula: the ANALYZER. If your tokenizer splits

	"""Cohen's kappa: agreement corrected for chance. Never quote raw agreement alone.

	kappa = (p_o - p_e) / (1 - p_e)
	p_e = hj + (1-h)(1-j) chance agreement from the two positive rates

	90% raw agreement on a 90/10 split is kappa ~0.44. On 95/5 the same 90% is NEGATIVE.
	So a kappa without its base rate is not comparable to anyone else's. Mirrors nanorag lab L09.
	"""

	"""Reciprocal rank fusion: merge rankings without comparing their scores.

	Two retrievers' scores are not comparable (BM25 ~8-25, cosine [-1,1]) and any
	monotone transform leaves a ranking unchanged -- so only ranks are comparable.
	RRF is a positional voting rule, closer to Borda count than to a scoring rule.

	k is a damping constant, not a free parameter: at k=0 rank 1 is worth 2x rank 2
	and one confident retriever wins; at k=60 the weights are nearly flat and what
	matters is how MANY retrievers ranked a document. Cormack et al. (2009); the
	result is insensitive across roughly 30-120. Mirrors nanorag lab L05.

	"""Attribute a failed RAG answer to the stage that failed. Three booleans, four verdicts.

	retrieval miss gold never reached the candidate pool -> index, analyzer, encoder, N
	packing loss reached the pool, not the prompt -> k, reranker, packing constraint
	generation failure in the prompt, answer still wrong -> generator / prompt
	right by accident answer right, gold never packed -> the model knew it already;
	retrieval was decorative

	Run it over fifty failures. The DISTRIBUTION is the work plan. Mirrors nanorag lab L11.
	"""

	"""Paired bootstrap for a retrieval metric: is this delta real?

	Resamples QUESTIONS with replacement, keeping both arms on the same questions --
	pairing removes per-question difficulty, which is usually the largest variance
	term. Stdlib only. Mirrors nanorag/metrics.py and lab L08.

	The two knobs people confuse:
	n how much uncertainty there is (your eval set; only this narrows the interval)
	n_boot how carefully you estimate it (stabilises the digits; does NOT narrow it)

	"""H× — cost a multi-agent topology before you build it.

	H× = tokens(topology) / tokens(one agent doing the same task)

	Multi-agent designs are chosen on a whiteboard and paid for in tokens. Three costs are
	invisible on the diagram: context re-sent on every handoff, the orchestrator's own
	reasoning turns, and the MERGE call — which carries every specialist's output and is
	the single most expensive call in the topology. Specialist count hits the total twice.

	From the L.A.B. Simulator, lab MAS-02 and drill MAS-101; framework: the Handoff Multiplier

	"""A release gate for an AI agent, in forty lines, that can actually say no.

	Three properties most gates lack:
	1. It exits non-zero. A gate that only warns is a report, and teams stop reading reports.
	2. Safety and citation bars are ABSOLUTE. A build at 99% overall with one policy-
	contradicting answer is blocked — and that breach is listed FIRST, because CI logs
	are read top-down under pressure.
	3. It never raises. A missing or unreadable metric is a block, not a crash: a crashed
	eval stage must not read as a pass, and a crashed gate has an ambiguous exit code.

	"""A tool return contract the model cannot misread.

	The bug this prevents: a policy search finds nothing and returns [].
	The model reads [] as "nothing applies" — the OPPOSITE of "I found nothing" —
	and writes "there are no restrictions on this fare". No error is raised.

	Rule: an exception is safer than an empty list, because an exception cannot be
	mistaken for data. So every outcome gets an explicit status, and every non-ok
	outcome carries advice that FORBIDS the wrong conclusion and names the next action.

	"""The two-line check that catches the most common agent-loop bug.

	Symptom: the model asks for the same tool twice, with identical arguments, forever.
	Cause: you appended the toolResult but not the assistant message that requested it.
	From the model's side it never asked, so the reply has nothing to attach to.

	The invariant: every toolUse in history has exactly one toolResult with the same id.
	Run it in development after every turn; it fails at the cause, not a turn later.

	From the L.A.B. Simulator, lab AGL-02: