Provenance, Preservation & Ground Truth
In Module 1 you designed a system; in Module 2 you audited its honesty. This time you audit its provenance — whether the truth behind its outputs is preserved, traceable, and recoverable. The grade is in the reasoning, not in finding a flawless system.
Time: ~45–60 min · Due: before the Module 4 session · Submit: post your write-up in the cohort channel.
Pick one system you know that produces polished outputs from some underlying data — a dashboard, a search/RAG tool, an extraction pipeline, a scoring or ranking system.
⚠️ The sleep tracker audited below is off-limits as your submission — it's the worked demo. Pick something else.
In a ~300–400 word audit, answer five things:
- Source vs derived. Name the source of truth. Then list the derived artifacts it produces. Which output feels authoritative but is actually a derived claim?
- What wins on disagreement? If a derived artifact and the source disagreed, which is right — and would anyone notice? Is there a way to check, or has the artifact quietly become the de-facto truth?
- Preserve-never-filter. Does the system preserve the source, or does a ranking/filter/transform destroy or hide anything? Name one place a wrongly-dropped item would be lost for good.
- Traceability. Pick one specific output and trace it home — to the page, passage, or records it came from. Where does the chain break?
- Recovery. If a derivation were silently wrong for three weeks, could you scope the blast radius and re-derive from a preserved source — or is it your catastrophe?
- Pick a polished output you personally trust. What's the source of truth behind it, and would anyone notice if the output and the source disagreed?
- When is it genuinely fine to discard a source after deriving from it — and when is it a trap? What distinguishes the two?
- You're replacing a 30-year-old system whose requirements were never written down. What is "correct" for the replacement — and what does that imply you must not do yet?
- Where have you seen two copies of the "same" fact disagree? Who should have owned it, and how would single-ownership have prevented the drift?
Reply to one classmate's audit. Find the single place where a derived artifact has quietly become the de-facto source of truth — the upward "traces back" arrow is cut — and say how you'd restore traceability and a recovery point.
Everything below is just these, applied to one system. Keep them in front of you:
- The map is not the territory. Every output — a score, a summary, a chart, an extraction — is a claim about a source, never the source itself.
- The source wins on disagreement. And the real danger isn't being wrong; it's that nobody can check, because the source wasn't preserved or traced.
- Preserve, never silently filter. A derived view may reorder or highlight — never delete or hide. Preservation comes before processing.
- Every claim points home. Traceability buys you verifiability and recovery; grounding forbids fabrication. An untraceable claim is one you can't responsibly act on.
- Choose a ground truth; give every fact one owner. When there's no external label, something has to be the truth — pick it deliberately, or the system treats its own output as truth.
Let's run the audit on something that is not your assignment: a wearable sleep tracker that shows a nightly Sleep Score and a stage breakdown (deep / REM / light / awake), computed from raw sensor data (motion, heart rate, temperature). Watch the moves, then make them on your own system.
1 · Source vs derived. The source of truth is the raw sensor time-series for the night. The derived artifacts are the sleep stages, the total-sleep-time number, the weekly trend, and the headline "Sleep Score: 85." The one that feels authoritative but is most derived is that score — it reads like a measured fact, but it's an inference stacked on stage estimates that are themselves inferred from noisy signals. It's a map several layers removed from the territory.
2 · What wins on disagreement — and would anyone notice? Say the app reports 7h12m asleep, but you spent an hour lying awake reading. Reality wins — but the app has no "I was actually awake" input, so the derived score silently becomes the de-facto truth. Nobody notices, because there's nothing in the app to check it against. In practice, the upward "traces back" arrow is already cut.
3 · Preserve-never-filter. Does the app keep the raw sensor stream, or only the nightly summary? Most consumer trackers keep only the derived summary and discard or never expose the raw signal. That's the anti-pattern: a destructive filter where the summary replaces the source. A mis-scored night can never be re-examined. A preserve-never-filter design would keep the raw stream and treat the score as a view over it.
4 · Traceability. Take "deep sleep: 1h20m" and try to trace it to the exact signal windows the algorithm scored as deep. In most apps you can't — the number points to nothing. The chain breaks right at the boundary between the summary and the discarded raw data. A traceable design lets you drill from the number to the minutes that produced it.
5 · Recovery. Suppose a firmware update mis-scaled heart rate and corrupted the staging for three weeks. Could you fix the historical scores? Only if the raw data was preserved. If the app kept only summaries, those three weeks are permanently wrong and unrecoverable — a catastrophe. If it kept the raw stream, you re-run the corrected algorithm over the preserved nights — a cleanup.
Notice what just happened: I separated source from derived, found what wins on disagreement (and that nobody could notice), spotted the destructive-filter anti-pattern, traced one number and found the break, and ran the recovery counterfactual. That's the whole of Part A. Your job is to make those five moves on your system.
This is the module in two arrows: derive down, traces back up. Your audit is really one question per artifact — does the upward arrow actually exist? The anti-pattern deletes the source, and every "traces back" arrow ends up pointing at nothing.
flowchart TD
SRC["SOURCE OF TRUTH<br/>preserved raw signal<br/>authoritative · wins on disagreement"]
D1["Derived<br/>sleep stages"]
D2["Derived<br/>total sleep time"]
D3["Derived<br/>Sleep Score 85"]
AUD([Human / audit<br/>verify · re-derive · recover])
SRC == derive ==> D1
SRC == derive ==> D2
SRC == derive ==> D3
D1 -. traces back .-> SRC
D2 -. traces back .-> SRC
D3 -. traces back .-> SRC
AUD -- verify against source --> SRC
style SRC stroke:#F5A623,stroke-width:2px
style AUD stroke:#C97E12,stroke-width:1.5px
Copy this into your doc and replace each blank. Keep it tight — most of the marks are in points 2 and 4.
SYSTEM (one line): __________________________________________
1. SOURCE vs DERIVED
Source of truth: ___________________________________
Derived artifacts: _________________________________
Feels authoritative but is derived: ________________
2. WHAT WINS ON DISAGREEMENT
If artifact and source disagree, right one is: _____
Would anyone notice? Can they check? _______________
3. PRESERVE-NEVER-FILTER
Is the source preserved, or filtered/destroyed? ____
One place a dropped item is lost for good: _________
4. TRACEABILITY
One output: ________________________________________
Trace it home to: __________________________________
Where the chain breaks: ____________________________
5. RECOVERY
After a 3-week silent error, could you re-derive? __
Catastrophe or cleanup, and why: ___________________
If you want a diagram, copy the Mermaid block above and relabel SRC and the derived nodes for your system — it renders as a flowchart in your Gist.
Part B — starter prompts (answer in your own words; don't just restate these):
- Ask what raw thing sits underneath the polished output — and whether there's any path from the number back to it. If there isn't, you've found your answer.
- Ask two questions of the source: is it the only authoritative copy, and could the derivation be wrong? Two yeses means discarding it is a trap.
- If the spec was never written down, what's left serving as the spec? Once you name that, ask what you therefore can't switch off yet.
- Name the two copies. Then ask which one updates and which should only ever reflect — the divergence lives in the gap between those answers.
- Mistaking a confident output for the source. A clean dashboard or a single score feels like truth; it's a derived claim.
- Assuming traceability because "the data's in there somewhere." Check you can actually point one claim home, specifically.
- Calling destructive filtering "cleanup." A dropped source is gone — and the filter's errors go with it.
- Skipping the recovery test. "Could I re-derive after a three-week silent error?" is the question that exposes everything.
- No defined ground truth. With nothing chosen as the reference, a system drifts toward treating its own output as true.
- Separates the source of truth from derived artifacts — doesn't treat a confident-looking output as the source.
- States what wins on disagreement and whether anyone could notice or check.
- Checks preserve-never-filter and names a real place a source could be lost.
- Traces one claim home and identifies exactly where the chain breaks.
Good luck — bring the audit you'd trust when a derivation has already gone wrong and you need something to fall back to.