A survey of public benchmarks, datasets, and open-source evaluation harnesses for transcript redaction and PII de-identification, with a focus on what exists for spoken / conversational / ASR transcripts versus clean written text — and what's still missing.
Scope note: "redaction eval" here means a benchmark or harness with ground-truth labels that lets you score how well a system removes PII/PHI (precision, recall, F1/F2, leakage rate), not just a redaction tool on its own.
| Name | Type | Domain | Spoken / ASR? | Multilingual | Indirect / quasi-ID leakage | Metrics | License | Link |
|---|---|---|---|---|---|---|---|---|
| i2b2 / n2c2 2014 de-id | Labeled dataset | Clinical notes (written) | ❌ Written narratives | ❌ EN | NER P/R/F1 | DUA / restricted | HF card · portal | |
| TAB (Text Anonymization Benchmark) | Dataset + harness | Legal — 1,268 ECHR court cases | ❌ Clean formal text | ❌ EN | ✅ DIRECT / QUASI / NO_MASK | ER_di, ER_qi, weighted precision (IC = −log Pr), F2 | Open (repo) | GitHub · CL 2022 paper |
| CallCenterEN | Labeled dataset | Real call-center calls — 91,706 transcripts, 10,448 audio hrs | ✅ Genuine spoken (audio withheld) | ❌ EN (IN/PH/US accents) | (redacted transcripts; harness unconfirmed) | CC BY-NC 4.0 | arXiv 2507.02958 · HF | |
| AI4Privacy (200k / 300k / 500k) | Synthetic labeled dataset | Template-generated synthetic text | ❌ Synthetic written | ✅ up to ~EN/ES/IT + more | span P/R/F1, SPriV | non-commercial / restricted | 200k · 300k · 500k | |
| MultiGraSCCo | Labeled dataset | Clinical (MT-translated German GraSCCo) | ❌ Written clinical | ✅ 10 langs / 6 families / 3 scripts | ✅ Indirect Personal Identifiers (k-anonymity framing) | span-level + IPI categories | CC BY 4.0 (Zenodo) | arXiv 2603.08879 · Zenodo |
| Presidio Evaluator (presidio-research) | Eval harness | Synthetic templates (BYO data) | ❌ Text-only | ❌ direct entities only | precision / recall / F-measure, error analysis | MIT | GitHub · PyPI | |
| ETUDE | Eval harness | Clinical de-id (6 tools, 4 corpora) | ❌ EN clinical notes | ❌ EN | P/R/F1 (+F2), exact/partial/contained matching | Open (Codeberg/GitHub) | JMIR 2024 | |
| PRvL | Models + harness | Self-hosted LLM redaction on AI4Privacy | ❌ Synthetic written | ✅ EN/ES/IT | span P/R/Acc (Span-Correct vs Label-Exact), SPriV = unmasked PII / output tokens | Open (open-source LLMs) | arXiv 2508.05545 | |
| Lightweight-LM PII study | Eval study | AI4Privacy-200k + real Discord chat | ❌ EN | char/entity P/R/F1, strict/relaxed boundaries, SPriV = FN / gold tokens | Paper (open) | arXiv 2512.18608 |
Legend: ✅ yes / strong ·
- i2b2 / n2c2 2014 — the canonical clinical de-id benchmark. Ground-truth PHI labels, but framed as NER over written clinical narratives (Partners Healthcare RPDR). Not spoken. Access is gated behind a data-use agreement.
- TAB — the best-designed redaction-specific benchmark. 1,268 English ECHR court cases, hand-annotated with direct vs. quasi-identifiers, plus a real eval harness (
evaluation.py). Domain is clean formal legal prose, so it's a methodology gold standard, not a spoken-data source. - CallCenterEN — the rare benchmark built on actual spoken conversations: 91,706 real agent-customer call transcripts, 10,448 audio hours, mixed accents. Audio is withheld for biometric privacy; transcripts are PII-redacted. The closest thing to a real-world ASR redaction corpus.
- AI4Privacy — large synthetic labeled PII corpora (200k/300k/500k) that most recent LLM-redaction papers train and eval on. Cheap, scalable ground truth — but template-generated written text, not real conversation.
- MultiGraSCCo — addresses two gaps at once: 10 languages across 3 writing systems, and an explicit indirect / quasi-identifier schema. But it's machine-translated clinical text, so still not spoken.
- Presidio Evaluator — the most practical off-the-shelf harness (MIT). Scores Presidio/spaCy/Flair/Azure for P/R/F. Ships only synthetic template ground truth, is text-only and English-by-default, and has no conversational/ASR handling.
- ETUDE — extensible end-to-end harness for clinical de-id; evaluates 6 tools across 4 corpora with exact/partial/contained span matching. English clinical notes only.
- PRvL — open-source self-hostable redaction models + harness; introduces SPriV as a privacy-leakage metric (proportion of PII left unmasked).
- Lightweight-LM PII study — the single clearest empirical proof of the gap: redaction accuracy collapses on real conversational (Discord) text (T5 ~0.97 → 0.788; Mistral 0.876), attributed to noise/slang/irregular syntax.
Everyone converges on precision / recall / F1, usually with F2 favored because in redaction a missed identifier (false negative) is far worse than an over-redaction (false positive). Beyond that:
- Leakage metrics are proliferating without a standard. Each paper rolls its own "SPriV":
- PRvL:
unmasked PII tokens / total output tokens - Lightweight-LM study:
FN / total gold tokens - TAB: entity-level recall split into
ER_di(direct) andER_qi(quasi), plus information-content-weighted precision (IC = −log Pr(token)), so masking a rare/identifying token counts more than a common one.
- PRvL:
- Span matching varies: exact vs. partial vs. fully-contained (ETUDE), strict vs. relaxed boundaries (lightweight-LM), span-correct vs. label-exact (PRvL). Numbers are not comparable across papers without normalizing this.
1. Almost nothing targets ASR noise directly. CallCenterEN is sourced from spoken audio, but no public benchmark deliberately models word-error-rate, disfluencies, speaker overlap, or transcription errors in its ground truth. The one study that touches real conversational text (Discord) shows a large accuracy drop — but that's a finding, not a reusable benchmark. There is no harness for evaluating the audio → ASR → redaction pipeline jointly (e.g., does a phone number survive being mis-transcribed?).
2. The good methodology lives in the wrong domains. TAB (legal) and MultiGraSCCo (clinical) are the only benchmarks that seriously model quasi-identifier / indirect leakage — exactly what matters for conversational data, where someone is re-identifiable from "my daughter's school" + "the plant I work at" without ever stating a name. But both are clean written text. Nobody has ported quasi-identifier annotation onto spoken/conversational transcripts.
3. Harnesses are English-text-first. Presidio Evaluator and ETUDE — the two most usable harnesses — are English and text-only. MultiGraSCCo brings multilingual labels but no general harness. There's no multilingual, conversation-aware eval harness off the shelf.
4. Leakage metrics aren't standardized. Three different "SPriV" definitions already exist. Without a community-standard leakage metric (and standardized span-matching), cross-paper comparison is unreliable.
5. Ground truth is mostly synthetic or clinical. The scalable corpora (AI4Privacy) are template-synthetic; the realistic ones are clinical/legal. The one genuinely-spoken corpus (CallCenterEN) is non-commercial (CC BY-NC 4.0), withholds audio, and it's unconfirmed whether it ships a scoring harness or un-redacted gold labels at all.
There is mature tooling to score PII redaction on clean English written text, strong methodology for quasi-identifier leakage in legal/clinical domains, and exactly one real spoken-conversation corpus — but no open, multilingual, ASR-noise-aware benchmark with a reusable harness and a standardized leakage metric for the case most people actually care about: redacting messy real conversation.
- i2b2/n2c2 2014 — https://huggingface.co/datasets/bigbio/n2c2_2014_deid · https://portal.dbmi.hms.harvard.edu/projects/n2c2-2014/ · https://n2c2.dbmi.hms.harvard.edu/challenge/2014-deidentification-heart-disease
- TAB — https://github.com/NorskRegnesentral/text-anonymization-benchmark · https://aclanthology.org/2022.cl-4.19.pdf · https://arxiv.org/abs/2202.00443
- CallCenterEN — https://arxiv.org/abs/2507.02958 · https://huggingface.co/papers/2507.02958
- AI4Privacy — https://huggingface.co/datasets/ai4privacy/pii-masking-200k · https://huggingface.co/datasets/ai4privacy/pii-masking-300k · https://huggingface.co/datasets/ai4privacy/open-pii-masking-500k-ai4privacy
- MultiGraSCCo — https://arxiv.org/pdf/2603.08879 · https://doi.org/10.5281/zenodo.19489040
- Presidio Evaluator — https://github.com/microsoft/presidio-research · https://pypi.org/project/presidio-evaluator/ · https://github.com/microsoft/presidio
- ETUDE — https://pmc.ncbi.nlm.nih.gov/articles/PMC11167315/
- PRvL — https://arxiv.org/pdf/2508.05545
- Lightweight-LM PII study — https://arxiv.org/html/2512.18608
Several core sources (PRvL, CallCenterEN, lightweight-LM study, MultiGraSCCo) are 2025–2026 arXiv preprints, not yet peer-reviewed; some artifact/access links were anonymized review URLs at submission. The most stable sources are peer-reviewed TAB (Computational Linguistics 2022) and ETUDE (JMIR 2024) plus the established i2b2/n2c2 sets. A claim that n2c2 2014 regroups 18 HIPAA categories into "6 main / 25 sub" did not survive verification — treat fine-grained category counts for that set as unconfirmed. "AI4Privacy is synthetic-written" and "Presidio has no ASR support" are accurate as used but not absolute (both expose pluggable multilingual hooks).
Compiled from a fan-out multi-source search with adversarial claim verification (24/25 claims confirmed by 3-vote majority). Generated with Claude Code.