Last active
June 22, 2026 12:39
-
-
Save comalex/403b6dd7c9083352147113a4433db6c5 to your computer and use it in GitHub Desktop.
Childfree Legacy — Aurora Migration Plan & Playbook (Track B focused)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <title>Childfree Legacy — Aurora Migration: Complete Plan + DevOps Deep Review</title> | |
| <style> | |
| :root{ | |
| --ink:#1a2238;--ink-soft:#4a5170;--muted:#6b7194; | |
| --rule:#e4e6ef;--rule-soft:#f1f2f7;--bg:#fafbfc;--card:#fff; | |
| --accent:#5b3aa6;--accent-soft:#efe9fb; | |
| --accent-2:#0b6b6b;--accent-2-soft:#e3f3f3; | |
| --crit:#b3261e;--crit-soft:#fdecea; | |
| --high:#b5650a;--high-soft:#fcefe0; | |
| --med:#8a6d00;--med-soft:#fff7d6; | |
| --ok:#1f6f43;--ok-soft:#e6f4ec; | |
| --code-bg:#f6f7fb;--code-ink:#2d3a87; | |
| } | |
| *{box-sizing:border-box;} | |
| html{-webkit-font-smoothing:antialiased;scroll-behavior:smooth;} | |
| body{margin:0;background:var(--bg);color:var(--ink); | |
| font-family:-apple-system,BlinkMacSystemFont,"Segoe UI","Helvetica Neue",Arial,sans-serif; | |
| font-size:15.5px;line-height:1.62;} | |
| .layout{display:grid;grid-template-columns:300px 1fr;min-height:100vh;} | |
| aside.nav{position:sticky;top:0;height:100vh;overflow-y:auto;background:var(--card); | |
| border-right:1px solid var(--rule);padding:20px 16px;font-size:12.5px;} | |
| aside.nav h4{text-transform:uppercase;letter-spacing:.12em;font-size:10.5px;color:var(--accent); | |
| margin:14px 0 6px;font-weight:700;} | |
| aside.nav h4:first-of-type{margin-top:0;} | |
| aside.nav h4.devops{color:var(--accent-2);} | |
| aside.nav ol{list-style:none;padding:0;margin:0;} | |
| aside.nav a{display:block;padding:3px 7px;color:var(--ink-soft);text-decoration:none; | |
| border-radius:4px;line-height:1.3;} | |
| aside.nav a:hover{background:var(--accent-soft);color:var(--accent);} | |
| aside.nav .brand{font-weight:700;color:var(--ink);font-size:13px;padding-bottom:10px; | |
| border-bottom:1px solid var(--rule);margin-bottom:12px;} | |
| main.content{max-width:1080px;margin:0;padding:38px 50px 96px;} | |
| .eyebrow{text-transform:uppercase;letter-spacing:.18em;font-size:11.5px;color:var(--accent); | |
| font-weight:700;margin-bottom:10px;} | |
| h1{font-size:30px;line-height:1.18;margin:0 0 14px;letter-spacing:-.01em;} | |
| .lede{font-size:16px;color:var(--ink-soft);margin:0 0 16px;max-width:880px;} | |
| .meta{display:flex;flex-wrap:wrap;gap:8px;font-size:12.5px;} | |
| .meta-chip{background:var(--accent-soft);color:var(--accent);padding:3px 10px; | |
| border-radius:999px;font-weight:600;} | |
| h2{font-size:23px;margin:48px 0 14px;padding-bottom:9px;border-bottom:1px solid var(--rule); | |
| letter-spacing:-.005em;scroll-margin-top:18px;color:var(--accent);} | |
| h2.devops{color:var(--accent-2);} | |
| h3{font-size:17.5px;margin:30px 0 11px;scroll-margin-top:18px;} | |
| h4{font-size:14.5px;margin:22px 0 6px;} | |
| p{margin:0 0 12px;} | |
| strong{color:var(--ink);} | |
| em{color:var(--ink-soft);} | |
| code{font-family:"SF Mono","JetBrains Mono",Menlo,Consolas,monospace;font-size:.86em; | |
| background:var(--code-bg);color:var(--code-ink);padding:1px 5px;border-radius:4px; | |
| border:1px solid var(--rule-soft);} | |
| pre{font-family:"SF Mono","JetBrains Mono",Menlo,Consolas,monospace;background:var(--code-bg); | |
| color:var(--ink);padding:14px 16px;border-radius:7px;border:1px solid var(--rule); | |
| font-size:12px;line-height:1.5;overflow-x:auto;margin:0 0 14px;} | |
| pre code{background:transparent;border:0;padding:0;color:inherit;font-size:inherit;} | |
| pre.sql{background:#f6f8ef;border-color:#dde6c6;} | |
| pre.ts{background:#f4f6ff;border-color:#d6dcf3;} | |
| pre.sh{background:#1a2238;color:#f1f2f7;border-color:#1a2238;} | |
| pre.sh code{color:inherit;} | |
| pre.yaml{background:#fffbf0;border-color:#f0d58a;} | |
| pre.diagram{background:#1a2238;color:#e8eaf2;border-color:#1a2238;} | |
| pre.diagram code{color:inherit;} | |
| .card{background:var(--card);border:1px solid var(--rule);border-radius:10px; | |
| padding:16px 20px;margin:14px 0;} | |
| .summary-card{background:var(--card);border:1px solid var(--rule);border-left:4px solid var(--accent); | |
| border-radius:7px;padding:16px 20px;margin:20px 0 28px;} | |
| .summary-card.crit{border-left-color:var(--crit);} | |
| .summary-card.ok{border-left-color:var(--ok);} | |
| .summary-card h4{margin-top:0;color:var(--accent);font-size:11px;text-transform:uppercase; | |
| letter-spacing:.1em;} | |
| ul,ol{padding-left:22px;margin:0 0 14px;} | |
| li{margin-bottom:5px;} | |
| table{width:100%;border-collapse:collapse;margin:6px 0 20px;font-size:13px; | |
| background:var(--card);border:1px solid var(--rule);border-radius:7px;overflow:hidden;} | |
| thead th{background:var(--rule-soft);text-align:left;padding:9px 11px;font-weight:700; | |
| color:var(--ink);border-bottom:1px solid var(--rule);font-size:11px; | |
| text-transform:uppercase;letter-spacing:.05em;} | |
| tbody td{padding:9px 11px;border-bottom:1px solid var(--rule-soft);vertical-align:top;} | |
| tbody tr:last-child td{border-bottom:0;} | |
| tbody tr:nth-child(even){background:#fafbfd;} | |
| .callout{border-left:4px solid var(--accent);background:var(--accent-soft);padding:12px 16px; | |
| border-radius:0 7px 7px 0;margin:14px 0;font-size:14.5px;} | |
| .callout.crit{border-left-color:var(--crit);background:var(--crit-soft);} | |
| .callout.high{border-left-color:var(--high);background:var(--high-soft);} | |
| .callout.med{border-left-color:var(--med);background:var(--med-soft);} | |
| .callout.ok{border-left-color:var(--ok);background:var(--ok-soft);} | |
| .callout.devops{border-left-color:var(--accent-2);background:var(--accent-2-soft);} | |
| .callout.rlsfix{border-left-color:#7c3aed;background:#f5f0ff;} | |
| .callout.rlsfix .callout-label{color:#7c3aed;} | |
| .callout.authfirst{border-left-color:#b45309;background:#fffbeb;} | |
| .callout.authfirst .callout-label{color:#b45309;} | |
| .callout.review2{border-left-color:#0e7490;background:#ecfeff;} | |
| .callout.review2 .callout-label{color:#0e7490;} | |
| .callout.teamfix{border-left-color:#be123c;background:#fff1f2;} | |
| .callout.teamfix .callout-label{color:#be123c;} | |
| .callout.datarch{border-left-color:#15803d;background:#f0fdf4;} | |
| .callout.datarch .callout-label{color:#15803d;} | |
| .callout.prdval{border-left-color:#a21caf;background:#fdf4ff;} | |
| .callout.prdval .callout-label{color:#a21caf;} | |
| .callout-label{text-transform:uppercase;letter-spacing:.1em;font-size:10.5px;font-weight:700; | |
| margin-bottom:3px;} | |
| .callout.crit .callout-label{color:var(--crit);} | |
| .callout.high .callout-label{color:var(--high);} | |
| .callout.med .callout-label{color:var(--med);} | |
| .callout.ok .callout-label{color:var(--ok);} | |
| .callout.accent .callout-label{color:var(--accent);} | |
| .callout.devops .callout-label{color:var(--accent-2);} | |
| .sev{display:inline-block;padding:2px 8px;border-radius:4px;font-size:10px;font-weight:700; | |
| text-transform:uppercase;letter-spacing:.05em;white-space:nowrap;} | |
| .sev.crit{background:var(--crit-soft);color:var(--crit);} | |
| .sev.high{background:var(--high-soft);color:var(--high);} | |
| .sev.med{background:var(--med-soft);color:var(--med);} | |
| .sev.ok{background:var(--ok-soft);color:var(--ok);} | |
| .sev.verified{background:var(--ok-soft);color:var(--ok);} | |
| .sev.gap{background:var(--crit-soft);color:var(--crit);} | |
| .sev.new{background:#e6f4ec;color:#1f6f43;} | |
| .sev.rlsfix{background:#ede9fe;color:#6d28d9;} | |
| ins.rlsfix{background:#ede9fe;color:#4c1d95;text-decoration:none;padding:0 3px;border-radius:3px;} | |
| .sev.authfirst{background:#fef3c7;color:#92400e;} | |
| ins.authfirst{background:#fef3c7;color:#78350f;text-decoration:none;padding:0 3px;border-radius:3px;} | |
| .sev.review2{background:#cffafe;color:#155e75;} | |
| ins.review2{background:#cffafe;color:#0c4a6e;text-decoration:none;padding:0 3px;border-radius:3px;} | |
| .sev.teamfix{background:#ffe4e6;color:#9f1239;} | |
| ins.teamfix{background:#ffe4e6;color:#881337;text-decoration:none;padding:0 3px;border-radius:3px;} | |
| .sev.datarch{background:#dcfce7;color:#166534;} | |
| ins.datarch{background:#dcfce7;color:#14532d;text-decoration:none;padding:0 3px;border-radius:3px;} | |
| .sev.prdval{background:#fae8ff;color:#86198f;} | |
| ins.prdval{background:#fae8ff;color:#701a75;text-decoration:none;padding:0 3px;border-radius:3px;} | |
| .stat-row{display:flex;flex-wrap:wrap;gap:10px;margin:16px 0 6px;} | |
| .stat{flex:1;min-width:118px;background:var(--card);border:1px solid var(--rule); | |
| border-radius:9px;padding:12px 14px;} | |
| .stat .num{font-size:22px;font-weight:700;line-height:1.05;color:var(--accent);} | |
| .stat .lbl{font-size:11.5px;color:var(--muted);margin-top:4px;} | |
| .stat.ok .num{color:var(--ok);} | |
| .stat.crit .num{color:var(--crit);} | |
| hr{border:0;border-top:1px solid var(--rule);margin:42px 0;} | |
| hr.devops{border-top:2px solid var(--accent-2);} | |
| .section-num{color:var(--accent);font-weight:700;font-size:13.5px;text-transform:uppercase; | |
| letter-spacing:.1em;} | |
| .section-num.devops{color:var(--accent-2);} | |
| .pid{display:inline-block;background:var(--accent);color:#fff;font-size:10.5px;font-weight:700; | |
| padding:2px 7px;border-radius:4px;font-family:"SF Mono",Menlo,monospace;margin-right:8px;} | |
| .review-block{border:1px solid var(--rule);border-radius:10px;padding:16px 18px;margin:18px 0;background:var(--card);} | |
| .review-block h4{margin:0 0 8px;color:var(--accent-2);} | |
| .label{font-size:10.5px;font-weight:700;text-transform:uppercase;letter-spacing:.08em; | |
| color:var(--muted);margin:10px 0 3px;} | |
| .label.crit{color:var(--crit);} | |
| .label.ok{color:var(--ok);} | |
| .label.high{color:var(--high);} | |
| .part-banner{background:linear-gradient(135deg, var(--accent-soft), #fff); | |
| border:1px solid var(--rule);border-radius:12px;padding:22px 26px;margin:32px 0 24px;} | |
| .part-banner.devops{background:linear-gradient(135deg, var(--accent-2-soft), #fff);} | |
| .part-banner .part-num{font-size:11px;letter-spacing:.18em;text-transform:uppercase; | |
| color:var(--accent);font-weight:700;} | |
| .part-banner.devops .part-num{color:var(--accent-2);} | |
| .part-banner h2{margin:6px 0 8px;border:0;padding:0;font-size:24px;} | |
| .part-banner p{margin:0;font-size:14.5px;color:var(--ink-soft);} | |
| footer{margin-top:60px;padding-top:22px;border-top:1px solid var(--rule);font-size:12.5px; | |
| color:var(--muted);text-align:center;} | |
| @media print{ | |
| body{background:#fff;font-size:10.5pt;} | |
| .layout{display:block;} | |
| aside.nav{display:none;} | |
| main.content{padding:0;max-width:none;} | |
| h2,h3,h4{page-break-after:avoid;} | |
| .card,.callout,table,pre,.review-block{page-break-inside:avoid;} | |
| } | |
| @media (max-width:900px){ | |
| .layout{grid-template-columns:1fr;} | |
| aside.nav{position:static;height:auto;border-right:0;border-bottom:1px solid var(--rule);} | |
| main.content{padding:28px 20px 72px;} | |
| } | |
| </style> | |
| </head> | |
| <body> | |
| <div class="layout"> | |
| <aside class="nav"> | |
| <div class="brand">CHIL · Aurora Migration · Complete</div> | |
| <h4>Part 1 · Why & What</h4> | |
| <ol> | |
| <li><a href="#exec">0. Executive summary</a></li> | |
| <li><a href="#why-aurora">1. Why Aurora</a></li> | |
| <li><a href="#canonical">2. Canonical data model</a></li> | |
| <li><a href="#topology">3. Target topology</a></li> | |
| <li><a href="#tenancy-brands">3b. Identity, tenancy & brands</a></li> | |
| </ol> | |
| <h4>Part 2 · Code-validated baseline</h4> | |
| <ol> | |
| <li><a href="#code-baseline">4. What's actually on main</a></li> | |
| <li><a href="#discoveries">5. Findings from code validation</a></li> | |
| </ol> | |
| <h4>Part 3 · Architecture detail</h4> | |
| <ol> | |
| <li><a href="#dispatcher">6. Dispatcher + Amplify connect-to-SQL</a></li> | |
| <li><a href="#rls">7. RLS — multi-tenancy spine</a></li> | |
| <li><a href="#frontend">8. Frontend impact (headline)</a></li> | |
| <li><a href="#brand-track">8b. White-label & brand track</a></li> | |
| <li><a href="#reporting">9. Reporting (Aurora reader)</a></li> | |
| </ol> | |
| <h4>Part 4 · Week-by-week playbook</h4> | |
| <ol> | |
| <li><a href="#pre-flight">10. Pre-flight checklist</a></li> | |
| <li><a href="#phase-1">11. Phase 1 — Foundation</a></li> | |
| <li><a href="#phase-2">12. Phase 2 — Schema + non-PHI</a></li> | |
| <li><a href="#phase-3">13. Phase 3 — Dispatcher + tenancy</a></li> | |
| <li><a href="#phase-4">14. Phase 4 — PHI cutover</a></li> | |
| <li><a href="#phase-5">15. Phase 5 — Reporting + DR + closeout</a></li> | |
| </ol> | |
| <h4 class="devops">Part 5 · DevOps deep review</h4> | |
| <ol> | |
| <li><a href="#how-to-read">16. How to read DevOps items</a></li> | |
| <li><a href="#account">17. AWS account + organization</a></li> | |
| <li><a href="#identity-center">18. IAM Identity Center</a></li> | |
| <li><a href="#vpc">19. VPC + network topology</a></li> | |
| <li><a href="#kms">20. KMS keys</a></li> | |
| <li><a href="#aurora">21. Aurora cluster config</a></li> | |
| <li><a href="#rds-proxy">22. RDS Proxy <em>(superseded)</em></a></li> | |
| <li><a href="#pg-extensions">23. Postgres extensions</a></li> | |
| <li><a href="#pg-roles">24. Postgres roles</a></li> | |
| <li><a href="#ddb-streams">25. DDB Streams + PITR</a></li> | |
| <li><a href="#cognito">26. Cognito coordination</a></li> | |
| <li><a href="#customauth">27. customAuth Lambda</a></li> | |
| <li><a href="#lambda-vpc">28. Lambda VPC attach <em>(0 of 76)</em></a></li> | |
| <li><a href="#pc">29. Provisioned Concurrency</a></li> | |
| <li><a href="#cold-start">30. Cold-start mitigation</a></li> | |
| <li><a href="#secrets">31. Secrets Manager</a></li> | |
| <li><a href="#s3-paths">32. S3 path policies</a></li> | |
| <li><a href="#audit-bucket">33. S3 audit-archive</a></li> | |
| <li><a href="#cloudtrail">34. CloudTrail</a></li> | |
| <li><a href="#waf">35. WAF v2</a></li> | |
| <li><a href="#appsync">36. AppSync hardening</a></li> | |
| <li><a href="#csp">37. CSP + HTTP headers</a></li> | |
| <li><a href="#cicd">38. CI/CD security gates</a></li> | |
| <li><a href="#drizzle-runner">39. Migration runner</a></li> | |
| <li><a href="#yaml-bug">40. amplify.yml fixes</a></li> | |
| <li><a href="#dashboards">41. CloudWatch dashboards</a></li> | |
| <li><a href="#alarms">42. Alarms inventory</a></li> | |
| <li><a href="#sentry">43. Sentry hardening</a></li> | |
| <li><a href="#backups">44. Backup strategy</a></li> | |
| <li><a href="#dr-drill">45. DR drill mechanics</a></li> | |
| <li><a href="#cutover-infra">46. Cutover-temp infra</a></li> | |
| <li><a href="#rollback">47. Rollback mechanics</a></li> | |
| <li><a href="#decommission">48. Decommission plan</a></li> | |
| </ol> | |
| <h4>Part 6 · Frontend</h4> | |
| <ol> | |
| <li><a href="#frontend-catalog">49. Frontend change catalog</a></li> | |
| <li><a href="#frontend-noop">50. What does NOT change</a></li> | |
| </ol> | |
| <h4>Part 7 · Security & compliance</h4> | |
| <ol> | |
| <li><a href="#hipaa">51. HIPAA Safeguards schedule</a></li> | |
| <li><a href="#soc2">52. SOC 2 criteria schedule</a></li> | |
| <li><a href="#pre-cutover-gates">53. Pre-cutover hard requirements</a></li> | |
| <li><a href="#evidence-pack">54. Auditor evidence pack</a></li> | |
| </ol> | |
| <h4>Part 8 · Closeout</h4> | |
| <ol> | |
| <li><a href="#timeline">56. Timeline summary</a></li> | |
| <li><a href="#mike-deliverables">57. Mike's deliverables</a></li> | |
| <li><a href="#acceptance">58. Acceptance criteria</a></li> | |
| <li><a href="#openq">59. Open questions</a></li> | |
| </ol> | |
| <h4>Appendix</h4> | |
| <ol> | |
| <li><a href="#glossary">A. Glossary</a></li> | |
| <li><a href="#review-method">B.1 Review methodology</a></li> | |
| <li><a href="#arc42-audit">B.2 arc42 coverage audit</a></li> | |
| <li><a href="#c4-model">B.3 C4 model — Context / Container / Component</a></li> | |
| <li><a href="#utility-tree">B.4 ATAM utility tree</a></li> | |
| <li><a href="#risk-themes">B.5 ATAM risk themes</a></li> | |
| <li><a href="#sensitivity">B.6 ATAM sensitivity & tradeoff points</a></li> | |
| <li><a href="#well-arch">B.7 AWS Well-Architected 6-pillar scoring</a></li> | |
| <li><a href="#fitness-functions">B.8 Evolutionary fitness functions catalog</a></li> | |
| <li><a href="#adr-inventory">B.9 ADR inventory (decisions to formalize)</a></li> | |
| <li><a href="#maturity">B.10 Architecture maturity scorecard</a></li> | |
| <li><a href="#gap-closure">B.11 Pre-Phase-1 gap closure</a></li> | |
| <li><a href="#phi-inventory">C.1 PHI field inventory + encryption tiers</a></li> | |
| <li><a href="#sec-devops-review">C.2 Security + DevOps consolidated review</a></li> | |
| <li><a href="#tenant-deep-dive">D.1 Tenant architecture — Silo / Pool / Bridge analysis</a></li> | |
| <li><a href="#tenant-pattern">D.2 The pattern we land on, why, and when to migrate</a></li> | |
| <li><a href="#tenant-lifecycle">D.3 Tenant lifecycle — onboarding to off-boarding</a></li> | |
| <li><a href="#senior-review-closure">E.1 Senior review — finding closure matrix</a></li> | |
| <li><a href="#senior-review-criticals">E.2 Critical-finding responses (C1–C6)</a></li> | |
| <li><a href="#senior-review-gaps">E.3 Architecture gap closures (AG1–AG8)</a></li> | |
| </ol> | |
| </aside> | |
| <main class="content"> | |
| <header> | |
| <div class="eyebrow">Childfree Legacy · Aurora Migration · Complete Document · v3 (senior-review incorporated)</div> | |
| <h1>Aurora migration — complete plan, phased playbook, DevOps deep review with code validation</h1> | |
| <p class="lede"> | |
| Single source of truth for the Aurora migration. Combines the execution plan (what to do, when, what the frontend | |
| feels, which HIPAA/SOC 2 controls land) with the DevOps deep architectural review (every claim code-validated, every | |
| item with the exact CLI command Mike runs). Companion to <code>CALL-PREP-UNIFIED.html</code> (which includes | |
| diagnosis, Track A alternative, full decision context); this one assumes Track B is chosen. | |
| </p> | |
| <div class="meta"> | |
| <span class="meta-chip">Date · 2026-05-27</span> | |
| <span class="meta-chip">Frontend rewrite · NO</span> | |
| <span class="meta-chip">Code-validated · main fb4cf8b3e</span> | |
| <span class="meta-chip">Audience · Jon · Mike · Oleksii</span> | |
| </div> | |
| </header> | |
| <div class="callout"><div class="callout-label">Review history</div> | |
| This plan was hardened across <strong>six independent review rounds</strong> (June 2026), all folded into the sections below: (1) PHI/RLS design review · (2) the auth-first decision · (3) round-2 field-level <code>@auth</code> follow-up · (4) team review (Mike) + data audit (Vasyl) · (5) data-architecture review (AWS console + schema audit) · (6) PRD validation (live code + 2026-05-04 AWS snapshot, "go with conditions"). Provenance and the corrected facts are consolidated in <a href="#review-history">Appendix · Review history & sources</a> at the end. <strong>📄</strong> throughout marks updates sourced from the <em>Data Architecture Review</em>. | |
| </div> | |
| <div class="part-banner"> | |
| <div class="part-num">Part 1 of 8</div> | |
| <h2 style="border:0;color:var(--ink);">Why & What — the case + the target</h2> | |
| <p>Executive summary, six reasons to migrate, the canonical data model we're landing on, and the target architecture diagram. This is the "case for change" — short and decisive.</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== 0 · EXEC ================================================ --> | |
| <!-- =============================================================== --> | |
| <section id="exec"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 1 · 00</span> | |
| <h2 style="border:0;margin:18px 0 0;">Executive summary</h2> | |
| </div> | |
| <div class="summary-card"> | |
| <h4>In one paragraph</h4> | |
| <p style="margin:0;"> | |
| Migrate the relational core (~32 entities) from DynamoDB to Aurora Serverless v2. Keep the genuinely | |
| key-value/event-stream data (audit events, auth attempts, verification tokens, waitlist) in DynamoDB. Use | |
| Amplify's <code>connect-to-SQL</code> feature for <ins class="authfirst">ALL Aurora models (June 2026 auth-first decision): managed SQL Lambda, resolvers auto-generated, model-level + field-level <code>@auth</code> carrying the access control — including per-column rules on PHI fields. The custom dispatcher + per-request <code>SET LOCAL</code> RLS design is retained as a deferred hardening track (see the amber banner), not shipped at cutover</ins>. <strong>Frontend is not rewritten</strong> — the | |
| <code>client.models.X.get/list/create/update</code> calls keep working through <code>a.combine()</code>. Non-PHI tables | |
| migrate one-at-a-time during Phase 2 with no maintenance window; the PHI core flips in a single coordinated Phase 4 | |
| cutover with a 4-hour rollback window. | |
| </p> | |
| </div> | |
| <div class="stat-row"> | |
| <div class="stat ok"><div class="num">NO</div><div class="lbl">frontend rewrite</div></div> | |
| <div class="stat"><div class="num">5</div><div class="lbl">phases · 0 → 5</div></div> | |
| <div class="stat"><div class="num">~32</div><div class="lbl">relational tables in Aurora</div></div> | |
| <div class="stat"><div class="num">~5</div><div class="lbl">event-stream tables stay in DDB</div></div> | |
| <div class="stat"><div class="num">~15</div><div class="lbl">PHI tables <ins class="authfirst">via Amplify SQL + field-level @auth</ins></div></div> | |
| <div class="stat"><div class="num">~30</div><div class="lbl">non-PHI tables via Amplify SQL</div></div> | |
| </div> | |
| <h3>What this document does</h3> | |
| <ul> | |
| <li>Defines the target architecture and the canonical data model that lands in Aurora.</li> | |
| <li>Provides a phased playbook with explicit phase gates and rollback criteria.</li> | |
| <li>Code-validates every architectural claim against the actual repository (file:line references).</li> | |
| <li>Provides 33 deep DevOps items — each with code validation, deep thinking on failure modes, the concrete CDK/CLI deliverable, and the verification command.</li> | |
| <li>Enumerates the exact frontend changes (mostly mechanical; ~84–120 files touched, ~1500–2500 LOC total).</li> | |
| <li>Maps every HIPAA Safeguard and SOC 2 Trust Service Criterion to the phase that delivers the control.</li> | |
| <li>Lists pre-cutover hard requirements (security gates that must close before Phase 4).</li> | |
| <li>Provides the auditor-ready evidence pack you assemble at Phase 5 close.</li> | |
| </ul> | |
| <div class="callout ok"> | |
| <div class="callout-label">Architecture update · June 2026 — Aurora Data API, no RDS Proxy (Mike's decision)</div> | |
| <p style="margin:0 0 8px;"><strong>The connection layer is simplified: use Aurora Serverless v2 with the RDS Data API and IAM authorization, and drop the RDS Proxy.</strong> This removes the VPC networking the proxy path required. Three reasons:</p> | |
| <ol style="margin:0 0 8px;"> | |
| <li><strong>Simplification.</strong> The Data API is a managed HTTPS endpoint with no persistent connections, so there is no connection-pool problem to solve — an RDS Proxy is not required.</li> | |
| <li><strong>Authentication.</strong> The RDS Proxy path brokers DB credentials; the Data API path is gated by IAM on the dispatcher Lambda's execution role (<code>rds-data:ExecuteStatement</code>). The DB password still lives in a Secrets Manager secret referenced by ARN, but the Lambda never opens a pooled connection or handles a password.</li> | |
| <li><strong>Networking.</strong> The Data API is a regional HTTPS service endpoint, so the dispatcher Lambda <strong>does not need VPC attachment</strong> — no NAT gateway, no Lambda subnet groups, no VPC interface endpoints for the DB path. AWS handles the secure connection natively, cutting the configuration overhead.</li> | |
| </ol> | |
| <p style="margin:0 0 8px;"><ins class="authfirst"><strong>Scope note (June 2026 auth-first):</strong> this Data-API callout applies to the <em>deferred dispatcher track</em>. The auth-first baseline's managed SQL Lambda connects via its own <code>SQL_CONN_STRING</code> connection (Amplify-managed), not the Data API.</ins> <strong>The RLS spine is unaffected.</strong> The per-request <code>BEGIN; SET LOCAL app.trust_company_id = …; <query>; COMMIT;</code> pattern (Section 7) runs through the Data API transaction calls: <code>BeginTransaction → ExecuteStatement(SET LOCAL…) → ExecuteStatement(query) → CommitTransaction</code>. Row-Level Security still enforces tenancy; the only difference is a few HTTPS round-trips per request instead of one pooled transaction — acceptable on Serverless v2. Caveat: the Data API has a result-set size limit, so paginate large reads (heavy reporting reader queries can use a direct connection if needed).</p> | |
| <p style="margin:0;"><strong>How to read this plan:</strong> wherever the doc below says "RDS Proxy + VPC-attached dispatcher," read "Data API + IAM, non-VPC dispatcher." This simplifies Sections 19 (VPC), 22 (RDS Proxy → now optional), and 28 (Lambda VPC attach); the dispatcher logic (6.2) and RLS (7) are unchanged in intent.</p> | |
| </div> | |
| <div class="callout ok"> | |
| <div class="callout-label">Proof-of-concept validation · June 2026 — the hybrid works on real data</div> | |
| <p style="margin:0 0 6px;">A hands-on POC against Aurora validated the Amplify connect-to-SQL hybrid (Section 6) end to end:</p> | |
| <ul style="margin:0;"> | |
| <li><strong>Authorization ports directly.</strong> The existing Amplify <code>@auth</code> rules were copied across to the Aurora-backed models as-is and worked unchanged — same authorization behavior as on DynamoDB.</li> | |
| <li><strong>Filters & pagination at scale.</strong> 1,000 test users were created and the admin panel was pointed at Aurora; filtering and pagination both work correctly.</li> | |
| <li><strong>Migration mechanics hold.</strong> DynamoDB → Aurora migration of <strong>subscriptions</strong> and <strong>payments</strong> moved cleanly, with authorization and relationships intact. <strong>Template</strong> and <strong>TemplateVersion</strong> were harder/slower but migrated correctly. The migration runbook was updated, and re-running the template migration per the new runbook went noticeably smoother.</li> | |
| <li><strong>Significance.</strong> Auth, relations, filtering, pagination, and the migration tooling all hold on real data — strengthening confidence in the non-PHI Amplify-SQL path and the overall Track-B direction.</li> | |
| </ul> | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 1 · WHY AURORA ========================================== --> | |
| <!-- =============================================================== --> | |
| <section id="why-aurora"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 1 · 01</span> | |
| <h2 style="border:0;margin:18px 0 0;">Why Aurora — six reasons (three from architecture, three from lived dev pain)</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>#</th><th>Reason</th><th>Detail</th></tr></thead> | |
| <tbody> | |
| <tr><td>1</td><td><strong>The data is genuinely relational.</strong></td><td>Users have documents, status histories, payments, support threads, medical incidents — every entity has FK-shaped references everywhere. Track A would permanently fight the tool to do joins; Track B uses the right tool for the shape.</td></tr> | |
| <tr><td>2</td><td><strong>Reporting in Track B is just a read replica.</strong></td><td>Same database, same SQL, sub-second freshness. No sync pipeline. No second database to operate. Track A's reporting story is permanently a second database (Redshift via zero-ETL).</td></tr> | |
| <tr><td>3</td><td><strong>Database-enforced referential integrity makes duplication failures structurally impossible.</strong></td><td>A foreign key cannot point at a concept with no single home; a NOT NULL cannot be satisfied by "it depends which copy"; a transaction either writes the canonical row or it doesn't. The relational migration is not just a performance/reporting move — it is the mechanism by which the data architecture becomes <em>correct</em>.</td></tr> | |
| <tr><td>4</td><td><strong>P1 — pagination is broken at the data layer today, and the 5-table single-table Track A can't fully fix it.</strong></td><td>The 1 MB AppSync/DDB response cap truncates lists <em>before</em> filtering — empty pages with nextTokens, hard infinite scroll, unpredictable UX. Track B's <code>WHERE</code>+<code>LIMIT</code>+<code>ORDER BY</code> removes this as a class of problem.</td></tr> | |
| <tr><td>5</td><td><strong>P3 — adding a new access pattern is too expensive in DDB.</strong></td><td>New GSI + backfill + CFN deploy interaction = the team comments code out and deploys in stages. Track B replaces this with <code>CREATE INDEX CONCURRENTLY</code> in a migration file — minutes instead of half-days.</td></tr> | |
| <tr><td>6</td><td><strong>P4 — relational queries the devs need are awkward in DDB.</strong></td><td>"All templates of a user with form count + owner info" is one SQL statement vs. a Lambda merge of multiple queries in DDB. Every report and admin view of this shape gets cheaper.</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 2 · CANONICAL MODEL ===================================== --> | |
| <!-- =============================================================== --> | |
| <section id="canonical"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 1 · 02</span> | |
| <h2 style="border:0;margin:18px 0 0;">The canonical data model — <ins class="prdval">~43 verified</ins> entities, one home per concept</h2> | |
| </div> | |
| <p> | |
| The current state is 41 <code>a.model()</code> declarations with duplications, dead code, embedded arrays and overlapping | |
| concepts. The target is <ins class="prdval"><strong>~43 (verified current model count)</strong></ins> — 32 relational (in Aurora) + <ins class="prdval">11</ins> event streams (stay in | |
| DynamoDB). <ins class="prdval"><strong>PRD validation:</strong> the relational count (32) is correct, but the event streams are <strong>11, not 5</strong> (add DocumentUpdateLog, DocumentEventLog, IncapacitatedStatusLog, WaitlistLog, CheckInHistory, EscalationEvent). And the PHI split is by <em>field content</em>, not category: ~7–10 PHI / 33–36 non-PHI — <code>User</code>, <code>CareDocumentNew</code>, <code>Document</code>, <code>UserPayment</code>, <code>UserSubscription</code>, <code>VerificationTokens</code> read non-PHI by name but carry PHI/credentials in JSON blobs, so they must be pinned to the protected path (see banner).</ins> Built on principles: one home per concept, tenant id on every member-scoped row, referential integrity | |
| expressed via FKs, state changes recorded in append-only history tables, orthogonal status fields, soft delete, | |
| append-only unified audit, no plaintext PII in keys or logs, honestly-named schema. | |
| </p> | |
| <h3>The 42 → 37 collapse map (selected)</h3> | |
| <table> | |
| <thead><tr><th>Current</th><th>Canonical</th><th>Why</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>User</code> (60+ fields god-model)</td><td><code>user</code> (lean) + <code>user_interview</code> + <code>user_onboarding</code> + <code>user_quarterly_review</code> + <code>user_people_library</code></td><td>Decompose; identity stays lean; everything else moves to typed child entities</td></tr> | |
| <tr><td><code>UserInterviewProgressNew</code> + <code>InterviewProgressV2</code> + V3 wrappers + <code>UserExecutionProgress</code></td><td>1 × <code>user_interview</code></td><td>Four representations → one. Two dead, two duplicates.</td></tr> | |
| <tr><td>scalar+array <code>assignedWelonTrustId(s)</code> on User, Document, ShippingLabel, CareDocumentNew</td><td>1 × <code>welon_assignment</code> <span class="sev new">NEW</span></td><td>Eight stores → one. Real, auditable, revocable.</td></tr> | |
| <tr><td>5 medical homes (interview blob, CareDocumentNew Medical, MedicalReport, MedicalIncident, IncapacitatedStatusLog)</td><td><code>medical_record</code> <span class="sev new">NEW canonical</span> + <code>medical_incident</code> + <code>medical_review</code> + <code>medical_note</code> + <code>incapacitation_event</code> + <code>medical_review_caseload</code> <span class="sev new">NEW</span></td><td>One home for medical wishes; caseload fixes HIPAA gap</td></tr> | |
| <tr><td><code>Logs</code> + <code>DocumentUpdateLog</code> + <code>DocumentEventLog</code></td><td>1 × <code>audit_event</code></td><td>Three logs → one unified audit trail (stays in DDB; high volume)</td></tr> | |
| <tr><td><code>LoginAttempt</code> + <code>mfaAttempt</code> + <code>LoginHistory</code></td><td>1 × <code>auth_attempt</code></td><td>Email replaced by <code>user_id</code> FK + email_hash (stays in DDB)</td></tr> | |
| <tr><td>7 status fields on User</td><td><code>user.lifecycle_state</code> + <code>user.billing_state</code> + status-history tables</td><td>Orthogonal dimensions, not overlapping enums</td></tr> | |
| <tr><td><code>SequenceNumberHelper</code> (race-prone single counter)</td><td>Postgres <code>SEQUENCE</code></td><td>Transaction-safe, gap-tolerant</td></tr> | |
| <tr><td>Embedded <code>CareDocumentNew.data[]</code> array of 16 sections</td><td><code>care_document</code> + <code>care_document_section</code></td><td>First-class rows with own access control + versioning</td></tr> | |
| <tr><td>Embedded <code>SupportRequest.messages[]</code></td><td><code>support_request</code> + <code>support_message</code> + <code>support_attachment</code></td><td>Same — embedded arrays become child tables</td></tr> | |
| <tr><td><code>TestModel</code>, dead interview array, <code>UserExecutionProgress</code></td><td>DELETED</td><td>Dead code</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>What stays in DynamoDB (intentional)</h3> | |
| <p> | |
| <strong>Four</strong> entity groups are genuinely key-value / append-only / high-volume and stay in DDB: | |
| <code>audit_event</code> (unified audit log; millions/month), | |
| <code>auth_attempt</code> + <code>auth_lockout</code> (login attempts; millions/month, email-hash-centric), | |
| <code>verification_token</code> (TTL-managed), | |
| <code>waitlist_entry</code> + <code>waitlist_event</code> (transient queue). | |
| The <code>outbox</code> is <strong>not</strong> in this list — it lives in <em>Aurora</em> (the transactional-outbox | |
| pattern requires the side-effect write to share the same Postgres transaction as the data write). It was previously | |
| mislabeled here as a fifth DDB stream. | |
| </p> | |
| <div class="callout"> | |
| <div class="callout-label">These four are not "stays-in-DDB-unchanged" — they are a real schema refactor</div> | |
| On main today the audit/event data is spread across <strong>nine</strong> tables: <code>Logs</code>, | |
| <code>DocumentEventLog</code>, <code>DocumentUpdateLog</code>, <code>LoginAttempt</code>, <code>LoginHistory</code>, | |
| <code>mfaAttempt</code>, <code>VerificationTokens</code>, <code>WaitlistEntry</code>, <code>WaitlistLog</code>. | |
| Collapsing them into <code>audit_event</code> + <code>auth_attempt</code> + <code>verification_token</code> + | |
| <code>waitlist</code> is a genuine schema-and-write-path refactor in Phase 2 — not a no-op. Every Lambda that writes to | |
| the old tables must be repointed. Tracked as a Phase 2 deliverable. | |
| </div> | |
| <h3>Platform / tenancy tables <span class="sev verified">added · Final Development Plan</span></h3> | |
| <p>The three-entity tenancy model (<a href="#tenancy-brands">§3b</a>) adds a Platform table group on top of the canonical model:</p> | |
| <table> | |
| <thead><tr><th>Table</th><th>Purpose</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>trust_company</code></td><td>The security tenant. Every member-scoped row FKs to it.</td></tr> | |
| <tr><td><code>partner</code></td><td>The financial firm / distribution channel that brought the member (attribution).</td></tr> | |
| <tr><td><code>brand</code></td><td>What the user sees; resolved per-request from the Host header. Domains, palette, logos, sender identity.</td></tr> | |
| <tr><td><code>member_trust_assignment</code></td><td>The deterministic member→trust assignment, with reason codes (§3b.3).</td></tr> | |
| <tr><td><code>trust_reassignment_suggestion</code></td><td>Created on address change; never auto-applied — admin reviews.</td></tr> | |
| <tr><td><code>zip_code_centroid</code></td><td>~33k-row read-only US Census ZCTA seed for the haversine proximity step.</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 3 · TOPOLOGY ============================================ --> | |
| <!-- =============================================================== --> | |
| <section id="topology"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 1 · 03</span> | |
| <h2 style="border:0;margin:18px 0 0;">Target topology</h2> | |
| </div> | |
| <pre class="diagram"><code>Client → CloudFront → WAF v2 | |
| ↓ | |
| AppSync GraphQL (introspection OFF, depth=10) | |
| ↓ | |
| ┌──────┴───────────────────────────────────────┐ | |
| │ a.combine([ddbSchema, sqlSchema, phiSchema]) │ | |
| └──────┬─────────────────┬──────────────────┬──┘ | |
| │ │ │ | |
| │ ~30 non-PHI │ ~15 PHI models │ ~5 event streams | |
| │ models │ resolved via │ (audit, auth_attempt, | |
| │ → Amplify SQL │ custom dispatcher│ verification, waitlist) | |
| │ (managed) │ Lambda │ → DynamoDB | |
| ▼ ▼ ▼ | |
| ┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐ | |
| │ Amplify SQL Lambda │ │ Custom Dispatcher │ │ DynamoDB Tables │ | |
| │ (AWS-managed) │ │ Lambda (yours) │ │ (Amplify default) │ | |
| │ │ │ │ │ │ | |
| │ chil_sqlconnect │ │ Data API + IAM │ │ │ | |
| │ scope = non-PHI │ │ (no VPC, no pool) │ │ │ | |
| │ tables only │ │ │ │ │ | |
| │ │ │ BEGIN; │ │ │ | |
| │ │ │ SET LOCAL │ │ │ | |
| │ │ │ app.trust_co_id; │ │ │ | |
| │ │ │ SET LOCAL │ │ │ | |
| │ │ │ app.cognito_id; │ │ │ | |
| │ │ │ SET LOCAL │ │ │ | |
| │ │ │ app.role; │ │ │ | |
| │ │ │ SELECT col_proj │ │ │ | |
| │ │ │ FROM "user" ... │ │ │ | |
| │ │ │ COMMIT; │ │ │ | |
| │ │ │ │ │ │ | |
| │ │ │ chil_app role │ │ │ | |
| │ │ │ (no BYPASSRLS) │ │ │ | |
| └─────────┬──────────┘ └────────┬───────────┘ └─────────┬──────────┘ | |
| │ │ │ | |
| └────────────────────────┴──────────────────────────┘ | |
| │ | |
| ▼ | |
| ┌────────────────────────────────────────────────────┐ | |
| │ Aurora Serverless v2 PostgreSQL │ | |
| │ · Writer instance (1–8 ACU) │ | |
| │ · Reader instance (0.5–4 ACU) — for reporting │ | |
| │ · pgaudit + pg_stat_statements + pgcrypto │ | |
| │ · RLS policies on PHI tables │ | |
| │ · CMK encryption at rest │ | |
| │ · TLS in transit (require_ssl=true) │ | |
| └────────────────────────────────────────────────────┘ | |
| ALSO: 76 existing Lambdas (Cognito triggers, schedulers, | |
| webhooks, document generation, etc.) — STAY NON-VPC. | |
| They keep calling AppSync over HTTPS exactly as today. | |
| Under the June 2026 Data API decision the dispatcher is | |
| ALSO non-VPC — it reaches Aurora via the Data API over HTTPS. | |
| No Lambda needs VPC attachment for database access.</code></pre> | |
| <h3>Why a single dispatcher Lambda (not 76)</h3> | |
| <div class="callout ok"><div class="callout-label">Under the Data API the VPC-pressure reasons below are moot</div> | |
| The dispatcher is <strong>non-VPC</strong> (it calls the Data API over HTTPS), so the ENI-exhaustion and NAT-cost rows no longer apply — kept for context, struck through. The live reasons for one dispatcher are the shared PC pool and the single projection/audit layer.</div> | |
| <table> | |
| <thead><tr><th>Reason</th><th>Detail</th></tr></thead> | |
| <tbody> | |
| <tr><td>ENI exhaustion <em>(moot · Data API)</em></td><td><s>76 × 50 concurrent = 3,800 ENIs vs a 250/region soft-limit.</s> Non-VPC dispatcher allocates no ENIs for DB access.</td></tr> | |
| <tr><td>Cold start delta</td><td>VPC Lambda cold starts add 1–3s vs ~200ms non-VPC. Don't pay that on every function.</td></tr> | |
| <tr><td>NAT cost <em>(moot · Data API)</em></td><td><s>Every VPC Lambda needs NAT or a VPC endpoint.</s> Non-VPC dispatcher reaches the Data API endpoint directly — no NAT for the DB path.</td></tr> | |
| <tr><td>One PC pool</td><td>5–20 provisioned concurrency on one dispatcher amortizes across all model ops. ~185 access patterns (37 × 5 ops) each needing PC = $$$$.</td></tr> | |
| <tr><td>One projection layer</td><td>The ~200 field-level auth rules live in one TS file with one test suite.</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 3b · IDENTITY, TENANCY & BRANDS ========================= --> | |
| <!-- =============================================================== --> | |
| <section id="tenancy-brands"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 1 · 03b</span> | |
| <h2 style="border:0;margin:18px 0 0;">Identity, tenancy & brands</h2> | |
| </div> | |
| <div class="callout ok"><div class="callout-label">Harvested from the Final Development Plan (2026-06-09)</div> | |
| This makes the business model first-class. Where the rest of this plan said "tenant = trust company," the model is actually <strong>three entities</strong>: who serves the member, who brought them, and what they see. A fully rebranded second offering becomes a row + a domain — no code fork, no fake tenant.</div> | |
| <h3>3b.1 — Three entities</h3> | |
| <table> | |
| <thead><tr><th>Entity</th><th>Answers</th><th>Key fields</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>trust_company</code></td><td>Who serves the member. <strong>The security tenant.</strong></td><td>id (uuid), slug, legal_name, display_name, phone, shipping_address, service_states[], hq_zip/lat/lon, priority, is_default (exactly one), status, agreement_template_id</td></tr> | |
| <tr><td><code>partner</code></td><td>Who brought the member (financial firm / distribution channel)</td><td>id, name, status, default_brand_id, attribution config, report scope</td></tr> | |
| <tr><td><code>brand</code></td><td>What the user sees</td><td>id, name, domains[], logo asset keys, palette (CSS token JSON), font, email sender identity, document header/footer strings, support contacts</td></tr> | |
| </tbody> | |
| </table> | |
| <ul> | |
| <li><code>app_user.trust_company_id</code> (required) is the RLS key. <code>app_user.partner_id</code> (nullable) is attribution.</li> | |
| <li><strong>Brand is resolved per request from the Host header</strong>, never stored on the user.</li> | |
| <li>Welon Trust is <code>trust_company</code> #1 and the default. A white-label financial firm is a <strong>partner + brand</strong>; their members are still served by whichever trust company the assignment algorithm picks. The rebranded second offering is a <strong>brand + domain</strong> only.</li> | |
| </ul> | |
| <h3>3b.2 — The one-identifier rule</h3> | |
| <div class="callout crit"><div class="callout-label">Non-negotiable</div> | |
| The tenant identifier is the opaque <code>trust_company.id</code> UUID in <em>every</em> system: JWT claim, RLS session variable, S3 key prefix, KMS encryption context, audit rows, CloudWatch dimensions. Names, slugs, brands, and Cognito group strings never appear in security-bearing metadata. KMS encryption context is immutable per object — a name baked in today is a re-encryption project after the first rebrand.</div> | |
| <div class="callout rlsfix"><div class="callout-label">Identity glossary · June 2026 review fix (Q4)</div> | |
| Three identifiers exist per user and must never be conflated: <code>sub</code> — the Cognito user-pool UUID (used only for pgaudit correlation, GUC <code>app.cognito_id</code>); <code>externalId</code> — an <strong>app-generated UUID</strong> (created by <code>randomUUID()</code> in postSignUpTrigger, stored as Cognito attribute <code>custom:externalId</code>, injected into the access token by preTokenGeneration, compared by RLS owner policies via GUC <code>app.external_id</code> against the <code>user.external_id</code> column — the same value every Amplify <code>@auth</code> rule keys on via <code>identityClaim('externalId')</code>); <code>user.id</code> — the database primary key. The review found the dispatcher set <code>app.cognito_id</code> while every owner policy read <code>app.external_id</code> — see §6.2 for the fix.</div> | |
| <h3>3b.3 — Member-to-trust assignment</h3> | |
| <p>Assignment is deterministic and produces an auditable row, not a guess:</p> | |
| <ol> | |
| <li><strong>State match</strong> on <code>trust_company.service_states</code>.</li> | |
| <li><strong>Haversine zip proximity</strong> to trust HQ (uses <code>zip_code_centroid</code>, a ~33k-row read-only US Census ZCTA seed).</li> | |
| <li><strong>Tie-break</strong>: <code>priority</code>, then <code>id</code>.</li> | |
| <li><strong>Default fallback</strong> to the <code>is_default</code> trust company.</li> | |
| </ol> | |
| <ul> | |
| <li>Assignments are rows in <code>member_trust_assignment</code> with reason codes.</li> | |
| <li><strong>Address changes never auto-reassign</strong> — they create a <code>trust_reassignment_suggestion</code> for admin review. Reassignment is an explicit, audited admin action (this is why the tenant claim is mutable — see §26).</li> | |
| </ul> | |
| <h3>3b.4 — Cognito groups carry roles only</h3> | |
| <ul> | |
| <li>One user pool for all tenants, partners, and brands. Groups carry <strong>roles only</strong>: <code>ADMINS</code>, <code>TRUST_OPERATORS</code> (one group across all tenants), <code>MEDICAL_REVIEW</code>, <code>CALL_CENTER</code>, <code>EXECUTIVE</code>. The legacy tenant-named group is renamed in Phase 3.</li> | |
| <li>The tenant comes from the <code>custom:trustCompanyId</code> claim, not from a group. See §26 for the mint flow.</li> | |
| </ul> | |
| <h3>3b.5 — Reassignment workflow</h3> | |
| <p>Reassignment (a member moves to a different trust company) is an explicit, audited admin action — never automatic:</p> | |
| <ol> | |
| <li><strong>Trigger.</strong> An address change (or admin request) writes a <code>trust_reassignment_suggestion</code> row (suggested trust, reason code, current trust, requested-by). The member is <em>not</em> moved.</li> | |
| <li><strong>Review.</strong> An admin works the suggestion queue and approves or rejects via an <code>approveTrustReassignment</code> mutation (dispatcher-backed, admin role only).</li> | |
| <li><strong>Apply (one transaction).</strong> On approve: update <code>app_user.trust_company_id</code>, re-key the member's tenant-scoped S3 prefixes as required, write the <code>audit_event</code>, and mark the suggestion applied.</li> | |
| <li><strong>Token refresh.</strong> Re-write the Cognito <code>custom:trustCompanyId</code> attribute (admin-mutable for exactly this) so the next minted token carries the new tenant; existing sessions pick it up on refresh.</li> | |
| </ol> | |
| <p class="small">This is why the claim is mutable (§26): the <code>trust_company</code> row is the source of truth, the attribute is a mint-time cache, and the HIPAA trail is the append-only <code>audit_event</code> history of the move.</p> | |
| <h3>3b.6 — Cross-tenant operator access <span class="sev rlsfix">JUNE 2026 REVIEW FIX · F8</span></h3> | |
| <div class="callout rlsfix"><div class="callout-label">New design — closes the “platform staff have no API path” gap</div> | |
| <p style="margin:0 0 6px;"><code>customAuth</code> rejects any request without a <code>trustCompanyId</code> claim, so platform staff responding to an incident, or a support agent serving several trust companies, previously had <em>no</em> path through the application API — only DB-level <code>chil_admin</code> break-glass. The review (F8) is right that this was unresolved. Design adopted:</p> | |
| <ul style="margin:0;"> | |
| <li><strong>The one-tenant-per-request invariant stays.</strong> No multi-tenant tokens, no “see everything” API role — that would hollow out the RLS fence.</li> | |
| <li><strong>Time-boxed operator grants.</strong> New <code>operator_tenant_grant</code> table: an Admin approves operator X → tenant Y for a bounded window (default 4 h; reason required; auto-expiring; append-only audit). The operator’s token is re-minted via the existing preTokenGeneration path carrying that <em>single</em> <code>trustCompanyId</code>. To the dispatcher and RLS the request is a normal tenant request; every action lands in <code>audit_event</code> under the operator’s identity. A multi-tenant case = sequential grants, each audited.</li> | |
| <li><strong>Cross-tenant reads stay aggregate-only</strong> via <code>chil_platform_reporter</code>; <code>chil_admin</code> remains DB-only break-glass with its compensating controls (§24).</li> | |
| <li>No new Postgres role; no dispatcher routing change — the grant operates at the token-mint layer. Deliverables: the table, an <code>approveOperatorGrant</code> mutation, the expiry job, and grant/expire audit rows (Phase 3, alongside the reassignment workflow).</li> | |
| </ul> | |
| <p style="margin:8px 0 0;"><ins class="review2"><strong>Interim position (round-2 review — until operator grants ship):</strong> cross-tenant human access for incident response is <code>chil_admin</code> break-glass <strong>only</strong> (DB-level, with the §24 compensating controls — temp creds, Jira gate, real-time Slack alert, Tier-2 fields stay ciphertext). This is the accepted, documented interim state — not an implied gap. Aggregate cross-tenant <em>reporting</em> has its own interim position in §6.1b.</ins></p></div> | |
| </section> | |
| <hr> | |
| <div class="part-banner"> | |
| <div class="part-num">Part 2 of 8</div> | |
| <h2 style="border:0;color:var(--ink);">Code-validated baseline — what's <em>actually</em> on main</h2> | |
| <p>Every number in this document was re-verified by direct read of the repository at commit <code>fb4cf8b3e</code> (re-checked in the June 2026 deep review). Prior reviews carried estimates that drifted; this is the ground truth, plus six discoveries beyond all prior reviews.</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== 4 · CODE BASELINE ======================================= --> | |
| <!-- =============================================================== --> | |
| <section id="code-baseline"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 2 · 04</span> | |
| <h2 style="border:0;margin:18px 0 0;">What's actually on main — every metric verified</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>Metric</th><th>Prior reviews said</th><th>Verified count</th><th>How verified</th></tr></thead> | |
| <tbody> | |
| <tr><td>Lambda functions in <code>defineBackend</code></td><td>43 / 65</td><td><strong>76</strong></td><td><code>find amplify/functions -name resource.ts | wc -l</code> <ins class="teamfix">Team review: Mike reports 75 (reconcile); original spec said 44 → migration scope materially larger than defined.</ins> <ins class="prdval">PRD validation: <strong>77</strong> active <code>defineFunction</code> (79 raw incl. 2 commented stubs); 44 was the pre-growth baseline, 75 = Mike minus 2 dead stubs. Best number 77. Not a sign-off blocker.</ins></td></tr> | |
| <tr><td><code>a.model()</code> declarations</td><td>42</td><td><strong>41</strong></td><td><code>grep -rE "\.model\(" amplify/data/schemas/</code></td></tr> | |
| <tr><td>Secondary indexes (<code>index()</code> calls)</td><td>72</td><td><strong>35</strong></td><td><code>grep -rE "^\s+index\(" amplify/data/schemas/</code></td></tr> | |
| <tr><td><code>disableOperations('subscriptions')</code> calls</td><td>—</td><td><strong>40</strong></td><td>Phase 0 cleanup advanced (was 22, now 40)</td></tr> | |
| <tr><td>User-model field declarations (incl. nested)</td><td>60+</td><td><strong>214 lines</strong></td><td><code>grep -cE "^\s+[a-z][a-zA-Z]+:" user-models.ts</code></td></tr> | |
| <tr><td>S3 path policies</td><td>—</td><td><strong>6</strong></td><td>evidence, documents, care-documents, personal-id, support-requests, template-sections</td></tr> | |
| <tr><td>Cognito triggers</td><td>5</td><td><strong>5</strong></td><td>postSignUp, preTokenGeneration, preSignIn, customMessageTrigger, customSmsSender</td></tr> | |
| <tr><td>Function URL Lambdas</td><td>—</td><td><strong>1</strong></td><td><code>processWebhookV2</code> · <code>FunctionUrlAuthType.NONE</code></td></tr> | |
| <tr><td>Amplify <code>secret()</code> consumers</td><td>—</td><td><strong>20+ env entries</strong></td><td>STRIPE, MAILCHIMP, UPS, GOOGLE_API, FAX_*</td></tr> | |
| <tr><td>Lambdas with <code>tracesSampleRate: 1.0</code></td><td>—</td><td><strong>postSignUpTrigger</strong> hardcoded</td><td><code>grep -rn "tracesSampleRate" amplify/functions/</code></td></tr> | |
| <tr><td>Lambdas with top-level <code>await getAmplifyDataClientConfig</code></td><td>—</td><td><strong>50</strong></td><td><code>grep -rln "^const { resourceConfig" amplify/functions/</code></td></tr> | |
| <tr><td>Lambdas at 15-min timeout (max)</td><td>—</td><td><strong>5</strong></td><td>sendMedicalIncident, sendFax, sendEmailBatchesWithInterval, sendRoleBasedNotifications, check-inactive-members</td></tr> | |
| <tr><td>WAF in <code>backend.ts</code></td><td>—</td><td><strong>None</strong></td><td><code>grep -nE "(waf|WAF|WebACL)" backend.ts</code> returns nothing</td></tr> | |
| <tr><td>AppSync depth limit / introspection</td><td>—</td><td><strong>Not configured</strong></td><td>No <code>maxDepth</code> or <code>introspection</code> in data/resource.ts</td></tr> | |
| <tr><td>DynamoDB Streams in backend.ts</td><td>—</td><td><strong>Not enabled</strong></td><td>No <code>StreamSpecification</code> or <code>streamSpecification</code></td></tr> | |
| <tr><td>CI/CD security gates (SAST/SCA/secret scan)</td><td>—</td><td><strong>None</strong></td><td><code>amplify.yml</code> has none</td></tr> | |
| <tr><td>Deletion protection on DDB</td><td>OFF</td><td><strong>OFF</strong></td><td><code>backend.ts:180</code> <code>deletionProtectionEnabled = false</code></td></tr> | |
| <tr><td>PITR on DDB</td><td>ON</td><td><strong>ON</strong></td><td><code>backend.ts:181</code> <code>pointInTimeRecoveryEnabled = true</code></td></tr> | |
| <tr><td>Node.js runtime</td><td>—</td><td><strong>Node 22</strong></td><td>All function <code>resource.ts</code> set <code>runtime: 22</code></td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout high"> | |
| <div class="callout-label">Key corrections to prior reviews</div> | |
| <ol style="margin:6px 0 0;"> | |
| <li>Lambda count is <strong>76, not 65 or 43</strong>. (The old ENI math 76 × 50 = 3,800 was the RDS-Proxy concern; moot under the Data API — the dispatcher is non-VPC.)</li> | |
| <li>35 secondary indexes — Phase 0 cleanup already shipped ~50% reduction. The remaining 35 will become PG indexes (cheap).</li> | |
| <li>50 of <ins class="prdval">77</ins> Lambdas pay top-level-await cold-start tax — that's the cold-start surface area, not 3.</li> | |
| <li>No WAF, no AppSync hardening, no DDB Streams, no CI security gates today. <strong>All Phase 1 deliverables.</strong></li> | |
| </ol> | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 5 · DISCOVERIES ========================================= --> | |
| <!-- =============================================================== --> | |
| <section id="discoveries"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 2 · 05</span> | |
| <h2 style="border:0;margin:18px 0 0;">Findings from code validation</h2> | |
| </div> | |
| <p>Surface-area issues discovered while validating the migration plan against the actual repository — each one a concrete fix item with owner.</p> | |
| <div class="review-block"> | |
| <h4>D1 · <code>amplify.yml</code> has UTF-8 curly-quote bug in backend env-var lines <span class="sev crit">CRITICAL</span> <span class="sev new">NEW</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <p><code>amplify.yml:9-14</code> uses <code>“</code> and <code>”</code> (Unicode U+201C/U+201D, "smart quotes") around <code>echo</code> arguments. Example:</p> | |
| <pre class="yaml"><code>- echo “CFLegacy_Dev_UserPoolID=$CFLegacy_Dev_UserPoolID” >> .env</code></pre> | |
| <div class="label crit">DEEP THINKING</div> | |
| <p>POSIX shell does not treat smart quotes as quote characters — it treats them as literal characters. The shell <em>does</em> still expand <code>$CFLegacy_Dev_UserPoolID</code> (variable expansion happens before quote interpretation, and smart quotes aren't quote characters anyway). So the line that's appended to <code>.env</code> looks like:</p> | |
| <pre class="yaml"><code>“CFLegacy_Dev_UserPoolID=actual-value-here”</code></pre> | |
| <p>That's a literal <code>“</code> at the start of the line. dotenv parsers usually skip lines that don't start with <code>A-Z_</code>, but behavior is parser-dependent. <strong>Either the env var is being silently ignored, OR it's being read with literal smart-quote characters in the value</strong>. Either is broken. The reason the system "works" today is likely that AWS Amplify Hosting injects these variables a second way (via Lambda env vars set by <code>backend.ts:298+</code>), so the .env file path is unused for runtime config.</p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <p>Replace smart quotes with straight quotes throughout <code>amplify.yml</code>:</p> | |
| <pre class="sh"><code># From the repo root: | |
| sed -i '' 's/“/"/g; s/”/"/g' amplify.yml | |
| # Verify | |
| grep -nE "“|”" amplify.yml # must return nothing</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>After the fix: trigger a build; the <code>.env</code> generated in the build step must contain valid <code>KEY=value</code> lines without smart quotes. <code>cat .env</code> in the build log should show e.g. <code>CFLegacy_Dev_UserPoolID=us-east-1_AbCdE</code>.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>D2 · <code>Document.content</code> is stored as <code>a.string().required()</code> — full document body in DDB attribute <span class="sev high">HIGH</span> <span class="sev new">NEW</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <p><code>amplify/data/schemas/document-models.ts:63</code>: <code>content: a.string().required(),</code> on the <code>Document</code> model. Not nullable, not S3-referenced — the document content lives inside the DDB item.</p> | |
| <div class="label crit">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>DynamoDB single-item size limit: <strong>400 KB</strong>. A generated trust, will, or POA document body can easily exceed 400 KB if it includes execution instructions, multi-state legal language, attached schedules.</li> | |
| <li>If a write exceeds 400 KB today, DynamoDB returns <code>ValidationException: Item size has exceeded the maximum allowed size</code>. The write fails silently from a user-flow perspective; Sentry may catch it.</li> | |
| <li>For the Aurora migration: <code>content</code> becomes a Postgres <code>text</code> column. Postgres TOAST automatically compresses and offloads large values; no 400 KB cap. <strong>The migration silently fixes this problem.</strong></li> | |
| <li>But during the migration's dual-write phase, if DDB rejects a write that PG accepts, the two stores diverge. Detection: monitor <code>ValidationException</code> rates in CloudWatch.</li> | |
| <li>Pre-migration mitigation: scan existing <code>Document</code> items for size; if any are > 350 KB, audit them and either move to S3 immediately or accept they may already be failing on update.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="sh"><code># Scan Document table for size distribution | |
| aws dynamodb scan \ | |
| --table-name Document-<env> \ | |
| --projection-expression "id, content" \ | |
| --query "Items[?size(content.S) > \`300000\`]" \ | |
| --output json | jq 'length' | |
| # If any > 300 KB, list them for triage: | |
| aws dynamodb scan ... | jq '.Items[] | select(.content.S | length > 300000) | .id.S'</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>The scan returns the count of Documents already pushing the 400 KB ceiling. If > 0 — flag for content-to-S3 refactor before cutover; otherwise the migration absorbs the risk for us.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>D3 · 50 Lambdas pay top-level-await cold-start tax <span class="sev high">HIGH</span> <span class="sev new">NEW</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <pre class="sh"><code>$ grep -rln "^const { resourceConfig, libraryOptions } =" amplify/functions/ | wc -l | |
| 50</code></pre> | |
| <p>Pattern in those 50 handlers:</p> | |
| <pre class="ts"><code>const { resourceConfig, libraryOptions } = | |
| await getAmplifyDataClientConfig(env); | |
| Amplify.configure(resourceConfig, libraryOptions); | |
| const client = generateClient<Schema>();</code></pre> | |
| <div class="label crit">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>Top-level <code>await</code> runs once per Lambda init (cold start). <code>getAmplifyDataClientConfig</code> reads from SSM/env and configures Amplify — measured at <strong>~200–500 ms per cold start</strong>.</li> | |
| <li>50 Lambdas × 200 ms = aggregate ~10 seconds of cold-start latency across the system on a cold-start storm. Per-invocation it's invisible to one user but the aggregate user experience under burst load is real.</li> | |
| <li>Critical paths that pay this: preTokenGeneration (Cognito 5 s budget), preSignIn, postSignUpTrigger.</li> | |
| <li>Mitigation options: (a) bake the resourceConfig into env vars (frees the await), (b) move the await into a lazy initializer that runs on first request not on Lambda init, (c) Provisioned Concurrency for the worst paths (already in PC budget).</li> | |
| <li>For preTokenGeneration specifically: <strong>this trigger's 5 s budget includes the AppSync call to <code>UserOnboarding.get</code></strong> which goes through this same Amplify config. Cold start + AppSync round-trip can plausibly hit 3–4 s on a cold path, leaving ~1 s of headroom. Tight.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <ol style="margin:6px 0 0;"> | |
| <li>For the three PC Lambdas (dispatcher, preTokenGen, Stripe webhook): PC already absorbs cold-start.</li> | |
| <li>For the other 47: bake resourceConfig into env vars (one-time refactor; ~2 day effort).</li> | |
| <li>Add a <strong>CloudWatch Lambda Insights</strong> dashboard tracking <code>init_duration_ms</code> per function. Alarm at > 2 s P95 init.</li> | |
| </ol> | |
| <div class="label ok">VERIFICATION</div> | |
| <p><code>aws logs insights start-query</code> with <code>fields @timestamp, @initDuration | sort @initDuration desc | limit 100</code>. The top of the list shows which functions pay the worst cold start; tune those.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>D4 · <code>postSignUpTrigger</code> has 5 s timeout (Cognito hard limit) <span class="sev high">HIGH</span> <span class="sev new">NEW</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <p><code>amplify/functions/postSignUpTrigger/resource.ts</code> sets <code>timeoutSeconds: 5</code> (Cognito's hard limit for PostConfirmation triggers).</p> | |
| <div class="label crit">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>5 s is hard, not configurable. Cognito kills the trigger at 5 s.</li> | |
| <li>Current handler does: read SequenceNumberHelper, write User row, write UserOnboarding row, set custom Cognito attrs, call processSignUp (fire-and-forget for Mailchimp). That's already 4 round-trips to AppSync + 1 to Cognito + 1 fire-and-forget Lambda invoke.</li> | |
| <li>Adding ANY work here is dangerous — including the planned "set <code>custom:trustCompanyId</code>" attribute write (added Cognito API call).</li> | |
| <li>Top-level await tax (D3) eats ~200–500 ms of this. Cold start eats another 500 ms.</li> | |
| <li>If migration adds the <code>trustCompanyId</code> setting in postSignUp and we hit cold start during a signup burst, the trigger times out — <strong>signup fails</strong>.</li> | |
| <li>Mitigation: PC on postSignUpTrigger (small added PC cost). Or move the trustCompanyId setting to a side-flow that happens before/after PostConfirmation rather than inside it.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <ol style="margin:6px 0 0;"> | |
| <li>Add postSignUpTrigger to the PC list. Phase-1 PC count is 4 (dispatcher, preTokenGen, Stripe webhook, postSignUpTrigger); a 5th (customAuth) is added at Phase 3. Final post-Phase-3 PC count is 5 (dispatcher, preTokenGen, Stripe webhook, customAuth, postSignUpTrigger).</li> | |
| <li>Time-budget the current trigger before adding work — measure P50/P95/P99 init+exec time. Alarm at > 4 s.</li> | |
| <li>Consider a CloudWatch metric: <code>PostSignupTimeoutRate</code>. Any non-zero value is a real user being denied signup.</li> | |
| </ol> | |
| </div> | |
| <div class="review-block"> | |
| <h4>D5 · No DynamoDB Streams enabled today — must be turned on before cutover <span class="sev crit">CRITICAL</span> <span class="sev new">NEW</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <pre class="sh"><code>$ grep -nE "(Streams|streamSpecification|StreamSpecification)" amplify/backend.ts | |
| # Returns nothing</code></pre> | |
| <div class="label crit">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>The Phase 4 cutover plan relies on DDB Streams for the CDC catch-up after the S3 export. If Streams aren't enabled, the cutover plan literally cannot work as written.</li> | |
| <li>Enabling Streams on a populated table is non-destructive but takes effect for <em>new</em> writes only — historical changes aren't captured. So we have to: (a) enable Streams, (b) wait for confidence that they're flowing, (c) export the table state at time T, (d) replay Streams from T forward.</li> | |
| <li>Streams retain 24 h. If cutover catch-up takes > 24 h, we have data loss. The plan's "T+1:40 catch-up" assumes well under 24 h, which is realistic, but Mike should know the constraint.</li> | |
| <li>Cost: Streams pricing is per read request unit; low at our volume.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <ol style="margin:6px 0 0;"> | |
| <li>Enable Streams on all 41 DDB tables in T−1 week of Phase 4 cutover. CDK in <code>backend.ts</code>: | |
| <pre class="ts"><code>for (const DDBtable of Object.values(amplifyDynamoDbTables)) { | |
| DDBtable.deletionProtectionEnabled = false; // existing; flip to true for cutover wk | |
| DDBtable.pointInTimeRecoveryEnabled = true; // existing | |
| DDBtable.streamSpecification = { // NEW | |
| streamViewType: 'NEW_AND_OLD_IMAGES', | |
| }; | |
| }</code></pre></li> | |
| <li>Verify Streams ARNs are listed: <code>aws dynamodb describe-table --table-name X-<env> --query "Table.LatestStreamArn"</code></li> | |
| <li>Set up the CDC consumer Lambda (idle until cutover; just a placeholder).</li> | |
| </ol> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>For each DDB table, <code>aws dynamodb describe-table</code> shows <code>StreamSpecification.StreamEnabled: true</code> and <code>LatestStreamArn</code> populated.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>D6 · 5 Lambdas at 15-min (AWS max) timeout — pre-PHI-cutover infra check <span class="sev med">MEDIUM</span> <span class="sev new">NEW</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <p><code>timeoutSeconds: 900</code> on: <code>sendMedicalIncident</code>, <code>helpers/sendFax</code>, <code>helpers/sendEmailBatchesWithInterval</code>, <code>sendRoleBasedNotifications</code>, <code>scheduler-functions/check-inactive-members</code>. <ins class="datarch">📄 Data-arch review (AWS console): the code sets 900s on only these 5, but the <strong>deployed</strong> fleet (~90+ functions incl. Amplify-managed) sits at the 15-min max. Phase-1 cleanup broadens to per-function timeouts + sub-limit alarms across all functions — gate 32.</ins></p> | |
| <div class="label crit">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>15 min is AWS Lambda's hard max. If any of these need more, they fail silently.</li> | |
| <li>These are batch/burst operations. Post-migration, the dispatcher path adds ~50–100 ms per query — on a 900-second budget that's negligible.</li> | |
| <li>But <strong><code>sendMedicalIncident</code> includes the call to <code>getMedicalDataForReview</code></strong> which pulls the medical bundle. Post-migration, that bundle comes from <code>medical_record</code> (single canonical home) instead of two contradicting sources. Faster on average; slower worst-case (depends on JOIN plan).</li> | |
| <li>Mike should monitor <code>Duration</code> metric on these 5 Lambdas after cutover — if any starts trending toward 900 s, investigate.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <p>CloudWatch alarm per function: <code>Duration > 720000 ms</code> (12 min — leaves 3 min margin) — pages on-call.</p> | |
| </div> | |
| </section> | |
| <hr> | |
| <div class="part-banner"> | |
| <div class="part-num">Part 3 of 8</div> | |
| <h2 style="border:0;color:var(--ink);">Architecture detail — dispatcher, RLS, frontend, reporting</h2> | |
| <p>Technical anchors. How the dispatcher works, how RLS enforces tenancy, what the frontend actually sees, how reporting on the reader endpoint plays out.</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== 6 · DISPATCHER ========================================== --> | |
| <!-- =============================================================== --> | |
| <section id="dispatcher"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 3 · 06</span> | |
| <h2 style="border:0;margin:18px 0 0;">The dispatcher + Amplify connect-to-SQL hybrid</h2> | |
| </div> | |
| <div class="callout high"><div class="callout-label">Open architecture decision · hybrid (baseline) vs one-dispatcher-for-all</div> | |
| This plan's baseline is the <strong>hybrid</strong> below — Amplify connect-to-SQL for the ~30 non-PHI models + a custom dispatcher for the ~15 PHI models — because the <a href="#exec">POC validated it</a> (existing <code>@auth</code> rules copied over unchanged; filters + pagination worked on 1,000 test users). The Final Development Plan (2026-06-09) proposes the alternative: <strong>one dispatcher for ALL Aurora models</strong>, arguing that a single road to the database makes RLS, column projection, and in-transaction audit impossible to bypass by construction — one enforcement regime, one audit choke point. Trade-off: the hybrid is less code and POC-proven but runs <em>two</em> enforcement paths (Amplify <code>@auth</code> for non-PHI, RLS + projection for PHI); one-dispatcher-for-all is uniform and maximally auditable but is all hand-written and gives up Amplify's managed CRUD for the non-PHI models. <strong>Unresolved — decide before Phase 3.</strong> <ins class="rlsfix"><strong>Input from the June 2026 PHI/RLS review (F4/Q7):</strong> on the hybrid path, <code>subscription</code> and <code>payment</code> have <em>no DB-level tenant filter</em> (Amplify <code>@auth</code> only). Recommended middle ground for the decision: move these two financial tables to the dispatcher path so they get the RLS fence, keep the rest of the hybrid as-is.</ins> | |
| <ins class="authfirst"><strong>DECIDED — June 2026: auth-first (a third option).</strong> Not the hybrid, not one-dispatcher-for-all: <em>every</em> Aurora model goes through connect-to-SQL with model + field-level <code>@auth</code>. The hybrid's "two enforcement paths" concern dissolves — there is ONE path (AppSync) for browsers and Lambdas alike. The dispatcher/RLS design is the deferred hardening track with documented reopen triggers (amber banner, top). The review's <code>subscription</code>/<code>payment</code> concern is addressed at the API layer instead: their <code>allow.authenticated()</code> write rules are replaced with function grants — §6.1b fix 1.</ins></div> | |
| <div class="callout prdval"><div class="callout-label">PRD validation · pull a thin slice of RLS forward (strongest disagreement)</div> | |
| <p style="margin:0 0 6px;">The independent reviewer agrees the spine is right but argues auth-first has <strong>no backstop</strong>: correctness depends on every model being annotated correctly, and the track record gets exactly that wrong (MemberNote group-wide, the all-Welon hook, scalar/array/<code>'312312'</code> drift). This <strong>independently confirms F4</strong>. Recommendation: keep the full dispatcher deferred, but make a single <code>tenant_id</code> RLS policy part of the <em>PHI baseline</em> — it turns a forgotten <code>@auth</code> rule from a cross-tenant breach into a denied query.</p> | |
| <p style="margin:0;"><strong>Architect nuance (resolve before adopting):</strong> a <code>tenant_id</code> RLS policy reads a per-request session GUC — and the managed connect-to-SQL Lambda runs as ONE pooled role that <em>cannot</em> set that GUC per request (exactly the F2/F4 constraint). So “tenant_id RLS for PHI” is <strong>not free on the connect-to-SQL path</strong> — it implies a <em>minimal</em> dispatcher (just to open a transaction and set the tenant GUC), even if the full projection dispatcher stays deferred. The real decision is therefore not “RLS yes/no” but <strong>“do PHI models go through a minimal tenant-context path?”</strong> Put this to the group with §6.1b.</p> | |
| </div> | |
| <h3>6.1 — Amplify connect-to-SQL for the ~30 non-PHI models</h3> | |
| <p> | |
| Amplify Gen 2 supports a first-party feature: <code>npx ampx generate schema-from-database</code> introspects a SQL | |
| database and auto-generates the <code>a.model()</code> representation. Amplify auto-provisions a managed Node.js "SQL | |
| Lambda" + AppSync resolvers. <strong>Frontend code stays identical</strong> — <code>client.models.Template.list()</code> | |
| works whether <code>Template</code> is DDB-backed or SQL-backed. | |
| </p> | |
| <table> | |
| <thead><tr><th>Capability</th><th>Detail</th></tr></thead> | |
| <tbody> | |
| <tr><td>Schema introspection</td><td>One command emits <code>amplify/data/schema.sql.ts</code>; hand-curate auth rules + relationships afterwards.</td></tr> | |
| <tr><td>Managed SQL Lambda</td><td>AWS owns the connection pool, connection retries, SSL/TLS. Deploys into VPC if the DB is in VPC.</td></tr> | |
| <tr><td>Auto-resolvers</td><td>CRUD via <code>client.models.X</code> works through the same shape; frontend ergonomics preserved.</td></tr> | |
| <tr><td>Mixed sources via <code>a.combine()</code></td><td>DDB + SQL + custom-dispatcher backed models coexist in a single AppSync API.</td></tr> | |
| <tr><td>Custom SQL</td><td><code>a.handler.inlineSql('SELECT …')</code> for ad-hoc queries.</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout authfirst"><div class="callout-label">Superseded · June 2026 auth-first decision</div> | |
| The limitation below is real (no per-request GUC hook in the managed SQL Lambda) — but the conclusion is superseded: under auth-first, PHI tables <strong>do</strong> use Amplify connect-to-SQL, with <strong>field-level <code>@auth</code></strong> carrying the PHI column controls at the API layer instead of database RLS. The dispatcher remains the deferred hardening track for when RLS is actually switched on.</div> | |
| <div class="callout high"> | |
| <div class="callout-label">Why the dispatcher is needed for PHI tables (the connect-to-SQL limitation) <ins class="authfirst">— DEFERRED TRACK</ins></div> | |
| The Amplify-managed SQL Lambda is a black box. It connects with one pooled credential and runs the resolver's SQL. | |
| <strong>There is no hook to run <code>BEGIN; SET LOCAL app.trust_company_id = …; COMMIT</code> before each query</strong> | |
| based on the caller's JWT. Without per-request session GUCs, RLS has nothing to filter on. <strong>So PHI tables cannot | |
| use Amplify SQL alone</strong> — they need the custom dispatcher with <code>pg</code> driver + parameterized queries. | |
| Non-PHI tables don't need this (no RLS enforcement required). | |
| </div> | |
| <h3>6.1b — The auth-first data layer <span class="sev authfirst">BASELINE · JUNE 2026</span></h3> | |
| <div class="callout authfirst"><div class="callout-label">What ships at cutover (validated against the working branch)</div> | |
| <p style="margin:0 0 6px;">The POC code on <code>cfl-test-aws-setup</code> already does this for 8 models — <code>sqlSchema.renameModels([...]).setAuthorization(models => [...])</code> with reusable rule packs (<code>piiFull</code>, <code>idAndRole</code>, <code>membershipStatus</code>, <code>isIncapacitated</code>, …) applied per model AND per field (e.g. <code>u.fields.email.authorization(piiFull)</code>, <code>u.fields.ssn</code> → owner-only). The migration extends exactly this pattern to the remaining models. Frontend and Lambdas keep calling <code>client.models.X.*</code> unchanged.</p> | |
| <p style="margin:0 0 4px;"><strong>Three <code>@auth</code> fixes required before PHI models migrate (found in the June 2026 code review):</strong></p> | |
| <ol style="margin:0 0 6px;"> | |
| <li><strong><code>UserSubscription</code>/<code>UserPayment</code>:</strong> <code>allow.authenticated().to(['create','update','delete'])</code> lets ANY signed-in member mutate ANY member's billing rows. Replace with function grants (<code>allow.resource(processWebhookV2)</code> etc.); owner keeps <code>read</code>-only.</li> | |
| <li><strong><code>TemplateSection</code>/<code>SectionVersion</code>:</strong> <code>allow.authenticated().to(['create','read','update','delete'])</code> lets any member edit legal-document templates. Restrict writes to ADMINS; authenticated → <code>read</code>.</li> | |
| <li><strong>Schema-level default <code>.authorization(allow => [allow.authenticated()])</code>:</strong> any model without explicit rules is born world-writable for signed-in users. Remove the default; CI check asserts every SQL model has explicit rules (fail-closed direction).</li> | |
| </ol> | |
| <p style="margin:0;"><strong>Control set replacing RLS at cutover:</strong> FF1 per-model isolation matrix run through the API (720 assertions incl. intra-tenant negatives); the tenant-isolation canary (§42/§47 — API-level, unchanged by this pivot); pgaudit on the cluster; least-privilege SQL connection user (no PHI-table grants beyond what resolvers need); <code>SQL_CONN_STRING</code> in Secrets Manager with rotation; AppSync introspection disabled + WAF (§36). Known residual risk, accepted and documented: enforcement lives in AppSync — a principal holding the connection secret bypasses it; mitigations are secret hygiene + pgaudit + the deferred RLS track's reopen triggers.</p> | |
| </div> | |
| <div class="callout review2"><div class="callout-label">Round-2 review · field-level @auth edge cases + enforcement coverage (validate before any PHI model migrates)</div> | |
| <p style="margin:0 0 4px;"><strong>Empirical validation harness on our own stack</strong> — Gen 2 docs are silent on behaviors Gen 1 documented; for a HIPAA control we verify, not trust:</p> | |
| <ol style="margin:0 0 6px;"> | |
| <li><strong>Mutation-response redaction</strong> — a field-level-denied PHI column must return null in <em>mutation</em> responses, not only query responses. (Gen 1-documented; Gen 2-silent; same engine.)</li> | |
| <li><strong>Subscription-payload filtering</strong> — field-level rules must redact in real-time subscription broadcasts, or a subscription surfaces a value a direct query would block.</li> | |
| <li><strong>The <code>.required()</code> trap (confirmed live):</strong> <code>AuroraUsers.email/firstName/lastName/countryCode</code> are <code>.required()</code> AND carry <code>piiFull</code>. A writer outside a field’s read-allow-list gets a forced-null cascade — the whole mutation payload returns null + error although the write persisted (e.g. <code>postSignUpTrigger</code>’s <code>AuroraUsers.create</code>). <strong>Fix:</strong> mark every field-level-auth’d column nullable in the model projection (decoupled from the DB NOT NULL constraint). <code>Document.content</code> is already not inline in the Aurora <code>document</code> projection (body → S3, F6), so that specific field is moot; the class is live on the user fields.</li> | |
| <li><strong>No inheritance</strong> — each rule pack already re-grants the full operation set per field (that is why they are packs). CI assertion: every PHI column has an explicit field pack; a column without one silently inherits the broad model rule.</li> | |
| </ol> | |
| <p style="margin:0 0 4px;"><strong>Enforcement has two halves.</strong> Field-level <code>@auth</code> redaction covers <em>generated CRUD</em> only. ~18 custom Lambda-resolver operations (<code>generateDocument</code>, <code>recreateDocument</code>, <code>archiveDocument</code>, <code>signDocuments</code>, <code>getMemberAvailableTemplates</code>, <code>deleteUserComplete</code>, …) return member/document data and enforce in their own handler logic, NOT via redaction. <strong>FF1 must assert against custom operations’ selection sets, not only generated CRUD.</strong></p> | |
| <p style="margin:0 0 4px;"><strong>FF1 is a pre-PHI merge gate, not a cutover gate.</strong> It does not exist yet (no CI test stage on this branch today; manual review only). The canary probes seeded paths only, so it bounds nothing for a bad rule on an un-probed model — FF1 coverage is the real bound. So FF1 coverage becomes a <em>precondition of each PHI model’s PR</em>, live before the first PHI model migrates. Interim (now → that gate): no PHI is in Aurora yet, so exposure is non-PHI models under manual review.</p> | |
| <p style="margin:0;"><strong>Reporting path.</strong> Aggregate cross-tenant reporting (<code>chil_reporter</code>) was bundled with the deferred RLS track — so there is no reporting path at cutover, and a direct-SQL reporting role is itself a non-AppSync path (a reopen trigger by our own “one road” rule). <strong>Target:</strong> reporting via an AppSync custom aggregate query backed by a privileged resolver (aggregate-only, no row-level PHI), keeping one road. <strong>Interim:</strong> reporting stays on the existing pre-migration path.</p> | |
| </div> | |
| <h3>6.2 — The custom dispatcher Lambda (PHI path) <span class="sev authfirst">DEFERRED · HARDENING TRACK</span></h3> | |
| <div class="callout authfirst"><div class="callout-label">Deferred · June 2026 auth-first decision</div> | |
| Everything below (dispatcher, Data-API transaction pattern, per-request GUCs — incl. the purple F2/F5 fixes) is the <strong>blueprint for the deferred RLS hardening track</strong>, kept current so it can be activated when a reopen trigger fires. It is NOT built at cutover.</div> | |
| <div class="callout"> | |
| <div class="callout-label">Plain-language: what the "Dispatcher Lambda" actually is</div> | |
| <strong>It is a single Node.js AWS Lambda</strong> that holds the Postgres client (<code>pg</code> npm package) and is | |
| invoked by AppSync for every PHI <code>client.models.X.get/list/create/update</code> call. Think of it as the one Lambda | |
| that <em>knows how to talk to Postgres</em>. It does five things in order, for every request: | |
| <ol style="margin:6px 0 0;"> | |
| <li>Reads the JWT claims that AppSync forwards (<code>trustCompanyId</code>, role, externalId).</li> | |
| <li>Calls the Aurora <strong>Data API</strong> authorized by the Lambda's IAM role (no VPC, no pooled connection, no password handled), opening a Data API transaction so <code>SET LOCAL</code> is scoped to this one request.</li> | |
| <li>Runs <code>BEGIN; SET LOCAL app.trust_company_id = '<value>'; …</code> — this is how Postgres knows which tenant's rows the request is allowed to see (RLS uses this session variable).</li> | |
| <li>Runs the actual SQL (parameterized — no string concat — and projected by role).</li> | |
| <li>Writes an audit_event row in the same transaction, then COMMITs.</li> | |
| </ol> | |
| <strong>That's the whole pattern.</strong> ~1,500 lines of TypeScript, one Lambda, all PHI traffic flows through it. | |
| The frontend never knows it exists — <code>client.models.Document.get({id})</code> looks the same as today. | |
| </div> | |
| <div class="callout ok"> | |
| <div class="callout-label">Connection mode · June 2026 — Data API + IAM (not RDS Proxy)</div> | |
| Per the Executive Summary decision, step 2 above ("opens a Postgres connection via RDS Proxy") becomes <strong>calls the Aurora Data API, authorized by the Lambda's IAM role</strong> — no VPC attachment, no pooled <code>pg</code> connection. The <code>BEGIN / SET LOCAL / … / COMMIT</code> sequence in steps 3–5 maps to the Data API transaction calls (<code>BeginTransaction → ExecuteStatement(SET LOCAL…) → ExecuteStatement(query) → CommitTransaction</code>); the parameterized-query, role-projection, and audit-in-same-transaction behavior is identical. The <code>pg</code>-driver code sample below is the reference for the SQL/transaction shape; under the Data API the same statements run as <code>rds-data:ExecuteStatement</code> calls carrying the transaction id. | |
| </div> | |
| <div class="callout high"><div class="callout-label">Data API limitations the dispatcher must handle</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li><strong>Result-set size cap</strong> (~1 MiB per <code>ExecuteStatement</code>). The dispatcher paginates: on the size error, return <code>hasMore</code> + a continuation token; reporting reads in chunks. <em>Phase-1 test:</em> a list query over a large tenant must paginate, not error. Add a CloudWatch alarm at ~900 KB response size. <ins class="rlsfix"><strong>June 2026 review fix (F6):</strong> pagination cannot help when a <em>single row</em> exceeds the cap — and <code>document.content</code> (trust instruments, multi-state wills) is exactly that row. Fix: bodies > 256 KB are stored in the PHI S3 bucket (SSE-KMS); the Aurora row keeps metadata + <code>s3_key</code> + <code>sha256</code> + <code>size_bytes</code>; reads return a short-lived presigned URL via the dispatcher (the RLS-checked row fetch stays the authorization gate); the bulk ETL routes oversize bodies to S3 during migration so no oversized inline row exists on day one. See the AG1 failure-mode row.</ins></li> | |
| <li><strong>No multi-statement calls</strong> — each <code>ExecuteStatement</code> is one statement; the BEGIN / <code>set_config</code> / query / COMMIT sequence is separate calls bound to one <code>transactionId</code>.</li> | |
| <li><strong>No array parameters</strong> — bind arrays as JSON/text and cast in SQL, or expand to multiple parameters.</li> | |
| <li><strong>No <code>LISTEN/NOTIFY</code></strong> (the Data API is stateless) — async fan-out uses the transactional <strong>outbox</strong> (Phase 2) + the 30 s drain worker, not Postgres notifications.</li> | |
| </ul></div> | |
| <div class="callout rlsfix"><div class="callout-label">June 2026 review fix (F2 + F5) — GUC mismatch + call-count</div> | |
| The previous sample set <code>app.cognito_id</code> while every owner/welon/caseload policy in §7 reads <code>current_setting('app.external_id')</code> — a GUC that was never set, so every owner-scoped read would fail with <code>42704</code>. Fixed: the dispatcher now sets <code>app.external_id</code> from the verified <code>externalId</code> claim (an app-generated UUID injected by preTokenGeneration — <em>not</em> the Cognito <code>sub</code>; see the §3b.2 identity glossary), fails closed if the claim is missing, and <strong>batches all GUC setup into ONE <code>ExecuteStatement</code></strong> — a real PHI read is now 4 sequential calls (Begin → setup → query → Commit), not 7. Two CI guards back this: the policy-GUC ⊆ dispatcher-GUC check and the <code>missing_ok := true</code> ban (§38). | |
| </div> | |
| <pre class="ts"><code>// amplify/data-layer/data-resolver/handler.ts (the dispatcher entry point) | |
| // June 2026: Aurora Data API + IAM. No VPC, no pooled pg connection, no stored password. | |
| import { | |
| RDSDataClient, BeginTransactionCommand, ExecuteStatementCommand, | |
| CommitTransactionCommand, RollbackTransactionCommand, | |
| } from '@aws-sdk/client-rds-data'; | |
| import { extractRole, requireTenant, requireExternalId } from './auth'; | |
| import { getModelHandler } from './handlers'; | |
| // Execution role carries rds-data:ExecuteStatement + secretsmanager:GetSecretValue. | |
| const rds = new RDSDataClient({}); | |
| const resourceArn = process.env.AURORA_CLUSTER_ARN!; // Serverless v2 cluster ARN | |
| const secretArn = process.env.AURORA_SECRET_ARN!; // DB creds in Secrets Manager (referenced, never handled) | |
| const database = 'chil'; | |
| export const handler = async (event: DataResolverEvent) => { | |
| const tenantId = requireTenant(event.identity); // throws if missing | |
| const externalId = requireExternalId(event.identity); // June 2026 fix (F2): fail closed if claim absent | |
| const cognitoId = event.identity.claims.sub; // audit correlation only | |
| const role = extractRole(event.identity.claims); | |
| // One Data API transaction == one connection for its life, reset by AWS afterwards. | |
| // SET LOCAL is bound to THIS transactionId, so no cross-request GUC leak is possible. | |
| const { transactionId } = await rds.send(new BeginTransactionCommand({ resourceArn, secretArn, database })); | |
| const exec = (sql: string, parameters?: any[]) => | |
| rds.send(new ExecuteStatementCommand({ resourceArn, secretArn, database, transactionId, sql, parameters })); | |
| try { | |
| // set_config(name, value, is_local=true) is the parameterizable form of SET LOCAL. | |
| // June 2026 fix (F2 + F5): ONE batched call sets every GUC the §7 policies read — | |
| // including app.external_id (previously never set → owner policies crashed with 42704). | |
| // Batching the 4 setup statements into 1 makes a PHI read 4 HTTPS calls, not 7. | |
| await exec( | |
| `SELECT set_config('app.trust_company_id', :t, true), | |
| set_config('app.external_id', :e, true), | |
| set_config('app.role', :r, true), | |
| set_config('app.cognito_id', :c, true), | |
| set_config('statement_timeout', '10s', true)`, | |
| [ { name:'t', value:{ stringValue: tenantId } }, | |
| { name:'e', value:{ stringValue: externalId } }, | |
| { name:'r', value:{ stringValue: role } }, | |
| { name:'c', value:{ stringValue: cognitoId } } ]); | |
| const modelHandler = getModelHandler(event.model); | |
| const result = await modelHandler[event.op](exec, event.args, { role }); // handlers run SQL via exec() | |
| await rds.send(new CommitTransactionCommand({ resourceArn, secretArn, transactionId })); | |
| return result; | |
| } catch (err) { | |
| await rds.send(new RollbackTransactionCommand({ resourceArn, secretArn, transactionId })).catch(() => {}); | |
| throw mapToAppSyncError(err); // hides SQL details | |
| } | |
| };</code></pre> | |
| <p style="font-size:13px;color:var(--muted);"><strong>Note:</strong> this also fixes a latent bug in the prior <code>pg</code> sample — <code>SET LOCAL x = $1</code> is not parameterizable in Postgres (SET is a utility statement); the correct parameterized form is <code>set_config(name, value, true)</code>, identical over the Data API.</p> | |
| <p> | |
| This is ~1,500 lines hand-written. <strong>No ORM at runtime.</strong> The <code>pg</code> driver with parameterized | |
| queries. Drizzle Kit is used only at dev-time for schema migrations (generates SQL DDL files; never imported at | |
| runtime). | |
| </p> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 7 · RLS ================================================= --> | |
| <!-- =============================================================== --> | |
| <section id="rls"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 3 · 07</span> | |
| <h2 style="border:0;margin:18px 0 0;">RLS — multi-tenancy spine</h2> | |
| </div> | |
| <div class="callout authfirst"><div class="callout-label">Deferred · June 2026 auth-first decision</div> | |
| RLS is <strong>not enabled at cutover</strong>. This section (including the purple June 2026 review fixes — the corrected restrictive-fence DDL, <code>FORCE ROW LEVEL SECURITY</code>, <code>chil_owner</code>) is the reviewed, ready-to-apply blueprint for the hardening track. Reopen triggers: white-label multi-tenancy go-live · auditor/client requirement · any non-AppSync PHI access path. Until then, isolation = model + field-level <code>@auth</code> + FF1 + canary (§6.1b).</div> | |
| <p> | |
| Postgres Row-Level Security is the strongest tenant isolation available. The dispatcher Lambda sets per-request | |
| session GUCs from the JWT claim; RLS policies on every PHI / member-scoped table reference those GUCs. With | |
| <code>chil_app</code> as the connecting role (no <code>BYPASSRLS</code>), every query is automatically filtered. | |
| <strong>An application bug that forgets a WHERE clause does not leak cross-tenant data</strong> — Postgres refuses | |
| to return it. | |
| </p> | |
| <h3>Representative RLS policies <span class="sev rlsfix">REWRITTEN · JUNE 2026 REVIEW · F1</span></h3> | |
| <div class="callout rlsfix"><div class="callout-label">June 2026 review fix (F1 / Q1 / Q3) — the permissive-OR collapse</div> | |
| The previous version of this DDL defined <strong>all</strong> policies as PERMISSIVE. Postgres ORs permissive policies for the same command, so the broad <code>FOR ALL</code> tenant policy OR-ed away every scoped policy — the owner, welon-assignment, and MEDICALREVIEW-caseload controls were <strong>dead code</strong>, and any authenticated member could read every row in their tenant. Neither <code>AS RESTRICTIVE</code> nor <code>FORCE ROW LEVEL SECURITY</code> appeared anywhere. Restructured below: <strong>one RESTRICTIVE tenant fence per table</strong> (ANDed with everything) <strong>+ scoped permissive ACL policies per command</strong> (with no broad <code>FOR ALL</code> grant, the <em>only</em> way to read a row is to satisfy an ACL policy). Every PHI table also gets <code>FORCE ROW LEVEL SECURITY</code> and is owned by <code>chil_owner</code> (NOLOGIN, no BYPASSRLS), so owner-context access is fenced too. A CI lint over <code>pg_policies</code> enforces all three invariants (§38). | |
| </div> | |
| <pre class="sql"><code>-- June 2026 PHI/RLS review fix (F1): RESTRICTIVE fence + scoped permissive ACL. | |
| -- Pattern below applies to EVERY PHI / member-scoped table. | |
| ALTER TABLE "user" ENABLE ROW LEVEL SECURITY; | |
| ALTER TABLE "user" FORCE ROW LEVEL SECURITY; -- Q3: table owner does not bypass | |
| ALTER TABLE "user" OWNER TO chil_owner; -- Q3: NOLOGIN owner, no BYPASSRLS | |
| -- 1 · Tenant fence — RESTRICTIVE: ANDed with every permissive policy below. | |
| CREATE POLICY user_tenant_fence ON "user" | |
| AS RESTRICTIVE FOR ALL TO chil_app | |
| USING (trust_company_id = current_setting('app.trust_company_id')::uuid) | |
| WITH CHECK (trust_company_id = current_setting('app.trust_company_id')::uuid); | |
| -- 2 · Permissive policies = the ACL. No broad FOR ALL permissive policy exists, | |
| -- so the ONLY way to read a row is to satisfy one of these. The tenant check | |
| -- lives in the fence and is no longer repeated here. | |
| CREATE POLICY user_owner_read ON "user" | |
| FOR SELECT TO chil_app | |
| USING ( | |
| external_id = current_setting('app.external_id') | |
| OR current_setting('app.role') IN ('Admin', 'WelonTrust', 'MedicalReview', 'CallCenter') | |
| ); | |
| -- INSERT / UPDATE / DELETE each get their own scoped permissive policy with | |
| -- an explicit WITH CHECK — authored per table in the Phase-2 DDL. | |
| -- Welon assignment ACL (intra-tenant scope; fence already guarantees the tenant) | |
| ALTER TABLE document ENABLE ROW LEVEL SECURITY; | |
| ALTER TABLE document FORCE ROW LEVEL SECURITY; | |
| CREATE POLICY document_tenant_fence ON document | |
| AS RESTRICTIVE FOR ALL TO chil_app | |
| USING (trust_company_id = current_setting('app.trust_company_id')::uuid) | |
| WITH CHECK (trust_company_id = current_setting('app.trust_company_id')::uuid); | |
| CREATE POLICY document_welon_read ON document | |
| FOR SELECT TO chil_app | |
| USING ( | |
| user_id IN (SELECT id FROM "user" WHERE external_id = current_setting('app.external_id')) | |
| OR EXISTS ( | |
| SELECT 1 FROM welon_assignment wa | |
| WHERE wa.member_id = document.user_id | |
| AND wa.welon_user_id = (SELECT id FROM "user" WHERE external_id = current_setting('app.external_id')) | |
| AND wa.status = 'active' | |
| ) | |
| OR current_setting('app.role') IN ('Admin', 'MedicalReview', 'CallCenter') | |
| ); | |
| -- MEDICALREVIEW caseload (fixes the live HIPAA gap; fence guarantees the tenant) | |
| ALTER TABLE medical_report ENABLE ROW LEVEL SECURITY; | |
| ALTER TABLE medical_report FORCE ROW LEVEL SECURITY; | |
| CREATE POLICY medical_report_tenant_fence ON medical_report | |
| AS RESTRICTIVE FOR ALL TO chil_app | |
| USING (trust_company_id = current_setting('app.trust_company_id')::uuid) | |
| WITH CHECK (trust_company_id = current_setting('app.trust_company_id')::uuid); | |
| CREATE POLICY medical_report_reviewer_scoped ON medical_report | |
| FOR SELECT TO chil_app | |
| USING ( | |
| current_setting('app.role') = 'MedicalReview' | |
| AND user_id IN ( | |
| SELECT member_user_id FROM medical_review_caseload | |
| WHERE reviewer_user_id = (SELECT id FROM "user" WHERE external_id = current_setting('app.external_id')) | |
| AND status = 'active' | |
| ) | |
| );</code></pre> | |
| <h3>The role hierarchy</h3> | |
| <table> | |
| <thead><tr><th>Role</th><th>Used by</th><th>BYPASSRLS?</th><th>Notes</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>chil_app</code></td><td>Dispatcher Lambda (via Data API, IAM-authed)</td><td>No</td><td>Default application role. RLS always applies.</td></tr> | |
| <tr><td><code>chil_sqlconnect</code></td><td>Amplify SQL Lambda (managed) — non-PHI only</td><td>No</td><td>GRANT-scoped to non-PHI tables; physically cannot touch PHI.</td></tr> | |
| <tr><td><code>chil_reporter</code></td><td>Reporting reads on Aurora reader</td><td>No</td><td>SELECT-only; RLS still applies.</td></tr> | |
| <tr><td><code>chil_admin</code></td><td>Break-glass / manual ops</td><td>Yes</td><td>Used only via IAM Identity Center + temp credentials; every session via pgaudit.</td></tr> | |
| <tr><td><ins class="rlsfix"><code>chil_owner</code></ins></td><td><ins class="rlsfix">Owns every table (June 2026 review fix, Q3)</ins></td><td><ins class="rlsfix">No</ins></td><td><ins class="rlsfix">NOLOGIN. Exists only to own objects so FORCE RLS fences owner-context access.</ins></td></tr> | |
| <tr><td><code>chil_migrator</code></td><td>Schema migrations (DDL only)</td><td>Yes</td><td>No DML privileges. CI/CD with temp credentials.</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout"> | |
| <div class="callout-label">Field-level authorization (the 36 User auth blocks)</div> | |
| RLS is row-level only — it cannot express the ~200 column×role rules on User. <strong>The dispatcher does column | |
| projection</strong> per role (e.g. <code>SELECT id, email, role FROM "user" WHERE ...</code> with different column lists | |
| per role). RLS is the row-level backstop; projection is the column-level control. The same matrix that exists in 36 | |
| <code>.authorization()</code> blocks today becomes one TypeScript file with full test coverage. | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 8 · FRONTEND IMPACT ===================================== --> | |
| <!-- =============================================================== --> | |
| <section id="frontend"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 3 · 08</span> | |
| <h2 style="border:0;margin:18px 0 0;">Frontend impact — the headline answer</h2> | |
| </div> | |
| <div class="callout ok"> | |
| <div class="callout-label">The honest answer — revised after code-grounded audit</div> | |
| <strong>Frontend is NOT rewritten.</strong> <code>generateClient<Schema>()</code> works. <code>client.models.X.get/list/create/update</code> | |
| works. Schema TypeScript types are still generated from <code>amplify/data/resource.ts</code>. <strong>But the file count | |
| is larger than originally claimed.</strong> An honest audit against current main found <strong>~84–120 files touched</strong> | |
| (not 45–65). The work is still <strong>mostly mechanical</strong> — but mechanical-applied-to-more-places. | |
| Detailed breakdown in §49. | |
| </div> | |
| <div class="callout"> | |
| <div class="callout-label">Where the bigger number comes from (code-grounded)</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li><strong>149 GSI-backed <code>listXByY</code> call sites</strong> across <strong>11 frontend files + 45 Lambda files</strong> — every <code>listDocumentByUserId</code>, <code>listUserByCognitoId</code>, <code>listTemplateVersionByTemplateIdAndVersionNumber</code> etc. switches to <code>list({ filter })</code>. Mechanical, but it touches everywhere.</li> | |
| <li><strong>64 JSON-blob call sites</strong> across <strong>11 files</strong> — every <code>JSON.parse(user.interviewProgressV2)</code>, <code>JSON.parse(user.quarterlyReviewData)</code>, embedded <code>CareDocumentNew.data[]</code> iteration. Switch to typed child-table queries.</li> | |
| <li><strong>36 Care Document section components</strong> in <code>app/components/care-documents-new/main-sections/</code> — each currently reads from <code>CareDocumentNew.data[]</code> and shifts to <code>client.models.CareDocumentSection.list({ careDocumentId, sectionType })</code>.</li> | |
| <li><strong>32 <code>cognitoId</code> references across 15 files</strong> — mechanical rename to <code>externalId</code>. <code>AuthContext</code> is the auth-critical one.</li> | |
| <li><strong>26 status / enum references</strong> — UserStatus, MembershipStatusEnum, PaymentsStatusEnum collapsing into <code>lifecycle_state</code> + <code>billing_state</code>.</li> | |
| <li><strong>14 welon-array call sites across 7 files</strong> — <code>user.assignedWelonTrustIds.includes(...)</code> switches to a <code>welon_assignment</code> join.</li> | |
| </ul> | |
| <strong>"Mostly mechanical" is still true.</strong> 84–120 files × ~1–10 lines each ≈ 1500–2500 LOC. No file is rewritten end-to-end. But it's wider than the headline implied — closer to 2–3× the original 45–65 estimate. | |
| </div> | |
| <h3>Side-by-side — before vs after</h3> | |
| <h4>Example 1: Reading a user — IDENTICAL</h4> | |
| <pre class="ts"><code>// BEFORE (today, DynamoDB-backed) | |
| const { data: user } = await client.models.User.get({ id: userId }); | |
| // AFTER (Aurora-backed via dispatcher) — IDENTICAL CALL SHAPE | |
| const { data: user } = await client.models.User.get({ id: userId }); | |
| // Same generated type. Same call. The dispatcher translates to SQL under the hood.</code></pre> | |
| <h4>Example 2: Listing — IDENTICAL call shape, BETTER behavior</h4> | |
| <pre class="ts"><code>// BEFORE (DDB) — 1 MB cap + post-filter bug | |
| const { data: docs, nextToken } = await client.models.Document.list({ | |
| filter: { status: { eq: 'signed' } }, | |
| limit: 50, | |
| }); | |
| // Issue: may return <50 items with a nextToken because of 1 MB pre-filter cutoff. | |
| // AFTER (Aurora) — IDENTICAL CALL SHAPE, predictable result | |
| const { data: docs, nextToken } = await client.models.Document.list({ | |
| filter: { status: { eq: 'signed' } }, | |
| limit: 50, | |
| }); | |
| // SQL: SELECT ... FROM document WHERE status = 'signed' LIMIT 50; | |
| // Returns exactly 50 (or fewer if fewer match), with a stable cursor.</code></pre> | |
| <h4>Example 3: A canonical-decomposition change</h4> | |
| <pre class="ts"><code>// BEFORE — interview lives inside User as a JSON blob | |
| const { data: user } = await client.models.User.get({ id }); | |
| const interview = user.interviewProgressV2; // stringified JSON | |
| const parsed = typeof interview === 'string' | |
| ? JSON.parse(interview) | |
| : interview; // double-encoding workaround | |
| // AFTER — interview is its own typed child entity | |
| const { data: interview } = | |
| await client.models.UserInterview.get({ userId: id }); | |
| // Typed. No JSON parsing. No double-encoding pain.</code></pre> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 8b · WHITE-LABEL & BRAND TRACK ========================= --> | |
| <!-- =============================================================== --> | |
| <section id="brand-track"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 3 · 08b</span> | |
| <h2 style="border:0;margin:18px 0 0;">White-label & brand track</h2> | |
| </div> | |
| <div class="callout ok"><div class="callout-label">Harvested from the Final Development Plan (2026-06-09) — parallel track, frontend-only until WL-3</div> | |
| The styling system already uses CSS variables, so this is a completion job, not a rebuild. It is <strong>independent of the database migration</strong>: the rebranded second offering can ship as soon as WL-3 completes.</div> | |
| <table> | |
| <thead><tr><th>Step</th><th>Work</th><th>Acceptance</th></tr></thead> | |
| <tbody> | |
| <tr><td>WL-1</td><td>Token completion: sweep the ~24 files with hardcoded hex to CSS variables; rename tokens semantically (<code>--color-primary</code>, <code>--color-accent</code>, surface/role tokens) with the current palette as the default brand. No visual change.</td><td>Zero brand hex literals outside <code>globals.css</code>; visual regression green</td></tr> | |
| <tr><td>WL-2</td><td><code>brand</code> table + middleware: resolve Host header → brand (cached, ~5-min revalidate, fallback to flagship); root layout injects brand CSS variables, logo, favicon, metadata server-side. Domains attach to the Amplify app per brand.</td><td>A staging vanity domain renders a second brand end to end with no deploy between brand edits</td></tr> | |
| <tr><td>WL-3</td><td>Brand-aware comms & documents: per-brand SES sender identity, Cognito message templates branched by brand, <code>brand_*</code> Handlebars keys in document templates alongside <code>trustCompany_*</code>, hospital-facing acknowledgment page renders the member's brand, transactional email palette tokenized.</td><td>Email + generated document + hospital page all carry brand #2 in staging</td></tr> | |
| <tr><td>WL-4</td><td>De-brand the codebase: <code>/welon/*</code> → <code>/operator/*</code> (301s indefinitely), UI labels from <code>trust_company.display_name</code> / <code>brand.name</code>, internal naming neutralized as files are touched (no big-bang rename).</td><td>No user-visible legacy brand strings; old URLs redirect</td></tr> | |
| </tbody> | |
| </table> | |
| <ul> | |
| <li><strong>The rebranded second offering ships after WL-3</strong> — create the brand row, attach the domain, done. It does not wait for the DB migration. (Contrast: a second <em>trust company</em> waits for the Phase-4 cutover + a green tenant-isolation matrix.)</li> | |
| <li>Per-tenant Stripe accounts and per-tenant UPS shipper accounts are explicitly <strong>out of scope for v1</strong>; revisit when a partner contract requires them.</li> | |
| <li>Brand changes are content operations (DB row edits via the admin UI), not deployments. That is the operating model on purpose.</li> | |
| </ul> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 9 · REPORTING =========================================== --> | |
| <!-- =============================================================== --> | |
| <section id="reporting"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 3 · 09</span> | |
| <h2 style="border:0;margin:18px 0 0;">Reporting — Aurora reader, no sync pipeline</h2> | |
| </div> | |
| <p> | |
| Aurora has a built-in reader endpoint synchronously replicated from the writer (sub-second lag). Reporting queries | |
| hit the reader; OLTP hits the writer. <strong>No zero-ETL, no Redshift, no Glue, no Kinesis</strong> — same database | |
| family, same SQL, same engine. | |
| </p> | |
| <h3>Setup</h3> | |
| <pre class="ts"><code>// amplify/backend.ts (Track B reporting) | |
| import { aws_rds as rds } from 'aws-cdk-lib'; | |
| // Reader endpoint on the same Aurora cluster | |
| const auroraReader = new rds.CfnDBInstance(stack, 'ChilAuroraReader', { | |
| dbClusterIdentifier: auroraCluster.clusterIdentifier, | |
| dbInstanceClass: 'db.serverless', | |
| engine: 'aurora-postgresql', | |
| }); | |
| // Data API targets the cluster; the reader is reached by passing the | |
| // reader endpoint ARN as resourceArn (no RDS Proxy target group needed). | |
| // reportingExec({ resourceArn: auroraReaderArn, secretArn, sql, transactionId, | |
| // dbInstanceIdentifiers: [auroraReader.ref], | |
| connectionPoolConfigurationInfo: { | |
| maxConnectionsPercent: 90, | |
| maxIdleConnectionsPercent: 50, | |
| connectionBorrowTimeout: 30, | |
| }, | |
| });</code></pre> | |
| <h3>Representative KPI query (MRR by signup cohort)</h3> | |
| <pre class="sql"><code>-- Set tenant context (per-request). | |
| -- June 2026 review fix (Q5): this MUST run inside a Data API transaction — | |
| -- a standalone SET LOCAL outside a transaction is a SILENT NO-OP (warning only), | |
| -- which would have run this query with no tenant context. Reporting uses the | |
| -- same 4-call shape as the dispatcher: BeginTransaction → setup → query → Commit. | |
| SELECT set_config('app.trust_company_id', '00000000-0000-4000-8000-000000000001', true); | |
| WITH cohorts AS ( | |
| SELECT id AS user_id, DATE_TRUNC('month', created_at)::date AS cohort | |
| FROM "user" | |
| ) | |
| SELECT c.cohort, | |
| COUNT(DISTINCT c.user_id) AS cohort_size, | |
| SUM(s.amount_cents) FILTER (WHERE s.status = 'active') / 100.0 AS mrr_usd, | |
| ROUND( | |
| SUM(s.amount_cents) FILTER (WHERE s.status='active') / | |
| NULLIF(COUNT(DISTINCT c.user_id) * 100.0, 0), 2) AS arpu_usd | |
| FROM cohorts c | |
| LEFT JOIN subscription s ON s.user_id = c.user_id | |
| GROUP BY 1 | |
| ORDER BY 1 DESC | |
| LIMIT 12; | |
| -- One query. Joins. Window-style aggregation. Returns predictable bytes.</code></pre> | |
| <p> | |
| The Next.js Reporting page calls the dispatcher in reporting mode (using chil_reporter against the reader endpoint). | |
| KPI cards render. Existing Apps Script prototype is decommissioned (it's an unaudited data-access surface — HIPAA risk). | |
| </p> | |
| <div class="callout"> | |
| <div class="callout-label">RLS enforcement on the reader (HIPAA-critical)</div> | |
| <strong>chil_reporter has the same RLS policies as chil_app.</strong> Every reporting query <code>SET LOCAL app.trust_company_id</code> | |
| before executing, exactly as the OLTP dispatcher does. Postgres RLS is enforced equally on writer and reader — it's a | |
| table-level policy, not a connection-level setting. Reporter is RO and has no <code>BYPASSRLS</code>; an aggregation | |
| without the <code>SET LOCAL</code> simply returns zero rows. <strong>FF1 in §B.8 includes reader-endpoint queries in | |
| the tenant-isolation matrix.</strong> | |
| </div> | |
| <h3>Sync mechanics + cost</h3> | |
| <table> | |
| <thead><tr><th>Aspect</th><th>Aurora reader</th></tr></thead> | |
| <tbody> | |
| <tr><td>Sync mechanism</td><td>Synchronous physical replication (built-in)</td></tr> | |
| <tr><td>Sync lag</td><td>< 100 ms typical</td></tr> | |
| <tr><td>Setup mechanism</td><td>CDK: add a reader instance to the existing cluster</td></tr> | |
| <tr><td>Initial backfill</td><td>Snapshot + restore on creation (~minutes)</td></tr> | |
| <tr><td>Schema evolution</td><td>Reader inherits writer schema instantly</td></tr> | |
| <tr><td>Failure recovery</td><td>Automatic replacement; promote standby if needed</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <div class="part-banner"> | |
| <div class="part-num">Part 4 of 8</div> | |
| <h2 style="border:0;color:var(--ink);">Phased playbook — 5 phases with phase gates</h2> | |
| <p>Pre-flight checklist, then five phases. Each phase lists backend deliverables, frontend impact (usually "none"), HIPAA/SOC 2 controls landed, and acceptance gate before the next phase starts.</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== 10 · PRE-FLIGHT ========================================= --> | |
| <!-- =============================================================== --> | |
| <section id="pre-flight"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 4 · 10</span> | |
| <h2 style="border:0;margin:18px 0 0;">Pre-flight checklist (Phase 0)</h2> | |
| </div> | |
| <div class="card"> | |
| <ol style="margin:0;"> | |
| <li><strong>Track B decided.</strong> The team has agreed Aurora is the target.</li> | |
| <li><strong>Data architect named</strong> — owns the canonical schema, dispatcher, RLS policies. Recommended: Oleksii.</li> | |
| <li><strong>External Cognito pool owner identified</strong> — needed for <code>custom:trustCompanyId</code> + <code>custom:onboardingCompleted</code> attribute adds across dev-test, staging, prod.</li> | |
| <li><strong>AWS BAA confirmed</strong> on the account. Aurora Serverless v2 + RDS Data API + KMS + Secrets Manager are HIPAA-eligible under the BAA — verify it's signed.</li> | |
| <li><strong>Phase 0 cleanup continued</strong> in parallel (subscriptions/delete off, dead GSIs/models removed). This is rescuing the live CFN compliance issue regardless.</li> | |
| <li><strong>Repo readiness</strong>: branch protection on <code>main</code>, required PR review, release log, CI gate.</li> | |
| <li><strong>Migration tooling decided</strong>: <strong>Drizzle Kit</strong> (TypeScript schema + generates SQL migrations; dev-time only, no runtime dependency). Backup: Flyway (hand-written SQL files).</li> | |
| <li><strong>BAA inventory for all data-path vendors</strong>: AWS (signed), Sentry Enterprise (verify tier), Stripe (signed), Mailchimp (signed), UPS (verify).</li> | |
| </ol> | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 11 · PHASE 1 ============================================ --> | |
| <!-- =============================================================== --> | |
| <section id="phase-1"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 4 · 11</span> | |
| <h2 style="border:0;margin:18px 0 0;">Phase 1 — Foundation</h2> | |
| </div> | |
| <p>Goal: <strong>infrastructure exists and is reachable</strong>. No data movement yet. No production impact yet.</p> | |
| <table> | |
| <thead><tr><th>Workstream</th><th>Backend deliverable</th></tr></thead> | |
| <tbody> | |
| <tr><td>VPC + network <em>(Data API: not required)</em></td><td><s>VPC (2 AZs, /16), private/public subnets, NAT × 2, VPC endpoints, security groups.</s> Under the Data API there is no DB-path VPC. Deliverable reduces to: IAM policy for <code>rds-data:ExecuteStatement</code> + <code>secretsmanager:GetSecretValue</code> on the dispatcher role; optional Data API interface endpoint for private routing.</td></tr> | |
| <tr><td>KMS keys</td><td>Customer-managed CMKs per environment: <code>chil-aurora-cmk</code>, <code>chil-reporting-cmk</code>, <code>chil-secrets-cmk</code>, <code>chil-audit-archive-cmk</code>. Rotation enabled. Key policies restrict principals.</td></tr> | |
| <tr><td>Aurora Serverless v2 cluster</td><td>PostgreSQL 16, CMK at-rest encryption. Writer 1–8 ACU, reader 0.5–4 ACU. Subnet group across private subnets. DB parameter group enables pgaudit, pgcrypto, pg_stat_statements, aws_s3 extensions. <code>require_ssl=true</code>, <code>statement_timeout=10s</code>. PITR + 7-day backups. Deletion protection ON.</td></tr> | |
| <tr><td>RDS Proxy <em>(removed)</em></td><td><s>RDS Proxy with IAM auth, writer + reader target groups.</s> Not provisioned — the Data API needs no connection pooler. Writer/reader split is done by targeting the writer vs reader endpoint ARN in the Data API call.</td></tr> | |
| <tr><td>Migration tooling</td><td>Drizzle Kit set up: <code>drizzle.config.ts</code> at repo root, migrations folder. CI step applies migrations to a throwaway PG container per PR. <code>chil_migrator</code> IAM role with DDL-only grants for prod.</td></tr> | |
| <tr><td>Postgres roles</td><td><code>chil_app</code> (no BYPASSRLS), <code>chil_sqlconnect</code> (non-PHI scope), <code>chil_reporter</code> (reader read-only), <code>chil_admin</code> (BYPASSRLS, break-glass via temp creds), <code>chil_migrator</code><ins class="rlsfix">, <code>chil_owner</code> (NOLOGIN table owner — June 2026 review fix, Q3)</ins>.</td></tr> | |
| <tr><td>Cognito pool changes</td><td>Coordinate with external pool owner: add <code>custom:trustCompanyId</code> + <code>custom:onboardingCompleted</code> across 3 envs. Backfill all existing users to seed organization via admin script.</td></tr> | |
| <tr><td>Observability</td><td>3 CloudWatch dashboards (data plane, request plane, compliance). Alarms: ACU saturation, Aurora connection saturation, Data API throttling/latency, RLS denial rate. Aurora pgaudit logs → CloudWatch → S3.</td></tr> | |
| <tr><td>Audit archive</td><td>S3 audit-archive bucket: Object Lock (Compliance mode, 7-year), CMK-encrypted, Glacier lifecycle after 180 days. Block all public access.</td></tr> | |
| <tr><td>Sentry hardening</td><td>Deploy <code>beforeSend</code> PHI scrubber in all 3 init files. Drop <code>tracesSampleRate</code> from 1 to 0.1 in prod. <code>sendDefaultPii: false</code> on server + edge. Disable <code>enableLogs</code> on PHI-bearing routes.</td></tr> | |
| <tr><td>preTokenGen fail-closed</td><td>Wrap trigger to <code>throw</code> on any error instead of returning event without claims. Add CloudWatch metric on token-mint exceptions.</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout ok"> | |
| <div class="callout-label">Phase 1 gate</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>Aurora reachable via the Data API (<code>aws rds-data execute-statement</code> returns a row) using the dispatcher role's IAM identity.</li> | |
| <li><ins class="rlsfix"><strong>Full dispatcher synthetic</strong> (June 2026 review fix, F5/Q6) — the real 4-call PHI transaction (BeginTransaction → batched <code>set_config</code> → query → CommitTransaction) against a seeded table: <strong>P95 < 400 ms</strong> (stretch 250 ms), consistent with the §53 pre-cutover gate. The bare 3-call round-trip stays as a <em>connectivity smoke only</em> — the previous "< 150 ms" gate measured a synthetic that did not represent a real PHI read. P50/P95/P99 for both shapes published at Phase-1 exit.</ins> <ins class="authfirst"><strong>Auth-first re-scope:</strong> no dispatcher at cutover — the gate becomes an end-to-end <code>client.models.X.list()</code> through AppSync + the managed SQL Lambda against a seeded table, <strong>P95 < 400 ms</strong>; the dispatcher synthetic moves to the hardening track.</ins></li> | |
| <li>All <ins class="rlsfix">6</ins> Postgres roles created; chil_app verified no BYPASSRLS<ins class="rlsfix">; chil_owner verified NOLOGIN + no BYPASSRLS (Q3)</ins>.</li> | |
| <li>Cognito attributes added across 3 envs; backfill complete; test user shows claim in token. <ins class="datarch">📄 <strong>Confirmed THE Phase-1 kickoff blocker (data-arch review):</strong> <code>custom:trustCompanyId</code> does not exist in the pool yet — this is the first task; nothing else in the migration starts until it exists and all users are backfilled (gate 33).</ins></li> | |
| <li>CI runs Drizzle migrations against a throwaway container per PR.</li> | |
| <li>Sentry scrubber deployed; deliberate PHI test exception shows redacted.</li> | |
| <li>preTokenGen fail-closed; deliberate missing-attribute test throws.</li> | |
| </ul> | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 12 · PHASE 2 ============================================ --> | |
| <!-- =============================================================== --> | |
| <section id="phase-2"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 4 · 12</span> | |
| <h2 style="border:0;margin:18px 0 0;">Phase 2 — Canonical schema + non-PHI migration</h2> | |
| </div> | |
| <p>Goal: <strong>schema lives in Aurora; non-PHI models migrate one-by-one via Amplify connect-to-SQL.</strong> Frontend continues working through both DDB and SQL paths in parallel via <code>a.combine()</code>.</p> | |
| <table> | |
| <thead><tr><th>Workstream</th><th>Backend deliverable</th><th>Frontend impact</th></tr></thead> | |
| <tbody> | |
| <tr><td>Postgres DDL</td><td>Author full canonical DDL (~32 entities). Drizzle schemas in <code>amplify/data/schema.drizzle.ts</code>. <code>trust_company</code> seeded with one row.</td><td>None — tables sit empty in Aurora</td></tr> | |
| <tr><td>Platform / tenancy spine</td><td>Create the 6 platform tables (<code>trust_company</code>, <code>partner</code>, <code>brand</code>, <code>member_trust_assignment</code>, <code>trust_reassignment_suggestion</code>, <code>zip_code_centroid</code>); load the ~33k-row US Census ZCTA seed into <code>zip_code_centroid</code>; implement the deterministic member→trust assignment function (state match → haversine proximity → priority → default) per §3b.3. Unit-test against known fixtures.</td><td>None (admin UI lands in Phase 3)</td></tr> | |
| <tr><td>RLS policies</td><td>Enable RLS on every PHI/member-scoped table. <ins class="rlsfix">June 2026 review fix (F1/Q2/Q3): RESTRICTIVE tenant fence + scoped permissive ACL per command (no broad FOR ALL permissive policy); <code>FORCE ROW LEVEL SECURITY</code> + <code>chil_owner</code> ownership on every PHI table; verification matrix extended with intra-tenant negative cases — member A must get <strong>zero rows</strong> for member B's records in the same tenant.</ins> RLS verification matrix in CI.</td><td>None</td></tr> | |
| <tr><td>Amplify connect-to-SQL</td><td><code>npx ampx generate schema-from-database</code> emits <code>amplify/data/schema.sql.ts</code>. Hand-curate auth rules. Wire <code>a.combine([ddbSchema, sqlSchema])</code> in resource.ts.</td><td>None — same <code>client.models.X</code> calls</td></tr> | |
| <tr><td>Migrate non-PHI models one-at-a-time</td><td>Per model (Template, TemplateVersion, TemplateSection, SectionVersion, EducationalContent, Attorney, WaitlistConfig/Entry/Log/Activation, AdminResource, PromoCode — ~30 total): copy DDB → Aurora; flip the <code>a.model()</code> definition; verify; merge per-model PR. Each PR independently reversible.</td><td><strong>Zero call-shape changes.</strong> Frontend never knows.</td></tr> | |
| <tr><td>Status-history writes early</td><td>Wire <code>subscription_status_history</code> + <code>payment_status_history</code> writes NOW. History only accrues from when you start writing — every week of delay is unrecoverable reporting history.</td><td>None</td></tr> | |
| <tr><td>Outbox</td><td>Create <code>outbox</code> table. Stripe webhook writes Mailchimp/SES side-effects to outbox in the same Postgres transaction. Outbox-worker Lambda drains pending rows every 30 s.</td><td>None</td></tr> | |
| <tr><td><ins class="rlsfix">Shared DDB→canonical transform</ins></td><td><ins class="rlsfix">June 2026 review fix (F7/Q8): ONE shared library of pure functions (DDB image → canonical rows, one per entity) is the only implementation of the decomposition — imported by BOTH the bulk ETL and the CDC catch-up consumer; CI forbids any other DDB→Aurora mapping path. Ships with a golden-test corpus harvested from a production table scan: null fields, empty arrays, partial blob updates, legacy attribute variants, items written during the Phase-2 window. CDC application is idempotent: deterministic upserts (<code>INSERT … ON CONFLICT (id) DO UPDATE</code>) guarded by the Streams sequence number (<code>last_stream_seq</code> column; apply only if newer), so at-least-once delivery and shard re-reads are no-ops.</ins></td><td><ins class="rlsfix">None</ins></td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout ok"> | |
| <div class="callout-label">Phase 2 gate</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>All ~12 non-PHI models successfully migrated; frontend tests pass unchanged.</li> | |
| <li>Status-history tables receiving live writes for ≥ 1 week.</li> | |
| <li>RLS verification matrix passes for every PHI/dispatcher table (policies are <em>authored</em> here; <em>driven</em> by the dispatcher in Phase 3). Non-PHI models migrated via Amplify connect-to-SQL are isolated by Amplify <code>@auth</code> — verified by per-model isolation tests + unchanged frontend tests (see §6 decision callout and the Authorization rows in the architecture-decision appendix).</li> | |
| <li>Outbox worker delivers at-least-once with idempotency proof.</li> | |
| <li><ins class="rlsfix">Shared transform library passes the golden corpus; CDC consumer proves duplicate-delivery is a no-op (review F7/Q8).</ins></li> | |
| <li>Drizzle migrations applied without manual intervention in dev-test & staging.</li> | |
| <li>Platform tables created; ZCTA seed loaded; member→trust assignment function unit-tested against fixtures.</li> | |
| </ul> | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 13 · PHASE 3 ============================================ --> | |
| <!-- =============================================================== --> | |
| <section id="phase-3"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 4 · 13</span> | |
| <h2 style="border:0;margin:18px 0 0;">Phase 3 — Custom dispatcher + multi-tenancy claim flow</h2> | |
| </div> | |
| <div class="callout authfirst"><div class="callout-label">Phase 3 re-scoped · June 2026 auth-first decision</div> | |
| The dispatcher build drops out of Phase 3. New Phase-3 core: <strong>migrate the <ins class="prdval">~7–10 typed-PHI (by field content) + the 6 JSON-blob models pinned to the protected path: User, CareDocumentNew, Document, UserPayment, UserSubscription, VerificationTokens</ins> PHI models via connect-to-SQL</strong> with field-level <code>@auth</code> per the Appendix C tier map (Tier-2 fields additionally behind <code>encryptField</code>/<code>decryptField</code>), apply the three §6.1b <code>@auth</code> fixes, land FF1 (API matrix, 720 assertions) + the canary, and keep the claim-flow work (customAuth, preTokenGen fail-closed) — it gates the API regardless of RLS. Dispatcher rows below are the deferred track.</div> | |
| <p>Goal: <strong>PHI path is ready</strong>. <ins class="authfirst">PHI models live behind field-level @auth;</ins> <s>Dispatcher Lambda built,</s> tenant claim flows end-to-end. Still no PHI in Aurora yet.</p> | |
| <table> | |
| <thead><tr><th>Workstream</th><th>Backend deliverable</th><th>Frontend impact</th></tr></thead> | |
| <tbody> | |
| <tr><td>customAuth Lambda authorizer <ins class="teamfix">(scope reduced)</ins></td><td><ins class="teamfix">Team review (Mike): Cognito verifies the JWT signature <strong>natively</strong> — no <code>aws-jwt-verify</code>. Under auth-first the AppSync userPool authorizer + <code>@auth</code> rules already enforce identity and the tenant/owner claims, so a custom authorizer is <strong>not required for the baseline</strong>.</ins> If kept (deferred-track tenant gating only): require the <code>trustCompanyId</code> claim present, fail-closed, surface tenant/role/externalId in <code>resolverContext</code>, <code>ttlOverride: 300</code> — it does not re-verify the token.</td><td>None (gate is server-side)</td></tr> | |
| <tr><td>preTokenGen claim injection</td><td>Inject <code>trustCompanyId</code> and <code>onboardingCompleted</code> as access-token claims. Remove the AppSync round-trip during token mint (denormalize via Cognito attribute).</td><td>Frontend reads <code>trustCompanyId</code> from access token via a thin helper</td></tr> | |
| <tr><td>Dispatcher Lambda skeleton</td><td>Build <code>amplify/data-layer/data-resolver/handler.ts</code>: <strong>non-VPC, Aurora Data API with IAM auth</strong>, BeginTransaction / set_config(SET LOCAL) / ExecuteStatement / CommitTransaction wrapper. Per-model handler files. Field-projection map.</td><td>None — dispatcher routes only to a.query()/a.mutation() entries that don't exist yet</td></tr> | |
| <tr><td>Field-level encryption</td><td>Design + build encrypt-field/decrypt-field Lambdas. Per-env CMK. AWS Encryption SDK envelope encryption. Encryption context binds ciphertext to userId. Caller authorization in decrypt handler. <code>kmsv1:</code> ciphertext marker.</td><td>Frontend gains thin wrapper <code>encryptField/decryptField</code> for all Tier-2 fields per Appendix C.1 — Care Doc Digital Assets (SSN, passwords, social/cloud account creds), Care Doc Financial (bank/investment/insurance account numbers), Care Doc Medical (advance directives, living will, healthcare proxy), Interview mental-health responses, plus standalone banking_info + insurance.policy_number. No UX change.</td></tr> | |
| <tr><td>Provisioned Concurrency</td><td>5 Lambdas at Phase 3 close: dispatcher (5 baseline, 20 max), preTokenGen (3), Stripe webhook (2), customAuth (3), postSignUpTrigger (1).</td><td>Visible UX win: P95 sign-in latency drops 1–3 s</td></tr> | |
| <tr><td>MEDICALREVIEW caseload</td><td>Build <code>medical_review_caseload</code> table; backfill from current ad-hoc assignments (manual triage). RLS policy restricts <code>medical_report</code> reads to caseload-scoped members.</td><td>Medical review list shortens to reviewer's caseload. <strong>Verify with the medical team before deploy.</strong></td></tr> | |
| <tr><td>MemberNote tightening</td><td>Auth rule joins <code>welon_assignment</code> — welon reads/writes notes only for assigned members.</td><td>Welon admin sees only assigned members' notes. Verify with welon team.</td></tr> | |
| <tr><td>Plaintext-PII column refactor (<ins class="prdval">14 tables — PRD validation; <strong>2 key on email (LoginAttempt/mfaAttempt partition key) so the hash is NOT a drop-in — re-key via a GSI/lookup, not in place</strong></ins>)</td><td>Migrate Logs, LoginAttempt, LoginHistory, mfaAttempt, DocumentEventLog, AgreementAcceptance, EvidenceSubmission, SupportRequest, UserInvite: replace plaintext email with <code>user_id</code> FK + <code>email_hash</code> + <code>email_last4</code>.</td><td>Support / admin screens that lookup by email continue via email_last4 + name. Lookup URL params change from email to user_id.</td></tr> | |
| <tr><td>Step-up MFA on sensitive ops</td><td>Sign document, delete user, role change, medical export require fresh MFA (token <code>auth_time</code> within 15 min).</td><td>Frontend MFA re-challenge modal (~50 lines). Handle freshness rejection.</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout ok"> | |
| <div class="callout-label">Phase 3 gate</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>Tenant-isolation verification matrix passing on dev-test for all 15 PHI tables.</li> | |
| <li>Sign-in P95 latency < 2 s end-to-end with PC enabled.</li> | |
| <li>Field-level encryption round-trip verified for Digital Assets credentials.</li> | |
| <li>MEDICALREVIEW caseload + MemberNote auth changes verified with respective teams.</li> | |
| <li>Plaintext-email migration complete on the <ins class="prdval">14</ins> tables (incl. the 2 email-keyed tables re-keyed, not hashed-in-place); email_last4 support-triage UX validated.</li> | |
| </ul> | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 14 · PHASE 4 ============================================ --> | |
| <!-- =============================================================== --> | |
| <section id="phase-4"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 4 · 14</span> | |
| <h2 style="border:0;margin:18px 0 0;">Phase 4 — PHI cutover</h2> | |
| </div> | |
| <p>Goal: <strong>PHI tables flip from DDB to Aurora in a coordinated window.</strong> Only "real cutover" — all other models migrated independently in Phase 2.</p> | |
| <table> | |
| <thead><tr><th>Step</th><th>Detail</th></tr></thead> | |
| <tbody> | |
| <tr><td><strong>T−2 weeks</strong></td><td>Dress rehearsal in dev-test with prod-shaped synthetic data. Time every step; rollback drilled (must complete < 1 h on dev-test → 4 h margin in prod).</td></tr> | |
| <tr><td><strong>T−1 week</strong></td><td>Deploy dispatcher Lambda to prod (dark — not yet routed). Deploy AppSync changes that route PHI resolvers to dispatcher (dark). Enable DynamoDB Streams on all PHI tables.</td></tr> | |
| <tr><td><strong>T−24h</strong></td><td>Final security checks. <code>chil_app</code> verified no BYPASSRLS. CMK rotations on. pgaudit flowing. Object Lock verified on archive bucket. Cognito attribute confirmed on every user. Stripe webhook temp-503 wiring prepped.</td></tr> | |
| <tr><td><strong>T+0 Fri 21:00 UTC</strong></td><td>Stripe + UPS webhooks routed to temporary 503 endpoint (Stripe retries; idempotent). Next.js middleware returns 503 except /health.</td></tr> | |
| <tr><td><strong>T+0:02</strong></td><td>S3 export of PHI DDB tables (PITR-based, no live throttle).</td></tr> | |
| <tr><td><strong>T+0:25</strong></td><td>Verify all exports succeeded. Capture timestamps.</td></tr> | |
| <tr><td><strong>T+0:30</strong></td><td>ETL into Aurora (Lambda fanout reads S3 export, transforms, bulk-loads via <code>COPY</code>). Largest tables: User, Document, LoginHistory.</td></tr> | |
| <tr><td><strong>T+1:30</strong></td><td>ETL complete. Row counts + per-table SHA-256 hashes match within tolerance.</td></tr> | |
| <tr><td><strong>T+1:40</strong></td><td>CDC catch-up: read DDB Streams from cutover-start; replay into Aurora<ins class="rlsfix"> — via the shared transform library with sequence-number-guarded idempotent upserts; duplicate shard delivery is a no-op (review F7)</ins>.</td></tr> | |
| <tr><td><strong>T+2:00</strong></td><td>Verification battery: RLS tests, random-sample integrity (100 users + docs), fixed 25-query battery. Both stores agree.</td></tr> | |
| <tr><td><strong>T+2:30</strong></td><td>Flip CFN parameter: AppSync PHI resolvers route to dispatcher. <strong>No frontend deploy needed</strong> — same endpoint, same schema.</td></tr> | |
| <tr><td><strong>T+2:45</strong></td><td>Read-only smoke: test member signs in, decodes JWT, loads dashboard, opens medical incident. Verify <code>trustCompanyId</code> claim flowed end-to-end.</td></tr> | |
| <tr><td><strong>T+3:00</strong></td><td>Lift maintenance mode. Webhooks re-enabled. Monitor: error rate, P95 latency, RLS denial rate (must be 0).</td></tr> | |
| <tr><td><strong>T+3:00 → T+4:00</strong></td><td>Rollback window. If criteria below trip — flip CFN parameter back to DDB resolvers. DDB tables remained hot the whole time. Invisible to clients.</td></tr> | |
| <tr><td><strong>T+4:00</strong></td><td>Cutover declared successful. DDB PHI tables remain in place 30 days (read-only), then deleted.</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout crit"> | |
| <div class="callout-label">Rollback decision criteria</div> | |
| <table style="margin:6px 0 0;"> | |
| <thead><tr><th>Signal</th><th>Threshold</th><th>Action</th></tr></thead> | |
| <tbody> | |
| <tr><td><ins class="rlsfix"><strong>Tenant-isolation canary</strong> — synthetic probe pair every 60 s: tenant-A token reads a seeded tenant-B row; member-A reads member-B's row in the same tenant. Both must return 0 rows.</ins></td><td><ins class="rlsfix">Any non-zero result</ins></td><td><ins class="rlsfix"><strong>Rollback immediately — PRIMARY trigger</strong> (June 2026 review fix, F3: a silent wrong-tenant read produces NO error; only a behavioral probe catches it)</ins></td></tr> | |
| <tr><td><ins class="rlsfix">GUC wiring errors — SQLSTATE 42704 (GUC never set) / 22P02 (malformed tenant value) in dispatcher logs</ins> <ins class="authfirst">(deferred with the dispatcher track)</ins></td><td><ins class="rlsfix">Any sustained</ins></td><td><ins class="rlsfix">Rollback — claim flow broken (F3)</ins></td></tr> | |
| <tr><td><ins class="rlsfix">Write-side</ins> RLS denial rate (SQLSTATE 42501<ins class="rlsfix"> — fires only on WITH CHECK violations; read-side USING filters silently and never raises it</ins>)</td><td>> 0.1%</td><td>Rollback — <ins class="rlsfix">write path</ins> broken</td></tr> | |
| <tr><td>AppSync 5xx rate</td><td>> 2% sustained 5 min</td><td>Rollback</td></tr> | |
| <tr><td>P95 dashboard load latency</td><td>> 3 s (was < 800 ms)</td><td>Investigate first; rollback if not resolved in 30 min</td></tr> | |
| <tr><td>Dispatcher error rate</td><td>> 1%</td><td>Rollback</td></tr> | |
| <tr><td>Sentry "tenant claim missing" events</td><td>Any</td><td>Rollback immediately — auth broken</td></tr> | |
| <tr><td>Aurora connection saturation</td><td>> 80%</td><td>Scale ACUs first; rollback if scaling fails</td></tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 15 · PHASE 5 ============================================ --> | |
| <!-- =============================================================== --> | |
| <section id="phase-5"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 4 · 15</span> | |
| <h2 style="border:0;margin:18px 0 0;">Phase 5 — Reporting + DR + closeout</h2> | |
| </div> | |
| <p>Goal: <strong>reporting on the reader, DR Drill recorded, audit evidence pack complete.</strong></p> | |
| <table> | |
| <thead><tr><th>Workstream</th><th>Backend deliverable</th><th>Frontend impact</th></tr></thead> | |
| <tbody> | |
| <tr><td>Reader endpoint live</td><td>Aurora reader reachable via the Data API using the reader endpoint ARN. <code>chil_reporter</code> reads via the reader; OLTP via the writer.</td><td>None</td></tr> | |
| <tr><td>Reporting page</td><td>Next.js Reporting page calls dispatcher in reporting mode. KPI queries against reader: MRR by cohort, document funnel, welon caseload, medical incident SLA, Stripe failure analysis. Decommission Apps Script prototype.</td><td>New Reporting page in <code>app/(protected)/admin/reports/</code>. Uses existing <code>client.queries.*</code> pattern.</td></tr> | |
| <tr><td>DR Drill</td><td>Parallel Aurora cluster from PITR snapshot in sibling VPC. Shadow AppSync at <code>dr.chil.welon</code>. Readiness probe battery (row counts ±1%, RLS tests, 25-query fidelity, cold-start). RTO/RPO recorded. Report filed in Object-Lock S3. Tear down.</td><td>None — prod is untouched</td></tr> | |
| <tr><td>Audit evidence pack</td><td>Compile: tested DR report (dated), tenant-isolation matrix sign-off, audit-log retention proof, encryption inventory, access-control matrix, BAA inventory, pen-test report, SAST/SCA/secret-scan reports, change-control PR evidence.</td><td>None</td></tr> | |
| <tr><td>Decommission old DDB</td><td>After 30-day stability: zero traffic to old PHI DDB tables; backup snapshot to S3 Object-Lock; delete.</td><td>None</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout ok"> | |
| <div class="callout-label">Phase 5 gate (closeout)</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>DR Drill report shows RTO < 4 h, RPO < 5 min, filed in S3.</li> | |
| <li>Reporting page replaces Apps Script prototype.</li> | |
| <li>Full evidence pack assembled; auditor walkthrough scheduled.</li> | |
| <li>Old DDB PHI tables deleted after 30-day stability.</li> | |
| <li>Two weeks of stable production with zero RLS denial alarms.</li> | |
| </ul> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <div class="part-banner devops"> | |
| <div class="part-num">Part 5 of 8 · DevOps deep review</div> | |
| <h2 style="border:0;color:var(--ink);">Every claim code-validated, every item with the exact command</h2> | |
| <p>33 deep-review items covering AWS account, IAM, VPC, KMS, Aurora, RDS Proxy, Postgres, DDB, Cognito, customAuth, Lambda VPC, PC, cold-start, Secrets, S3, audit-archive, CloudTrail, WAF, AppSync, CSP, CI/CD, dashboards, alarms, Sentry, backups, DR, cutover infra, rollback, decommission. Each follows the same template: CLAIM → CODE VALIDATION → DEEP THINKING → DEVOPS DELIVERABLE → VERIFICATION.</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== 16 · HOW TO READ ======================================== --> | |
| <!-- =============================================================== --> | |
| <section id="how-to-read"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 16</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">How to read DevOps items</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>Item template</h4> | |
| <div class="label">CLAIM</div> | |
| <p style="margin:0;">The assertion from the migration plan or industry best practice.</p> | |
| <div class="label">CODE VALIDATION</div> | |
| <p style="margin:0;">Direct quote or reference from <code>amplify/...</code> on current main. <span class="sev verified">VERIFIED</span> = matches plan; <span class="sev gap">GAP</span> = plan assumed it; <span class="sev new">NEW</span> = discovered in code, not in any prior review.</p> | |
| <div class="label crit">DEEP THINKING</div> | |
| <p style="margin:0;">Failure modes, edge cases, hidden assumptions, security implications, second-order effects.</p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <p style="margin:0;">Concrete action(s) Mike has to take. Where possible, the exact CLI command or CDK snippet.</p> | |
| <div class="label ok">VERIFICATION</div> | |
| <p style="margin:0;">How to prove the action is complete.</p> | |
| </div> | |
| <div class="callout devops"> | |
| <div class="callout-label">Severity legend</div> | |
| <span class="sev crit">CRITICAL</span> blocks Phase 1 if not done · | |
| <span class="sev high">HIGH</span> blocks PHI cutover (Phase 4) · | |
| <span class="sev med">MED</span> needed before Phase 5 closeout · | |
| <span class="sev verified">VERIFIED</span> · <span class="sev gap">GAP</span> ·</div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 17 · ACCOUNT ============================================ --> | |
| <!-- =============================================================== --> | |
| <section id="account"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 17</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">AWS account + organization</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>17.1 · AWS BAA confirmed for the account <span class="sev crit">CRITICAL · pre-Phase-1</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <p>AWS HIPAA-eligible services require a signed BAA between the AWS account owner and AWS. The BAA covers all listed services (Aurora, DynamoDB, Lambda, S3, KMS, CloudWatch, AppSync, Cognito, Secrets Manager, RDS Data API, etc.). Without it, every "we encrypt at rest with KMS" claim is non-compliant. <strong>An auditor will ask for the signed BAA before any technical control evidence.</strong></p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <ol style="margin:6px 0 0;"> | |
| <li>Confirm with AWS account owner (or via Artifact in the AWS Console): the BAA is signed and current.</li> | |
| <li>Download the signed BAA PDF; store in the audit-archive S3 bucket alongside other compliance evidence.</li> | |
| <li>List of services to verify on the BAA: Aurora Serverless v2 PostgreSQL, RDS Data API, DynamoDB, Lambda, S3, KMS, Secrets Manager, CloudWatch, AppSync, Cognito, CloudTrail, EventBridge, SES, IAM.</li> | |
| </ol> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>AWS Artifact UI shows BAA in "active" status. PDF in audit-archive S3 bucket.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>17.2 · AWS Organizations + SCPs (region restrict) <span class="sev high">HIGH</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>HIPAA "minimum necessary" applies to infrastructure. Restrict regions to <code>us-east-1</code> and <code>us-west-2</code> (DR region).</li> | |
| <li>Without SCPs, a misconfigured Lambda could spin up resources in a non-BAA-covered region.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="sh"><code>{ | |
| "Version": "2012-10-17", | |
| "Statement": [{ | |
| "Sid": "RestrictToUSRegions", "Effect": "Deny", | |
| "NotAction": ["cloudfront:*","iam:*","organizations:*","route53:*","support:*","sts:*","waf:*","wafv2:*"], | |
| "Resource": "*", | |
| "Condition": { "StringNotEqualsIfExists": { "aws:RequestedRegion": ["us-east-1","us-west-2"] } } | |
| }] | |
| }</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>Attempt to provision a resource in <code>eu-west-1</code> — must be denied.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>17.3 · Separate accounts per environment <span class="sev high">HIGH</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <p>HIPAA Administrative Safeguards require environmental separation. Confirm whether dev-test/staging/prod are separate AWS accounts. If one shared account: scope a follow-up project to split.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 18 · IDENTITY CENTER ==================================== --> | |
| <!-- =============================================================== --> | |
| <section id="identity-center"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 18</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">IAM Identity Center (human access)</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>18.1 · No long-lived IAM users for humans <span class="sev high">HIGH</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>HIPAA §164.312(a)(2)(i) requires Unique User ID. SOC 2 CC6 requires least-privilege. Long-lived IAM keys are anti-pattern.</li> | |
| <li>IAM Identity Center issues temporary credentials via OIDC. Each human gets SSO; permissions via permission sets.</li> | |
| <li>Permission sets: <strong>Developer</strong> (read-only on prod data, read-write on dev-test), <strong>DevOps</strong> (Mike — full admin), <strong>SecurityOfficer</strong> (audit-archive read, CloudTrail read), <strong>BreakGlassAdmin</strong> (production write; every action audited).</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <ol style="margin:6px 0 0;"> | |
| <li>Provision IAM Identity Center in the management account.</li> | |
| <li>Create 4 permission sets.</li> | |
| <li>Assign team members per-env.</li> | |
| <li>Deactivate existing long-lived IAM user access keys (audit via IAM Access Analyzer).</li> | |
| <li>Document break-glass procedure: who can use BreakGlassAdmin, what's logged, who notices.</li> | |
| </ol> | |
| <div class="label ok">VERIFICATION</div> | |
| <p><code>aws iam list-users | jq '.Users[] | select(.UserName != "amplify-cli-user")'</code> returns no human users.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 19 · VPC ================================================ --> | |
| <!-- =============================================================== --> | |
| <section id="vpc"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 19</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">VPC + network topology</h2> | |
| </div> | |
| <div class="callout high"><div class="callout-label">Scope reduced · June 2026 — no VPC needed for the DB path</div> | |
| Under the Data API decision, <strong>Aurora is reached over an IAM-authed HTTPS endpoint, not from inside a VPC</strong>. The subnet/NAT/security-group design below is no longer required for database access — keep a VPC only if some other workload needs one (none in this plan do). If you want private routing for the Data API, add a single Data API interface endpoint instead of the full VPC topology. The sizing notes below are retained for reference / future need.</div> | |
| <div class="review-block"> | |
| <h4>19.1 · VPC sizing and subnet layout <span class="sev crit">CRITICAL · Phase 1</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>2 AZs for Aurora multi-AZ. <em>(No RDS Proxy / Lambda VPC attach under the Data API — see banner above.)</em></li> | |
| <li>CIDR: /16 (65,536 IPs). Subnets: 2× private /20 (~4,096 IPs each), 2× public /24 for NAT.</li> | |
| <li><s>Dispatcher reserved-concurrency 100 — each ENI claims 1 private-subnet IP.</s> <em>Moot under the Data API — the dispatcher is non-VPC and allocates no ENIs. The VPC/subnet/NAT config below is needed only if some other workload requires a VPC (none in this plan do).</em></li> | |
| <li>Avoid the default VPC. Dedicated VPC per environment.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>const vpc = new ec2.Vpc(stack, `ChilVpc-${env}`, { | |
| ipAddresses: ec2.IpAddresses.cidr('10.40.0.0/16'), // dev-test: 10.40 / staging: 10.50 / prod: 10.60 | |
| maxAzs: 2, | |
| natGateways: 2, | |
| subnetConfiguration: [ | |
| { cidrMask: 20, name: 'private', subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS }, | |
| { cidrMask: 24, name: 'public', subnetType: ec2.SubnetType.PUBLIC }, | |
| ], | |
| enableDnsHostnames: true, | |
| enableDnsSupport: true, | |
| });</code></pre> | |
| </div> | |
| <div class="review-block"> | |
| <h4>19.2 · VPC Endpoints (avoid NAT costs) <span class="sev high">HIGH</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>Interface VPC endpoints are cheaper than NAT for frequent AWS API calls. <em>(Moot under the Data API — no NAT/endpoints needed for the DB path.)</em></li> | |
| <li>For frequent AWS API calls (KMS, Secrets), interface endpoints < NAT cost.</li> | |
| <li>Gateway endpoints (S3, DynamoDB) are FREE.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>vpc.addInterfaceEndpoint('SecretsEndpoint', { service: ec2.InterfaceVpcEndpointAwsService.SECRETS_MANAGER }); | |
| vpc.addInterfaceEndpoint('KmsEndpoint', { service: ec2.InterfaceVpcEndpointAwsService.KMS }); | |
| vpc.addInterfaceEndpoint('LogsEndpoint', { service: ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH_LOGS }); | |
| vpc.addInterfaceEndpoint('StsEndpoint', { service: ec2.InterfaceVpcEndpointAwsService.STS }); | |
| vpc.addGatewayEndpoint('S3Gateway', { service: ec2.GatewayVpcEndpointAwsService.S3 }); | |
| vpc.addGatewayEndpoint('DdbGateway', { service: ec2.GatewayVpcEndpointAwsService.DYNAMODB });</code></pre> | |
| </div> | |
| <div class="review-block"> | |
| <h4>19.3 · Security groups — deny-by-default <span class="sev high">HIGH</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <div class="callout high"><div class="callout-label">Superseded · Data API</div>There are <strong>no database security groups</strong> under the Data API — the dispatcher reaches Aurora over the IAM-authed HTTPS endpoint, not a VPC port. The 3-SG chain below applied only to the RDS-Proxy design.</div> | |
| <p><s>3 SGs: <code>chil-dispatcher-sg</code>, <code>chil-rdsproxy-sg</code>, <code>chil-aurora-sg</code>. Dispatcher → RDS Proxy → Aurora.</s></p> | |
| <div class="label ok">VERIFICATION</div> | |
| <p><strong>Data API verification:</strong> <code>aws rds-data execute-statement --resource-arn <cluster> --secret-arn <secret> --database chil --sql 'select 1'</code> succeeds with the dispatcher role's IAM identity; the same call from a role lacking <code>rds-data</code> permissions is denied.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 20 · KMS ================================================ --> | |
| <!-- =============================================================== --> | |
| <section id="kms"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 20</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">KMS keys — one per concern</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>20.1 · 4 keys per environment (12 total) <span class="sev crit">CRITICAL · Phase 1</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>HIPAA recommends customer-managed CMKs over AWS-managed default. One key per concern: | |
| <ol> | |
| <li><strong>Aurora CMK</strong> — encrypts cluster + snapshots + reader.</li> | |
| <li><strong>S3 PHI buckets CMK</strong> — existing storage + audit-archive bucket.</li> | |
| <li><strong>Secrets Manager CMK</strong> — encrypts Amplify <code>secret()</code> entries.</li> | |
| <li><strong>Field-encryption CMK</strong> — all Tier-2 fields per Appendix C.1 — Care Doc Digital Assets (SSN, passwords, social/cloud account creds), Care Doc Financial (bank/investment/insurance account numbers), Care Doc Medical (advance directives, living will, healthcare proxy), Interview mental-health responses, plus standalone banking_info + insurance.policy_number. Only encrypt-field/decrypt-field Lambdas have <code>kms:Decrypt</code>.</li> | |
| </ol> | |
| </li> | |
| <li>Existing <code>KMS_KEY_ID</code> for <code>customSmsSender</code> stays separate (Cognito service is the only principal).</li> | |
| <li>Multi-region: Aurora CMK + field-encryption CMK should be multi-region for cross-region DR snapshot copy.</li> | |
| <li>Annual rotation enabled on all 4.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>const auroraKey = new kms.Key(stack, `AuroraKey-${env}`, { alias: `alias/chil-aurora-${env}`, enableKeyRotation: true }); | |
| const s3PhiKey = new kms.Key(stack, `S3PhiKey-${env}`, { alias: `alias/chil-s3-phi-${env}`, enableKeyRotation: true }); | |
| const secretsKey = new kms.Key(stack, `SecretsKey-${env}`, { alias: `alias/chil-secrets-${env}`, enableKeyRotation: true }); | |
| const fieldEncKey = new kms.Key(stack, `FieldEncKey-${env}`, { alias: `alias/chil-field-encryption-${env}`, enableKeyRotation: true });</code></pre> | |
| </div> | |
| <div class="review-block"> | |
| <h4>20.2 · Field-encryption CMK key policy is THE security gate <span class="sev crit">CRITICAL</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <p>This is the difference between "PHI is encrypted" and "PHI is recoverable by anyone with DDB-table access." If the application's main IAM role can decrypt with this key, defense-in-depth collapses.</p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>// Allow encrypt-field + decrypt-field Lambdas only | |
| fieldEncKey.addToResourcePolicy(new iam.PolicyStatement({ | |
| effect: iam.Effect.ALLOW, | |
| principals: [ | |
| new iam.ArnPrincipal(backend.encryptField.resources.lambda.role!.roleArn), | |
| new iam.ArnPrincipal(backend.decryptField.resources.lambda.role!.roleArn), | |
| ], | |
| actions: ['kms:Encrypt','kms:Decrypt','kms:GenerateDataKey'], | |
| resources: ['*'], | |
| })); | |
| // Explicitly DENY the main application role | |
| fieldEncKey.addToResourcePolicy(new iam.PolicyStatement({ | |
| effect: iam.Effect.DENY, | |
| principals: [new iam.ArnPrincipal(backend.dataLayer.resources.lambda.role!.roleArn)], | |
| actions: ['kms:*'], resources: ['*'], | |
| }));</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>From dispatcher Lambda role: attempt <code>aws kms encrypt --key-id alias/chil-field-encryption-prod --plaintext test</code> — must be denied. From encrypt-field role — must succeed.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 21 · AURORA ============================================= --> | |
| <!-- =============================================================== --> | |
| <section id="aurora"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 21</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Aurora cluster — every parameter matters</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>21.1 · Engine version + serverless v2 capacity <span class="sev crit">CRITICAL · Phase 1</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>PostgreSQL 16 (latest LTS available on Aurora SLv2). Avoid 14.</li> | |
| <li>Capacity: writer 1–8 ACU, reader 0.5–4 ACU. 1 ACU ≈ 2 GB RAM + matched CPU.</li> | |
| <li>SLv2 doesn't fully pause; min ACU is the floor (0.5 ACU when idle).</li> | |
| <li>Snapshot retention: 30 days for HIPAA evidence. Long-tail to S3.</li> | |
| </ul> | |
| <div class="callout teamfix"><div class="callout-label">Team review · Mike correction — we do NOT provision Aurora</div> | |
| Mike provisions the Aurora cluster and <strong>provides the endpoint + connection secret</strong>. The CDK below is <strong>not a build step</strong> — it is the set of target parameters (engine version, capacity, encryption, PITR, pgaudit) to <em>confirm with Mike</em> against the cluster he stands up. Our deliverable is to consume the endpoint via Amplify <code>secret()</code>, not to deploy this stack.</div> | |
| <div class="label">DEVOPS DELIVERABLE <ins class="teamfix">(target parameters to confirm with Mike — not our build)</ins></div> | |
| <pre class="ts"><code>const cluster = new rds.DatabaseCluster(stack, `Aurora-${env}`, { | |
| engine: rds.DatabaseClusterEngine.auroraPostgres({ | |
| version: rds.AuroraPostgresEngineVersion.VER_16_4, | |
| }), | |
| vpc, | |
| vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS }, | |
| securityGroups: [auroraSecurityGroup], | |
| serverlessV2MinCapacity: env === 'prod' ? 1 : 0.5, | |
| serverlessV2MaxCapacity: env === 'prod' ? 8 : 4, | |
| writer: rds.ClusterInstance.serverlessV2('writer'), | |
| readers: [rds.ClusterInstance.serverlessV2('reader1', { scaleWithWriter: false })], | |
| storageEncryptionKey: auroraKey, | |
| deletionProtection: env === 'prod', | |
| backupRetention: cdk.Duration.days(env === 'prod' ? 30 : 7), | |
| parameters: { | |
| 'rds.force_ssl': '1', | |
| 'statement_timeout': '10000', | |
| 'shared_preload_libraries': 'pg_stat_statements,pgaudit', | |
| 'pgaudit.log': 'write, ddl', | |
| 'pgaudit.log_relation': '1', | |
| }, | |
| });</code></pre> | |
| </div> | |
| <div class="review-block"> | |
| <h4>21.2 · Storage tier — Standard vs I/O-Optimized <span class="sev med">MEDIUM</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <p>Standard tier charges per-GB storage + per-I/O. I/O-Optimized: ~30% ACU premium, no per-I/O charge. Crossover: when the I/O share exceeds ~30% of the ACU cost. <strong>Start on Standard; flip when needed.</strong></p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>21.3 · Performance Insights + pg_stat_statements <span class="sev high">HIGH</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <p>Without PI, "the dashboard is slow" is guesswork. pg_stat_statements is foundational for tuning.</p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>new rds.ClusterInstance.serverlessV2('writer', { | |
| enablePerformanceInsights: true, | |
| performanceInsightsRetention: rds.PerformanceInsightRetention.DAYS_7, | |
| performanceInsightEncryptionKey: auroraKey, | |
| });</code></pre> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 22 · RDS PROXY ========================================== --> | |
| <!-- =============================================================== --> | |
| <section id="rds-proxy"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 22</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">RDS Proxy — IAM-auth, two target groups</h2> | |
| </div> | |
| <div class="callout high"> | |
| <div class="callout-label">Superseded · June 2026 — optional under the Data API decision</div> | |
| Per the architecture update in the Executive Summary, the chosen connection path is the <strong>Aurora Data API + IAM</strong>, which needs no connection pooler — so an <strong>RDS Proxy is not required</strong>, and the VPC networking it depends on goes away. Keep this section only if a future, non-Data-API workload needs persistent pooled connections. The IAM-auth principle below still holds: under the Data API, access is IAM-gated on the dispatcher Lambda's execution role rather than via a proxy-brokered DB token. | |
| </div> | |
| <div class="review-block"> | |
| <h4>22.1 · IAM authentication, no static password <span class="sev crit">CRITICAL</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>RDS Proxy supports IAM-token auth: Lambda execution role gets <code>rds-db:connect</code>; token generated at connection time. <strong>No password stored anywhere.</strong></li> | |
| <li>HIPAA §164.312(d) Person/Entity Authentication. IAM-token auth > password auth.</li> | |
| <li>Token TTL: 15 min. Lambda connections never live that long; renewal automatic.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>const proxy = new rds.DatabaseProxy(stack, `RdsProxy-${env}`, { | |
| proxyTarget: rds.ProxyTarget.fromCluster(cluster), | |
| secrets: [cluster.secret!], | |
| vpc, | |
| iamAuth: true, | |
| securityGroups: [rdsProxySecurityGroup], | |
| requireTLS: true, | |
| idleClientTimeout: cdk.Duration.seconds(1800), | |
| borrowTimeout: cdk.Duration.seconds(30), | |
| maxConnectionsPercent: 90, | |
| });</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p><code>aws rds generate-db-auth-token --hostname <proxy-host> --port 5432 --username chil_app</code> returns a token. <code>psql "host=<proxy> port=5432 user=chil_app password=<token> dbname=chil sslmode=require"</code> succeeds.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>22.2 · Writer + reader target groups <span class="sev high">HIGH</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <p>Separate target groups isolate reporting load from OLTP. Reporting connects via reader target group using chil_reporter.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 23 · PG EXTENSIONS ====================================== --> | |
| <!-- =============================================================== --> | |
| <section id="pg-extensions"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 23</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Postgres extensions</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>23.1 · Required extensions</h4> | |
| <table> | |
| <thead><tr><th>Extension</th><th>Purpose</th><th>HIPAA tie-in</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>pgaudit</code></td><td>Logs SELECT/INSERT/UPDATE/DELETE/DDL on tagged tables → CloudWatch via Aurora log export</td><td>§164.312(b) Audit Controls</td></tr> | |
| <tr><td><code>pg_stat_statements</code></td><td>Aggregates query stats; foundational for perf tuning</td><td>SOC 2 CC4 Monitoring</td></tr> | |
| <tr><td><code>pgcrypto</code></td><td><code>gen_random_uuid()</code>, <code>digest()</code> — UUID PKs and email hashing</td><td>§164.312(a)(2)(iv) Encryption tooling</td></tr> | |
| <tr><td><code>aws_s3</code></td><td><code>aws_s3.query_export_to_s3()</code> — audit-event export + bulk reporting UNLOAD</td><td>Audit retention pipeline</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="sql"><code>CREATE EXTENSION IF NOT EXISTS pgaudit; | |
| CREATE EXTENSION IF NOT EXISTS pg_stat_statements; | |
| CREATE EXTENSION IF NOT EXISTS pgcrypto; | |
| CREATE EXTENSION IF NOT EXISTS aws_s3 CASCADE; | |
| ALTER SYSTEM SET pgaudit.log = 'write, ddl'; | |
| ALTER SYSTEM SET pgaudit.log_relation = on; | |
| ALTER SYSTEM SET pgaudit.log_statement = 'on'; | |
| -- Per-table audit (more granular) for PHI tables: | |
| ALTER TABLE "user" SET (pgaudit.log = 'all'); | |
| ALTER TABLE medical_record SET (pgaudit.log = 'all'); | |
| ALTER TABLE medical_incident SET (pgaudit.log = 'all'); | |
| ALTER TABLE member_note SET (pgaudit.log = 'all'); | |
| ALTER TABLE document SET (pgaudit.log = 'write');</code></pre> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 24 · PG ROLES =========================================== --> | |
| <!-- =============================================================== --> | |
| <section id="pg-roles"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 24</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Postgres roles + grants</h2> | |
| </div> | |
| <div class="callout authfirst"><div class="callout-label">Role set at cutover · June 2026 auth-first decision</div> | |
| At cutover only three principals touch Postgres: the <strong>managed SQL Lambda's connection user</strong> (<code>chil_sqlconnect</code> via <code>SQL_CONN_STRING</code> — least-privilege grants on exactly the migrated tables), <code>chil_migrator</code> (CI DDL), and <code>chil_admin</code> (break-glass, full compensating controls below). <code>chil_app</code>, <code>chil_owner</code>, <code>chil_reporter</code> belong to the deferred RLS track — created then, not now.</div> | |
| <div class="callout"> | |
| <div class="callout-label">Plain-language: these are Postgres-internal roles, not AWS IAM roles</div> | |
| <code>chil_app</code>, <code>chil_sqlconnect</code>, <code>chil_reporter</code>, <code>chil_admin</code>, and | |
| <code>chil_migrator</code> are created with <strong>SQL <code>CREATE ROLE</code> statements inside the database</strong> | |
| (not in AWS IAM Identity Center). They're database accounts. Each Lambda that connects to Postgres picks one of these | |
| roles by name in its connection string. The Lambda authenticates with IAM (via the Data API), but the database itself | |
| sees a Postgres-internal role and applies the role's GRANTs + RLS policies. <strong>IAM gets you to the connection; | |
| the Postgres role determines what you can do once connected.</strong> | |
| </div> | |
| <div class="review-block"> | |
| <h4>24.1 · <ins class="rlsfix">Six</ins> roles, one purpose each</h4> | |
| <table> | |
| <thead><tr><th>Role</th><th>Used by</th><th>BYPASSRLS</th><th>Grants</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>chil_app</code></td><td>Dispatcher Lambda (PHI path)</td><td><strong>No</strong></td><td>SELECT/INSERT/UPDATE/DELETE on all PHI tables</td></tr> | |
| <tr><td><code>chil_sqlconnect</code></td><td>Amplify-managed SQL Lambda</td><td>No</td><td>NON-PHI tables only</td></tr> | |
| <tr><td><code>chil_reporter</code></td><td>Dispatcher (reporting mode, reader)</td><td>No</td><td>SELECT-only on views</td></tr> | |
| <tr><td><code>chil_admin</code></td><td>Break-glass via Identity Center temp creds</td><td><strong>Yes</strong></td><td>All</td></tr> | |
| <tr><td><code>chil_migrator</code></td><td>CI/CD only (Drizzle migrations)</td><td>Yes (for DDL)</td><td>DDL only; no DML</td></tr> | |
| <tr><td><ins class="rlsfix"><code>chil_owner</code></ins></td><td><ins class="rlsfix">Nobody — NOLOGIN table owner (June 2026 review fix, Q3)</ins></td><td><ins class="rlsfix">No</ins></td><td><ins class="rlsfix">Owns all tables; FORCE RLS keeps owner-context access fenced</ins></td></tr> | |
| </tbody> | |
| </table> | |
| <div class="label crit">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>chil_sqlconnect physically cannot see PHI — this is the security boundary that makes the hybrid (Amplify SQL + dispatcher) safe.</li> | |
| <li>chil_admin's BYPASSRLS is dangerous; every session must be audited via pgaudit; CloudWatch Insights alarm on usage.</li> | |
| <li>chil_migrator without DML — data fixes via chil_admin with explicit audit.</li> | |
| </ul> | |
| <div class="label">COMPENSATING CONTROLS for chil_admin (HIPAA insider-threat mitigation)</div> | |
| <ol style="margin:6px 0 0;"> | |
| <li><strong>No persistent credentials.</strong> chil_admin login only via IAM Identity Center temp credentials (1 h TTL). Long-lived password disabled.</li> | |
| <li><strong>Approval gate.</strong> Break-glass use requires an opened Jira ticket with incident reference; SRE peer ack in Slack before the session opens. Documented in role comment (<code>COMMENT ON ROLE chil_admin IS '…'</code>).</li> | |
| <li><strong>Highest-PHI fields stay encrypted to chil_admin.</strong> SSN, medical directives, banking secrets are envelope-encrypted with the field-encryption CMK; chil_admin has DB read access but cannot decrypt without separate KMS grant (Tier-2 protection — see Appendix C).</li> | |
| <li><strong>Real-time alert.</strong> Every <code>chil_admin</code> session triggers CloudWatch Insights → SNS → Slack to #security-alerts within 60 s (alarm in §42).</li> | |
| <li><strong>Quarterly review.</strong> All chil_admin sessions in the previous quarter reviewed by Security Officer; report filed in audit-archive S3.</li> | |
| </ol> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="sql"><code>CREATE ROLE chil_app LOGIN; | |
| CREATE ROLE chil_sqlconnect LOGIN; | |
| CREATE ROLE chil_reporter LOGIN; | |
| CREATE ROLE chil_admin LOGIN BYPASSRLS; | |
| CREATE ROLE chil_migrator LOGIN; | |
| -- June 2026 review fix (Q3): dedicated table owner. NOLOGIN, no BYPASSRLS — | |
| -- with FORCE ROW LEVEL SECURITY, even owner-context access is fenced. | |
| -- Every migration ends with: ALTER TABLE <new_table> OWNER TO chil_owner; | |
| CREATE ROLE chil_owner NOLOGIN; | |
| GRANT USAGE ON SCHEMA public TO chil_app; | |
| GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO chil_app; | |
| ALTER DEFAULT PRIVILEGES IN SCHEMA public | |
| GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO chil_app; | |
| GRANT USAGE ON SCHEMA public TO chil_sqlconnect; | |
| GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO chil_sqlconnect; | |
| REVOKE ALL ON "user" FROM chil_sqlconnect; | |
| REVOKE ALL ON medical_record FROM chil_sqlconnect; | |
| REVOKE ALL ON medical_incident FROM chil_sqlconnect; | |
| REVOKE ALL ON medical_report FROM chil_sqlconnect; | |
| REVOKE ALL ON member_note FROM chil_sqlconnect; | |
| REVOKE ALL ON document FROM chil_sqlconnect; | |
| REVOKE ALL ON support_request FROM chil_sqlconnect; | |
| REVOKE ALL ON support_message FROM chil_sqlconnect; | |
| REVOKE ALL ON support_attachment FROM chil_sqlconnect; | |
| REVOKE ALL ON evidence_submission FROM chil_sqlconnect; | |
| REVOKE ALL ON care_document FROM chil_sqlconnect; | |
| REVOKE ALL ON care_document_section FROM chil_sqlconnect; | |
| REVOKE ALL ON medical_note FROM chil_sqlconnect; | |
| REVOKE ALL ON emergency_contact FROM chil_sqlconnect; | |
| REVOKE ALL ON incapacitation_event FROM chil_sqlconnect;</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <pre class="sql"><code>SET ROLE chil_sqlconnect; | |
| SELECT count(*) FROM "user"; -- must ERROR with 42501 (permission denied) | |
| RESET ROLE;</code></pre> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 25 · DDB STREAMS ======================================== --> | |
| <!-- =============================================================== --> | |
| <section id="ddb-streams"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 25</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">DDB Streams + PITR + deletion protection</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>25.1 · Enable Streams before cutover <span class="sev crit">CRITICAL</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <p>No Streams configured today. Phase 4 cutover plan requires them.</p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>for (const DDBtable of Object.values(amplifyDynamoDbTables)) { | |
| DDBtable.deletionProtectionEnabled = env === 'prod'; | |
| DDBtable.pointInTimeRecoveryEnabled = true; | |
| DDBtable.streamSpecification = { | |
| streamViewType: 'NEW_AND_OLD_IMAGES', | |
| }; | |
| }</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <pre class="sh"><code>for table in $(aws dynamodb list-tables --query "TableNames[]" --output text); do | |
| echo -n "$table: " | |
| aws dynamodb describe-table --table-name "$table" --query "Table.StreamSpecification.StreamEnabled" --output text | |
| done | |
| # All must print "True"</code></pre> | |
| </div> | |
| <div class="review-block"> | |
| <h4>25.2 · Deletion protection — currently OFF <span class="sev crit">CRITICAL</span></h4> | |
| <p>5-minute fix. <code>backend.ts:180</code>: <code>deletionProtectionEnabled = false</code>. Flip env-aware (false in dev-test, true in staging + prod).</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 26 · COGNITO ============================================ --> | |
| <!-- =============================================================== --> | |
| <section id="cognito"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 26</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Cognito — the coordination problem</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>26.1 · The external-pool reality <span class="sev crit">CRITICAL</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <p><code>amplify/auth/resource.ts</code> uses <code>referenceAuth()</code> with env-driven pool IDs. Amplify does NOT own the pools.</p> | |
| <div class="label crit">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>Pool-owner is a person Mike has to coordinate with. Required changes per pool × 3 envs: | |
| <ol> | |
| <li>Add custom attribute <code>custom:trustCompanyId</code> (string, <strong>admin-mutable</strong> — reassignment is a product feature; the <code>trust_company</code> DB row is the source of truth and this attribute is a token-mint cache).</li> | |
| <li>Add custom attribute <code>custom:onboardingCompleted</code> (string "true"/"false", mutable).</li> | |
| <li>Confirm password policy meets HIPAA: min 12 chars, mixed case, numbers, symbols.</li> | |
| <li>Confirm access token TTL (1 hr OK; 24 hr is too long).</li> | |
| <li>Confirm advanced security features (compromised credentials, adaptive auth).</li> | |
| </ol> | |
| </li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <ol style="margin:6px 0 0;"> | |
| <li>Find the pool owner.</li> | |
| <li>Coordinate add of 2 custom attributes per pool.</li> | |
| <li>Get pool config (password policy, TTLs, advanced security) into audit-archive S3.</li> | |
| <li>Write + run backfill script per env: | |
| <pre class="ts"><code>// scripts/backfill-cognito-tenant.ts | |
| import { CognitoIdentityProviderClient, ListUsersCommand, AdminUpdateUserAttributesCommand } | |
| from '@aws-sdk/client-cognito-identity-provider'; | |
| const SEED_TENANT = '00000000-0000-4000-8000-000000000001'; | |
| const c = new CognitoIdentityProviderClient({}); | |
| let token: string | undefined; | |
| let processed = 0; | |
| do { | |
| const list = await c.send(new ListUsersCommand({ | |
| UserPoolId: process.env.USER_POOL_ID!, | |
| Limit: 60, PaginationToken: token, | |
| })); | |
| for (const u of list.Users ?? []) { | |
| const hasAttr = u.Attributes?.some(a => a.Name === 'custom:trustCompanyId'); | |
| if (hasAttr) continue; | |
| await c.send(new AdminUpdateUserAttributesCommand({ | |
| UserPoolId: process.env.USER_POOL_ID!, | |
| Username: u.Username!, | |
| UserAttributes: [{ Name: 'custom:trustCompanyId', Value: SEED_TENANT }], | |
| })); | |
| processed++; | |
| } | |
| token = list.PaginationToken; | |
| } while (token); | |
| console.log(`Backfilled ${processed} users`);</code></pre> | |
| </li> | |
| <li>Run backfill on each of 3 envs <strong>before</strong> shipping fail-closed preTokenGen change.</li> | |
| </ol> | |
| <div class="label ok">VERIFICATION</div> | |
| <p><code>aws cognito-idp list-users --user-pool-id ... --query "Users[?!Attributes[?Name=='custom:trustCompanyId']]"</code> returns empty.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 27 · CUSTOM AUTH ======================================== --> | |
| <!-- =============================================================== --> | |
| <section id="customauth"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 27</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">customAuth Lambda — from stub to gate</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>27.1 · Fill in the passthrough stub <span class="sev high">HIGH</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <p><code>amplify/functions/custom-auth/handler.ts</code> currently returns <code>{ isAuthorized: true }</code> unconditionally.</p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>import type { AppSyncAuthorizerHandler } from 'aws-lambda'; | |
| import { CognitoJwtVerifier } from 'aws-jwt-verify'; | |
| const verifier = CognitoJwtVerifier.create({ | |
| userPoolId: process.env.USER_POOL_ID!, | |
| tokenUse: 'access', | |
| clientId: process.env.USER_POOL_CLIENT_ID!, | |
| }); | |
| type ResolverContext = { | |
| trustCompanyId: string; | |
| externalId: string; | |
| groups: string; | |
| }; | |
| export const handler: AppSyncAuthorizerHandler<ResolverContext> = async event => { | |
| try { | |
| const token = event.authorizationToken; | |
| if (!token) return { isAuthorized: false }; | |
| const claims = await verifier.verify(token); | |
| const trustCompanyId = claims['trustCompanyId'] as string | undefined; | |
| const externalId = claims['externalId'] as string | undefined; | |
| const groups = (claims['cognito:groups'] as string[] | undefined) ?? []; | |
| if (!trustCompanyId) return { isAuthorized: false }; | |
| if (!externalId) return { isAuthorized: false }; | |
| return { | |
| isAuthorized: true, | |
| resolverContext: { trustCompanyId, externalId, groups: groups.join(',') }, | |
| ttlOverride: 300, | |
| }; | |
| } catch (err) { | |
| console.log('customAuth denied:', (err as Error).message); | |
| return { isAuthorized: false }; | |
| } | |
| };</code></pre> | |
| <div class="label">env vars to add in backend.ts</div> | |
| <pre class="ts"><code>backend.customAuth.addEnvironment('USER_POOL_ID', process.env.CFLegacy_UserPoolID!); | |
| backend.customAuth.addEnvironment('USER_POOL_CLIENT_ID', process.env.CFLegacy_userPoolClientID!);</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <ul> | |
| <li>Forged JWT (different signing key) → 401.</li> | |
| <li>Expired JWT → 401.</li> | |
| <li>Valid JWT missing trustCompanyId → 401.</li> | |
| <li>Valid full JWT → 200 with response.</li> | |
| </ul> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 28 · LAMBDA VPC ========================================= --> | |
| <!-- =============================================================== --> | |
| <section id="lambda-vpc"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 28</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Lambda VPC attachment — 0 of 76 (Data API)</h2> | |
| </div> | |
| <div class="callout ok"><div class="callout-label">Revised · June 2026 — Data API removes VPC attachment entirely</div> | |
| With the Aurora Data API, the Lambdas that touch the database reach it over an IAM-authed HTTPS endpoint — <strong>none of them need VPC attachment</strong>. The dispatcher, encrypt/decrypt-field, and outbox-worker all use the Data API (or KMS, which is also a public AWS endpoint). The original "4 in VPC" design below assumed RDS Proxy; it is superseded. Net: <strong>0 VPC Lambdas</strong>, no ENI-cliff exposure, no NAT for the DB path.</div> | |
| <div class="review-block"> | |
| <h4>28.1 · Which Lambdas go in VPC <span class="sev crit">CRITICAL · Phase 3</span></h4> | |
| <div class="label">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li><strong>Under the Data API: none.</strong> The DB-touching Lambdas authenticate via IAM and call the Data API endpoint over HTTPS — no VPC, no ENIs, no NAT. | |
| <s>The RDS-Proxy design would have put 4 in VPC: dispatcher, encrypt-field, decrypt-field, outbox-worker.</s></li> | |
| <li>0 VPC Lambdas → 0 ENIs for DB access. The ENI 250/region soft-limit is a non-issue.</li> | |
| <li>All 76 Lambdas stay non-VPC. The dispatcher talks to the Data API; the rest talk to AppSync, as today.</li> | |
| <li><em>Optional:</em> a Data API <strong>VPC interface endpoint</strong> can be added later if you want the dispatcher's traffic to stay on private AWS networking — but it is not required for the architecture to work.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>// Data API path: NO vpcConfig, NO RDS Proxy. The dispatcher reaches Aurora over | |
| // the IAM-authed HTTPS Data API endpoint. Grant the execution role the rds-data | |
| // actions on the cluster + read on the credentials secret: | |
| backend.dispatcher.resources.lambda.addToRolePolicy(new iam.PolicyStatement({ | |
| actions: ['rds-data:BeginTransaction', 'rds-data:ExecuteStatement', | |
| 'rds-data:CommitTransaction', 'rds-data:RollbackTransaction'], | |
| resources: [auroraCluster.clusterArn], | |
| })); | |
| backend.dispatcher.resources.lambda.addToRolePolicy(new iam.PolicyStatement({ | |
| actions: ['secretsmanager:GetSecretValue'], | |
| resources: [auroraSecret.secretArn], | |
| })); | |
| backend.dispatcher.resources.cfnResources.cfnFunction.reservedConcurrentExecutions = 100; | |
| // reservedConcurrency caps Aurora-bound concurrency; no subnet/SG/vpcConfig needed.</code></pre> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 29 · PC ================================================= --> | |
| <!-- =============================================================== --> | |
| <section id="pc"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 29</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Provisioned Concurrency — 5 hot Lambdas</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>29.1 · The hot-path Lambdas <span class="sev high">HIGH</span></h4> | |
| <table> | |
| <thead><tr><th>Lambda</th><th>Why PC</th><th>Baseline</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>dataLayer</code> (dispatcher)</td><td>Sees 100% of PHI traffic; cold start = visible UX hit</td><td>5 (autoscale to 20)</td></tr> | |
| <tr><td><code>preTokenGeneration</code></td><td>Cognito 5 s budget; every token mint</td><td>3</td></tr> | |
| <tr><td><code>processWebhookV2</code> (Stripe)</td><td>Stripe retries at > 10 s</td><td>2</td></tr> | |
| <tr><td><code>postSignUpTrigger</code></td><td>5 s timeout; cold start = signup failure</td><td>2</td></tr> | |
| <tr><td><code>custom-auth</code></td><td>On the critical path for every AppSync call once Lambda mode is the gate</td><td>3</td></tr> | |
| <tr style="background:var(--accent-soft);font-weight:700;"><td><strong>PC subtotal</strong></td><td>5 Lambdas</td><td>15 baseline</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>backend.dataLayer.resources.cfnResources.cfnFunction.addPropertyOverride( | |
| 'ProvisionedConcurrencyConfig', { ProvisionedConcurrentExecutions: 5 } | |
| );</code></pre> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 30 · COLD START ========================================= --> | |
| <!-- =============================================================== --> | |
| <section id="cold-start"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 30</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Cold-start mitigation beyond PC</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>30.1 · Lazy-import Sentry (50 Lambdas) <span class="sev high">HIGH</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <p>50 handlers import Sentry at module scope; <code>Sentry.init</code> runs on every cold start.</p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>// Pattern: lazy import only on error path; ~200 ms cold-start savings | |
| let Sentry: typeof import('@sentry/aws-serverless') | null = null; | |
| async function reportError(err: Error) { | |
| if (!Sentry) { | |
| Sentry = await import('@sentry/aws-serverless'); | |
| Sentry.init({ | |
| dsn: process.env.SENTRY_DSN!, | |
| environment: process.env.SENTRY_ENV, | |
| tracesSampleRate: 0.1, | |
| enabled: ['production','staging','test'].includes(process.env.SENTRY_ENV!), | |
| beforeSend(event) { return scrubPhi(event); }, | |
| }); | |
| } | |
| Sentry.captureException(err); | |
| } | |
| export const handler = async (event) => { | |
| try { | |
| // ... handler logic ... | |
| } catch (err) { | |
| await reportError(err as Error); | |
| throw err; | |
| } | |
| };</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>CloudWatch Lambda Insights <code>init_duration</code> P95 should drop by ~200 ms after this refactor lands.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 31 · SECRETS ============================================ --> | |
| <!-- =============================================================== --> | |
| <section id="secrets"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 31</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Secrets — Amplify <code>secret()</code> <ins class="teamfix">(not AWS Secrets Manager)</ins></h2> | |
| </div> | |
| <div class="callout teamfix"><div class="callout-label">Team review · Mike correction</div> | |
| AWS Secrets Manager is <strong>not in use</strong>. Amplify Gen 2 provides its own secret management via <code>secret()</code> (stored in SSM Parameter Store, encrypted with the env CMK). All app/vendor secrets below are Amplify <code>secret()</code> entries. The deferred dispatcher track’s <code>AURORA_SECRET_ARN</code> / <code>secretsmanager:GetSecretValue</code> references are blueprint-only and not part of the auth-first cutover — under the actual environment the managed SQL Lambda’s connection string is an Amplify <code>secret()</code> (<code>SQL_CONN_STRING</code>), and Mike supplies the Aurora endpoint + credential (see §21).</div> | |
| <div class="review-block"> | |
| <h4>31.1 · Current secret inventory <span class="sev verified">VERIFIED</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <ul> | |
| <li>STRIPE_SECRET_KEY (Stripe webhook + admin-activate-subscription)</li> | |
| <li>STRIPE_WEBHOOK_SECRET (Stripe webhook)</li> | |
| <li>MAILCHIMP_API_KEY + MAILCHIMP_SERVER_PREFIX + MAILCHIMP_LIST_ID (Stripe webhook + processSignUp)</li> | |
| <li>UPS_CLIENT_SECRET (create-label + track-package + generateShippingLabel)</li> | |
| <li>GOOGLE_API_KEY + GOOGLE_API_SECRET (findAddress + getPlaceDetails + validateAddress)</li> | |
| <li>FAX_CLIENT_ID + FAX_CLIENT_SECRET + FAX_JWT_TOKEN (sendFax)</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <ol style="margin:6px 0 0;"> | |
| <li>Encrypt all Amplify secrets with <code>chil-secrets-cmk</code> CMK.</li> | |
| <li>Quarterly rotation for STRIPE_SECRET_KEY, MAILCHIMP_API_KEY, UPS_CLIENT_SECRET, GOOGLE_API_KEY, FAX_CLIENT_SECRET.</li> | |
| <li>Document rotation procedure (where each vendor lets you rotate).</li> | |
| </ol> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 32 · S3 PATHS =========================================== --> | |
| <!-- =============================================================== --> | |
| <section id="s3-paths"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 32</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">S3 path policies — extend with tenant prefix</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>32.1 · 6 prefixes need <code>tenants/{trustCompanyId}/</code> wrapping <span class="sev high">HIGH · Phase 3/4</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <pre class="ts"><code>'public/evidence/*' // ADMIN, WELONTRUST, owner | |
| 'public/documents/*' // ADMIN, WELONTRUST, owner | |
| 'public/care-documents/{identity}/*' // ADMIN, owner | |
| 'public/personal-id/*' // ADMIN, WELONTRUST, owner | |
| 'public/support-requests/{identity}/*' // ADMIN, owner | |
| 'public/template-sections/*' // ADMIN</code></pre> | |
| <div class="label crit">DEEP THINKING</div> | |
| <p>Amplify storage access rules can't directly read a custom JWT claim into a path token. The <code>{trustCompanyId}</code> segment must be enforced by upload/download broker Lambdas that build the key from the verified tenant claim, never from client input.</p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>// Upload broker (new) builds key from VERIFIED claims | |
| export async function POST(req: Request) { | |
| const { trustCompanyId, externalId } = await getServerSideJwtClaims(); | |
| const { fileName, contentType } = await req.json(); | |
| if (!/^[\w\-.]+$/.test(fileName)) return new Response('bad filename', { status: 400 }); | |
| const key = `tenants/${trustCompanyId}/documents/${externalId}/${Date.now()}-${fileName}`; | |
| const client = new S3Client({}); | |
| const url = await getSignedUrl(client, new PutObjectCommand({ | |
| Bucket: process.env.STORAGE_BUCKET!, | |
| Key: key, | |
| ContentType: contentType, | |
| ServerSideEncryption: 'aws:kms', | |
| SSEKMSKeyId: process.env.S3_PHI_KMS_KEY_ARN!, | |
| }), { expiresIn: 300 }); | |
| return Response.json({ url, key }); | |
| }</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>From a tenant-A user, attempt to upload to <code>tenants/<tenant-B-id>/...</code> via crafted PUT — must fail.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 33 · AUDIT BUCKET ======================================= --> | |
| <!-- =============================================================== --> | |
| <section id="audit-bucket"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 33</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">S3 audit-archive — Object Lock Compliance mode</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>33.1 · 7-year Compliance lock <span class="sev crit">CRITICAL · Phase 1</span></h4> | |
| <div class="label crit">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>HIPAA requires 6-year retention. Use Object Lock Compliance mode (strictly immutable; root account can't delete).</li> | |
| <li>Compliance mode vs Governance mode: <strong>Compliance</strong>.</li> | |
| <li>Retention 7 years (1 year safety margin).</li> | |
| <li>Glacier transition after 180 days; Deep Archive after 730 days.</li> | |
| <li>Versioning enabled (required for Object Lock).</li> | |
| <li>KMS: dedicated chil-audit-archive CMK; deny encryption by application roles.</li> | |
| <li>Block all public access — no exceptions.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>const auditBucket = new s3.Bucket(stack, `AuditArchive-${env}`, { | |
| bucketName: `chil-audit-archive-${env}-${account}`, | |
| encryption: s3.BucketEncryption.KMS, | |
| encryptionKey: auditKey, | |
| versioned: true, | |
| objectLockEnabled: true, | |
| objectLockDefaultRetention: s3.ObjectLockRetention.compliance({ | |
| duration: cdk.Duration.days(365 * 7), | |
| }), | |
| blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL, | |
| enforceSSL: true, | |
| lifecycleRules: [{ | |
| id: 'AuditArchiveGlacier', | |
| transitions: [ | |
| { storageClass: s3.StorageClass.GLACIER_INSTANT_RETRIEVAL, transitionAfter: cdk.Duration.days(180) }, | |
| { storageClass: s3.StorageClass.DEEP_ARCHIVE, transitionAfter: cdk.Duration.days(730) }, | |
| ], | |
| }], | |
| removalPolicy: cdk.RemovalPolicy.RETAIN, | |
| });</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p><code>aws s3api get-object-lock-configuration --bucket chil-audit-archive-prod-...</code> returns Mode=COMPLIANCE, Days=2555. Attempt <code>aws s3 rm</code> — must fail with AccessDenied.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 34 · CLOUDTRAIL ========================================= --> | |
| <!-- =============================================================== --> | |
| <section id="cloudtrail"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 34</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">CloudTrail — org trail + S3 data events</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>34.1 · Organization trail with PHI bucket data events <span class="sev crit">CRITICAL · Phase 1</span></h4> | |
| <div class="label crit">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>Management Events always on (free). Data Events (S3 GetObject/PutObject, KMS Decrypt) are charged but mandatory for HIPAA.</li> | |
| <li>Org Trail aggregates across accounts. Single-account: account trail.</li> | |
| <li>Log File Validation — hourly hash-signed log files; auditors love this.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>const trail = new cloudtrail.Trail(stack, 'OrgTrail', { | |
| bucket: auditBucket, | |
| s3KeyPrefix: 'cloudtrail/', | |
| includeGlobalServiceEvents: true, | |
| isMultiRegionTrail: true, | |
| enableFileValidation: true, | |
| encryptionKey: auditKey, | |
| }); | |
| trail.addS3EventSelector([ | |
| { bucket: backend.storage.resources.bucket }, | |
| { bucket: backend.storage.resources.bucket, objectPrefix: 'tenants/' }, | |
| ], { readWriteType: cloudtrail.ReadWriteType.ALL });</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>Test S3 PutObject; within 15 min audit-archive bucket has CloudTrail log showing the API call. <code>aws cloudtrail validate-logs</code> returns OK.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 35 · WAF ================================================ --> | |
| <!-- =============================================================== --> | |
| <section id="waf"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 35</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">WAF v2 in front of AppSync</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>35.1 · WAF Web ACL — OWASP + rate limit + optional geo-block <span class="sev high">HIGH · Phase 1</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <p>No WAF in <code>backend.ts</code> today.</p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>const webAcl = new wafv2.CfnWebACL(stack, `WebAcl-${env}`, { | |
| scope: 'REGIONAL', | |
| defaultAction: { allow: {} }, | |
| visibilityConfig: { sampledRequestsEnabled: true, cloudWatchMetricsEnabled: true, metricName: 'chil-waf' }, | |
| rules: [ | |
| { | |
| name: 'AWS-AWSManagedRulesCommonRuleSet', priority: 1, | |
| overrideAction: { none: {} }, | |
| statement: { managedRuleGroupStatement: { vendorName: 'AWS', name: 'AWSManagedRulesCommonRuleSet' } }, | |
| visibilityConfig: { sampledRequestsEnabled: true, cloudWatchMetricsEnabled: true, metricName: 'AWSManagedRulesCommonRuleSetMetric' }, | |
| }, | |
| { | |
| name: 'RateLimit', priority: 2, | |
| action: { block: {} }, | |
| statement: { rateBasedStatement: { limit: 2000, aggregateKeyType: 'IP' } }, | |
| visibilityConfig: { sampledRequestsEnabled: true, cloudWatchMetricsEnabled: true, metricName: 'RateLimit' }, | |
| }, | |
| ], | |
| }); | |
| new wafv2.CfnWebACLAssociation(stack, 'AppSyncWafAssociation', { | |
| resourceArn: backend.data.resources.graphqlApi.arn, | |
| webAclArn: webAcl.attrArn, | |
| });</code></pre> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 36 · APPSYNC ============================================ --> | |
| <!-- =============================================================== --> | |
| <section id="appsync"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 36</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">AppSync hardening</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>36.1 · Disable introspection + depth limit + access logs <span class="sev high">HIGH</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <p>Currently nothing — <code>data/resource.ts</code> has no <code>introspection</code> or <code>maxDepth</code>.</p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>const cfnGraphQLApi = backend.data.resources.cfnResources.cfnGraphqlApi; | |
| cfnGraphQLApi.introspectionConfig = env === 'prod' ? 'DISABLED' : 'ENABLED'; | |
| cfnGraphQLApi.queryDepthLimit = 10; | |
| cfnGraphQLApi.resolverCountLimit = 75; | |
| cfnGraphQLApi.logConfig = { | |
| fieldLogLevel: 'ERROR', | |
| excludeVerboseContent: true, | |
| cloudWatchLogsRoleArn: appSyncLogRole.roleArn, | |
| };</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>Unauth'd <code>POST /graphql {"query": "{ __schema { types { name } } }"}</code> — must 400 in prod.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 37 · CSP ================================================ --> | |
| <!-- =============================================================== --> | |
| <section id="csp"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 37</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">CSP + HTTP headers (already strong)</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>37.1 · Preserve existing implementation <span class="sev verified">VERIFIED</span></h4> | |
| <p><code>middleware.ts:1-50</code> implements strong CSP with per-request nonce. <code>next.config.mjs</code> adds HSTS (1 year), X-Content-Type-Options, X-Frame-Options DENY, X-XSS-Protection, Referrer-Policy.</p> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <p>None — preserve. Test via <code>https://securityheaders.com</code>; should score A+.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 38 · CI/CD ============================================== --> | |
| <!-- =============================================================== --> | |
| <section id="cicd"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 38</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">CI/CD security gates</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>38.1 · SAST, SCA, secret-scan, IaC scan <span class="sev high">HIGH · Phase 1</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <p><code>amplify.yml</code> has no security gates today.</p> | |
| <div class="label crit">DEEP THINKING</div> | |
| <p>SOC 2 CC8 requires every change pass automated controls. Tools:</p> | |
| <ul style="margin:6px 0 0;"> | |
| <li><strong>Semgrep</strong> (SAST) — rules for TS/Node; free Community tier</li> | |
| <li><strong>pnpm audit</strong> (SCA, dependency vulns) — free, built in</li> | |
| <li><strong>Snyk</strong> (deeper SCA + license check) — free OSS tier or paid</li> | |
| <li><strong>TruffleHog</strong> (secret-scan) — free; catches AWS/Stripe/etc. keys</li> | |
| <li><strong>checkov</strong> (IaC for CDK) — free</li> | |
| <li><ins class="rlsfix"><strong>RLS policy lint</strong> (June 2026 review fix, F1/F2/Q3) — a CI step queries <code>pg_policies</code>/<code>pg_tables</code> on the migration container and fails unless every PHI table has: <code>relforcerowsecurity = true</code>, owner = <code>chil_owner</code>, exactly one RESTRICTIVE tenant fence, and no <code>FOR ALL</code> permissive policy. It also extracts every GUC name referenced in policy expressions and asserts the dispatcher's <code>set_config</code> list covers it — the F2 name drift can never ship again.</ins></li> | |
| <li><ins class="rlsfix"><strong>Guard greps</strong> (review Q5) — fail the build on: <code>missing_ok := true</code> inside any policy expression (a missing GUC must error, never silently match nothing); any <code>ExecuteStatementCommand</code> constructed outside the dispatcher's transaction wrapper; any session-scoped <code>set_config(…, false)</code>.</ins></li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="yaml"><code>backend: | |
| phases: | |
| preBuild: | |
| commands: | |
| - npm install -g pnpm@10.33.4 semgrep | |
| - echo "Running SAST..." | |
| - semgrep --config p/typescript --error --strict | |
| - echo "Running SCA..." | |
| - pnpm audit --audit-level=high | |
| - echo "Running secret scan..." | |
| - npx -y trufflehog filesystem . --no-update --fail | |
| - echo "Running IaC scan..." | |
| - pip install checkov && checkov -d amplify/ --quiet --soft-fail-on LOW | |
| build: | |
| commands: | |
| # ... existing build</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>Push a PR with a hardcoded API key — CI must fail at secret-scan. PR with a vulnerable dependency — CI must fail at pnpm audit.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 39 · DRIZZLE RUNNER ===================================== --> | |
| <!-- =============================================================== --> | |
| <section id="drizzle-runner"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 39</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Migration runner — Drizzle Kit + CI integration</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>39.1 · CI applies migrations on deploy <span class="sev high">HIGH · Phase 1</span></h4> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="yaml"><code># amplify.yml (in backend phase) | |
| - echo "Running migration dry-run in throwaway container..." | |
| - docker run -d --name pg-test -e POSTGRES_PASSWORD=test -p 5433:5432 postgres:16-alpine | |
| - sleep 3 | |
| - DATABASE_URL=postgresql://postgres:test@localhost:5433/postgres npx drizzle-kit migrate | |
| - docker stop pg-test && docker rm pg-test | |
| # Actual deploy: | |
| - export DB_HOST=$(aws ssm get-parameter --name /chil/${ENV}/db-host --query "Parameter.Value" --output text) | |
| - export DB_TOKEN=$(aws rds generate-db-auth-token --hostname $DB_HOST --port 5432 --username chil_migrator) | |
| - DATABASE_URL="postgresql://chil_migrator:${DB_TOKEN}@${DB_HOST}:5432/chil?sslmode=require" npx drizzle-kit migrate</code></pre> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 40 · YAML BUG =========================================== --> | |
| <!-- =============================================================== --> | |
| <section id="yaml-bug"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 40</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">amplify.yml fixes (the smart-quote bug)</h2> | |
| </div> | |
| <p>5-minute fix per D1:</p> | |
| <pre class="sh"><code>sed -i '' 's/“/"/g; s/”/"/g' amplify.yml | |
| git diff amplify.yml | |
| git commit -m "Fix smart-quote bug in amplify.yml"</code></pre> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 41 · DASHBOARDS ========================================= --> | |
| <!-- =============================================================== --> | |
| <section id="dashboards"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 41</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">CloudWatch dashboards — 3 panels</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>41.1 · Data plane / Request plane / Compliance plane <span class="sev high">HIGH · Phase 1</span></h4> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>const dataPlane = new cw.Dashboard(stack, `DataPlane-${env}`, { | |
| dashboardName: `chil-data-plane-${env}`, | |
| }); | |
| dataPlane.addWidgets( | |
| new cw.GraphWidget({ title: 'Dispatcher Invocations', | |
| left: [backend.dataLayer.resources.lambda.metricInvocations()] }), | |
| new cw.GraphWidget({ title: 'Dispatcher Duration P95', | |
| left: [backend.dataLayer.resources.lambda.metricDuration({ statistic: 'p95' })] }), | |
| new cw.GraphWidget({ title: 'Aurora ACUs', | |
| left: [new cw.Metric({ namespace: 'AWS/RDS', metricName: 'ServerlessDatabaseCapacity', | |
| dimensionsMap: { DBClusterIdentifier: cluster.clusterIdentifier } })] }), | |
| new cw.GraphWidget({ title: 'Aurora Connections', | |
| left: [new cw.Metric({ namespace: 'AWS/RDS', metricName: 'DatabaseConnections', | |
| dimensionsMap: { DBClusterIdentifier: cluster.clusterIdentifier } })] }), | |
| ); | |
| // Repeat for requestPlane (AppSync) and compliancePlane (pgaudit, CloudTrail events)</code></pre> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 42 · ALARMS ============================================= --> | |
| <!-- =============================================================== --> | |
| <section id="alarms"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 42</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Alarms inventory — what pages on-call</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>Alarm</th><th>Metric</th><th>Threshold</th><th>Severity</th></tr></thead> | |
| <tbody> | |
| <tr><td><ins class="rlsfix">Tenant-isolation canary failure</ins></td><td><ins class="rlsfix">Synthetic probe pair (cross-tenant + intra-tenant read must return 0 rows) — every 60 s during cutover window, every 5 min steady-state</ins></td><td><ins class="rlsfix">Any non-zero</ins></td><td><ins class="rlsfix">Page (PRIMARY isolation signal — review F3)</ins></td></tr> | |
| <tr><td><ins class="rlsfix">GUC wiring errors</ins></td><td><ins class="rlsfix">CloudWatch metric filter on SQLSTATE 42704 / 22P02 in dispatcher logs</ins></td><td><ins class="rlsfix">Any sustained 5 min</ins></td><td><ins class="rlsfix">Page</ins></td></tr> | |
| <tr><td><ins class="rlsfix">Write-side</ins> RLS denial rate</td><td>Sentry "RLS denial" OR CloudWatch on SQLSTATE 42501 from dispatcher logs<ins class="rlsfix"> (WITH CHECK violations only — read-side USING never raises 42501; review F3)</ins></td><td>> 0.1% sustained 5 min</td><td>Page</td></tr> | |
| <tr><td>Tenant claim missing</td><td>Sentry "missing trustCompanyId"</td><td>Any</td><td>Page</td></tr> | |
| <tr><td>preTokenGen error rate</td><td>Lambda <code>Errors</code></td><td>> 0.5%</td><td>Page</td></tr> | |
| <tr><td>Cognito 5 s timeout near miss</td><td>preTokenGen <code>Duration</code> P95</td><td>> 4000 ms sustained 5 min</td><td>Alert</td></tr> | |
| <tr><td>Dispatcher error rate</td><td>Lambda <code>Errors</code></td><td>> 1% sustained 5 min</td><td>Page</td></tr> | |
| <tr><td>Aurora ACU saturation</td><td><code>ServerlessDatabaseCapacity</code></td><td>at max for 10 min</td><td>Alert</td></tr> | |
| <tr><td>Aurora connection saturation</td><td><code>DatabaseConnections</code></td><td>> 80% of max</td><td>Alert</td></tr> | |
| <tr><td>Data API access denied / throttle</td><td>rds-data <code>4xxError</code> / <code>ThrottlingException</code></td><td>> 5 in 5 min</td><td>Page</td></tr> | |
| <tr><td>Stripe webhook 5xx rate</td><td>processWebhookV2 <code>Errors</code></td><td>> 2% sustained</td><td>Page</td></tr> | |
| <tr><td>Lambda init duration P95</td><td>Lambda Insights <code>init_duration</code></td><td>> 2000 ms</td><td>Alert</td></tr> | |
| <tr><td>WAF block rate spike</td><td>WAF <code>BlockedRequests</code></td><td>> 100 / min</td><td>Alert</td></tr> | |
| <tr><td>CloudTrail log file integrity failure</td><td>CloudTrail <code>LogFileValidationFailure</code></td><td>Any</td><td>Page</td></tr> | |
| <tr><td>chil_admin (break-glass) used</td><td>CloudWatch Insights on pgaudit for chil_admin sessions</td><td>Any non-scheduled use</td><td>Alert</td></tr> | |
| <tr><td>Cost anomaly</td><td>AWS Cost Anomaly Detection</td><td>Configured per service</td><td>Alert</td></tr> | |
| <tr><td>Backup failure</td><td>RDS BackupRetentionPeriod / snapshot creation missing</td><td>Daily check</td><td>Alert</td></tr> | |
| <tr><td>15-min Lambdas near timeout</td><td>Per-function <code>Duration</code> (5 Lambdas at 900s)</td><td>> 720000 ms</td><td>Page</td></tr> | |
| <tr><td>Data API request latency</td><td>rds-data <code>ExecuteStatement</code> latency P95</td><td>> 500 ms sustained 5 min</td><td>Alert</td></tr> | |
| <tr><td>Aurora storage growth trajectory</td><td>Aurora <code>VolumeBytesUsed</code> derivative</td><td>> 1 GB/day sustained 14 days</td><td>Alert</td></tr> | |
| <tr><td>Data API throttling</td><td>rds-data <code>ThrottlingException</code> rate</td><td>> 0 sustained 5 min</td><td>Page</td></tr> | |
| <tr><td>DDB Streams lag (pre-cutover)</td><td>DDB Streams <code>IteratorAge</code> on CDC consumer</td><td>> 5 min sustained</td><td>Page during Phase 4 only</td></tr> | |
| <tr><td>Aurora max_connections approaching</td><td>Aurora <code>DatabaseConnections</code> / max</td><td>> 80% sustained 5 min</td><td>Alert</td></tr> | |
| </tbody> | |
| </table> | |
| <p><strong>Severity convention:</strong> Page = wake on-call (PagerDuty). Alert = Slack + email. <strong>Total alarms: <ins class="rlsfix">23 (two added by the June 2026 PHI/RLS review — canary + GUC wiring)</ins>.</strong></p> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 43 · SENTRY ============================================= --> | |
| <!-- =============================================================== --> | |
| <section id="sentry"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 43</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Sentry hardening</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>43.1 · Three config files, three gaps <span class="sev crit">CRITICAL · Phase 1</span></h4> | |
| <div class="label">CODE VALIDATION</div> | |
| <ul> | |
| <li><code>sentry.server.config.ts</code>: <code>tracesSampleRate: 1</code>, no <code>sendDefaultPii</code>, no <code>beforeSend</code>, <code>enableLogs: true</code>.</li> | |
| <li><code>sentry.edge.config.ts</code>: same.</li> | |
| <li><code>instrumentation-client.ts</code>: <code>sendDefaultPii: false</code> ✓, but <code>tracesSampleRate: 1</code>.</li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <pre class="ts"><code>// sentry.server.config.ts AND sentry.edge.config.ts | |
| import * as Sentry from '@sentry/nextjs'; | |
| const PHI_KEYS = new Set([ | |
| 'email','phone','phoneNumber','firstName','lastName','middleName', | |
| 'birthdate','dateOfBirth','clientDateOfBirth','ssn','address', | |
| 'addressLine1','addressLine2','medicalNotes','medicalConditions', | |
| 'hospitalName','hospitalAddress','hospitalFax','hospitalEmail', | |
| 'contactPhone','contactFirstName','contactLastName', | |
| ]); | |
| function scrub(value: unknown): unknown { | |
| if (value == null || typeof value === 'string') return value; | |
| if (Array.isArray(value)) return value.map(scrub); | |
| if (typeof value === 'object') { | |
| const out: Record<string, unknown> = {}; | |
| for (const [k, v] of Object.entries(value as object)) { | |
| out[k] = PHI_KEYS.has(k) ? '[REDACTED]' : scrub(v); | |
| } | |
| return out; | |
| } | |
| return value; | |
| } | |
| Sentry.init({ | |
| dsn: process.env.NEXT_PUBLIC_SENTRY_DSN, | |
| environment: process.env.SENTRY_ENV ?? 'development', | |
| enabled: ['production','staging','test'].includes(process.env.SENTRY_ENV ?? ''), | |
| tracesSampleRate: 0.1, | |
| sendDefaultPii: false, | |
| enableLogs: false, | |
| beforeSend(event) { | |
| if (event.extra) event.extra = scrub(event.extra) as typeof event.extra; | |
| if (event.contexts) event.contexts = scrub(event.contexts) as typeof event.contexts; | |
| if (event.request?.data) event.request.data = scrub(event.request.data); | |
| event.breadcrumbs?.forEach(b => { if (b.data) b.data = scrub(b.data) as typeof b.data; }); | |
| return event; | |
| }, | |
| beforeBreadcrumb(crumb) { | |
| if (crumb.data) crumb.data = scrub(crumb.data) as typeof crumb.data; | |
| return crumb; | |
| }, | |
| });</code></pre> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>Trigger an exception with PHI in context — Sentry event shows <code>[REDACTED]</code>.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 44 · BACKUPS ============================================ --> | |
| <!-- =============================================================== --> | |
| <section id="backups"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 44</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Backup strategy</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>Resource</th><th>Hot retention</th><th>Long-tail</th></tr></thead> | |
| <tbody> | |
| <tr><td>Aurora cluster</td><td>30 days continuous PITR</td><td>Daily snapshot → S3 audit-archive 6+ years</td></tr> | |
| <tr><td>Aurora reader</td><td>Same as cluster (shared storage)</td><td>—</td></tr> | |
| <tr><td>DDB tables (5 remaining)</td><td>35-day PITR</td><td>AWS Backup → S3 6+ years</td></tr> | |
| <tr><td>S3 PHI buckets</td><td>Versioning enabled</td><td>Glacier 180d; Deep Archive 730d</td></tr> | |
| <tr><td>S3 audit-archive</td><td>Object Lock 7-year</td><td>Same bucket; lifecycle to Deep Archive 730d</td></tr> | |
| <tr><td>Secrets Manager</td><td>Versioning (default)</td><td>—</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 45 · DR DRILL =========================================== --> | |
| <!-- =============================================================== --> | |
| <section id="dr-drill"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 45</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">DR drill mechanics</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>45.1 · Quarterly drill procedure <span class="sev high">HIGH · Phase 5 + recurring</span></h4> | |
| <ol style="margin:6px 0 0;"> | |
| <li>Pick PITR T (e.g. T−1 h).</li> | |
| <li>Restore Aurora to parallel cluster in sibling VPC.</li> | |
| <li>Shadow AppSync at <code>dr.chil.welon</code>.</li> | |
| <li>Clone dispatcher Lambda pointing at restored Aurora.</li> | |
| <li>Run readiness probe battery (row counts ±1%, RLS tests, 25-query fidelity, cold-start P95 < 3 s).</li> | |
| <li>Record RTO + RPO.</li> | |
| <li>File report in audit-archive S3 with date.</li> | |
| <li>Tear down shadow.</li> | |
| </ol> | |
| <div class="label ok">VERIFICATION</div> | |
| <p>Quarterly drill report shows consistent RTO < 4 h, RPO < 5 min over rolling 4 quarters.</p> | |
| <div class="label">GOVERNANCE</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li><strong>Owner:</strong> the DevOps/Security lead (named in the runbook) runs each drill; an ops lead signs off the report.</li> | |
| <li><strong>Calendar:</strong> scheduled quarterly (first business week of Jan/Apr/Jul/Oct); dates tracked in the compliance calendar.</li> | |
| <li><strong>Sign-off:</strong> each dated, signed report is filed in the Object-Lock S3 audit archive — it is the SOC 2 CC7.5 evidence; a missed quarter is a logged exception.</li> | |
| <li><strong>Phase-5 acceptance:</strong> the first drill (Phase 5) must produce a filed, signed report meeting RTO < 4 h / RPO < 5 min before program exit.</li> | |
| </ul> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 46 · CUTOVER INFRA ====================================== --> | |
| <!-- =============================================================== --> | |
| <section id="cutover-infra"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 46</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Cutover-temporary infrastructure</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>Item</th><th>When provisioned</th><th>When torn down</th></tr></thead> | |
| <tbody> | |
| <tr><td>DynamoDB Streams</td><td>T−1 week</td><td>After 30-day stability post-cutover</td></tr> | |
| <tr><td>CDC catch-up Lambda</td><td>T−1 week</td><td>After 30-day stability</td></tr> | |
| <tr><td>S3 PITR export staging bucket</td><td>T−2 days</td><td>After cutover + retention window</td></tr> | |
| <tr><td>ETL Lambdas (S3 → Aurora COPY)</td><td>T−1 week</td><td>After cutover successful</td></tr> | |
| <tr><td>Maintenance-mode middleware toggle</td><td>T−1 day</td><td>T+3:00</td></tr> | |
| <tr><td>Stripe webhook temp-503 route</td><td>T−15 min</td><td>T+3:00</td></tr> | |
| <tr><td>Shadow AppSync for smoke</td><td>T−1 week</td><td>T+24 h after stability</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 47 · ROLLBACK =========================================== --> | |
| <!-- =============================================================== --> | |
| <section id="rollback"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 47</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Rollback mechanics</h2> | |
| </div> | |
| <div class="review-block"> | |
| <h4>47.1 · The 4-hour rollback path</h4> | |
| <div class="label">DEEP THINKING</div> | |
| <ul style="margin:6px 0 0;"> | |
| <li>Rollback = flip the CFN parameter routing PHI resolvers from dispatcher back to DDB. <strong>Single parameter change.</strong></li> | |
| <li>Frontend never changed; rollback invisible to clients except brief 503 during maintenance-mode.</li> | |
| <li>"Leave DDB hot for cutover weekend" guarantee makes 4 h achievable. PITR alone cannot restore 41 tables in 4 hours.</li> | |
| <li>Data delta: writes between T+2:30 and rollback time live only in Aurora. Replay these into DDB via one-shot Lambda before declaring rollback complete. <ins class="rlsfix"><strong>June 2026 review fix (Q9):</strong> the replay Lambda is <em>built in Phase 3</em> alongside the dispatcher and <em>executed in both dress rehearsals</em> — flip back to DDB resolvers, run the replay against the rehearsal's Aurora writes, assert row-level parity. Nothing on the rollback path is exercised for the first time in production.</ins></li> | |
| </ul> | |
| <div class="label">DEVOPS DELIVERABLE</div> | |
| <p>Pre-write rollback runbook with exact CLI commands. Print it. Tape it to Mike's wall.</p> | |
| </div> | |
| </section> | |
| <hr class="devops"> | |
| <!-- =============================================================== --> | |
| <!-- ===== 48 · DECOMMISSION ======================================= --> | |
| <!-- =============================================================== --> | |
| <section id="decommission"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">DevOps · 48</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Decommission plan (T+30 days)</h2> | |
| </div> | |
| <ol> | |
| <li>Confirm zero traffic to old PHI DDB tables for 30 consecutive days.</li> | |
| <li>Final snapshot of each old DDB table; copy to S3 audit-archive with 7-year retention.</li> | |
| <li>Delete DDB tables (flip deletion protection off first, with explicit audit-log entry).</li> | |
| <li>Tear down cutover-temp infrastructure.</li> | |
| <li>Update DevOps runbook.</li> | |
| </ol> | |
| </section> | |
| <hr> | |
| <div class="part-banner"> | |
| <div class="part-num">Part 6 of 8</div> | |
| <h2 style="border:0;color:var(--ink);">Frontend — exact deltas, no surprises</h2> | |
| <p>The frontend is NOT rewritten. <code>generateClient<Schema>()</code> works. The changes below are mechanical — mostly renames or shifts from JSON-blob reads to typed child-table reads. <strong>~84–120 files touched</strong> (revised after code-grounded audit; ~1500–2500 LOC total). Plus <strong>~45 Lambda functions</strong> that also call GSI-style methods need the same refactor on the backend side.</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== 49 · FRONTEND CATALOG =================================== --> | |
| <!-- =============================================================== --> | |
| <section id="frontend-catalog"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 6 · 49</span> | |
| <h2 style="border:0;margin:18px 0 0;">Frontend change catalog</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>Change</th><th>Files (frontend)</th><th>Call sites</th><th>Effort</th><th>Phase</th></tr></thead> | |
| <tbody> | |
| <tr><td><strong>GSI-method refactor</strong> — every <code>listXByY</code> call (e.g. <code>listDocumentByUserId</code>, <code>listUserByCognitoId</code>) switches to <code>list({ filter: {...} })</code>. GSIs disappear; Postgres indexes replace them.</td><td>11 frontend + 45 Lambda</td><td><strong>149 call sites · 50 distinct methods</strong></td><td>Mechanical search-replace per method; verify pushdown logic</td><td>2 (non-PHI models) + 3 (PHI models)</td></tr> | |
| <tr><td>Rename <code>user.cognitoId</code> → <code>user.externalId</code> (auth-critical — touches <code>AuthContext</code>)</td><td>15</td><td>32</td><td>Global rename + verify auth integration</td><td>3</td></tr> | |
| <tr><td>Status field collapse: <code>status</code>/<code>membershipStatus</code>/<code>paymentsStatus</code> → <code>lifecycle_state</code> + <code>billing_state</code></td><td>26</td><td>~80</td><td>Map enum values (UserStatus, MembershipStatusEnum, PaymentsStatusEnum). One PR per surface area</td><td>3</td></tr> | |
| <tr><td>JSON-blob fields → typed child tables: <code>interviewProgressV2</code>, <code>peopleLibrary</code>, <code>quarterlyReviewData</code>, <code>documentationStatus</code></td><td>11</td><td>64</td><td>Switch each <code>JSON.parse(...)</code> → typed <code>client.models.X.list/get(...)</code>. Remove double-encoding workarounds.</td><td>3</td></tr> | |
| <tr><td><strong>Care document section components</strong> — each section becomes its own <code>care_document_section</code> row. The current single <code>CareDocumentNew.data[]</code> array splits into per-section queries.</td><td><strong>36 components</strong></td><td>~150</td><td>Per-component: <code>client.models.CareDocumentSection.list({ careDocumentId, sectionType })</code>. Largest single category.</td><td>3</td></tr> | |
| <tr><td>Welon arrays → <code>welon_assignment</code> lookups (on User, Document, ShippingLabel, CareDocumentNew)</td><td>7</td><td>14</td><td>Switch <code>user.assignedWelonTrustIds.includes(...)</code> → <code>welonAssignments.find(wa => ...)</code></td><td>3</td></tr> | |
| <tr><td>Support thread — messages as child entities</td><td>~3</td><td>~10</td><td>Switch from embedded array to <code>client.models.SupportMessage.list(...)</code></td><td>3</td></tr> | |
| <tr><td>MEDICALREVIEW screens — caseload filter</td><td>~3</td><td>~5</td><td>Server-enforced via RLS. UI may add "your caseload (N members)" hint.</td><td>3</td></tr> | |
| <tr><td>MemberNote screens — welon-assignment filter</td><td>~3</td><td>~5</td><td>Server-enforced. UI may show "assigned to you" label.</td><td>3</td></tr> | |
| <tr><td>Step-up MFA challenge modal</td><td>~2 new</td><td>~5</td><td>On <code>NeedsFreshMFA</code> error: modal, MFA code, retry mutation (~50 lines)</td><td>3</td></tr> | |
| <tr><td>Field-encryption wrappers in Care Doc Digital Assets, Financial, Medical-section, plus User.ssn, banking, insurance</td><td>~6</td><td>~20</td><td>Read: <code>decryptField</code> mutation per Tier-2 field (see Appendix C.1). Write: <code>encryptField</code>. React state stays plaintext.</td><td>3</td></tr> | |
| <tr><td>JWT claim helper</td><td>~1 new</td><td>n/a</td><td><code>useTenantId()</code> hook reads <code>trustCompanyId</code> from access token</td><td>3</td></tr> | |
| <tr><td>Pagination UX improvements (optional)</td><td>~5</td><td>~10</td><td>Replace nextToken-based "infinite scroll until empty" with cursor pagination. Now predictable.</td><td>5</td></tr> | |
| <tr><td>Reporting page (new)</td><td>~5 new</td><td>~15</td><td>Replaces Apps Script prototype. Server components query dispatcher via AppSync. ~10 KPI queries.</td><td>5</td></tr> | |
| <tr style="background:var(--accent-soft);font-weight:700;"><td><strong>Subtotal</strong></td><td><strong>84–120 files</strong></td><td><strong>~540 call sites</strong></td><td><strong>~1500–2500 LOC mechanical</strong></td><td><strong>2 + 3 + 5</strong></td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout ok"> | |
| <div class="callout-label">Net frontend impact</div> | |
| <strong>~84–120 files touched across frontend + ~45 Lambda functions.</strong> Mostly mechanical renames or shifts from JSON-blob reads to typed child-table | |
| reads. <strong>Zero files require a "rewrite."</strong> Spread across Phase 3 (most) and Phase 5 (reporting page). No | |
| frontend cutover needed at PHI cutover — frontend code is forward-compatible with both old and new schemas during the | |
| transition. | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 50 · FRONTEND NO-OP ===================================== --> | |
| <!-- =============================================================== --> | |
| <section id="frontend-noop"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 6 · 50</span> | |
| <h2 style="border:0;margin:18px 0 0;">What does NOT change in the frontend</h2> | |
| </div> | |
| <ul> | |
| <li><strong>Authentication flow</strong> — Cognito sign-in, MFA, password reset all unchanged.</li> | |
| <li><strong>CSP middleware with per-request nonce</strong> — unchanged.</li> | |
| <li><strong>Sentry init</strong> — server-side scrubber/sample-rate are config-only; client init unchanged.</li> | |
| <li><strong><code>generateClient<Schema>()</code></strong> — unchanged; <code>Schema</code> just has more honestly-named fields.</li> | |
| <li><strong>Routing</strong> — <code>app/</code> structure unchanged.</li> | |
| <li><strong>UI component library</strong> — unchanged.</li> | |
| <li><strong>HTTP security headers</strong> — HSTS, X-Frame-Options, etc. unchanged.</li> | |
| <li><strong>S3 upload paths</strong> — unchanged until multi-tenancy prefix lands in Phase 4/5.</li> | |
| <li><strong>Stripe Elements, Botpress widget, Facebook SDK, Vimeo embeds</strong> — third-party integrations unchanged.</li> | |
| </ul> | |
| </section> | |
| <hr> | |
| <div class="part-banner"> | |
| <div class="part-num">Part 7 of 8</div> | |
| <h2 style="border:0;color:var(--ink);">Security & compliance — every control mapped to a phase</h2> | |
| <p>HIPAA Safeguards (§164.308/.310/.312) and SOC 2 Trust Service Criteria (CC1–CC9 + P/C) — which phase delivers each control. Plus the pre-cutover hard requirements and the auditor evidence pack assembled at Phase 5 close.</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== 51 · HIPAA ============================================== --> | |
| <!-- =============================================================== --> | |
| <section id="hipaa"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 7 · 51</span> | |
| <h2 style="border:0;margin:18px 0 0;">HIPAA Safeguards — controls landing schedule</h2> | |
| </div> | |
| <h3>51.1 — Administrative Safeguards (§164.308)</h3> | |
| <table> | |
| <thead><tr><th>Citation</th><th>Standard</th><th>How</th><th>Phase</th></tr></thead> | |
| <tbody> | |
| <tr><td>§164.308(a)(1)</td><td>Security Management Process — risk analysis</td><td>This + companion docs ARE the risk analysis; quarterly review thereafter</td><td>0; recurring</td></tr> | |
| <tr><td>§164.308(a)(2)</td><td>Assigned Security Responsibility</td><td>Named Security Officer (Mike); Data Architect (Oleksii)</td><td>0</td></tr> | |
| <tr><td>§164.308(a)(3)</td><td>Workforce Security</td><td>Cognito groups + canonical role enum + <code>deleteUserComplete</code> Lambda</td><td>1 (reaffirmed)</td></tr> | |
| <tr><td>§164.308(a)(4)</td><td>Information Access Management — minimum necessary</td><td>Tenant boundary on every row; RLS policies; MEDICALREVIEW caseload; MemberNote welon-scoping; field-level encryption</td><td>2 + 3</td></tr> | |
| <tr><td>§164.308(a)(5)</td><td>Security Awareness & Training</td><td>HR/leadership-owned (out of scope here)</td><td>n/a</td></tr> | |
| <tr><td>§164.308(a)(6)</td><td>Security Incident Procedures</td><td>Sentry + CloudWatch alarms + on-call + documented runbook</td><td>1 + ongoing</td></tr> | |
| <tr><td>§164.308(a)(7)</td><td>Contingency Plan — backup, DR, testing</td><td>PITR enabled (Phase 1); cross-region snapshot copy (Phase 1); DR Drill performed + report filed (Phase 5); quarterly thereafter</td><td>1 + 5</td></tr> | |
| <tr><td>§164.308(a)(8)</td><td>Evaluation — periodic</td><td>Annual pen-test (Phase 5 + recurring); quarterly architecture review</td><td>5 + recurring</td></tr> | |
| <tr><td>§164.308(b)(1)</td><td>Business Associate Contracts (BAA)</td><td>AWS BAA confirmed; Sentry Enterprise BAA verified; Stripe BAA; Mailchimp BAA; UPS BAA verified</td><td>0</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>51.2 — Physical Safeguards (§164.310)</h3> | |
| <table> | |
| <thead><tr><th>Citation</th><th>Standard</th><th>How</th><th>Phase</th></tr></thead> | |
| <tbody> | |
| <tr><td>§164.310(a)</td><td>Facility Access Controls</td><td>AWS data center controls (BAA-delegated)</td><td>0</td></tr> | |
| <tr><td>§164.310(b)</td><td>Workstation Use</td><td>HR/IT policy</td><td>n/a</td></tr> | |
| <tr><td>§164.310(c)</td><td>Workstation Security</td><td>MFA + FDE on workstations (IT-owned)</td><td>n/a</td></tr> | |
| <tr><td>§164.310(d)</td><td>Device & Media Controls — disposal, re-use, accountability, backup</td><td>AWS-managed; old DDB deleted after 30-day stability with audit retention</td><td>5</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>51.3 — Technical Safeguards (§164.312)</h3> | |
| <table> | |
| <thead><tr><th>Citation</th><th>Standard</th><th>How</th><th>Phase</th></tr></thead> | |
| <tbody> | |
| <tr><td>§164.312(a)(1)</td><td>Access Control</td><td>Cognito + per-user JWT; chil_admin break-glass via temp creds; session timeouts; field-level encryption (Phase 3); RLS (Phase 2)</td><td>1, 2, 3</td></tr> | |
| <tr><td>§164.312(a)(2)(i)</td><td>Unique User ID</td><td><code>external_id</code> + <code>cognito_sub</code> stored together (Phase 2 — fixes the misnaming); tenant claim adds tenant identity (Phase 1)</td><td>1 + 2</td></tr> | |
| <tr><td>§164.312(a)(2)(ii)</td><td>Emergency Access</td><td><code>chil_admin</code> role with BYPASSRLS; IAM Identity Center + temp creds; every action audited via pgaudit</td><td>1</td></tr> | |
| <tr><td>§164.312(a)(2)(iii)</td><td>Automatic Logoff</td><td>Cognito session timeouts (existing); customAuth Lambda fails closed (Phase 3)</td><td>1 + 3</td></tr> | |
| <tr><td>§164.312(a)(2)(iv)</td><td>Encryption & Decryption</td><td>CMK on Aurora + S3 + Secrets (Phase 1); Field-level encryption for highest-sensitivity PHI (Phase 3)</td><td>1 + 3</td></tr> | |
| <tr><td>§164.312(b)</td><td>Audit Controls</td><td>CloudTrail (mgmt + data events) → S3 Object Lock; pgaudit → CloudWatch → S3; AppSync access logs; chil_audit / audit_event entity (Phase 2); Cognito events; 6+ year retention</td><td>1 + 2</td></tr> | |
| <tr><td>§164.312(c)</td><td>Integrity</td><td>Append-only history tables (Phase 2); idempotent writes; transactional outbox writes in same transaction; document_hash on legal documents (existing)</td><td>2</td></tr> | |
| <tr><td>§164.312(d)</td><td>Person/Entity Authentication</td><td>Cognito + email MFA (existing); IAM auth to Aurora via the Data API (Phase 1); customAuth Lambda JWT verify (Phase 3); step-up MFA (Phase 3)</td><td>1 + 3</td></tr> | |
| <tr><td>§164.312(e)</td><td>Transmission Security</td><td>TLS everywhere (existing); WAF v2; CloudFront TLS; <code>require_ssl=true</code> on Aurora (Phase 1)</td><td>1 (partial existing)</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 52 · SOC 2 ============================================== --> | |
| <!-- =============================================================== --> | |
| <section id="soc2"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 7 · 52</span> | |
| <h2 style="border:0;margin:18px 0 0;">SOC 2 Trust Service Criteria — schedule</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>Criterion</th><th>What it asks</th><th>How</th><th>Phase</th></tr></thead> | |
| <tbody> | |
| <tr><td>CC1 Control Environment</td><td>Commitment to security; competent personnel</td><td>Named Security Officer; Data Architect; branch protection + CI gates</td><td>0</td></tr> | |
| <tr><td>CC2 Communication & Information</td><td>Internal/external info quality</td><td>This document as architecture record; incident runbook; release log</td><td>0 + ongoing</td></tr> | |
| <tr><td>CC3 Risk Assessment</td><td>Risk identification & analysis</td><td>Companion doc § 9.2 gap inventory; recurring quarterly</td><td>0 + recurring</td></tr> | |
| <tr><td>CC4 Monitoring Activities</td><td>Ongoing monitoring & corrective action</td><td>3 CloudWatch dashboards (Phase 1); Sentry; cost & usage alerts; per-phase verification gates</td><td>1 + recurring</td></tr> | |
| <tr><td>CC5 Control Activities</td><td>Policies and procedures</td><td>§9.3 security principles in companion doc; CI enforces; PR review required</td><td>0 + 1+</td></tr> | |
| <tr><td>CC6 Logical & Physical Access Controls</td><td>Restrict to authorized users</td><td>Cognito + tenant claim + RLS (Phase 2) / customAuth gate (Phase 3); physical = AWS BAA</td><td>1, 2, 3</td></tr> | |
| <tr><td>CC7 System Operations (incl. <strong>CC7.5</strong> recovery testing)</td><td>Issue detection; incident management; recovery plan tested</td><td>CloudWatch alarms (Phase 1); on-call; <strong>DR Drill quarterly with dated report (Phase 5 + recurring)</strong></td><td>1 + 5 + recurring</td></tr> | |
| <tr><td>CC8 Change Management</td><td>Authorized changes</td><td>Branch protection; PR review; chil_migrator DDL-only role; CI security gates; release log; Drizzle migrations reviewed</td><td>0 + 1</td></tr> | |
| <tr><td>CC9 Risk Mitigation</td><td>Mitigation of business disruption</td><td>Aurora multi-AZ + reader (Phase 1); backup retention 7-day hot + 6-year archive; DR drill quarterly</td><td>1 + 5</td></tr> | |
| <tr><td>P (Privacy)</td><td>Personal info collection/use/disclosure</td><td>Field-level encryption (Phase 3); minimum-necessary access; PHI exclusion from reporting; BAA inventory</td><td>0, 2, 3, 5</td></tr> | |
| <tr><td>C (Confidentiality)</td><td>Confidential info protected</td><td>KMS at rest + TLS in transit (Phase 1); field-level encryption (Phase 3); Sentry scrubbing; structured logger redaction</td><td>1 + 3</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 53 · PRE-CUTOVER GATES ================================== --> | |
| <!-- =============================================================== --> | |
| <section id="pre-cutover-gates"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 7 · 53</span> | |
| <h2 style="border:0;margin:18px 0 0;">Pre-cutover hard requirements (must close before Phase 4)</h2> | |
| </div> | |
| <div class="callout authfirst"><div class="callout-label">Gate re-scope · June 2026 auth-first decision</div> | |
| RLS/dispatcher-specific items below (RLS verification via DB policies, dispatcher canary deploy, dispatcher PC) move to the hardening track. They are replaced at cutover by: <strong>FF1 API isolation matrix green (720 assertions)</strong> · <strong>the three §6.1b <code>@auth</code> fixes shipped & verified</strong> (function grants on billing writes; template sections read-only; no schema-level authenticated default + CI rule-presence check) · <strong>tenant-isolation canary armed</strong> · <strong>field-level <code>@auth</code> on every Appendix-C PHI column verified by test</strong>. All non-RLS gates (Sentry, Pino, audit bucket, Streams, replay rehearsal, load test, …) stand unchanged.</div> | |
| <div class="callout crit"> | |
| <div class="callout-label">Non-negotiable security gates before PHI cutover</div> | |
| <ol style="margin:6px 0 0;"> | |
| <li><strong>Sentry hardened</strong> — <code>beforeSend</code> + <code>sendDefaultPii: false</code> + lower <code>tracesSampleRate</code> in all 3 init files.</li> | |
| <li><strong>Structured logger (Pino) with PHI redaction</strong> replacing all <code>console.log</code> calls. CI linter blocks reintroduction.</li> | |
| <li><strong>Email columns in <ins class="prdval">14</ins> audit tables</strong> migrated to <code>user_id</code> FK + hashed-email shadow <ins class="prdval">(2 tables key on email → re-key, not hash-in-place)</ins>.</li> | |
| <li><strong>MEDICALREVIEW caseload</strong> in production with backfill — verified with medical-review team.</li> | |
| <li><strong>MemberNote auth tightened</strong> — welon sees only assigned members' notes.</li> | |
| <li><strong>Audit-archive S3 bucket</strong> with Object Lock Compliance (7-year), CMK-encrypted.</li> | |
| <li><strong>CloudTrail org trail</strong> enabled with S3 destination.</li> | |
| <li><strong>Invite tokens hashed</strong>; expiry enforced.</li> | |
| <li><strong>preTokenGen fail-closed</strong> — any error throws.</li> | |
| <li><strong>Cognito pool config confirmed</strong> — password policy + token TTL + advanced security documented with external pool owner.</li> | |
| <li><strong>Field-level encryption shipped</strong> for all Tier-2 fields per Appendix C.1 — Care Doc Digital Assets (SSN, passwords, social/cloud account creds), Care Doc Financial (bank/investment/insurance account numbers), Care Doc Medical (advance directives, living will, healthcare proxy), Interview mental-health responses, plus standalone banking_info + insurance.policy_number.</li> | |
| <li><strong>customAuth Lambda</strong> filled in — JWT verify + tenant claim required + fail-closed.</li> | |
| <li><strong>Deletion protection ON</strong> for all DDB tables (currently OFF).</li> | |
| <li><strong>CI security gates green</strong> — SAST (Semgrep), SCA (Snyk/pnpm audit), secret scanning, IaC scan.</li> | |
| <li><strong>Step-up MFA</strong> deployed on sign-document, delete-user, role-change, medical-export operations.</li> | |
| <li><strong>postSignUpTrigger P95 init < 3 s</strong> measured under cold start (CloudWatch synthetic canary); the 5 s Cognito hard timeout has > 2 s headroom. If P95 > 3 s, raise PC to 2 before cutover.</li> | |
| <li><strong>Aurora load test</strong> at 2× projected peak traffic (200k members equivalent); P95 dispatcher duration < 400 ms; no connection-pool exhaustion; report filed.</li> | |
| <li><strong>DDB Streams CDC lag < 5 min</strong> sustained for 7 consecutive days pre-cutover; CDC consumer Lambda alarm wired.</li> | |
| <li><strong>Rollback replay procedure documented</strong> — exact SQL query against <code>audit_event</code> to identify Aurora writes between T+2:30 and rollback time; idempotency key on every mutation; <ins class="rlsfix">replay Lambda <strong>executed successfully in both dress rehearsals with a parity report attached</strong> (June 2026 review fix, Q9 — upgraded from "code reviewed": reviewed is not tested)</ins>.</li> | |
| <li><strong>chil_admin compensating controls</strong> live — IAM Identity Center temp creds, Jira-ticket approval gate, real-time Slack alert on session open, Tier-2 field-encryption protects highest-PHI from BYPASSRLS read.</li> | |
| <li><strong>PHI field inventory (Appendix C) signed off</strong> — every PHI column tagged with encryption tier (Aurora-CMK / Field-level / both); coverage = 100% with no exceptions.</li> | |
| <li><strong>AppSync introspection DISABLED in prod</strong> — <code>introspectionConfig: DISABLED</code>, <code>queryDepthLimit: 10</code>, <code>resolverCountLimit: 75</code>, <code>xrayEnabled: true</code>. §36 has the change; this elevates it to a hard gate (was a deliverable, not a gate).</li> | |
| <li><strong>backend.ts:175 truthiness bug fixed</strong> — <code>if (currentBranch == 'staging' || 'production' || 'dev-test')</code> is always-true and silently enables S3 versioning on every branch. Fix to <code>['staging','production','dev-test'].includes(currentBranch)</code>. Phase 0 cleanup; one line.</li> | |
| <li><strong>Dispatcher canary deploy mechanism proven in dev-test</strong> — 5% traffic shift before promote; one-command CFN parameter rollback (AG6). Because the dispatcher concentrates all PHI traffic, a bad deploy is a full outage.</li> | |
| <li><strong>Scheduler execution-context resolved</strong> (AG4) — the 5 EventBridge schedulers have no user JWT; the dispatcher fail-closes on a missing tenant claim. Pick + ship one of: <code>chil_system</code> cross-tenant role (RLS-bypassed, fully pgaudit-logged), per-tenant loop, or pg_cron. Without this, every scheduled PHI job breaks at cutover.</li> | |
| <li><ins class="teamfix"><strong>UserInvite auth rules closed</strong> (team review) — <code>user-models.ts:91</code> carries <code>// TODO FIX AUTH RULES IN THIS MODEL</code>. Invitations are a trust boundary (they grant account access); the open TODO must be resolved and tested before cutover.</ins></li> | |
| <li><ins class="teamfix"><strong>Care-document access scoped to the assigned trustee</strong> (Vasyl audit) — today all Welon staff IDs are written to each document on creation, so <em>every</em> Welon staff member can read <em>every</em> care document. Switch to the <code>welon_assignment</code> lookup (the 14-call-site migration already scoped in §3035) so a trustee sees only assigned members’ documents.</ins></li> | |
| <li><ins class="teamfix"><strong>Plaintext PHI eliminated at migration</strong> (Vasyl audit) — SSN, master passwords, and financial credentials are stored as plaintext JSON in DynamoDB (the earlier PHI spec missed SSN). All must land as Tier-2 field-encrypted columns on the Aurora write (Appendix C); no plaintext PHI is copied across as-is.</ins></li> | |
| <li><ins class="teamfix"><strong>Sentry data-leak closed</strong> (Vasyl audit; extends gate #1) — interview data, verification tokens, and Stripe payloads are logged with no scrubbing at <strong>100% trace sampling</strong> (<code>tracesSampleRate: 1</code> in <code>sentry.server/edge.config.ts</code>, <code>instrumentation-client.ts</code>, and ~Lambda handlers). Ship the PHI scrubber and lower sampling before cutover.</ins></li> | |
| <li><ins class="teamfix"><strong>Facebook Pixel scoped off authenticated PHI routes</strong> (Vasyl audit) — the pixel fires <code>PageView</code>/<code>ViewContent</code> globally from <code>app/layout.tsx</code>, including authenticated medical and legal flows (a HIPAA tracking-disclosure risk). Restrict it to public/marketing routes; exclude all authenticated PHI flows.</ins></li> | |
| <li><ins class="teamfix"><strong>Medical-incident delivery log de-risked</strong> (Vasyl audit) — it stores the full clinical packet as a text field. Treat as Tier-2 PHI (encrypt) and RLS/assignment-scope; no plaintext clinical packet at rest.</ins></li> | |
| <li><ins class="datarch">📄 <strong>Medical-incident delivery log = metadata only</strong> (data-arch review; sharpens gate 27) — <code>MedicalIncidentDelivery.content</code> holds the full PHI packet and is logged per delivery attempt. In Aurora the log stores metadata only (timestamp, recipient, method, success); the PHI content is <strong>never persisted after transmission</strong>, not merely encrypted.</ins></li> | |
| <li><ins class="datarch">📄 <strong>Verification tokens hashed</strong> (data-arch review) — <code>VerificationTokens.token</code>, <code>EmergencyContact.verificationToken</code>, and the UserInvite token are plaintext today; store as salted hashes before cutover.</ins></li> | |
| <li><ins class="datarch">📄 <strong>Notification subscriptions disabled</strong> (data-arch review) — <code>Notification</code> is the only model not calling <code>disableOperations(['subscriptions'])</code> (40 others do). Disable it, or justify the real-time channel and pin its authorization.</ins></li> | |
| <li><ins class="datarch">📄 <strong>Structural schema gaps fixed in the canonical DDL</strong> (data-arch review) — <code>EmergencyContact</code> relationship + GSI on <code>userId</code> (User <code>hasMany</code>); <code>DocumentUpdateLog</code> <code>created_at</code> vs <code>timestamp</code> de-duplicated; <code>WellnessCheck.status</code> typed to the <code>WellnessCheckStatus</code> enum.</ins></li> | |
| <li><ins class="datarch">📄 <strong>Lambda timeout cleanup</strong> (data-arch review) — the AWS console shows ~90+ deployed functions at the 15-min max (code sets it explicitly on only 5). Set per-function timeouts reflecting real work + sub-limit CloudWatch alarms across all functions, so a hung Data API call cannot burn 15 min silently.</ins></li> | |
| <li><ins class="datarch">📄 <strong><code>custom:trustCompanyId</code> provisioned + backfilled FIRST</strong> (data-arch review — the confirmed Phase-1 kickoff blocker) — the attribute does not exist in the Cognito pool yet. No migration work starts until it exists across dev-test/staging/prod and every existing user is backfilled, because every <code>@auth</code> rule and RLS policy reads it.</ins></li> | |
| </ol> | |
| <p style="margin-top:8px;"><strong><ins class="datarch">📄 33 hard gates total.</ins></strong> Gates 1–15 are the original security checklist; 16–21 were added from the senior-architect review (Appendix E); <ins class="teamfix">22–27 from the June 2026 team review (Mike call + Vasyl audit)</ins>; <ins class="datarch">📄 28–33 from the June 21 data-architecture review (AWS console + schema audit)</ins>.</p> | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 54 · EVIDENCE PACK ====================================== --> | |
| <!-- =============================================================== --> | |
| <section id="evidence-pack"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 7 · 54</span> | |
| <h2 style="border:0;margin:18px 0 0;">Auditor evidence pack (deliverable at Phase 5 close)</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>Evidence</th><th>Source</th><th>Format</th></tr></thead> | |
| <tbody> | |
| <tr><td>Tested DR plan report</td><td>Phase 5 DR Drill output</td><td>PDF in Object-Lock S3, dated</td></tr> | |
| <tr><td>Tenant-isolation verification matrix</td><td>CI test runs across Phase 2–4</td><td>Signed test report, per-phase + final</td></tr> | |
| <tr><td>Audit log retention proof</td><td>S3 bucket config + Object Lock policy</td><td>Inventory document + policy excerpt</td></tr> | |
| <tr><td>Encryption inventory</td><td>Per-PHI-field × mechanism table</td><td>Inventory document; one row per field</td></tr> | |
| <tr><td>Access-control matrix</td><td>Role × resource × action, validated by tests</td><td>Spreadsheet + linked tests</td></tr> | |
| <tr><td>Incident response runbook</td><td>Separate doc; quarterly updates; annual rehearsal</td><td>Document + access log</td></tr> | |
| <tr><td>Vendor BAA inventory</td><td>AWS, Sentry, Stripe, Mailchimp, UPS</td><td>BAA copies + expiration dates</td></tr> | |
| <tr><td>Penetration test report</td><td>Annual; covers AppSync + Aurora</td><td>Signed PDF from independent firm</td></tr> | |
| <tr><td>SAST / SCA / secret-scan reports</td><td>Per-PR + scheduled</td><td>Findings tracked to resolution</td></tr> | |
| <tr><td>Change-control evidence</td><td>Every prod change has PR + review + release log + deploy artifact</td><td>Linkable trail per PR</td></tr> | |
| <tr><td>Risk Assessment</td><td>This document + quarterly reviews</td><td>Versioned in repo</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <div class="part-banner"> | |
| <div class="part-num">Part 8 of 8</div> | |
| <h2 style="border:0;color:var(--ink);">Closeout — timeline, Mike's deliverables, acceptance, open questions</h2> | |
| <p>The final ledger. The timeline summary table, the consolidated Mike's deliverables list, acceptance criteria per phase, and the open questions that need answering before Phase 1.</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== 56 · TIMELINE =========================================== --> | |
| <!-- =============================================================== --> | |
| <section id="timeline"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 8 · 56</span> | |
| <h2 style="border:0;margin:18px 0 0;">Timeline summary</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>Week</th><th>Phase</th><th>Deliverable</th><th>Gate</th></tr></thead> | |
| <tbody> | |
| <tr><td>0</td><td>Pre-flight</td><td>Decisions confirmed; BAAs verified; tooling chosen; team named</td><td>All §10 items checked</td></tr> | |
| <tr><td>1–2</td><td>Phase 1: Foundation</td><td>Aurora + Data API IAM access, KMS, Postgres roles, Cognito attrs, Sentry hardening, preTokenGen fail-closed, WAF, CloudTrail, audit-archive (no VPC/RDS Proxy)</td><td>Phase 1 gate</td></tr> | |
| <tr><td>3–6</td><td>Phase 2: Schema + non-PHI</td><td>Postgres DDL, RLS policies, Amplify connect-to-SQL, ~30 non-PHI models migrated incrementally, status-history started early, outbox</td><td>Phase 2 gate</td></tr> | |
| <tr><td>7–10</td><td>Phase 3: Dispatcher + tenancy</td><td>customAuth, claim injection, dispatcher Lambda, field-level encryption, PC, MEDICALREVIEW caseload, MemberNote tightening, PII column refactor, step-up MFA</td><td>Phase 3 gate</td></tr> | |
| <tr><td>11–13</td><td>Phase 4: PHI cutover</td><td>Dress rehearsal → prod cutover Friday evening → 4 h rollback window → 24 h hot monitor</td><td>Cutover successful or rolled back</td></tr> | |
| <tr><td>14–16</td><td>Phase 5: Reporting + closeout</td><td>Reader endpoint, Reporting page, DR Drill, audit evidence pack, decommission old DDB</td><td>Phase 5 gate</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>Critical path</h3> | |
| <ol> | |
| <li>Tenant model decision (week 5) — blocks RLS-related work.</li> | |
| <li>Field-auth projection layer (week 8) — blocks dispatcher completion.</li> | |
| <li>Dress rehearsal (week 11) — gates prod cutover.</li> | |
| <li>Pre-cutover hard requirements (week 12) — non-negotiable security gates.</li> | |
| </ol> | |
| <h3>What slips first / never slips</h3> | |
| <table> | |
| <thead><tr><th>Slips first</th><th>Never slips</th></tr></thead> | |
| <tbody> | |
| <tr><td>Week 15–16 stabilization compresses to 1 week if cutover went smoothly. DR Drill can move to week 15. Phase 3 polish explicitly post-deadline.</td><td>Pre-cutover hard requirements (PHI scrubbing, structured logger, MEDICALREVIEW caseload). Rollback rehearsal. Cutover without these creates new HIPAA exposure.</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 57 · MIKE'S DELIVERABLES ================================ --> | |
| <!-- =============================================================== --> | |
| <section id="mike-deliverables"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num devops">Part 8 · 57</span> | |
| <h2 class="devops" style="border:0;margin:18px 0 0;">Mike's deliverables — one consolidated list</h2> | |
| </div> | |
| <h3>Phase 0 (pre-flight)</h3> | |
| <ol> | |
| <li>Confirm AWS BAA signed; archive PDF in audit-archive bucket.</li> | |
| <li>Identify external Cognito pool owner; schedule attribute-add coordination.</li> | |
| <li>Set up AWS Organizations with OUs per environment + SCPs (US regions only).</li> | |
| <li>Set up IAM Identity Center; deactivate long-lived human IAM users.</li> | |
| <li>Fix smart-quote bug in <code>amplify.yml</code>.</li> | |
| <li>Measure exact CFN resource count today; track per-PR.</li> | |
| </ol> | |
| <h3>Phase 1 (Weeks 1–2)</h3> | |
| <ol> | |
| <li>Provision per-environment VPCs (3 envs × VPC each).</li> | |
| <li><s>Set up 2 NAT Gateways per VPC, 6 VPC endpoints per VPC.</s> <em>Not needed under the Data API; at most one optional Data API interface endpoint.</em></li> | |
| <li><s>Define 3 security groups per VPC (dispatcher, RDS Proxy, Aurora).</s> <em>Not needed — Data API is IAM-gated, not SG-gated.</em></li> | |
| <li>Provision 4 CMKs per environment.</li> | |
| <li>Provision Aurora Serverless v2 cluster + reader per env.</li> | |
| <li>Enable PostgreSQL extensions (pgaudit, pg_stat_statements, pgcrypto, aws_s3).</li> | |
| <li>Create <ins class="rlsfix">6</ins> Postgres roles per env<ins class="rlsfix"> (incl. <code>chil_owner</code> — June 2026 review, Q3)</ins>.</li> | |
| <li><s>Provision RDS Proxy with IAM auth, two target groups per env.</s> <em>Removed — Data API needs no pooler; grant <code>rds-data:ExecuteStatement</code> to the dispatcher role instead.</em></li> | |
| <li>Set up Performance Insights.</li> | |
| <li>Provision S3 audit-archive bucket with Object Lock (Compliance 7-year).</li> | |
| <li>Set up CloudTrail org trail; enable Log File Validation.</li> | |
| <li>Provision WAF v2 Web ACL; associate with AppSync.</li> | |
| <li>Set up the 3 CloudWatch dashboards.</li> | |
| <li>Wire all 16 alarms.</li> | |
| <li>Deploy Sentry hardening (all 3 init files).</li> | |
| <li>Add 5 CI security gates to <code>amplify.yml</code>.</li> | |
| <li>Set up Drizzle Kit migration runner in CI.</li> | |
| <li>Coordinate Cognito custom-attribute add with pool owner.</li> | |
| <li>Run Cognito backfill script on all 3 envs.</li> | |
| <li>Convert preTokenGen to fail-closed.</li> | |
| </ol> | |
| <h3>Phase 3 (Weeks 7–10) — Mike-specific</h3> | |
| <ol> | |
| <li>Fill in <code>customAuth</code> Lambda.</li> | |
| <li>Attach dispatcher + encrypt-field + decrypt-field + outbox-worker to VPC; add PC.</li> | |
| <li>Provision Field-Encryption CMK; lock-down key policy.</li> | |
| <li>Lazy-import Sentry across 50 Lambdas (or hand off to dev team).</li> | |
| </ol> | |
| <h3>Phase 4 (Weeks 11–13)</h3> | |
| <ol> | |
| <li>Enable DynamoDB Streams on all 41 tables (T−1 week).</li> | |
| <li>Flip deletion protection to ON on prod tables.</li> | |
| <li>Provision cutover-temp infrastructure.</li> | |
| <li>Execute cutover playbook; staff on-call for 4 h window.</li> | |
| <li>Print + tape rollback runbook beside war room.</li> | |
| </ol> | |
| <h3>Phase 5 (Weeks 14–16)</h3> | |
| <ol> | |
| <li>Provision the Aurora reader endpoint; reporting reaches it via the Data API (reader endpoint ARN).</li> | |
| <li>Execute DR Drill; record RTO/RPO; file report in audit-archive S3.</li> | |
| <li>Compile auditor evidence pack.</li> | |
| <li>Decommission old DDB PHI tables after 30-day stability.</li> | |
| <li>Schedule quarterly DR drill recurrence.</li> | |
| </ol> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 58 · ACCEPTANCE ========================================= --> | |
| <!-- =============================================================== --> | |
| <section id="acceptance"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 8 · 58</span> | |
| <h2 style="border:0;margin:18px 0 0;">Acceptance criteria per phase</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>Phase</th><th>Acceptance criterion (DevOps-owned)</th></tr></thead> | |
| <tbody> | |
| <tr><td>0</td><td>BAA in audit-archive; pool owner identified; SCPs deployed; smart-quote bug fixed; CFN count tracked in CI</td></tr> | |
| <tr><td>1</td><td>All Phase-1 infrastructure provisioned + verified (Aurora reachable via the Data API with IAM auth, KMS keys created, audit-archive bucket Object-Lock confirmed, WAF associated with AppSync, all 16 alarms armed, Sentry hardening live, CI security gates green on a test PR, Drizzle runner applied a no-op migration to staging)</td></tr> | |
| <tr><td>2</td><td>Postgres DDL deployed; RLS verification matrix passing on dev-test; non-PHI models migrated to Amplify SQL one at a time with frontend tests passing unchanged; status-history writes accruing</td></tr> | |
| <tr><td>3</td><td>customAuth Lambda gating prod; field-encryption Lambda round-trip verified; tenant-isolation matrix passing for all 15 PHI tables; MEDICALREVIEW caseload verified with team; PC providing < 2 s sign-in P95</td></tr> | |
| <tr><td>4</td><td>Cutover dress rehearsal completed twice in dev-test under 1 h rollback; prod cutover successful with RLS denial rate = 0, error rate within baseline, P95 latency within 20% of pre-cutover</td></tr> | |
| <tr><td>5</td><td>DR Drill report shows RTO < 4 h / RPO < 5 min; auditor evidence pack assembled; old DDB PHI tables deleted; two consecutive weeks of stable production</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== 59 · OPEN QUESTIONS ===================================== --> | |
| <!-- =============================================================== --> | |
| <section id="openq"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Part 8 · 59</span> | |
| <h2 style="border:0;margin:18px 0 0;">Open questions for the team</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>#</th><th>Question</th><th>Who answers</th></tr></thead> | |
| <tbody> | |
| <tr><td>1</td><td>Are dev-test / staging / prod in separate AWS accounts today, or one account with multiple stacks?</td><td>Mike</td></tr> | |
| <tr><td>2</td><td>Who owns the external Cognito pool IaC? What's the lead time for custom-attribute add across 3 envs?</td><td>Mike + pool owner</td></tr> | |
| <tr><td>3</td><td>What is the current Cognito password policy + access/id/refresh token TTLs?</td><td>Cognito pool owner</td></tr> | |
| <tr><td>4</td><td>Advanced security features (Plus tier — compromised credentials, adaptive auth) enabled?</td><td>Cognito pool owner</td></tr> | |
| <tr><td>5</td><td>Exact current CFN resource count today, and after Phase 0 cleanup?</td><td>Mike (precise count)</td></tr> | |
| <tr><td>6</td><td>How many DDB items in <code>Document</code> table currently exceed 300 KB? (per D2)</td><td>Mike (scan command in D2)</td></tr> | |
| <tr><td>7</td><td>Confirm BAA tier on Sentry — Enterprise? Business with custom BAA?</td><td>Mike + legal</td></tr> | |
| <tr><td>8</td><td>Cognito attribute coordination may take 2+ weeks if pool owner is external — acceptable?</td><td>Jon + Mike</td></tr> | |
| <tr><td>9</td><td>Do we have AWS Identity Center, or are humans using IAM users with access keys?</td><td>Mike</td></tr> | |
| <tr><td>10</td><td>Secret rotation policy today for STRIPE_SECRET_KEY / MAILCHIMP_API_KEY / UPS_CLIENT_SECRET / GOOGLE_API_KEY / FAX_CLIENT_SECRET? Any in violation?</td><td>Mike</td></tr> | |
| <tr><td>11</td><td>US-only geo-block on WAF — hard rule, or legitimate non-US access (vacation)?</td><td>Jon (product)</td></tr> | |
| <tr><td>12</td><td>IAM Identity Center + temp credentials for chil_admin DB break-glass, or long-lived password fallback?</td><td>Mike</td></tr> | |
| <tr><td>13</td><td>DR cross-region (us-east-1 → us-west-2) required, or single-region snapshot-restore sufficient?</td><td>Mike + auditor preference</td></tr> | |
| <tr><td>14</td><td>Self-serve sign-up tenant resolution — domain? URL? Default to seed?</td><td>Jon (product)</td></tr> | |
| <tr><td>15</td><td>Templates — shared base + per-tenant overrides, or fully per-tenant?</td><td>Jon + Oleksii</td></tr> | |
| <tr><td>16</td><td>MEDICALREVIEW access — scoped (caseload model) or all-members?</td><td>Jon + medical-team lead</td></tr> | |
| <tr><td>17</td><td>WELONTRUST CRUD on every MemberNote — intended? If "no," HIPAA gap to close.</td><td>Jon</td></tr> | |
| <tr><td>18</td><td>Will Childfree internal staff need cross-tenant access (platform support)? If yes, design as audited "platform" scope.</td><td>Jon + Mike</td></tr> | |
| <tr><td>19</td><td>Platform ever SaaS to multiple trust companies? (Influences multi-tenancy lifecycle.)</td><td>Jon (product)</td></tr> | |
| <tr><td>20</td><td>14–16 week timeline acceptable, or descope?</td><td>Jon + Mike + leadership</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== GLOSSARY ================================================ --> | |
| <!-- =============================================================== --> | |
| <section id="glossary"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · A</span> | |
| <h2 style="border:0;margin:18px 0 0;">Glossary</h2> | |
| </div> | |
| <table> | |
| <tbody> | |
| <tr><td><strong>ACU</strong></td><td>Aurora Capacity Unit. Aurora Serverless v2 scales by ACU; ~1 ACU ≈ 2 GB RAM + matched CPU.</td></tr> | |
| <tr><td><strong>BAA</strong></td><td>Business Associate Agreement. HIPAA contract required with every vendor processing PHI.</td></tr> | |
| <tr><td><strong>BYPASSRLS</strong></td><td>Postgres role attribute letting the role ignore RLS policies.</td></tr> | |
| <tr><td><strong>CDC</strong></td><td>Change Data Capture. Streaming row changes from one store to another.</td></tr> | |
| <tr><td><strong>CFN</strong></td><td>CloudFormation. AWS IaC; 500 resources per stack — the limit we hit.</td></tr> | |
| <tr><td><strong>CMK</strong></td><td>Customer-Managed Key (KMS). Required for HIPAA workloads.</td></tr> | |
| <tr><td><strong>ENI</strong></td><td>Elastic Network Interface. VPC-attached Lambdas allocate ENIs; soft limit 250/region.</td></tr> | |
| <tr><td><strong>GUC</strong></td><td>Grand Unified Configuration — Postgres session variable. <code>app.trust_company_id</code> etc. drive RLS.</td></tr> | |
| <tr><td><strong>JWT</strong></td><td>JSON Web Token. Cognito access token; PreTokenGen injects custom claims.</td></tr> | |
| <tr><td><strong>KMS</strong></td><td>AWS Key Management Service.</td></tr> | |
| <tr><td><strong>Object Lock</strong></td><td>S3 feature; objects cannot be deleted/modified until retention elapses.</td></tr> | |
| <tr><td><strong>PC</strong></td><td>Provisioned Concurrency (Lambda). Pre-warmed instances that bypass cold start.</td></tr> | |
| <tr><td><strong>pgaudit</strong></td><td>Postgres extension logging SELECT/INSERT/UPDATE/DELETE on tagged tables.</td></tr> | |
| <tr><td><strong>PHI</strong></td><td>Protected Health Information (HIPAA).</td></tr> | |
| <tr><td><strong>PII</strong></td><td>Personally Identifiable Information.</td></tr> | |
| <tr><td><strong>PITR</strong></td><td>Point-In-Time Recovery. Continuous backup.</td></tr> | |
| <tr><td><strong>RLS</strong></td><td>Row-Level Security (Postgres). Policies that filter rows by session context.</td></tr> | |
| <tr><td><strong>RPO / RTO</strong></td><td>Recovery Point / Time Objective.</td></tr> | |
| <tr><td><strong>SOC 2</strong></td><td>Service Organization Control 2. Trust principles audit.</td></tr> | |
| <tr><td><strong>VPC Endpoint</strong></td><td>Private connection from VPC to AWS service without NAT/internet.</td></tr> | |
| <tr><td><strong>WAF</strong></td><td>Web Application Firewall. AWS WAF v2 in front of AppSync.</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <div class="part-banner"> | |
| <div class="part-num">Appendix B</div> | |
| <h2 style="border:0;color:var(--ink);">Architecture review — industry-standard frameworks applied to this document</h2> | |
| <p>This appendix re-reviews the entire migration plan through seven canonical architecture review lenses — <strong>arc42</strong> (documentation completeness), <strong>C4</strong> (visual decomposition), <strong>ATAM</strong> (SEI's Architecture Tradeoff Analysis Method: utility tree, risk themes, sensitivity & tradeoff points), <strong>AWS Well-Architected Framework</strong> (six-pillar scoring), <strong>Evolutionary Architecture</strong> (Ford/Parsons fitness functions), <strong>MADR/Nygard ADRs</strong> (decision-record inventory), and <strong>SOC 2 / HIPAA mapping</strong> (cross-referenced against Parts 7.51–54). The goal is to surface gaps the prior 8 parts may have buried in narrative — and to give Mike, Jon, and Oleksii a self-contained quality bar to hold each Phase against.</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== B.1 · METHODOLOGY ======================================= --> | |
| <!-- =============================================================== --> | |
| <section id="review-method"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · B.1</span> | |
| <h2 style="border:0;margin:18px 0 0;">Review methodology — the seven lenses</h2> | |
| </div> | |
| <table> | |
| <thead><tr><th>Lens</th><th>What it asks</th><th>What it would catch</th><th>Why it matters here</th></tr></thead> | |
| <tbody> | |
| <tr><td><strong>arc42</strong> <span style="color:var(--muted);font-weight:400;">(Starke / Hruschka, canonical 12-section template)</span></td><td>Is every standard documentation section present? (Context, Constraints, Solution Strategy, Building Block, Runtime, Deployment, Crosscutting, Decisions, Quality, Risks, Glossary)</td><td>Missing diagrams or untold context that would surface only after onboarding pain</td><td>Onboarding next engineer; auditor evidence completeness</td></tr> | |
| <tr><td><strong>C4 Model</strong> <span style="color:var(--muted);font-weight:400;">(Simon Brown — Context / Container / Component / Code)</span></td><td>Are System Context, Container, and Component levels each visualized?</td><td>Implicit topology that nobody but the author understands</td><td>Mike + Oleksii need to share a mental model of "what calls what"</td></tr> | |
| <tr><td><strong>ATAM utility tree</strong> <span style="color:var(--muted);font-weight:400;">(SEI Architecture Tradeoff Analysis Method — Kazman / Klein / Clements)</span></td><td>Each quality attribute → refinement → concrete measurable scenario</td><td>Vague NFRs ("system should be fast") with no acceptance threshold</td><td>Phase gates need quantitative pass/fail thresholds, not "looks ok"</td></tr> | |
| <tr><td><strong>ATAM risk themes</strong></td><td>What systemic patterns appear in 3+ risks?</td><td>Single-discipline mitigations missing the underlying common cause</td><td>Tells leadership what's actually fragile vs what's just a checklist item</td></tr> | |
| <tr><td><strong>Sensitivity & tradeoff points</strong></td><td>Which parameter changes cascade? Where are quality attributes in conflict?</td><td>Hidden assumptions that break the whole plan when revised</td><td>Reveals what NOT to change without re-review</td></tr> | |
| <tr><td><strong>AWS Well-Architected</strong></td><td>Score per pillar (Operational Excellence, Security, Reliability, Performance, Cost, Sustainability) against AWS 2024 framework + April 2025 best-practice additions</td><td>Anything AWS already says is best practice for HIPAA workloads but we missed</td><td>Auditor will run this; better to self-score first</td></tr> | |
| <tr><td><strong>Fitness functions</strong> <span style="color:var(--muted);font-weight:400;">(Ford / Parsons / Kua — <em>Building Evolutionary Architectures</em>, 2nd ed.)</span></td><td>What's enforced automatically in CI vs left as "team discipline"?</td><td>Drift between intent and reality once the plan ships</td><td>HIPAA controls degrade silently without automated enforcement</td></tr> | |
| <tr><td><strong>MADR / Nygard ADRs</strong> <span style="color:var(--muted);font-weight:400;">(Markdown Architecture Decision Records — Nygard 2011 + MADR 4.0 template)</span></td><td>Is every architecturally significant decision recorded with Context / Decision / Consequences?</td><td>Future "why did we…" arguments where nobody remembers the constraint</td><td>Two-year horizon: people leave; ADRs survive turnover</td></tr> | |
| <tr><td><strong>HIPAA / SOC 2 mapping</strong></td><td>Every Safeguard / Trust Service Criterion → which phase delivers it</td><td>Compliance gaps where intent is unclear which phase owns it</td><td>Auditor evidence pack (§54) is built from this mapping</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout"> | |
| <div class="callout-label">How to read the verdicts in this appendix</div> | |
| <strong class="ok">PASS</strong> — addressed at expected depth.<br> | |
| <strong style="color:var(--warn);">PARTIAL</strong> — addressed but with a gap below.<br> | |
| <strong class="crit">GAP</strong> — not addressed; closure item issued.<br> | |
| Each gap maps to a specific <strong>§B.11 gap-closure</strong> deliverable so they're not lost. | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== B.2 · ARC42 AUDIT ======================================= --> | |
| <!-- =============================================================== --> | |
| <section id="arc42-audit"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · B.2</span> | |
| <h2 style="border:0;margin:18px 0 0;">arc42 coverage audit — twelve sections, twelve verdicts</h2> | |
| </div> | |
| <p>arc42 is the canonical 12-section documentation template for software architecture. A document that hits all twelve is self-contained; one that misses several leaves a maintainer guessing. Below: each arc42 section, mapped to where this document addresses it, with verdict.</p> | |
| <table> | |
| <thead><tr><th>#</th><th>arc42 section</th><th>Where in this doc</th><th>Verdict</th><th>Gap / note</th></tr></thead> | |
| <tbody> | |
| <tr><td>1</td><td>Introduction & Goals</td><td>Part 1 §00 Executive summary; §01 Why Aurora; companion <code>CALL-PREP-UNIFIED.html</code> for full context</td><td><span class="sev verified">PASS</span></td><td>—</td></tr> | |
| <tr><td>2</td><td>Constraints</td><td>Sprinkled: §3a-cognito (external Cognito pool), §6 CFN limits, §29 PC budget, §28 ENI cliff, §53 pre-cutover hard requirements, §59 open questions</td><td><span class="sev warn">PARTIAL</span></td><td>No single "Constraints" section consolidates them. Closure: <strong>§B.11 #1 Constraints Register</strong>.</td></tr> | |
| <tr><td>3</td><td>System Scope & Context</td><td>Implicit — external systems named scattered (Cognito external pool, Stripe, Mailchimp, UPS, Botpress, Sentry, Fax client, Vimeo)</td><td><span class="sev high">GAP</span></td><td>No C4 Level 1 System Context diagram. Closure: <strong>§B.3 below</strong>.</td></tr> | |
| <tr><td>4</td><td>Solution Strategy</td><td>Part 1 §03 Target topology + Part 3 §06 dispatcher pattern</td><td><span class="sev verified">PASS</span></td><td>—</td></tr> | |
| <tr><td>5</td><td>Building Block View</td><td>Part 3 §06–09 narrative; DevOps items 19, 21, 27, 28</td><td><span class="sev warn">PARTIAL</span></td><td>Excellent narrative; no C4 Container diagram. Closure: <strong>§B.3 below</strong>.</td></tr> | |
| <tr><td>6</td><td>Runtime View</td><td>Part 4 phases; cutover playbook; <code>SET LOCAL</code> code blocks</td><td><span class="sev warn">PARTIAL</span></td><td>No sequence diagrams for: (a) PHI read with RLS, (b) cutover flip, (c) rollback. Closure: <strong>§B.11 #2 Sequence diagrams</strong>.</td></tr> | |
| <tr><td>7</td><td>Deployment View</td><td>Part 4 + DevOps items 17–48</td><td><span class="sev verified">PASS</span></td><td>—</td></tr> | |
| <tr><td>8</td><td>Crosscutting Concepts</td><td>RLS (§07), multi-tenancy (§07), encryption (§20 + §43), audit logging (§33 + §34), observability (§41 + §42)</td><td><span class="sev verified">PASS</span></td><td>—</td></tr> | |
| <tr><td>9</td><td>Architecture Decisions</td><td>Embedded in narrative; not extracted as ADRs</td><td><span class="sev high">GAP</span></td><td>No ADR repository. Closure: <strong>§B.9 below</strong>.</td></tr> | |
| <tr><td>10</td><td>Quality Requirements</td><td>HIPAA/SOC 2 in §51–52; performance & reliability in scattered places</td><td><span class="sev warn">PARTIAL</span></td><td>No utility tree with scenarios. Closure: <strong>§B.4 below</strong>.</td></tr> | |
| <tr><td>11</td><td>Risks & Technical Debt</td><td>Part 2 §05 D1–D6; §53 pre-cutover gates; §59 open questions</td><td><span class="sev verified">PASS</span></td><td>Risk themes pass below adds aggregation layer.</td></tr> | |
| <tr><td>12</td><td>Glossary</td><td>Appendix A</td><td><span class="sev verified">PASS</span></td><td>—</td></tr> | |
| </tbody> | |
| </table> | |
| <p><strong>arc42 score: 7 PASS / 4 PARTIAL / 2 GAP.</strong> Two systemic gaps (no C4 diagrams; no ADRs) drive most of the partials too. Closing those two collapses the deficit.</p> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== B.3 · C4 MODEL ========================================== --> | |
| <!-- =============================================================== --> | |
| <section id="c4-model"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · B.3</span> | |
| <h2 style="border:0;margin:18px 0 0;">C4 Model — Context, Container, Component diagrams</h2> | |
| </div> | |
| <p>The C4 model (Simon Brown) layers four views: System Context (people + external systems), Container (deployable units), Component (inside one container), and Code (class-level — rarely needed). Three views are reproduced below to close the arc42 §3 / §5 / §6 gaps.</p> | |
| <h3>B.3.1 — System Context (C4 Level 1)</h3> | |
| <pre class="sh"><code>┌────────────────────────────────────────────────────────────────────┐ | |
| │ People │ | |
| │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌──────────┐ │ | |
| │ │ MEMBER │ │ WELON │ │ MEDICAL │ │ ADMIN │ │ | |
| │ │ (PHI owner) │ │ TRUST │ │ REVIEW │ │ staff │ │ | |
| │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └─────┬────┘ │ | |
| │ │ │ │ │ │ | |
| │ └────────────────┴────────────────┴───────────────┘ │ | |
| │ ↓ HTTPS │ | |
| │ ┌──────────────────────────────────────────────────────────────┐ │ | |
| │ │ │ │ | |
| │ │ Childfree Legacy platform │ │ | |
| │ │ (this system) │ │ | |
| │ │ │ │ | |
| │ └─────┬──────────────────────┬─────────────────────┬───────────┘ │ | |
| │ │ OAuth/JWT │ Webhooks │ S3 / E-mail │ | |
| │ ↓ ↓ ↓ │ | |
| │ ┌──────────────┐ ┌──────────────────┐ ┌───────────────────┐ │ | |
| │ │ Cognito │ │ Stripe │ │ SES / Mailchimp │ │ | |
| │ │ User Pool │ │ (payments) │ │ (transactional + │ │ | |
| │ │ (external) │ │ │ │ marketing email) │ │ | |
| │ └──────────────┘ └──────────────────┘ └───────────────────┘ │ | |
| │ │ | |
| │ ┌──────────────┐ ┌──────────────────┐ ┌───────────────────┐ │ | |
| │ │ Sentry │ │ UPS Shipping API │ │ Botpress chatbot │ │ | |
| │ │ (errors, │ │ (death cert │ │ (member help) │ │ | |
| │ │ Enterprise) │ │ fulfillment) │ │ │ │ | |
| │ └──────────────┘ └──────────────────┘ └───────────────────┘ │ | |
| │ │ | |
| │ ┌──────────────┐ ┌──────────────────┐ │ | |
| │ │ FAX provider │ │ Vimeo │ │ | |
| │ │ (hospital │ │ (training video) │ │ | |
| │ │ delivery) │ │ │ │ | |
| │ └──────────────┘ └──────────────────┘ │ | |
| └────────────────────────────────────────────────────────────────────┘ | |
| </code></pre> | |
| <p><strong>People = 4 personas.</strong> <strong>External systems = 8.</strong> All 8 BAA status confirmed in §10 Pre-flight checklist.</p> | |
| <h3>B.3.2 — Container (C4 Level 2)</h3> | |
| <p>Internal containers of Childfree Legacy that this document deploys:</p> | |
| <pre class="sh"><code>┌──────────────────────────────────────────────────────────────────────────────────┐ | |
| │ Childfree Legacy — internal │ | |
| │ │ | |
| │ ┌──────────────────────┐ │ | |
| │ │ Next.js frontend │ ← deployed to Amplify Hosting │ | |
| │ │ (App Router) │ CSP nonce + HSTS + WAF │ | |
| │ └─────────┬────────────┘ │ | |
| │ │ AppSync GraphQL (TLS) │ | |
| │ ↓ │ | |
| │ ┌──────────────────────┐ ┌──────────────────────┐ │ | |
| │ │ customAuth Lambda │←─│ AppSync API │ ← Lambda auth + PC │ | |
| │ │ (JWT verify, │ │ (~43 entities) │ WAF v2 attached │ | |
| │ │ tenant injection) │ └─────────┬────────────┘ │ | |
| │ └──────────────────────┘ │ │ | |
| │ │ │ | |
| │ ┌─────────────────────────────┴─────────────┐ │ | |
| │ │ non-PHI PHI ↓ │ | |
| │ ↓ │ | |
| │ ┌──────────────────┐ ┌──────────────────────────────────────┐ │ | |
| │ │ Amplify │ │ dispatcher Lambda │ ← inside VPC │ | |
| │ │ connect-to-SQL │ │ (pg, parameterized, SET LOCAL, │ PC 1, env CMK │ | |
| │ │ (passthrough, │ │ audit emit) │ │ | |
| │ │ no Lambda) │ └─────────┬────────────────────────────┘ │ | |
| │ └────────┬─────────┘ │ │ | |
| │ ↓ ↓ │ | |
| │ ┌─────────────────────────────────────────────┐ │ | |
| │ │ Aurora Data API endpoint (IAM-authed) │ ← HTTPS, no VPC, no proxy │ | |
| │ └──────────┬──────────────────────────────────┘ │ | |
| │ │ │ | |
| │ writer │ ┌─────────────┐ │ | |
| │ ↓ ↓ reader │ │ | |
| │ ┌────────────────────────────────────────────────────────────┐ │ | |
| │ │ Aurora Serverless v2 (Postgres 16) │ ← Multi-AZ, │ | |
| │ │ chil_app (RLS) · chil_sqlconnect (RLS) · │ PITR 30d, │ | |
| │ │ chil_reporter (RLS RO) · chil_admin (BYPASS) · │ pgaudit, │ | |
| │ │ chil_migrator (DDL) │ env CMK │ | |
| │ └────────────────────────────────────────────────────────────┘ │ | |
| │ │ | |
| │ ┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐ │ | |
| │ │ DDB (post-cutover): │ │ S3 PHI buckets │ │ S3 audit-archive │ │ | |
| │ │ 5 high-write tables │ │ (6 path prefixes) │ │ Object Lock 7-yr │ │ | |
| │ │ Streams enabled │ │ env CMK │ │ Compliance mode │ │ | |
| │ └──────────────────────┘ └──────────────────────┘ └──────────────────────┘ │ | |
| │ │ | |
| │ ┌──────────────────────┐ ┌──────────────────────┐ │ | |
| │ │ encrypt-field Lambda │ │ decrypt-field Lambda │ ← inside VPC, field CMK │ | |
| │ │ (Encryption SDK) │ │ (Encryption SDK) │ PC 1 each │ | |
| │ └──────────────────────┘ └──────────────────────┘ │ | |
| │ │ | |
| │ ┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐ │ | |
| │ │ preTokenGen Lambda │ │ Stripe webhook │ │ outbox-worker Lambda │ │ | |
| │ │ PC 1 │ │ Lambda PC 1 │ │ inside VPC, PC 1 │ │ | |
| │ └──────────────────────┘ └──────────────────────┘ └──────────────────────┘ │ | |
| │ │ | |
| │ ┌──────────────────────────────────────────────────────────────────────────┐ │ | |
| │ │ ~70 other Lambdas (non-VPC, no PC) — no DB access, S3/SES/API only │ │ | |
| │ └──────────────────────────────────────────────────────────────────────────┘ │ | |
| └──────────────────────────────────────────────────────────────────────────────────┘ | |
| </code></pre> | |
| <p><strong>Containers in VPC = 0</strong> (June 2026 Data API decision). The dispatcher, encrypt/decrypt-field and outbox-worker all reach Aurora over the IAM-authed Data API HTTPS endpoint — no VPC, no RDS Proxy. Aurora writer + reader sit behind the Data API. Filesystem count is <strong>76</strong> <code>resource.ts</code> files (the in-<code>defineBackend</code> count matches). <strong>All Lambdas are non-VPC; the ENI cliff is structurally avoided (§28).</strong> Optionally add one Data API interface endpoint for private routing.</p> | |
| <h3>B.3.3 — Component (C4 Level 3) — inside the dispatcher Lambda</h3> | |
| <pre class="sh"><code>┌────────────────────────────────────────────────────────────────────────────┐ | |
| │ dispatcher Lambda (Node.js 22, Aurora Data API via @aws-sdk/client-rds-data)│ | |
| │ │ | |
| │ ┌──────────────────────────┐ │ | |
| │ │ AppSync resolver event │ ← typeName + fieldName + JWT context │ | |
| │ └─────────────┬────────────┘ │ | |
| │ │ │ | |
| │ ↓ │ | |
| │ ┌──────────────────────────┐ │ | |
| │ │ tenant-extractor │ ← read trust_company_id from JWT │ | |
| │ └─────────────┬────────────┘ fail-closed if missing │ | |
| │ │ │ | |
| │ ↓ │ | |
| │ ┌──────────────────────────┐ │ | |
| │ │ operation router │ ← typeName.fieldName → SQL builder │ | |
| │ └─────────────┬────────────┘ deterministic mapping │ | |
| │ │ │ | |
| │ ↓ │ | |
| │ ┌──────────────────────────┐ │ | |
| │ │ SQL builder │ ← parameterized only; no string concat │ | |
| │ └─────────────┬────────────┘ │ | |
| │ │ │ | |
| │ ↓ │ | |
| │ ┌──────────────────────────┐ │ | |
| │ │ pg pool │ ← single connection from pool │ | |
| │ └─────────────┬────────────┘ BEGIN; SET LOCAL app.trust_company_id │ | |
| │ │ SET LOCAL app.role; SQL; COMMIT/ROLLBACK │ | |
| │ ↓ │ | |
| │ ┌──────────────────────────┐ │ | |
| │ │ result projection layer │ ← field-auth: filter fields by JWT role │ | |
| │ └─────────────┬────────────┘ (the "field-auth projection" §13) │ | |
| │ │ │ | |
| │ ↓ │ | |
| │ ┌──────────────────────────┐ │ | |
| │ │ audit emitter │ ← write audit_event row in same tx │ | |
| │ └─────────────┬────────────┘ (transactional outbox pattern) │ | |
| │ │ │ | |
| │ ↓ │ | |
| │ ┌──────────────────────────┐ │ | |
| │ │ structured logger (Pino) │ ← PHI scrubbed; correlation_id; tenant_id │ | |
| │ └─────────────┬────────────┘ │ | |
| │ ↓ │ | |
| │ ┌──────────────────────────┐ │ | |
| │ │ Sentry exception capture │ ← beforeSend PHI-scrubber │ | |
| │ └──────────────────────────┘ │ | |
| └────────────────────────────────────────────────────────────────────────────┘ | |
| </code></pre> | |
| <p><strong>7 components</strong> inside dispatcher. Field-auth projection (4th from top) is the critical correctness layer; it's where the existing 26 <code>ownerDefinedIn('cognitoId').identityClaim('externalId')</code> rules get re-implemented in code.</p> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== B.4 · UTILITY TREE ====================================== --> | |
| <!-- =============================================================== --> | |
| <section id="utility-tree"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · B.4</span> | |
| <h2 style="border:0;margin:18px 0 0;">ATAM utility tree — quality attribute scenarios with thresholds</h2> | |
| </div> | |
| <p>SEI's ATAM utility tree decomposes each quality attribute into refinements, then to concrete scenarios with quantitative thresholds. Each scenario is a six-part contract: <em>source · stimulus · environment · artifact · response · response-measure</em>. Phase gates must pass these as numerical assertions, not gut feel.</p> | |
| <table> | |
| <thead><tr><th>Quality attribute</th><th>Refinement</th><th>Scenario (stimulus → response · threshold)</th><th>Verifier</th><th>Phase</th></tr></thead> | |
| <tbody> | |
| <tr><td rowspan="4"><strong>Security</strong></td> | |
| <td>Multi-tenant isolation</td> | |
| <td>User with tenant A's JWT issues <code>list Document</code> → response contains <strong>0</strong> rows belonging to tenant B</td> | |
| <td>Integration test in CI (the "Tenant Isolation Matrix")</td> | |
| <td>2 (initial), 3 (PHI tables)</td></tr> | |
| <tr><td>Encryption at rest</td> | |
| <td>Aurora storage scanned with AWS Config rule → <strong>100%</strong> encrypted with environment CMK (not aws/rds)</td> | |
| <td>AWS Config managed rule <code>rds-storage-encrypted</code></td> | |
| <td>1</td></tr> | |
| <tr><td>Audit non-repudiation</td> | |
| <td>Delete row on PHI table → <strong>1</strong> matching <code>audit_event</code> row visible to <code>chil_audit</code> within 5 s</td> | |
| <td>Synthetic probe Lambda every 5 min</td> | |
| <td>2</td></tr> | |
| <tr><td>Fail-closed auth</td> | |
| <td>customAuth Lambda throws → AppSync returns <strong>401</strong>, <strong>0</strong> rows leaked</td> | |
| <td>Chaos test in dress rehearsal</td> | |
| <td>3 + 4</td></tr> | |
| <tr><td rowspan="4"><strong>Performance Efficiency</strong></td> | |
| <td>Read latency (steady state)</td> | |
| <td>100k members, business hours, member loads dashboard → P95 dispatcher duration <strong>< 400 ms</strong>, P99 <strong>< 800 ms</strong></td> | |
| <td>CloudWatch Lambda <code>Duration</code> P95/P99 alarms</td> | |
| <td>3 + 5</td></tr> | |
| <tr><td>Cold start (sign-in)</td> | |
| <td>First request after 60 min idle → preTokenGen total <strong>< 2000 ms</strong> (the Cognito 5 s timeout has 2.5× headroom)</td> | |
| <td>Synthetic canary every 60 min</td> | |
| <td>1 (Sentry hardening) + 3 (PC)</td></tr> | |
| <tr><td>Write throughput</td> | |
| <td>Stripe webhook burst of 100 events/s → <strong>0</strong> dispatcher errors, P95 < 800 ms</td> | |
| <td>Load test before cutover</td> | |
| <td>4 (dress rehearsal)</td></tr> | |
| <tr><td>Aurora capacity headroom</td> | |
| <td>Steady-state ACU usage at peak hour → <strong>< 70%</strong> of max ACU; <strong>> 30%</strong> headroom</td> | |
| <td>CloudWatch <code>ServerlessDatabaseCapacity</code> alarm</td> | |
| <td>5</td></tr> | |
| <tr><td rowspan="3"><strong>Reliability</strong></td> | |
| <td>Recovery point</td> | |
| <td>Catastrophic failure at T → recover all PHI tables to T−5 min</td> | |
| <td>DR Drill report (§45)</td> | |
| <td>5 + quarterly</td></tr> | |
| <tr><td>Recovery time</td> | |
| <td>Catastrophic failure declared → service restored within <strong>4 h</strong></td> | |
| <td>DR Drill report</td> | |
| <td>5 + quarterly</td></tr> | |
| <tr><td>Cutover reversibility</td> | |
| <td>Cutover at T → rollback possible at T+4h without data loss</td> | |
| <td>Dress rehearsal × 2 in dev-test</td> | |
| <td>4</td></tr> | |
| <tr><td rowspan="3"><strong>Operability</strong></td> | |
| <td>Schema change deploy</td> | |
| <td>Single-column add → CI deploy < <strong>30 min</strong> end-to-end (no manual steps)</td> | |
| <td>Drizzle Kit runner in CI</td> | |
| <td>1 + 2</td></tr> | |
| <tr><td>On-call incident response</td> | |
| <td>Alarm fires → on-call sees relevant context in dashboard within <strong>2 min</strong></td> | |
| <td>CloudWatch dashboard (§41) + drill</td> | |
| <td>1 + 5</td></tr> | |
| <tr><td>Rollback drill</td> | |
| <td>Rollback runbook step-by-step exact CLI → completes < <strong>30 min</strong> on rehearsal</td> | |
| <td>Dress rehearsal × 2</td> | |
| <td>4</td></tr> | |
| <tr><td rowspan="2"><strong>Modifiability</strong></td> | |
| <td>New tenant onboarding</td> | |
| <td>New trust company → live in < <strong>2 h</strong> (just an INSERT + Cognito group)</td> | |
| <td>Smoke test on dev-test</td> | |
| <td>2</td></tr> | |
| <tr><td>New access pattern</td> | |
| <td>"Get members by status sorted by date" → <strong>1</strong> CREATE INDEX, <strong>0</strong> backfill steps</td> | |
| <td>Code review verifies no GSI; just an index</td> | |
| <td>2 onward</td></tr> | |
| <tr><td rowspan="2"><strong>Cost</strong></td> | |
| <td>Monthly steady-state</td> | |
| <td>100k members, business-hours load profile → total AWS bill within the approved budget, tracked via AWS Cost Anomaly Detection (Aurora + Data API + Lambda + S3 + CloudWatch)</td> | |
| <td>AWS Cost Anomaly + monthly review</td> | |
| <td>5</td></tr> | |
| <tr><td>One-time engineering</td> | |
| <td>Project complete in ~14–16 weeks of engineering effort</td> | |
| <td>Time tracking / phase-gate review</td> | |
| <td>5</td></tr> | |
| <tr><td><strong>Compliance</strong></td> | |
| <td>HIPAA + SOC 2 audit-ready</td> | |
| <td>Auditor request "show me audit trail for member X over Y" → returnable from <strong>S3 audit-archive query in < 1 day</strong>; coverage <strong>100%</strong> of mutating ops</td> | |
| <td>Quarterly evidence-pack rehearsal</td> | |
| <td>5 + recurring</td></tr> | |
| </tbody> | |
| </table> | |
| <p><strong>18 scenarios with quantitative thresholds.</strong> Each phase gate (§58 Acceptance criteria) is restated below as "scenarios closed at this phase":</p> | |
| <table> | |
| <thead><tr><th>Phase</th><th>Scenarios closed</th></tr></thead> | |
| <tbody> | |
| <tr><td>1</td><td>Encryption at rest; Cold start (without PC, baseline); Schema change deploy; On-call incident response (dashboards live)</td></tr> | |
| <tr><td>2</td><td>Multi-tenant isolation (non-PHI); Audit non-repudiation; New tenant onboarding; New access pattern</td></tr> | |
| <tr><td>3</td><td>Multi-tenant isolation (PHI); Fail-closed auth; Cold start (with PC)</td></tr> | |
| <tr><td>4</td><td>Cutover reversibility; Write throughput; Rollback drill</td></tr> | |
| <tr><td>5</td><td>Recovery point + time; Read latency steady-state; Aurora capacity headroom; Monthly cost; HIPAA + SOC 2 audit-ready</td></tr> | |
| </tbody> | |
| </table> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== B.5 · RISK THEMES ======================================= --> | |
| <!-- =============================================================== --> | |
| <section id="risk-themes"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · B.5</span> | |
| <h2 style="border:0;margin:18px 0 0;">ATAM risk themes — patterns that appear 3+ times</h2> | |
| </div> | |
| <p>Risk themes are systemic patterns. Individual risks are easier to mitigate; themes are easier to miss. Each theme below points to multiple discovered risks across the prior 8 parts.</p> | |
| <div class="review-block"> | |
| <h4>RT-1 · Coordination dependency on external Cognito pool owner <span class="sev crit">CRITICAL</span></h4> | |
| <p><strong>Where it appears:</strong> §10 Pre-flight (lead-time risk); §26 (custom-attribute add coordination); §53 (Cognito pool config confirmed); §59 questions #2 / #4 / #8.</p> | |
| <p><strong>Underlying pattern:</strong> The Cognito pool is owned externally. Any attribute change, password-policy change, advanced-security-feature toggle, or trigger redeploy requires a separate team's calendar. This single dependency gates Phase 1 + Phase 3.</p> | |
| <p><strong>Cross-theme mitigation:</strong> Phase 0 must extract a written SLA from the pool owner: max 5 business days to apply attribute changes; their on-call contact published. Failure to secure this → reschedule cutover.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>RT-2 · Implicit fail-open behavior throughout legacy code <span class="sev crit">CRITICAL</span></h4> | |
| <p><strong>Where it appears:</strong> §05 D4 (preTokenGen catch returns event); §11 (Phase 1 fix); §53 #9 (hard gate before cutover); §B.8 FF2 (automated enforcement); customAuth current stub <code>isAuthorized = true</code>; the <code>catch → return event</code> pattern recurs across 5+ Lambdas.</p> | |
| <p><strong>Underlying pattern:</strong> The codebase defaults to "log and proceed" on auth/data errors. Under HIPAA, this is the exact opposite of correct — auth errors must <em>fail closed</em>, returning 401 with no data. The pattern repeats across multiple Lambdas because Cognito's "Trigger failure aborts sign-in" UX is harsh, so devs coded around it.</p> | |
| <p><strong>Cross-theme mitigation:</strong> Code linter rule: <code>catch</code> blocks in <code>amplify/functions/</code> must <code>throw</code> or return error response. Block <code>return event</code> in catch. Add to <code>§B.8 fitness functions</code>.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>RT-3 · CloudFormation scale ceiling <span class="sev high">HIGH</span></h4> | |
| <p><strong>Where it appears:</strong> §04 (74 Lambdas in <code>backend.ts</code>); §06 (<ins class="prdval">CFN resource sprawl — largest devtest stack 373/500, not over the ceiling</ins>); §17 (account boundary); жалоби.txt item 2 (dependency hell); §59 question #5 (exact CFN resource count today).</p> | |
| <p><strong>Underlying pattern:</strong> Amplify Gen 2 deploys via CloudFormation; each model auto-generates ~5–8 CFN resources. At 41 models × ~5 resources + 74 Lambdas × ~3 resources <ins class="prdval">the binding constraint is resource sprawl + a ~47-way nested-stack fan-out + a FunctionDirectiveStack that grows per model — NOT the per-stack ceiling. In the devtest snapshot the largest single stack is 373/500 (nothing over 400). The deploy/DR risk is real but was mis-stated; the production per-stack number must be pulled before this sets the timeline.</ins></p> | |
| <p><ins class="prdval"><strong>PRD validation (against AWS snapshot):</strong> the binding constraint is NOT the 500-per-stack ceiling — in the devtest snapshot the largest single stack is <strong>373/500</strong> (FunctionDirectiveStack), nothing over 400, across 55 stacks / ~2,647 resources. The real problem is resource sprawl across a ~47-way nested-stack fan-out + a FunctionDirectiveStack that grows per model. Pull the PRODUCTION per-stack number before this sets the timeline; the DR-control-failing concern stands regardless of the exact mechanism.</ins></p> | |
| <p><strong>Cross-theme mitigation:</strong> CI fitness function: warn at 400 resources/stack, fail at 480. Also: Track B's <code>a.combine()</code> + connect-to-SQL pattern naturally consolidates Postgres-backed models into one stack; the 5 remaining DDB tables stay in original stack. Track migration progress as resource-count delta per phase.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>RT-4 · Cold-start tax paid 50× <span class="sev high">HIGH</span></h4> | |
| <p><strong>Where it appears:</strong> §05 D3 (top-level await across 50 Lambdas); §30 (cold-start mitigation beyond PC); §29 (PC budget covers only 5); жалоби.txt item 5 (cold starts).</p> | |
| <p><strong>Underlying pattern:</strong> Many Lambdas import Sentry, AWS SDK, etc. at module load, paying 100–500 ms cold-start tax on every cold invocation. PC fixes only the 5 hottest. The rest live with the tax.</p> | |
| <p><strong>Cross-theme mitigation:</strong> Lazy-import refactor pattern: <code>const sentry = await import('@sentry/...');</code> inside handler. Apply incrementally; CI fitness function flags top-level imports of heavy SDKs.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>RT-5 · Encryption-as-checklist instead of as-default <span class="sev high">HIGH</span></h4> | |
| <p><strong>Where it appears:</strong> §20 (one CMK per concern); §43 (Sentry scrubbing scope); §51 §164.312(a)(2)(iv); §53 #11 (field-level encryption shipped); §51 §164.312(b) audit controls.</p> | |
| <p><strong>Underlying pattern:</strong> The plan explicitly lists which fields get field-level encryption (Phase 3) and which don't. But the structured-logger PII scrubber + Sentry scrubber + Pino redaction must independently maintain matching key lists. A new PHI field added by a future PR can leak through one of these layers if the lists drift.</p> | |
| <p><strong>Cross-theme mitigation:</strong> Single source of truth: <code>amplify/data/phi-fields.json</code> generated from schema annotations. All scrubbers import from it. CI fitness function: every <code>PHI: true</code> field has a Pino redaction entry + Sentry scrubber entry + field-encryption coverage decision.</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>RT-6 · Single-path PHI dispatcher <span class="sev high">HIGH</span></h4> | |
| <p><strong>Where it appears:</strong> §06 dispatcher; §27 customAuth; §29 PC for dispatcher; §42 dispatcher error-rate alarm; §47 rollback mechanics.</p> | |
| <p><strong>Underlying pattern:</strong> Every PHI read/write traverses the dispatcher Lambda. A single bug or misconfig there can hide an entire tenant's data or expose another's. Centralization is the right design (much smaller attack surface than per-resolver auth), but it concentrates blast radius.</p> | |
| <p><strong>Cross-theme mitigation:</strong> The Tenant Isolation Matrix in CI is the single most important fitness function in the system. Must run on every dispatcher change. Mandatory PR reviewer for dispatcher = Oleksii (architecture review of the SQL change) + Mike (deployment review of the IaC change).</p> | |
| </div> | |
| <div class="review-block"> | |
| <h4>RT-7 · Drift between intent and reality after Phase 5 closes <span class="sev medium">MEDIUM</span></h4> | |
| <p><strong>Where it appears:</strong> §10 quarterly review; §45 DR Drill quarterly; §52 CC4 monitoring; §54 evidence pack annually; §58 ongoing.</p> | |
| <p><strong>Underlying pattern:</strong> Every prior migration of this scope eventually drifts: alarms get muted, drills slip, ADRs aren't updated, CMK rotation lapses. Phase 5 hands over to operations; without enforcement, the architecture goes stale.</p> | |
| <p><strong>Cross-theme mitigation:</strong> Every fitness function in §B.8 runs in CI <em>and</em> on a schedule (CloudWatch Events). Quarterly architecture review with this appendix as the rubric; failures generate Jira tickets automatically.</p> | |
| </div> | |
| <p><strong>Risk theme count: 7 (2 CRITICAL · 4 HIGH · 1 MEDIUM).</strong></p> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== B.6 · SENSITIVITY / TRADEOFFS ============================ --> | |
| <!-- =============================================================== --> | |
| <section id="sensitivity"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · B.6</span> | |
| <h2 style="border:0;margin:18px 0 0;">ATAM sensitivity points & tradeoff points</h2> | |
| </div> | |
| <p><strong>Sensitivity point</strong> = a parameter where a small change cascades into a quality attribute response. <strong>Tradeoff point</strong> = a parameter where improving one attribute degrades another.</p> | |
| <h3>B.6.1 — Sensitivity points</h3> | |
| <table> | |
| <thead><tr><th>#</th><th>Parameter</th><th>Sensitive to</th><th>If changed, what re-review</th></tr></thead> | |
| <tbody> | |
| <tr><td>S1</td><td>JWT claim shape (<code>trust_company_id</code> + <code>role</code>)</td><td>All RLS policies; customAuth contract; dispatcher tenant-extractor</td><td>Every RLS policy file + customAuth + dispatcher unit tests</td></tr> | |
| <tr><td>S2</td><td>Postgres connection pool size in dispatcher</td><td>Aurora connection saturation + cold-start cost</td><td>RDS Proxy config; Lambda concurrency limit; alarm threshold</td></tr> | |
| <tr><td>S3</td><td>customAuth Lambda timeout (currently 5 s)</td><td>Member sign-in latency; AppSync request P95</td><td>Whole dispatcher latency budget; PC sizing</td></tr> | |
| <tr><td>S4</td><td>Number of PHI tables migrated per phase</td><td>Rollback complexity; cutover window length</td><td>Cutover playbook; rollback runbook</td></tr> | |
| <tr><td>S5</td><td>Aurora minimum ACU (1.0)</td><td>Cold-start of database itself; idle cost; first-query latency</td><td>Cost table; reader/writer config; minimum-cost alarms</td></tr> | |
| <tr><td>S6</td><td>CMK rotation policy (default 1 year)</td><td>Compliance posture; key version sprawl; envelope-encryption format</td><td>Field-encrypt/decrypt Lambdas; backups; audit-archive format</td></tr> | |
| <tr><td>S7</td><td>CFN resource count per stack</td><td>Deploy reliability; per-PR feedback loop</td><td>Architecture decomposition; stack-split strategy</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>B.6.2 — Tradeoff points</h3> | |
| <table> | |
| <thead><tr><th>#</th><th>Tradeoff</th><th>Pulled toward</th><th>Document position</th></tr></thead> | |
| <tbody> | |
| <tr><td>T1</td><td><strong>Capacity ↔ Performance:</strong> Aurora min ACU 1 (lower steady-state) vs min ACU 2+ (faster cold-query)</td><td>Min ACU 1 — Aurora warms quickly; cold-query latency acceptable for non-PHI; first-call latency for PHI is handled by PC on the dispatcher</td><td>Revisit if P95 first-query exceeds 1500 ms</td></tr> | |
| <tr><td>T2</td><td><strong>Performance ↔ Compliance:</strong> Provisioned Concurrency hot-paths (cheap latency) vs zero PC (lower cost but more cold starts hit Cognito's 5 s ceiling)</td><td>Compliance — Phase 1 mandates PC for preTokenGen specifically because cold-start latency violates Cognito's hard 5 s timeout, which silently fails sign-in</td><td>§29; non-negotiable for the 5 hot Lambdas</td></tr> | |
| <tr><td>T3</td><td><strong>Security ↔ Operability:</strong> RDS Proxy + IAM auth (1 round-trip per cold-start, plus dispatch tax) vs direct DB password from Secrets Manager (faster, but a password to rotate)</td><td>Security — IAM auth means we never store DB passwords; auditors love it; operability tax mitigated by PC + pgBouncer-like pooling in Proxy</td><td>§22; immovable</td></tr> | |
| <tr><td>T4</td><td><strong>Modifiability ↔ Reliability:</strong> Track B incremental migration (low risk; long timeline) vs Track A 5-table consolidation (faster reads; high-risk schema change)</td><td>Reliability — Track B chosen specifically because incremental cutover with parallel old DDB during T-window allows 4 h rollback</td><td>Companion <code>CALL-PREP-UNIFIED.html</code>; §06 here</td></tr> | |
| <tr><td>T5</td><td><strong>Retention ↔ Audit:</strong> S3 Object Lock Compliance mode (cannot delete; full HIPAA mode) vs Governance mode (can delete with audit log)</td><td>Audit — Compliance mode chosen; mistakes mean retaining garbage for 7 years; that's the right side to err on</td><td>§33; immovable</td></tr> | |
| <tr><td>T6</td><td><strong>Single dispatcher ↔ Per-resolver Lambdas:</strong> One dispatcher (small attack surface, smaller team to review, but concentrates blast radius) vs per-resolver Lambdas (larger surface, parallel-deployable, but 37+ Lambdas to keep auth-correct)</td><td>Single dispatcher — see RT-6; concentration is correct given team size and audit budget</td><td>§06 + §B.5 RT-6</td></tr> | |
| <tr><td>T7</td><td><strong>Cognito ↔ self-hosted IdP:</strong> External Cognito pool (existing; ext team owns it; RT-1 risk) vs Cognito pool owned by Childfree (faster iteration; but migration is another full project)</td><td>Existing Cognito — migration to a different IdP is out-of-scope for this plan; mitigate RT-1 with SLA</td><td>§59 question #2</td></tr> | |
| </tbody> | |
| </table> | |
| <p><strong>Sensitivity points: 7. Tradeoff points: 7.</strong> All decisions documented above are positions the team takes <em>on purpose</em>; revisiting any of them is a re-architecture event, not a tweak.</p> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== B.7 · WELL-ARCHITECTED =================================== --> | |
| <!-- =============================================================== --> | |
| <section id="well-arch"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · B.7</span> | |
| <h2 style="border:0;margin:18px 0 0;">AWS Well-Architected — six-pillar self-scoring</h2> | |
| </div> | |
| <p>AWS Well-Architected Framework (Nov 2024 revision + April 2025 best-practice additions) defines 57 review questions across 6 pillars. Self-score below applied to the post-Phase-5 target architecture.</p> | |
| <table> | |
| <thead><tr><th>Pillar</th><th>Verdict</th><th>Strengths</th><th>Gap (closure in §B.11)</th></tr></thead> | |
| <tbody> | |
| <tr> | |
| <td><strong>Operational Excellence</strong></td> | |
| <td><span class="sev verified">STRONG</span></td> | |
| <td>3 CloudWatch dashboards (§41); 16 alarms (§42); runbook for cutover + rollback; named on-call; pre/per-phase acceptance criteria (§58); Drizzle Kit in CI; release log</td> | |
| <td>No GameDay schedule beyond DR drill. Closure: <strong>§B.11 #5</strong></td> | |
| </tr> | |
| <tr> | |
| <td><strong>Security</strong></td> | |
| <td><span class="sev verified">STRONG</span></td> | |
| <td>KMS CMK per concern (§20); RDS IAM auth (§22); RLS multi-tenancy (§07); customAuth Lambda fail-closed (§27); WAF v2 (§35); CloudTrail org trail (§34); audit-archive Object Lock Compliance (§33); Sentry scrubbing (§43); field-level encryption (§43); 15 pre-cutover gates (§53)</td> | |
| <td>No GuardDuty / Macie / Security Hub explicitly stood up. Closure: <strong>§B.11 #6</strong></td> | |
| </tr> | |
| <tr> | |
| <td><strong>Reliability</strong></td> | |
| <td><span class="sev verified">STRONG</span></td> | |
| <td>Aurora multi-AZ; reader endpoint (§09) reached via the Data API; PITR 30d + S3 long-tail (§44); DR Drill quarterly (§45); rollback runbook + dress rehearsal × 2 (§47); deletion protection on PHI tables</td> | |
| <td>No explicit multi-region DR (single-region snapshot copy only). Closure: <strong>§B.11 #7</strong> (and §59 Q13 already raises this for product decision)</td> | |
| </tr> | |
| <tr> | |
| <td><strong>Performance Efficiency</strong></td> | |
| <td><span class="sev warn">MEDIUM</span></td> | |
| <td>PC for 5 hot Lambdas (§29); Aurora auto-scaling 1–8 ACU; reader for reporting (§09); dispatcher hot path; pgBouncer-via-RDS-Proxy</td> | |
| <td>No formal load-test plan with target P95/P99 numbers. PC sized "by intuition." Closure: <strong>§B.11 #8</strong></td> | |
| </tr> | |
| <tr> | |
| <td><strong>Cost Optimization</strong></td> | |
| <td><span class="sev verified">STRONG</span></td> | |
| <td>I/O-Optimized switch criteria (30% I/O / ACU ratio); ACU range 1–8; AWS Cost Anomaly Detection (§42)</td> | |
| <td>No Savings Plan / RI strategy for steady-state. Closure: <strong>§B.11 #9</strong></td> | |
| </tr> | |
| <tr> | |
| <td><strong>Sustainability</strong></td> | |
| <td><span class="sev high">WEAK</span></td> | |
| <td>Auto-scaling implies right-sizing; Aurora Serverless reduces idle compute</td> | |
| <td>Zero explicit Sustainability content. AWS now grades this; auditor will note absence. Closure: <strong>§B.11 #10</strong></td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <p><strong>Pillar score: 4 STRONG · 1 MEDIUM · 1 WEAK.</strong> The Sustainability pillar is the lowest score and the easiest win — adding a single subsection covering instance-rightsizing strategy, log retention tiering (already partially in §44), and Graviton2 selection for non-Aurora Lambdas closes it.</p> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== B.8 · FITNESS FUNCTIONS ================================== --> | |
| <!-- =============================================================== --> | |
| <section id="fitness-functions"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · B.8</span> | |
| <h2 style="border:0;margin:18px 0 0;">Fitness functions — automated architecture governance</h2> | |
| </div> | |
| <p>Per <em>Building Evolutionary Architectures</em> (Ford / Parsons / Kua, 2nd ed.): a fitness function is an automated, objective measure of an architectural characteristic, evaluated continuously. The 13 below transform this document's "we will…" promises into machine-enforced guards.</p> | |
| <table> | |
| <thead><tr><th>#</th><th>Fitness function</th><th>Target</th><th>Where it runs</th><th>Maps to</th></tr></thead> | |
| <tbody> | |
| <tr><td>FF1</td><td><strong>Tenant isolation matrix</strong> — every PHI table × every tenant role × CRUD <ins class="rlsfix">× row-scope</ins> = correct response. <ins class="rlsfix">June 2026 review fix (Q2): a third axis adds the intra-tenant negative cases the matrix previously missed — member A requesting member B's rows in the same tenant must return zero rows (this is exactly the case the F1 permissive-OR collapse would have leaked, and the old matrix would have passed it).</ins></td><td><ins class="rlsfix">15 PHI tables × 4 roles × 4 ops × 3 row-scopes (own row · other row same tenant · other tenant) = 720 assertions pass</ins></td><td>CI integration suite (Phase 2 onward); on every <ins class="authfirst">data-layer</ins> PR</td><td>§07 RLS · RT-6 <ins class="authfirst">· §6.1b — under auth-first this matrix runs through the API (AppSync + @auth) and is the PRIMARY isolation control, not a backstop</ins> <ins class="review2">· round-2 review: FF1 is a <strong>pre-PHI merge gate</strong> (coverage required before each PHI model’s PR; it does not exist yet) and must assert against <strong>custom Lambda-resolver operations’</strong> selection sets, not only generated CRUD — field-level redaction does not reach custom resolvers</ins></td></tr> | |
| <tr><td>FF2</td><td><strong>Fail-closed catch handlers</strong> — no <code>return event</code> from catch in <code>amplify/functions/</code></td><td>0 matches in grep</td><td>CI lint stage (Semgrep rule)</td><td>§53 · RT-2</td></tr> | |
| <tr><td>FF3</td><td><strong><ins class="rlsfix">Write-side</ins> RLS denial rate</strong> — production dispatcher logs SQLSTATE 42501 events <ins class="rlsfix">(WITH CHECK violations only — June 2026 review fix F3: read-side USING filtering never raises 42501; read-side isolation is verified by the tenant-isolation canary probes in §42/§47, the primary signal)</ins></td><td>= 0 over 24 h rolling</td><td>CloudWatch Logs Insights every 5 min</td><td>§42 alarm <ins class="authfirst">· deferred with RLS — at cutover the canary (FF1-live) is the production isolation signal</ins></td></tr> | |
| <tr><td>FF4</td><td><strong>CFN resource count per stack</strong></td><td>Warn ≥ 400; fail ≥ 480; max 500 (AWS hard limit)</td><td>CI post-synth</td><td>RT-3</td></tr> | |
| <tr><td>FF5</td><td><strong>Encryption at rest</strong> — all Aurora storage + S3 buckets use environment CMK (never aws/rds, aws/s3)</td><td>100% encrypted with environment CMK</td><td>AWS Config rules <code>rds-storage-encrypted</code>, <code>s3-default-encryption-kms</code></td><td>§20 · §51 §164.312(a)(2)(iv)</td></tr> | |
| <tr><td>FF6</td><td><strong>PHI fields ↔ scrubber coverage</strong> — every <code>PHI: true</code> field in schema appears in Pino redaction + Sentry scrubber</td><td>100% coverage</td><td>CI unit test reads schema + scrubber configs and diffs</td><td>RT-5 · §43</td></tr> | |
| <tr><td>FF7</td><td><strong>Heavy top-level imports</strong> — Sentry, AWS SDK, encryption SDK loaded at handler top-level in > 10 Lambdas</td><td>≤ 5 (intentional list maintained)</td><td>CI AST scan</td><td>RT-4 · §30</td></tr> | |
| <tr><td>FF8</td><td><strong>Tenant claim presence in JWT</strong> — every authenticated request to dispatcher has non-null trust_company_id</td><td>= 100% over 24 h</td><td>CloudWatch metric on customAuth Lambda</td><td>RT-2 · §27</td></tr> | |
| <tr><td>FF9</td><td><strong>chil_admin / chil_migrator usage</strong> — break-glass and DDL roles used only during scheduled windows</td><td>0 unscheduled use over 30 d</td><td>CloudWatch Insights on pgaudit; alarms (§42)</td><td>§24 · §51 §164.312(a)(2)(ii)</td></tr> | |
| <tr><td>FF10</td><td><strong>Lambda init duration P95</strong></td><td>< 2000 ms across all 76 Lambdas</td><td>CloudWatch Insights on Lambda Insights <code>init_duration</code></td><td>§42 alarm · §29 PC budget</td></tr> | |
| <tr><td>FF11</td><td><strong>Audit non-repudiation coverage</strong> — every mutating dispatcher op writes audit_event in same tx</td><td>100% of writes vs <code>audit_event</code> count per 5 min > 0.95 ratio</td><td>CloudWatch derived metric</td><td>§07 · §51 §164.312(b)</td></tr> | |
| <tr><td>FF12</td><td><strong>CMK rotation enabled</strong> on all 4 environment CMKs</td><td>All 4 CMKs have rotation enabled</td><td>AWS Config managed rule <code>cmk-backing-key-rotation-enabled</code></td><td>S6 sensitivity · §51 §164.312(a)(2)(iv)</td></tr> | |
| <tr><td>FF13</td><td><strong>CloudTrail integrity</strong> — Log File Validation passes</td><td>0 validation failures in 90 d</td><td>CloudTrail managed alarm (§42)</td><td>§34 · §51 §164.312(b)</td></tr> | |
| <tr><td>FF14 <span class="sev verified">SUPERSEDED</span></td><td><s><strong>RDS Proxy pinning ceiling.</strong></s> <strong>Superseded June 2026</strong> — Data API has no pooled connection to pin. Replaced by: Data API throttling + latency alarms (§42).</td><td>n/a</td><td>§42 Data API alarms</td><td>ADR-013 (superseded) · §B.6 S2</td></tr> | |
| <tr><td>FF15</td><td><strong>Tenant claim in every minted JWT</strong> — synthetic canary signs in + decodes the access token</td><td>100% of minted tokens carry non-null <code>trustCompanyId</code></td><td>CloudWatch synthetic canary every 5 min</td><td>C4 · §27 · RT-1</td></tr> | |
| <tr><td>FF16</td><td><strong>KMS decrypt request rate</strong> — Encryption SDK decrypts per day (cost + quota guard)</td><td>Alarm if > threshold/day → evaluate per-Lambda decrypt cache (TTL < 5 min)</td><td>CloudWatch on KMS <code>Decrypt</code> request count</td><td>C6 · ADR-009</td></tr> | |
| </tbody> | |
| </table> | |
| <p><strong>16 fitness functions.</strong> 9 run in CI on every PR; 7 run continuously in production via CloudWatch / Config. Together they enforce every claim made in this document. FF14–FF16 were added from the senior-architect review (Appendix E).</p> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== B.9 · ADR INVENTORY ====================================== --> | |
| <!-- =============================================================== --> | |
| <section id="adr-inventory"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · B.9</span> | |
| <h2 style="border:0;margin:18px 0 0;">ADR inventory — decisions that should be formalized</h2> | |
| </div> | |
| <p>Per Nygard / MADR / Y-Statement conventions: every architecturally significant decision deserves a record (Title · Status · Context · Decision · Consequences). The 11 below are the decisions <em>made</em> in this document but not yet extracted as ADRs. Phase 1 deliverable: write each as a one-page MADR file in <code>docs/adr/</code>.</p> | |
| <table> | |
| <thead><tr><th>ID</th><th>Title</th><th>Status</th><th>Where decided in this doc</th></tr></thead> | |
| <tbody> | |
| <tr><td>ADR-001</td><td>Use Aurora Serverless v2 PostgreSQL (not RDS PostgreSQL, not DocumentDB, not Aurora Provisioned)</td><td>Accepted</td><td>§00 · §01</td></tr> | |
| <tr><td>ADR-002</td><td>Use Postgres Row-Level Security (RLS) as the multi-tenancy spine, not application-layer policy</td><td>Accepted</td><td>§07</td></tr> | |
| <tr><td>ADR-003</td><td>Custom dispatcher Lambda (single, <strong>non-VPC via Data API</strong> — revised June 2026 from VPC-attached) over AppSync Aurora HTTP Resolver or per-resolver Lambdas</td><td>Accepted (revised)</td><td>§06 · RT-6 · T6</td></tr> | |
| <tr><td>ADR-004</td><td>Five Postgres roles (chil_app, chil_sqlconnect, chil_reporter, chil_admin, chil_migrator) — not three, not seven</td><td>Accepted</td><td>§24</td></tr> | |
| <tr><td>ADR-005</td><td>Drizzle Kit for migrations (dev-time only; pg driver at runtime, no ORM)</td><td>Accepted</td><td>§39</td></tr> | |
| <tr><td>ADR-006</td><td>Track B incremental migration with Amplify <code>connect-to-SQL</code> + <code>a.combine()</code>, not Track A 5-table consolidation</td><td>Accepted</td><td>Companion <code>CALL-PREP-UNIFIED.html</code>; T4 here</td></tr> | |
| <tr><td>ADR-007</td><td><s>RDS Proxy with two target groups (writer + reader) and IAM auth</s> → <strong>Superseded June 2026: Aurora Data API + IAM, no RDS Proxy.</strong> Writer/reader split is done via the Data API targeting the writer vs reader endpoint.</td><td>Superseded</td><td>§22 · T3</td></tr> | |
| <tr><td>ADR-008</td><td>S3 audit-archive Object Lock <strong>Compliance</strong> (not Governance) for 7-year retention</td><td>Accepted</td><td>§33 · T5</td></tr> | |
| <tr><td>ADR-009</td><td>Field-level encryption via AWS Encryption SDK envelope encryption (NOT pgcrypto in-database) — <strong>now with alternatives-considered</strong> (see E.2 C6)</td><td>Accepted</td><td>§20 · §43 · E.2</td></tr> | |
| <tr><td>ADR-010</td><td>Cognito pool stays externally owned; integrate, do not migrate</td><td>Accepted</td><td>§26 · §59 Q2 · T7</td></tr> | |
| <tr><td>ADR-011</td><td>Single VPC across environments via account separation, not per-tenant VPC</td><td>Accepted</td><td>§17 · §19</td></tr> | |
| <tr><td>ADR-012</td><td>Non-VPC Lambdas use env-var-injected secrets; VPC attachment reserved for DB-attached Lambdas only (prevents accidental ENI-cliff regressions)</td><td>Accepted</td><td>§28 · E.2 AG4-adjacent</td></tr> | |
| <tr><td><strong>ADR-013</strong></td><td><s><strong>RDS Proxy pinning stance</strong> — accept pinning, size <code>max_connections</code>.</s> <strong>Superseded June 2026:</strong> Aurora Data API chosen — no pooled connection to pin; <code>SET LOCAL</code> is transaction-scoped via the Data API. FF14 pinning alarm no longer applies; watch Data API throttling/latency instead.</td><td><strong>Superseded</strong></td><td>E.2 C1 · §B.6 S2</td></tr> | |
| </tbody> | |
| </table> | |
| <p><strong>13 ADRs to write.</strong> Y-Statement form (single sentence: "In the context of X, facing Y, we decided for Z, because of W, accepting downside V") is fastest; full MADR (annotated form) for ADR-002, ADR-003, ADR-006, <strong>ADR-013</strong> because those have the most consequences. Total effort: ~1.5 days for Oleksii. ADR-012 + ADR-013 added from the senior-architect review (Appendix E).</p> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== B.10 · MATURITY SCORECARD ================================ --> | |
| <!-- =============================================================== --> | |
| <section id="maturity"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · B.10</span> | |
| <h2 style="border:0;margin:18px 0 0;">Architecture maturity scorecard</h2> | |
| </div> | |
| <p>Composite scoring across all seven review lenses. <strong>5 = world-class</strong>; <strong>3 = production-ready</strong>; <strong>1 = needs work</strong>.</p> | |
| <table> | |
| <thead><tr><th>Lens</th><th>Score (post-Phase-5 target)</th><th>Notes</th></tr></thead> | |
| <tbody> | |
| <tr><td>arc42 documentation completeness</td><td><strong>4.0</strong></td><td>10/12 PASS once §B.3 + §B.9 close</td></tr> | |
| <tr><td>C4 model visual decomposition</td><td><strong>4.0</strong></td><td>All 3 levels now present (§B.3); diagrams ASCII-only — Structurizr / Mermaid would lift to 5</td></tr> | |
| <tr><td>ATAM quality attributes</td><td><strong>4.5</strong></td><td>18 scenarios with thresholds; phase gates explicit</td></tr> | |
| <tr><td>ATAM risk surface</td><td><strong>4.0</strong></td><td>7 themes named + mitigated; RT-1 (external Cognito) remains residual</td></tr> | |
| <tr><td>AWS Well-Architected</td><td><strong>4.0</strong></td><td>4 STRONG / 1 MEDIUM / 1 WEAK; Sustainability is the open pillar</td></tr> | |
| <tr><td>Fitness functions / governance</td><td><strong>4.5</strong></td><td>16 functions cataloged (FF14–16 from senior review); 100% automatable</td></tr> | |
| <tr><td>ADR record discipline</td><td><strong>3.0</strong></td><td>13 ADRs to write (incl. ADR-013 pinning); once written, lifts to 4.5</td></tr> | |
| <tr><td>HIPAA + SOC 2 control mapping</td><td><strong>4.5</strong></td><td>Every safeguard mapped to a phase + verifier; auditor evidence pack designed</td></tr> | |
| <tr style="background:var(--accent-soft);font-weight:700;"><td><strong>Composite (self-rating)</strong></td><td><strong>4.1 / 5</strong></td><td>"Production-ready, with named gap-closure items"</td></tr> | |
| <tr style="background:var(--accent-soft);font-weight:700;"><td><strong>Independent senior review</strong></td><td><strong>3.2 → 4.0 → 4.5</strong></td><td>3.2 today; 4.0 after the 6 criticals close; 4.5 after the 11 highs close (Appendix E). The gap from 4.1 is the criticals C1–C6.</td></tr> | |
| </tbody> | |
| </table> | |
| <p>The deficit lives in arc42-style documentation polish, ADR-record discipline, and the six critical findings in Appendix E. The reviewer's 3.2-today rating is the honest external number; this v3 closes or gates all six criticals, moving the fundable score toward 4.0+.</p> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== B.11 · GAP CLOSURE ====================================== --> | |
| <!-- =============================================================== --> | |
| <section id="gap-closure"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · B.11</span> | |
| <h2 style="border:0;margin:18px 0 0;">Pre-Phase-1 gap closure deliverables</h2> | |
| </div> | |
| <p>Every gap identified above maps to one of these 10 deliverables. None is large. All are owned by Oleksii unless noted. All produce artifacts that go into the auditor evidence pack (§54).</p> | |
| <table> | |
| <thead><tr><th>#</th><th>Deliverable</th><th>Closes</th><th>Effort</th><th>Owner</th></tr></thead> | |
| <tbody> | |
| <tr><td>1</td><td><strong>Constraints register</strong> — one-page MD listing all hard constraints (CFN 500, ENI 250, Cognito 5 s, BAA-eligible services list, US-only data residency, etc.)</td><td>arc42 §2</td><td>2 h</td><td>Oleksii</td></tr> | |
| <tr><td>2</td><td><strong>Sequence diagrams</strong> for PHI read with RLS · cutover flip · 4 h rollback. Mermaid in repo</td><td>arc42 §6 Runtime View</td><td>4 h</td><td>Oleksii</td></tr> | |
| <tr><td>3</td><td><strong>11 ADRs</strong> in <code>docs/adr/</code> per §B.9; MADR for ADR-002/3/6, Y-Statement for the rest</td><td>arc42 §9 / Decision discipline</td><td>1 day</td><td>Oleksii</td></tr> | |
| <tr><td>4</td><td><strong>Quality scenarios test harness</strong> — codify §B.4 utility tree as 18 named assertions in CI</td><td>arc42 §10; Fitness function coverage</td><td>2 days</td><td>Oleksii + Mike</td></tr> | |
| <tr><td>5</td><td><strong>GameDay schedule</strong> — quarterly chaos exercises beyond DR Drill (failed-claim test, RLS-bypass attempt, dispatcher OOM, cold-Cognito test)</td><td>Well-Architected Operational Excellence</td><td>1 day to design; recurring</td><td>Mike</td></tr> | |
| <tr><td>6</td><td><strong>GuardDuty + Macie + Security Hub</strong> stand-up with HIPAA + SOC 2 standard packs</td><td>Well-Architected Security; SOC 2 CC7</td><td>1 day</td><td>Mike</td></tr> | |
| <tr><td>7</td><td><strong>Multi-region DR decision & design</strong> — answer §59 Q13; if YES, design cross-region snapshot copy + DNS failover</td><td>Well-Architected Reliability</td><td>0.5 day decision + 2 days implementation if YES</td><td>Mike + Jon</td></tr> | |
| <tr><td>8</td><td><strong>Load test plan</strong> with target P95/P99 numbers per scenario (§B.4) · execute in dress rehearsal</td><td>Well-Architected Performance Efficiency</td><td>2 days plan + 1 day exec</td><td>Mike</td></tr> | |
| <tr><td>9</td><td><strong>Savings Plan / Reserved Instance analysis</strong> on steady-state Aurora + Lambda usage; commit if > 30% savings</td><td>Well-Architected Cost</td><td>0.5 day analysis</td><td>Mike</td></tr> | |
| <tr><td>10</td><td><strong>Sustainability subsection</strong> — Graviton2 for non-VPC Lambdas; CloudWatch log retention tiering (already in §44); document idle-ACU savings; commit to instance-rightsizing reviews quarterly</td><td>Well-Architected Sustainability</td><td>2 h documentation + 0.5 day Graviton migration</td><td>Mike</td></tr> | |
| <tr style="background:var(--accent-soft);"><td colspan="5"><strong>Added from senior-architect review (Appendix E) — these are blocking Phase-1 gates, not just gap-closure</strong></td></tr> | |
| <tr><td><strong>11</strong></td><td><strong>AWS feature-verification gate (C2)</strong> — written confirmation that Amplify <code>connect-to-SQL</code> is GA + BAA-eligible for our config; throwaway PoC wiring <code>a.combine([ddbSchema, sqlSchema])</code> with one non-PHI model on Aurora; verify auth rules + pagination + CFN resource impact. <strong>If it fails, all 41 models route through the dispatcher and Phase 2 re-baselines.</strong></td><td>De-risks Phase 2 foundation</td><td>2 days</td><td>Oleksii</td></tr> | |
| <tr><td><strong>12</strong></td><td><strong>External Cognito pool SLA (C4)</strong> — written doc: max 5-business-day attribute-change lead time, on-call contact, escalation path. Phase 1 is gated on receiving it in writing.</td><td>Unblocks Phase 1 + 3 (RT-1)</td><td>1 week (calendar dependency)</td><td>Mike</td></tr> | |
| <tr><td><strong>13</strong></td><td><strong>CFN root-cause evidence (C5)</strong> — pull <code>aws cloudformation describe-stack-events</code> for the most recent failed deploy; identify the failing substack + the resource type at the 500-limit; file the excerpt; choose remediation targeted to evidence <em>before</em> any Phase 1 infra. Do not assume schema bloat (RT-3).</td><td>Phase 1 starts on a working pipeline</td><td>1 day</td><td>Mike</td></tr> | |
| <tr><td><strong>14</strong></td><td><strong>Data-API IAM wiring verification</strong> <em>(was: RDS Proxy pinning ADR-013 — superseded)</em> — confirm the dispatcher role's <code>rds-data:*</code> + <code>secretsmanager:GetSecretValue</code> grants, and that no GUC leaks across Data API transactions (each is its own <code>transactionId</code>, connection reset between).</td><td>Confirms the Data-API tenancy path; closes the old pool-defeat / GUC-leak class structurally</td><td>0.5 day</td><td>Oleksii</td></tr> | |
| <tr><td><strong>15</strong></td><td><strong>Service Quotas review (H1)</strong> — confirm/raise VPC ENI-per-region (default 250), Lambda concurrency, RDS connections, KMS request rate before Phase 3.</td><td>Prevents Phase-3 ENI/quota outage</td><td>1 day</td><td>Mike</td></tr> | |
| </tbody> | |
| </table> | |
| <p><strong>Total gap-closure effort: ~15 person-days</strong> (10 original + ~5 from the senior review). Deliverables 11–15 are <strong>hard Phase-1 entry gates</strong>: do not provision Phase 1 infrastructure until #11 (connect-to-SQL PoC) and #13 (CFN root cause) pass — either can change the phased structure. Each produces a tangible artifact (file in the repo, dashboard, runbook entry, or test in CI) the auditor can verify.</p> | |
| <div class="callout ok"> | |
| <div class="callout-label">Post-closure self-score</div> | |
| With deliverables 1–10 complete the composite moves <strong>4.1 → 4.7 / 5.0</strong>. With the senior-review gates 11–15 | |
| and Appendix E closures folded in, the independent reviewer rates the plan <strong>3.2 today → 4.0 after the 6 criticals | |
| close → 4.5 after the 11 highs close</strong>. The remaining deficit lives in residual external dependencies (Cognito | |
| pool ownership, RT-1) and ADRs that cannot be finalized until they prove out in Phases 2–4. | |
| </div> | |
| </section> | |
| <hr> | |
| <div class="part-banner"> | |
| <div class="part-num">Appendix C</div> | |
| <h2 style="border:0;color:var(--ink);">Security + DevOps consolidated review</h2> | |
| <p>This appendix is the output of a security-architect lens + platform-engineer lens applied to the whole document. It produces (C.1) a per-field PHI inventory mapped to encryption tier, and (C.2) the consolidated punch list with closure status for every gap surfaced by the two reviews.</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== C.1 · PHI INVENTORY ===================================== --> | |
| <!-- =============================================================== --> | |
| <section id="phi-inventory"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · C.1</span> | |
| <h2 style="border:0;margin:18px 0 0;">PHI field inventory — every protected column, mapped to encryption tier</h2> | |
| </div> | |
| <p>Auditors require a per-field PHI map. The two tiers below define <strong>how</strong> a field is protected — every PHI field has Aurora-CMK at-rest (Tier 1, automatic) and a subset gets an additional envelope-encryption layer (Tier 2) so even a <code>chil_admin</code> BYPASSRLS read returns ciphertext.</p> | |
| <table> | |
| <thead> | |
| <tr><th>Tier</th><th>Mechanism</th><th>Who can decrypt</th><th>Use for</th></tr> | |
| </thead> | |
| <tbody> | |
| <tr><td><strong>Tier 1</strong></td><td>Aurora storage encryption with environment CMK</td><td>Any session with SELECT (subject to RLS)</td><td>All PHI by default. Protects against disk theft, snapshot exfil, cross-account read.</td></tr> | |
| <tr><td><strong>Tier 2</strong></td><td>Tier 1 <em>plus</em> AWS Encryption SDK envelope encryption with the field-encryption CMK (separate from storage CMK)</td><td>Only Lambdas with explicit KMS grant on field-encryption CMK <em>and</em> matching encryption context</td><td>Highest-sensitivity PHI: credentials, SSN, banking, medical directives. chil_admin BYPASSRLS still sees ciphertext.</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout rlsfix"><div class="callout-label">June 2026 review fix (F9) — Tier-2 ciphertext is non-queryable, by design + the escape hatch</div> | |
| Tier-2 ciphertext (AWS Encryption SDK, non-deterministic envelope encryption) cannot appear in <code>WHERE</code>, <code>ORDER BY</code>, or aggregations — intentionally: non-determinism is what makes a <code>chil_admin</code> BYPASSRLS read return noise. An audit of every current query pattern confirms nothing filters or sorts on a Tier-2 field (lookups run on <code>id</code>, <code>external_id</code>, email, name — all Tier 1 or plaintext). <strong>Escape hatch, pre-decided so nobody improvises it later:</strong> if a flow ever needs equality lookup on a Tier-2 field (e.g. find-member-by-SSN), add a <strong>blind index</strong> column — <code>ssn_bidx = HMAC-SHA256(key_bidx, normalized_value)</code> stored beside the ciphertext, with <code>key_bidx</code> held in KMS <em>separately</em> from the field-encryption CMK and grantable independently. Equality only; range queries and sorting on Tier-2 stay impossible by design (accepted product constraint). | |
| </div> | |
| <h3>C.1.1 — Field inventory</h3> | |
| <table> | |
| <thead><tr><th>Entity / column</th><th>PHI?</th><th>Tier</th><th>Rationale</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>user.firstName, lastName, middleName</code></td><td>Yes (HIPAA §164.514(b)(2)(i))</td><td>1</td><td>Identifier; RLS-protected; pgaudit-logged</td></tr> | |
| <tr><td><code>user.email</code></td><td>Yes (identifier)</td><td>1</td><td>RLS-protected. Hashed copy in <ins class="prdval">14</ins> audit tables (see §53 #3)</td></tr> | |
| <tr><td><code>user.phone, phoneNumber</code></td><td>Yes</td><td>1</td><td>RLS-protected</td></tr> | |
| <tr><td><code>user.dateOfBirth, birthdate</code></td><td>Yes (HIPAA §164.514(b)(2)(i)(C))</td><td>1</td><td>RLS-protected. <ins class="datarch">📄 Data-arch review: stored as a <em>string</em> today (Mike confirmed), no input timestamp → Aurora <code>birthdate DATE</code> column; people library becomes its own typed table with automatic timestamps.</ins></td></tr> | |
| <tr><td><code>user.ssn</code></td><td>Yes (highest sensitivity)</td><td><strong>2</strong></td><td>Field-level encryption + chil_admin cannot decrypt</td></tr> | |
| <tr><td><code>user.addressLine1, addressLine2, city, state, zip</code></td><td>Yes (geographic identifier)</td><td>1</td><td>RLS-protected</td></tr> | |
| <tr><td><code>medical_record.medical_directives</code></td><td>Yes (sensitive PHI)</td><td><strong>2</strong></td><td>Field-level encryption; only authorized medical-review Lambda decrypts</td></tr> | |
| <tr><td><code>medical_record.medical_conditions</code></td><td>Yes</td><td>1</td><td>RLS-protected + caseload-scoped via medical_review_caseload</td></tr> | |
| <tr><td><code>medical_record.medications, allergies, blood_type</code></td><td>Yes</td><td>1</td><td>RLS-protected</td></tr> | |
| <tr><td><code>medical_incident.*</code></td><td>Yes</td><td>1</td><td>RLS-protected; full pgaudit trail</td></tr> | |
| <tr><td><code>medical_review.*</code></td><td>Yes</td><td>1</td><td>RLS-protected; caseload-scoped</td></tr> | |
| <tr><td><code>hospital.name, address, fax, email, contactPhone</code></td><td>Yes</td><td>1</td><td>RLS-protected (provider directory entries)</td></tr> | |
| <tr><td><code>emergency_contact.firstName, lastName, phone, relationship</code></td><td>Yes</td><td>1</td><td>RLS-protected</td></tr> | |
| <tr><td colspan="4" style="background:var(--accent-soft);font-weight:700;">Care Document sections (today: embedded in <code>CareDocumentNew.data[]</code> JSON; after migration: <code>care_document_section</code> rows, one per section type)</td></tr> | |
| <tr><td><code>care_document_section[DigitalAssets].socialSecurityNumber</code></td><td>Yes (highest)</td><td><strong>2</strong></td><td>Field-level encryption; the canonical Tier-2 example. Today stored as plaintext in JSON blob — <strong>fix at migration</strong>. <ins class="teamfix">Confirmed by Vasyl’s audit (June 2026); the earlier PHI spec listed SSN as “not found” — it is present. Master passwords + financial credentials are plaintext in the same blob (gate 24).</ins></td></tr> | |
| <tr><td><code>care_document_section[DigitalAssets].passwordManagerPassword, passwordManagerEmail</code></td><td>Yes (highest)</td><td><strong>2</strong></td><td>Field-level encryption; the master credentials</td></tr> | |
| <tr><td><code>care_document_section[DigitalAssets].socialAccounts[].password, cloudAccounts[].password, otherAccounts[].password</code></td><td>Yes (highest)</td><td><strong>2</strong></td><td>Field-level encryption per credential entry — encryption context binds to (userId + accountType + accountId)</td></tr> | |
| <tr><td><code>care_document_section[Financial].bankAccounts[].accountNumber, routingNumber</code></td><td>Yes (highest)</td><td><strong>2</strong></td><td>Field-level encryption; chil_admin cannot decrypt</td></tr> | |
| <tr><td><code>care_document_section[Financial].investmentAccounts[].accountNumber</code></td><td>Yes (highest)</td><td><strong>2</strong></td><td>Field-level encryption</td></tr> | |
| <tr><td><code>care_document_section[Financial].lifeInsurance[].policyNumber</code></td><td>Yes (highest)</td><td><strong>2</strong></td><td>Field-level encryption</td></tr> | |
| <tr><td><code>care_document_section[Medical].advanceDirectives, livingWill, healthcareProxy</code></td><td>Yes (highest)</td><td><strong>2</strong></td><td>Field-level encryption — directives are most-sensitive PHI; restricted to medical-review caseload + decrypt grant</td></tr> | |
| <tr><td><code>care_document_section[Medical].medicalConditions, medications, allergies</code></td><td>Yes</td><td>1</td><td>RLS-protected; caseload-scoped (no Tier-2 because clinicians need fast read)</td></tr> | |
| <tr><td><code>care_document_section[Medical].hospitalPreferences, doctors[]</code></td><td>Yes</td><td>1</td><td>RLS-protected</td></tr> | |
| <tr><td><code>care_document_section[Legal].willLocation, attorneyContact</code></td><td>Yes</td><td>1</td><td>RLS-protected</td></tr> | |
| <tr><td><code>care_document_section[Funeral].arrangements, burialWishes, religiousPreferences</code></td><td>Yes</td><td>1</td><td>RLS-protected; sensitive personal preferences</td></tr> | |
| <tr><td><code>care_document_section[*].notes, freeText</code></td><td>Maybe (often contains PHI)</td><td>1</td><td>Free-text fields — structured-logger PHI scrubber runs at log time; never appears in error reports</td></tr> | |
| <tr><td colspan="4" style="background:var(--accent-soft);font-weight:700;">Interview responses (today: embedded in <code>user.interviewProgressV2</code> JSON; after migration: <code>user_interview_step</code> rows)</td></tr> | |
| <tr><td><code>user_interview_step.medical_history_response</code></td><td>Yes</td><td>1</td><td>RLS-protected; may contain medical-condition disclosure</td></tr> | |
| <tr><td><code>user_interview_step.family_history_response</code></td><td>Yes</td><td>1</td><td>RLS-protected; identifies relatives</td></tr> | |
| <tr><td><code>user_interview_step.mental_health_response</code></td><td>Yes (sensitive)</td><td><strong>2</strong></td><td>Field-level encryption — mental health responses are 42 CFR Part 2-style protected</td></tr> | |
| <tr><td><code>user_interview_step.financial_situation_response</code></td><td>Maybe (PII)</td><td>1</td><td>RLS-protected</td></tr> | |
| <tr><td><code>user_interview_step.free_text_responses (JSONB)</code></td><td>Maybe (PHI in free text)</td><td>1</td><td>RLS-protected; PHI scrubber on log lines</td></tr> | |
| <tr><td colspan="4" style="background:var(--accent-soft);font-weight:700;">Standalone PHI tables (not embedded; relational from day one)</td></tr> | |
| <tr><td><code>banking_info.account_number, routing_number</code></td><td>Yes (highest)</td><td><strong>2</strong></td><td>Field-level encryption; chil_admin cannot decrypt</td></tr> | |
| <tr><td><code>insurance.policy_number, group_number, providerName</code></td><td>Yes (highest for policy_number)</td><td><strong>2</strong> for policy/group · 1 for providerName</td><td>Field-level encryption on the secrets; provider name is RLS-protected</td></tr> | |
| <tr><td><code>document.content</code> (legal docs)</td><td>Yes (often)</td><td>1</td><td>RLS-protected; D2 calls for off-DDB storage for > 300 KB<ins class="rlsfix">; in Aurora, bodies > 256 KB live in S3 (pointer + sha256 in the row) so a single row can never breach the Data API ~1 MB result cap (review F6, §6.2)</ins></td></tr> | |
| <tr><td><code>support_message.body</code> (member-authored)</td><td>Maybe (often contains PHI)</td><td>1</td><td>RLS-protected; access scoped to thread + welon assignment</td></tr> | |
| <tr><td><code>member_note.body</code></td><td>Maybe</td><td>1</td><td>RLS-protected + welon-assignment-scoped (§53 #5)</td></tr> | |
| <tr><td><code>audit_event.detail (JSONB)</code></td><td>Maybe (depends on op)</td><td>1</td><td>RLS-protected; structured-logger PHI scrubber runs before writing</td></tr> | |
| <tr><td colspan="4" style="background:var(--accent-soft);font-weight:700;">Non-PHI / hashed PII (no encryption needed)</td></tr> | |
| <tr><td><code>auth_attempt.email_hash, email_last4</code></td><td>No (hashed PII)</td><td>n/a</td><td>One-way hash; not reversible PHI</td></tr> | |
| <tr><td><code>subscription.stripe_customer_id, stripe_subscription_id</code></td><td>No (Stripe-side PHI)</td><td>n/a</td><td>Stripe is BAA-covered; we hold opaque IDs only</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout crit"> | |
| <div class="callout-label">Migration-time fix: Care Document sections store credentials as PLAINTEXT today</div> | |
| <strong>Today</strong>, <code>CareDocumentNew.data[].DigitalAssets</code> + <code>data[].Financial</code> + | |
| <code>data[].Medical.advanceDirectives</code> are stored as plaintext JSON inside a DDB string attribute. They are PHI | |
| and they are <em>not encrypted at field level</em>. Aurora storage CMK protects them at rest, but a chil_admin BYPASSRLS | |
| read returns plaintext. <strong>The Phase 3 field-encryption Lambda must encrypt every Tier-2 field above as part of the | |
| non-PHI → PHI cutover.</strong> The frontend already sends and receives plaintext — the <code>encrypt-field</code> Lambda | |
| runs server-side at write; <code>decrypt-field</code> at read. No UX change. | |
| </div> | |
| <div class="callout ok"> | |
| <div class="callout-label">Coverage rule (enforced by FF6)</div> | |
| <strong>Every <code>PHI: true</code> field above is reflected in:</strong> the Pino redaction key list, the Sentry | |
| <code>beforeSend</code> scrubber's <code>PHI_KEYS</code> set, the structured-logger redact path, and (for Tier-2) the | |
| encrypt-field / decrypt-field Lambdas' allowed-fields list. <strong>FF6 in §B.8 compares all four lists against a single | |
| generated source (<code>amplify/data/phi-fields.json</code>) and fails CI if any drift.</strong> | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== C.2 · CONSOLIDATED REVIEW =============================== --> | |
| <!-- =============================================================== --> | |
| <section id="sec-devops-review"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · C.2</span> | |
| <h2 style="border:0;margin:18px 0 0;">Security + DevOps consolidated review — punch list</h2> | |
| </div> | |
| <p>Two reviews were applied to this document: a security-architect review (auth chain, multi-tenant isolation, encryption, audit trail, HIPAA/SOC 2 mapping, insider threat) and a DevOps/platform-engineer review (account topology, Aurora sizing, Lambda VPC, networking, CI/CD, observability, backup/DR, cutover, cost, rollback). Findings consolidated below with the closure status applied to this version of the document.</p> | |
| <h3>C.2.1 — Security findings</h3> | |
| <table> | |
| <thead><tr><th>Finding</th><th>Severity</th><th>Where</th><th>Closure</th></tr></thead> | |
| <tbody> | |
| <tr><td><strong>S1</strong> · PHI field inventory missing — auditor needs a per-field map to encryption tier</td><td>CRITICAL</td><td>§51–52 lacked it</td><td><span class="sev verified">CLOSED</span> — <strong>Appendix C.1 above</strong></td></tr> | |
| <tr><td><strong>S2</strong> · CI security gates incomplete — no DAST on AppSync; IaC scan limited</td><td>HIGH</td><td>§38, §B.8 FF4</td><td><span class="sev verified">CLOSED</span> — §38 now lists Semgrep + pnpm audit + TruffleHog + checkov (IaC). DAST added to Phase 1 deliverable: AppSync pre-prod scan with OWASP ZAP / Burp before §53 sign-off.</td></tr> | |
| <tr><td><strong>S3</strong> · Insider threat — chil_admin BYPASSRLS lacks compensating controls beyond pgaudit</td><td>HIGH</td><td>§24</td><td><span class="sev verified">CLOSED</span> — §24 now lists 5 compensating controls: temp creds only · Jira-ticket approval · Tier-2 encryption · real-time Slack alert · quarterly review</td></tr> | |
| <tr><td><strong>S4</strong> · Encryption-as-checklist vs encryption-as-default — selective field encryption ambiguous</td><td>HIGH</td><td>§20, §43, RT-5</td><td><span class="sev verified">CLOSED</span> — Appendix C.1 defines explicit Tier 1 / Tier 2 with mechanism + decryptor + use-case. RT-5 references the inventory.</td></tr> | |
| <tr><td><strong>S5</strong> · RLS on reporter role not explicitly documented — risk of cross-tenant data leak via reporting</td><td>HIGH</td><td>§09</td><td><span class="sev verified">CLOSED</span> — §09 now has explicit RLS-enforcement callout for chil_reporter on the reader endpoint, with FF1 coverage</td></tr> | |
| <tr><td><strong>S6</strong> · CDC catch-up time tolerance not validated — 24 h DDB Streams retention vs unknown actual catch-up</td><td>HIGH</td><td>§D5, §14, §47</td><td><span class="sev verified">CLOSED</span> — §42 alarm for DDB Streams lag > 5 min added; §53 #18 hard gate requires CDC lag < 5 min sustained for 7 days pre-cutover</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>C.2.2 — DevOps findings</h3> | |
| <table> | |
| <thead><tr><th>Finding</th><th>Severity</th><th>Where</th><th>Closure</th></tr></thead> | |
| <tbody> | |
| <tr><td><strong>D-OP-1</strong> · VPC endpoint list missing CloudWatch Logs, STS (forces NAT traffic)</td><td>HIGH</td><td>§19.2</td><td><span class="sev verified">CLOSED</span> — §19.2 already lists Logs + STS + KMS + Secrets + S3 + DDB gateways. Verified at line 1438–1442.</td></tr> | |
| <tr><td><strong>D-OP-2</strong> · Monitoring blind spots — NAT data transfer, Aurora storage growth, RDS Proxy saturation, DDB Streams lag</td><td>HIGH</td><td>§42 alarms</td><td><span class="sev verified">CLOSED</span> — 5 alarms added to §42: NAT data transfer spike · Aurora storage growth trajectory · RDS Proxy connection queue latency · DDB Streams lag · RDS Proxy max connections approaching. Total alarms now 21.</td></tr> | |
| <tr><td><strong>D-OP-3</strong> · postSignUpTrigger cold-start risk against 5 s Cognito timeout</td><td>HIGH</td><td>§D5</td><td><span class="sev verified">CLOSED</span> — §53 #16 hard gate requires P95 init < 3 s under cold start with synthetic canary; raise PC to 2 if not met</td></tr> | |
| <tr><td><strong>D-SZ-1</strong> · Aurora ACU peak range undocumented — no load-test data</td><td>MEDIUM</td><td>§21</td><td><span class="sev verified">CLOSED</span> — §53 #17 hard gate requires load test at 2× projected peak with report</td></tr> | |
| <tr><td><strong>D-SZ-2</strong> · Backup / Glacier long-tail storage not itemized</td><td>MEDIUM</td><td>ops</td><td>Outstanding — S3 audit-archive Glacier growth not itemized. Track in the cost model if/when the cost section is reinstated (the cost section was removed from this document).</td></tr> | |
| <tr><td><strong>D-SZ-3</strong> · RDS Proxy connection saturation behavior unspecified</td><td>HIGH</td><td>§22, §42</td><td><span class="sev verified">CLOSED</span> — §42 now alarms on RDS Proxy <code>QueryDatabaseResponseLatency</code> P95 > 100 ms (queue backup signal). Phase-1 deliverable: document <code>max_connections</code> on Aurora parameter group (recommend 200 baseline; tune from load test) and confirm RDS Proxy <code>connection_borrow_timeout</code> = 30 s (returns timeout, not 503).</td></tr> | |
| <tr><td><strong>D-INC-1</strong> · Field-encryption scope vague — canonical model doesn't mark encrypted columns</td><td>MEDIUM</td><td>§02</td><td><span class="sev verified">CLOSED</span> — Appendix C.1 above is the authoritative list with column-level granularity; FF6 enforces drift detection.</td></tr> | |
| <tr><td><strong>D-INC-2</strong> · Rollback data replay Lambda details missing — risk of duplicate or missing replays</td><td>CRITICAL</td><td>§47</td><td><span class="sev verified">CLOSED</span> — §53 #19 hard gate: rollback replay procedure documented with SQL query against <code>audit_event</code> + idempotency key + one-shot Lambda code reviewed before cutover.</td></tr> | |
| <tr><td><strong>D-INC-3</strong> · Lambda VPC-attachment pattern undocumented as ADR (future Lambdas may VPC-attach by accident)</td><td>MEDIUM</td><td>§28</td><td>Outstanding — add ADR-012 ("Non-VPC Lambdas use env-var-injected secrets; VPC reserved for DB-attached only") to §B.9 inventory.</td></tr> | |
| <tr><td><strong>D-COST</strong> · GuardDuty + Macie + Security Hub run-cost not itemized</td><td>LOW</td><td>§B.11 #6</td><td>Outstanding — security-tooling run-cost not itemized (cost section removed); gap-closure deliverable §B.11 #6 covers standing the tools up.</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>C.2.3 — Items preserved as strengths</h3> | |
| <table> | |
| <thead><tr><th>Strength</th><th>Where</th><th>Why preserve</th></tr></thead> | |
| <tbody> | |
| <tr><td><strong>Fail-closed architecture end-to-end</strong></td><td>§11 (preTokenGen), §27 (customAuth), §53 #9 + #12 (gates), RT-2 (theme), FF2 (lint enforcement)</td><td>Single strongest control against auth bypass — every layer enforced + automated</td></tr> | |
| <tr><td><strong>RLS + role segregation as multi-tenancy spine</strong></td><td>§07 (policies), §24 (roles), §09 (reader RLS), T3 (tradeoff), B.5 RT-6 (single-path risk)</td><td>Clean separation: no role has both BYPASSRLS and persistent credentials; ADR-002 + ADR-004 lock this in</td></tr> | |
| <tr><td><strong>Three-layer audit trail</strong></td><td>§07 (audit_event transactional outbox), §23 (pgaudit), §33 (S3 audit-archive Object Lock Compliance), §34 (CloudTrail Log File Validation)</td><td>Each layer tamper-resistant; coverage redundant; auditor-friendly</td></tr> | |
| <tr><td><strong>Phase-4 hard gates</strong></td><td>§53 (now 21 non-negotiable items)</td><td>Closes the silent-compliance-decay failure mode; auditor walks the list</td></tr> | |
| <tr><td><strong>DR drill quantified + scheduled</strong></td><td>§45 (mechanics), §B.4 (RPO/RTO scenarios)</td><td>Evidence-generating activity vs. shelf plan; addresses SOC 2 CC7.5</td></tr> | |
| <tr><td><strong>Cost transparency with breakeven analysis</strong></td><td>(cost detail removed from this document)</td><td>Net delta with DDB is small; reporting pipeline "comes free"; auditor verifiable</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>C.2.4 — Bottom line</h3> | |
| <div class="callout ok"> | |
| <div class="callout-label">Post-review verdict</div> | |
| Of the <strong>16 review findings</strong> (6 security · 10 DevOps), <strong>13 are CLOSED inline in this version of the | |
| document</strong> and 3 remain Outstanding (D-SZ-2 backup line item · D-INC-3 ADR-012 · D-COST GuardDuty/Macie/Security | |
| Hub cost). All 3 Outstanding items are documentation-only follow-ups; none gate Phase 1 or Phase 4. The 6 preserved | |
| strengths cover the controls auditors look for first: fail-closed auth, multi-tenancy isolation, redundant audit trail, | |
| hard pre-cutover gates, evidence-generating DR drills, and cost transparency. <strong>Document is ready for Phase 0 | |
| sign-off.</strong> | |
| </div> | |
| </section> | |
| <hr> | |
| <div class="part-banner"> | |
| <div class="part-num">Appendix D</div> | |
| <h2 style="border:0;color:var(--ink);">Tenant architecture — deep dive (AWS SaaS Lens applied)</h2> | |
| <p>Why we land on pooled Postgres + RLS rather than a per-tenant database; under what conditions we'd add an additional silo for a single enterprise tenant; how the tenant lifecycle works end-to-end. This appendix is the longest single answer to "is this really the right multi-tenancy architecture for us?"</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== D.1 · SILO / POOL / BRIDGE ============================== --> | |
| <!-- =============================================================== --> | |
| <section id="tenant-deep-dive"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · D.1</span> | |
| <h2 style="border:0;margin:18px 0 0;">Silo / Pool / Bridge — applied to our case</h2> | |
| </div> | |
| <div class="callout"> | |
| <div class="callout-label">What "tenant" means here</div> | |
| A <strong>tenant = a Trust Company</strong> (organization that buys the Childfree Legacy platform and onboards its | |
| members). Members of a trust are NOT tenants themselves — they're scoped within one trust. Welon staff is scoped | |
| within one trust. <strong><code>trustCompanyId</code> = the tenant boundary that every PHI read/write is scoped by.</strong> | |
| Today: 2 known trusts; product ambition could grow to ~5–20 over 2 years; not a "thousand-trust SaaS." | |
| </div> | |
| <h3>D.1.1 — The three canonical models</h3> | |
| <p>The AWS SaaS Lens of the Well-Architected Framework defines three tenant-isolation models. Each is a different | |
| position on the cost-vs-isolation spectrum.</p> | |
| <table> | |
| <thead><tr><th>Model</th><th>Database</th><th>Compute (Lambda)</th><th>Storage (S3)</th><th>Pros</th><th>Cons</th></tr></thead> | |
| <tbody> | |
| <tr> | |
| <td><strong>Silo</strong></td> | |
| <td>One DB per tenant. Strongest isolation. Tenant has dedicated Aurora cluster (or schema).</td> | |
| <td>Dedicated Lambda set per tenant (rare) or shared with strict identity-based ACL.</td> | |
| <td>Per-tenant S3 bucket(s)</td> | |
| <td>Tenant-level blast-radius isolation · easier per-tenant compliance certification · easier "we'll delete your data" assurance · easier per-tenant noisy-neighbor management</td> | |
| <td>Highest cost (N × cluster cost) · slowest onboarding · per-tenant schema drift if not careful · hardest to do cross-tenant analytics</td> | |
| </tr> | |
| <tr> | |
| <td><strong>Pool</strong></td> | |
| <td>One shared DB across all tenants. RLS policies + <code>SET LOCAL app.trust_company_id</code> per request.</td> | |
| <td>Shared Lambda pool with tenant claim in JWT.</td> | |
| <td>Shared S3 bucket with tenant prefix.</td> | |
| <td>Lowest cost (one cluster) · fastest onboarding (just an INSERT) · simplest analytics · simplest schema evolution</td> | |
| <td>RLS bug → cross-tenant leak · noisy-neighbor at DB layer · "delete tenant" is more work · auditor scrutiny on RLS coverage</td> | |
| </tr> | |
| <tr> | |
| <td><strong>Bridge</strong></td> | |
| <td>Pool for standard tenants + Silo for enterprise tenants who pay for isolation.</td> | |
| <td>Shared Lambda; tenant routing layer chooses target DB.</td> | |
| <td>Shared bucket with tenant prefix + dedicated bucket(s) for siloed tenants.</td> | |
| <td>Best of both — pool for cost efficiency, silo for big-customer demand · gradual silo migration possible</td> | |
| <td>Most operational complexity — two ops models · routing layer is a new failure point · CI test matrix doubles</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <h3>D.1.2 — Decision matrix scored for our case</h3> | |
| <p>Each row is a real characteristic of Childfree Legacy as of today. Scoring: <strong>5 = strong fit</strong>; | |
| <strong>1 = poor fit</strong>.</p> | |
| <table> | |
| <thead><tr><th>Characteristic</th><th>Silo</th><th>Pool (our choice)</th><th>Bridge</th></tr></thead> | |
| <tbody> | |
| <tr><td><strong>Tenant count today (2 trusts) and 2-year horizon (~5–20)</strong></td><td>3 — workable, but every new tenant adds an Aurora baseline cost</td><td><strong>5</strong> — pool is the obvious choice at this scale</td><td>4 — overkill for now; promote to bridge only when one enterprise tenant demands isolation</td></tr> | |
| <tr><td><strong>HIPAA per-tenant compliance posture</strong></td><td>5 — auditor walkthrough is trivial: "this tenant's data lives in this cluster"</td><td><strong>4</strong> — auditor needs the Tenant-Isolation Matrix (FF1) to be convinced; achievable with RLS + tests</td><td>5 — same as silo per-enterprise; pooled tenants like pool</td></tr> | |
| <tr><td><strong>Cost at our scale (per tenant)</strong></td><td>1 — each tenant gets a dedicated cluster floor before usage</td><td><strong>5</strong> — marginal cost per tenant near-zero until 50k+ members</td><td>2 — silo cost dominates the bill once one enterprise siloes</td></tr> | |
| <tr><td><strong>Onboarding speed</strong></td><td>2 — provision cluster + migrate schema + setup Cognito group + DNS = ~hours</td><td><strong>5</strong> — one INSERT + Cognito group + Stripe customer = ~seconds</td><td>3 — depends on tier</td></tr> | |
| <tr><td><strong>Schema evolution</strong></td><td>2 — N clusters to migrate, schema drift risk grows quadratically</td><td><strong>5</strong> — one Drizzle migration, one CI run</td><td>3 — must run migrations on N+1 schemas, harder to test</td></tr> | |
| <tr><td><strong>Cross-tenant analytics (platform reporting)</strong></td><td>1 — must ETL N clusters into a warehouse; cost compounds</td><td><strong>5</strong> — single reader endpoint; same SQL</td><td>2 — same ETL problem for the siloed subset</td></tr> | |
| <tr><td><strong>Noisy-neighbor risk</strong></td><td>5 — fully isolated</td><td>3 — Aurora is connection-pooled at RDS Proxy; per-tenant rate limiting at AppSync mitigates; statement_timeout caps query cost</td><td>4 — depends on which side the noisy tenant lands</td></tr> | |
| <tr><td><strong>Blast radius of a single bug</strong></td><td>5 — bug only affects one tenant</td><td>3 — bug in RLS or dispatcher affects all tenants; mitigated by FF1 + dispatcher PR gating</td><td>4 — silo half is safe; pool half not</td></tr> | |
| <tr><td><strong>"Delete tenant" flow (HIPAA right-to-erasure)</strong></td><td>5 — drop cluster, snapshot to audit-archive, done</td><td>3 — DELETE rows + verify cascade + retain in audit_event 6 years</td><td>4 — depends on tier</td></tr> | |
| <tr><td><strong>Audit complexity</strong></td><td>3 — each cluster has its own CloudTrail / pgaudit / backups</td><td><strong>5</strong> — single audit trail, single CMK rotation, single PITR config</td><td>3 — two flows to audit</td></tr> | |
| <tr><td><strong>Engineering team size to operate</strong></td><td>2 — N clusters means N times the operational surface; needs a platform team</td><td><strong>5</strong> — one cluster, one Mike</td><td>3 — between silo + pool ops cost</td></tr> | |
| <tr style="background:var(--accent-soft);font-weight:700;"><td><strong>Composite</strong></td><td><strong>2.8</strong></td><td><strong>4.4</strong></td><td><strong>3.4</strong></td></tr> | |
| </tbody> | |
| </table> | |
| <p>Pool wins decisively for our scale + tenant count + team size. The Bridge model is the upgrade path if a future enterprise tenant demands silo.</p> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== D.2 · THE PATTERN ======================================= --> | |
| <!-- =============================================================== --> | |
| <section id="tenant-pattern"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · D.2</span> | |
| <h2 style="border:0;margin:18px 0 0;">The pattern we land on, why, and when we'd migrate to Bridge</h2> | |
| </div> | |
| <h3>D.2.1 — The pooled pattern, end-to-end</h3> | |
| <table> | |
| <thead><tr><th>Layer</th><th>Tenant scoping mechanism</th><th>Failure mode if mechanism breaks</th></tr></thead> | |
| <tbody> | |
| <tr><td>Authentication</td><td>Cognito user pool (single pool, all tenants); tenant identity attached as <code>trustCompanyId</code> custom attribute on each user; injected as JWT claim by preTokenGen</td><td>Wrong tenant claim → wrong RLS context → leak (FF8 catches missing claim; preTokenGen fails-closed)</td></tr> | |
| <tr><td>API gateway</td><td>AppSync GraphQL; customAuth Lambda verifies JWT, refuses if <code>trustCompanyId</code> missing</td><td>Refused requests; never reaches data layer</td></tr> | |
| <tr><td>Authorization (non-PHI)</td><td>Amplify <code>connect-to-SQL</code> runs as the single <code>chil_sqlconnect</code> role and <strong>cannot set a per-request <code>app.trust_company_id</code> GUC</strong>, so database RLS does not auto-filter it. Non-PHI tenant isolation is enforced by <strong>Amplify <code>@auth</code></strong> (owner/group rules carrying the <code>trustCompanyId</code> claim — the POC-validated path). RLS is the spine for the <em>PHI / dispatcher</em> models only (§7). <em>This caveat disappears only if the team adopts one-dispatcher-for-all (§6 decision).</em></td><td>A non-PHI model shipped without its <code>@auth</code> tenant rule → leak (caught by the per-model isolation tests, not by an RLS policy)</td></tr> | |
| <tr><td>Authorization (PHI)</td><td>Dispatcher Lambda runs as <code>chil_app</code>; <code>BEGIN; SET LOCAL app.trust_company_id = '<value>'</code> on every request; RLS enforces</td><td>Dispatcher bug → leak (FF1, RT-6 mitigation: dispatcher PR mandatory Oleksii + Mike review)</td></tr> | |
| <tr><td>Reporting</td><td><code>chil_reporter</code> on reader endpoint; same RLS policies; one query at a time</td><td>Same as above; FF1 includes reader queries</td></tr> | |
| <tr><td>Storage (S3)</td><td>Bucket policies + per-tenant prefix: <code>s3://bucket/{trustCompanyId}/...</code>; CloudFront signed URLs include tenant prefix</td><td>Bucket policy bug → cross-tenant read; mitigated by IAM-conditioned aws:PrincipalTag matching tenant claim</td></tr> | |
| <tr><td>Encryption</td><td>One environment CMK; encryption context binds ciphertext to (userId + tenantId) so cross-tenant copy fails to decrypt</td><td>Wrong encryption context → decrypt fails (audit-visible)</td></tr> | |
| <tr><td>Audit</td><td>Single audit_event table with <code>trust_company_id</code> column; pgaudit; CloudTrail; S3 audit-archive Object Lock</td><td>n/a — audit always written; tenant-scoped queries on audit trail</td></tr> | |
| <tr><td>Observability</td><td>CloudWatch dashboards with tenant_id dimension on key metrics; cost-allocation tags include tenant</td><td>Per-tenant noisy-neighbor visible in real time</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>D.2.2 — Why pooled wins for us (versus alternatives we considered)</h3> | |
| <ol> | |
| <li><strong>Tenant count = 2 today, capped <~20 over 2 years.</strong> Silo's per-tenant cluster floor means a large fixed compute cost per tenant. Pool's marginal cost is near-zero until ~50k members per tenant.</li> | |
| <li><strong>Auditor preference matches.</strong> Healthcare SaaS auditors have seen pooled-with-RLS for years. They look for the Tenant Isolation Matrix (FF1) and the audit trail to confirm. Silo would simplify their job but cost us 10× more for no engineering benefit.</li> | |
| <li><strong>Cross-tenant reporting is built-in.</strong> Our product roadmap includes platform-wide KPIs (e.g., "trust onboarding velocity," "average member time-to-first-care-doc"). With pool, one SQL query on the reader. With silo, an ETL pipeline (extra cost + ops) before we can answer the question.</li> | |
| <li><strong>Single-cluster operational simplicity.</strong> One DR drill quarterly. One CMK rotation. One PITR config. One AWS Config rule set. Mike + Oleksii can hold this in their heads; silo's N-cluster ops would require a platform team.</li> | |
| <li><strong>HIPAA does not require silo.</strong> The Safeguards (§164.312) ask for strong technical controls; they are agnostic on whether tenants share a database. RLS + role segregation + encryption + audit trail is a complete answer.</li> | |
| </ol> | |
| <h3>D.2.3 — When we'd upgrade to Bridge (the trigger conditions)</h3> | |
| <p>Bridge means "pool for standard tier, silo for one specific tenant." Each trigger below would push a single enterprise tenant onto a dedicated Aurora cluster:</p> | |
| <ol> | |
| <li><strong>Tenant contractually demands data residency in a different AWS region.</strong> Our pool is us-east-1; if a tenant requires us-west-2 or eu-central-1 (no GDPR exposure planned but possible), they get a regional silo.</li> | |
| <li><strong>Tenant exceeds 30% of total dataset</strong> and contributes to noisy-neighbor. Past ~50k members in one tenant, query plans diverge; siloing them isolates their performance.</li> | |
| <li><strong>Tenant insists on holding their own KMS keys (CMK BYOK or external-key-store).</strong> Pool uses our environment CMK; tenant-controlled CMK requires a silo.</li> | |
| <li><strong>Tenant runs their own audit / compliance tooling that needs raw DB access</strong> (rare; usually a contract negotiation point). Silo gives them their own pgaudit stream.</li> | |
| <li><strong>Acquisition / spin-off</strong> where part of the tenant base detaches into a separate corporate entity.</li> | |
| </ol> | |
| <p><strong>Probability over 2 years: ~25%</strong> that we hit at least one trigger. Plan for it but don't build it until needed. The Bridge migration is well-trodden territory: spin up a dedicated cluster, replicate the schema, dual-write for a cutover window, drop the tenant from the pool's RLS scope.</p> | |
| <h3>D.2.4 — What would make us regret pool (red flags to monitor)</h3> | |
| <ul> | |
| <li>RLS denial rate (FF3) trending up — could mean policy drift after a schema change.</li> | |
| <li>Cross-tenant noisy-neighbor: P95 dispatcher latency by tenant diverges > 50%.</li> | |
| <li>Tenant Isolation Matrix (FF1) takes > 5 min to run as we add tables (slow tests mean fewer runs).</li> | |
| <li>A single tenant's row count exceeds 10× the next-biggest tenant — disk allocation patterns degrade.</li> | |
| <li>Auditor pushes back on "single CMK encrypts all tenants' data" — could need per-tenant CMK (silo trigger).</li> | |
| </ul> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== D.3 · TENANT LIFECYCLE ================================== --> | |
| <!-- =============================================================== --> | |
| <section id="tenant-lifecycle"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · D.3</span> | |
| <h2 style="border:0;margin:18px 0 0;">Tenant lifecycle — onboarding to off-boarding</h2> | |
| </div> | |
| <h3>D.3.1 — Onboard a new tenant</h3> | |
| <pre class="sh"><code>1. Insert tenant row: | |
| INSERT INTO trust_company (id, name, created_at, status) | |
| VALUES (uuid_generate_v4(), 'New Trust LLC', now(), 'active'); | |
| 2. Create Cognito group for tenant admins: | |
| aws cognito-idp create-group \ | |
| --user-pool-id $POOL --group-name "trust-admins-$TENANT_ID" | |
| 3. Provision initial trust-admin user(s) (sends invite email): | |
| POST /api/admin/invite-user | |
| { trustCompanyId, email, role: 'TRUST_ADMIN' } | |
| 4. Templates: clone shared base templates into tenant scope | |
| (or leave shared-only — depends on §59 Q15 product decision). | |
| 5. S3: confirm tenant prefix is allowed in the bucket policy | |
| (already wildcard; no per-tenant config change needed). | |
| 6. Stripe: create Stripe customer for billing. | |
| 7. Smoke test: | |
| - Sign in as trust admin | |
| - Verify trust_admin sees only their tenant's data | |
| - Tenant Isolation Matrix (FF1) re-runs in CI with new tenant | |
| Total time: ~5 minutes; mostly the invite email round-trip.</code></pre> | |
| <h3>D.3.2 — Add a member to an existing tenant</h3> | |
| <pre class="sh"><code>1. Trust admin clicks "Invite member" | |
| 2. App generates invite token, INSERT into user_invite (trustCompanyId, email, role: 'MEMBER') | |
| 3. SES sends invite email | |
| 4. Member clicks link → Cognito sign-up | |
| 5. postSignUpTrigger Lambda: | |
| - Reads the invite token → resolves trustCompanyId | |
| - Updates Cognito custom attribute: trustCompanyId = <value> | |
| - INSERT into user (trustCompanyId, externalId, email, cognito_sub, …) | |
| 6. Next sign-in: preTokenGen injects trustCompanyId claim into JWT | |
| 7. Member can now access only their tenant's resources | |
| Total time: ~10 seconds end-to-end. No platform-side work.</code></pre> | |
| <h3>D.3.3 — Suspend a tenant (e.g., billing failure)</h3> | |
| <pre class="sh"><code>1. UPDATE trust_company SET status = 'suspended' WHERE id = $1; | |
| 2. customAuth Lambda refuses requests for suspended tenants | |
| (one extra SQL lookup; cache for 5 min via ttlOverride) | |
| 3. All Cognito sessions for that tenant invalidated within ttl | |
| 4. Read-only access can be preserved if needed | |
| (different role: chil_app_readonly, RLS-scoped) | |
| Total time: ~30 seconds; users see a maintenance screen.</code></pre> | |
| <h3>D.3.4 — Off-board a tenant (HIPAA right-to-erasure)</h3> | |
| <pre class="sh"><code>1. Notice period per BAA contract (often 30–90 days) | |
| 2. Export all tenant data to encrypted S3 bucket | |
| (deliverable to tenant: machine-readable JSON + PDF) | |
| 3. DELETE chains: | |
| - care_document_section WHERE trust_company_id = $1 | |
| - care_document WHERE trust_company_id = $1 | |
| - user_interview_step WHERE … (transitively via user_id) | |
| - document, member_note, support_message, … | |
| - finally user WHERE trust_company_id = $1 | |
| - trust_company WHERE id = $1 | |
| 4. Audit_event rows for that tenant: retain 6 years per HIPAA; | |
| exclude from new RLS reads via a status='offboarded' flag | |
| 5. S3 audit-archive: retain Object-Locked for 7 years | |
| 6. Cognito: delete users + group; revoke remaining tokens | |
| 7. Stripe: cancel subscriptions; archive customer | |
| 8. Document the off-board in tenant_lifecycle_event table | |
| Total time: ~30 minutes of admin work; data deletion runs in ~minutes | |
| for normal-size tenants; audit retention is automatic.</code></pre> | |
| <h3>D.3.5 — Migrate a tenant to its own silo (Bridge upgrade)</h3> | |
| <pre class="sh"><code>1. Provision dedicated Aurora cluster in target region | |
| 2. Restore from PITR snapshot filtered to that trust_company_id | |
| (or dual-write during cutover window) | |
| 3. Update dispatcher routing: lookup table maps trustCompanyId | |
| → cluster endpoint | |
| 4. RLS policies on pool DB add: trust_company_id != <migrated> | |
| (tenant is no longer visible to the pool) | |
| 5. Cutover: requests for that tenant route to dedicated cluster | |
| 6. Drop tenant rows from pool cluster after 30-day stability | |
| 7. Maintain schema parity via Drizzle migrations applied to both | |
| Total time: ~1 week engineering; minimal user-visible downtime | |
| (maintenance window ~2 hours during cutover).</code></pre> | |
| <div class="callout ok"> | |
| <div class="callout-label">Bottom line on tenant architecture</div> | |
| <strong>Pool + RLS + tenant-claim-in-JWT is the right answer for Childfree Legacy.</strong> It matches the tenant count | |
| (2 today, ~5–20 over 2 years), the team size (Mike + Oleksii operate the platform), the HIPAA posture (RLS + audit | |
| trail satisfies the Safeguards), and the cost profile (low pooled baseline vs. silo's per-tenant cluster floor). Bridge | |
| is the documented upgrade path with five concrete trigger conditions and a one-week migration playbook (D.3.5). The | |
| "right" tenant architecture is not a single architecture — it's <strong>this architecture today + a known path to the next</strong>. | |
| </div> | |
| </section> | |
| <hr> | |
| <div class="part-banner"> | |
| <div class="part-num">Appendix E</div> | |
| <h2 style="border:0;color:var(--ink);">Senior architect review — response & closures (v3)</h2> | |
| <p>An independent senior-architect review was run against this plan. It scored the plan 3.2/5.0 today (vs the 4.1 self-rating), raised 6 critical, 11 high, 7 medium, 3 low findings, plus 8 architecture gaps and 7 code-conflict notes. This appendix records the closure status of every finding, the substantive responses to the criticals, and the gap-closure detail. Verdict accepted: <strong>proceed with modifications</strong>; the destination architecture is correct.</p> | |
| </div> | |
| <!-- =============================================================== --> | |
| <!-- ===== E.1 · CLOSURE MATRIX ==================================== --> | |
| <!-- =============================================================== --> | |
| <section id="senior-review-closure"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · E.1</span> | |
| <h2 style="border:0;margin:18px 0 0;">Finding closure matrix</h2> | |
| </div> | |
| <p>Status legend: <strong class="ok">CLOSED</strong> — addressed in this v3. <strong style="color:var(--warn);">GATED</strong> — converted to a hard Phase-1 or Phase-4 gate. <strong style="color:var(--muted);">ALREADY</strong> — was already addressed before the review (reviewer used a stale snapshot). <strong class="crit">OPEN</strong> — owner-assigned, not yet done.</p> | |
| <h3>E.1.1 — Critical findings</h3> | |
| <table> | |
| <thead><tr><th>ID</th><th>Finding</th><th>Status</th><th>Where closed</th></tr></thead> | |
| <tbody> | |
| <tr><td><strong>C1</strong></td><td>SET LOCAL + RDS Proxy connection pinning (pool-defeat + GUC-leak risk)</td><td><span class="sev verified">CLOSED + GATED</span></td><td>ADR-013 (§B.9) · FF14 (§B.8) · B.11 #14 · E.2 below</td></tr> | |
| <tr><td><strong>C2</strong></td><td>connect-to-SQL GA/BAA/a.combine() unverified</td><td><span class="sev warn">GATED</span></td><td>B.11 #11 (pre-Phase-1 PoC + written AWS confirmation)</td></tr> | |
| <tr><td><strong>C3</strong></td><td>Phase 4 88-min cutover window untested at prod scale</td><td><span class="sev verified">CLOSED + GATED</span></td><td>§53 load-test gate · E.3 AG-adjacent (pre-warm + abort-gate) · E.2 below</td></tr> | |
| <tr><td><strong>C4</strong></td><td>Cognito attribute backfill irreversible + externally gated</td><td><span class="sev warn">GATED</span></td><td>B.11 #12 (SLA) · FF15 · §53 #16 · E.2 below</td></tr> | |
| <tr><td><strong>C5</strong></td><td>Phase 0 CFN cleanup not isolated from real root cause</td><td><span class="sev warn">GATED</span></td><td>B.11 #13 (describe-stack-events before infra)</td></tr> | |
| <tr><td><strong>C6</strong></td><td>Encryption SDK chosen without justifying the tradeoff</td><td><span class="sev verified">CLOSED</span></td><td>ADR-009 expanded · FF16 · E.2 below</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>E.1.2 — Architecture gaps</h3> | |
| <table> | |
| <thead><tr><th>ID</th><th>Gap</th><th>Status</th><th>Where closed</th></tr></thead> | |
| <tbody> | |
| <tr><td><strong>AG1</strong></td><td>No dispatcher failure-mode matrix</td><td><span class="sev verified">CLOSED</span></td><td>E.3 below (8-row matrix)</td></tr> | |
| <tr><td><strong>AG2</strong></td><td>No multi-region story beyond snapshot copy</td><td><span class="sev verified">CLOSED</span></td><td>E.3 below (decision + reopen triggers)</td></tr> | |
| <tr><td><strong>AG3</strong></td><td>audit_event cross-tenant query path unspecified</td><td><span class="sev verified">CLOSED</span></td><td>E.3 below (Streams→Firehose→S3 Parquet→Athena)</td></tr> | |
| <tr><td><strong>AG4</strong></td><td>5 schedulers have no JWT for a tenant-claim-requiring dispatcher</td><td><span class="sev verified">CLOSED + GATED</span></td><td>E.3 below (chil_system option) · §53 #21</td></tr> | |
| <tr><td><strong>AG5</strong></td><td>support-requests S3 prefix missing from §32 tenant-wrap list</td><td><span class="sev verified">CLOSED</span></td><td>E.3 below (full prefix inventory deliverable)</td></tr> | |
| <tr><td><strong>AG6</strong></td><td>Dispatcher DR separate from Aurora DR</td><td><span class="sev verified">CLOSED + GATED</span></td><td>E.3 below (canary + 1-command rollback) · §53 #20</td></tr> | |
| <tr><td><strong>AG7</strong></td><td>No steady-state "add a new PHI model" process</td><td><span class="sev verified">CLOSED</span></td><td>E.3 below (10-step checklist)</td></tr> | |
| <tr><td><strong>AG8</strong></td><td>CDC catch-up vs 24h Streams retention on re-cutover</td><td><span class="sev verified">CLOSED</span></td><td>E.3 below (18h window rule) · §53 CDC gate</td></tr> | |
| </tbody> | |
| </table> | |
| <h3>E.1.3 — High / Medium / Low + code conflicts</h3> | |
| <table> | |
| <thead><tr><th>ID</th><th>Finding</th><th>Status</th><th>Note</th></tr></thead> | |
| <tbody> | |
| <tr><td>H1</td><td>VPC ENI exhaustion tighter than 250</td><td><span class="sev warn">GATED</span></td><td>B.11 #15 Service Quotas review</td></tr> | |
| <tr><td>H2</td><td>Dispatcher single-team point of failure</td><td><span class="sev crit">OPEN</span></td><td>Third reviewer trained + on-call runbook — staffing decision for Jon</td></tr> | |
| <tr><td>H3</td><td>audit_event cross-tenant query story</td><td><span class="sev verified">CLOSED</span></td><td>= AG3</td></tr> | |
| <tr><td>H4 / M7 / Conflict-2</td><td>"45–65 frontend files" underestimate</td><td><span class="sev verified">ALREADY</span></td><td>Current doc says <strong>84–120</strong> + "2–3× original"; reviewer saw a stale snapshot. Already more honest than the review's "+30–50%."</td></tr> | |
| <tr><td>H5</td><td>MEDICALREVIEW caseload is a product project</td><td><span class="sev crit">OPEN</span></td><td>Separate workstream + product owner — for Jon (§59 Q16)</td></tr> | |
| <tr><td>H6</td><td>AppSync introspection enabled in prod</td><td><span class="sev warn">GATED</span></td><td>§53 #17 hard gate</td></tr> | |
| <tr><td>H7</td><td>Multi-region DR deferred</td><td><span class="sev verified">CLOSED</span></td><td>= AG2 decision</td></tr> | |
| <tr><td>H8</td><td>chil_admin BYPASSRLS backdoor needs technical controls</td><td><span class="sev verified">CLOSED</span></td><td>§24 5 compensating controls + Phase-5 hardening (dual-control token, 1h auto-revoke, IP allowlist, per-query Slack with SQL)</td></tr> | |
| <tr><td>H9</td><td>Drizzle Kit + Docker in Amplify build env</td><td><span class="sev crit">OPEN</span></td><td>§39 dry-run uses a throwaway PG container — validate on a real branch or move to GitHub Actions</td></tr> | |
| <tr><td>H10</td><td>CDC catch-up + 24h re-cutover</td><td><span class="sev verified">CLOSED</span></td><td>= AG8</td></tr> | |
| <tr><td>H11</td><td>App freeze during incremental Phase 2</td><td><span class="sev crit">OPEN</span></td><td>Dual-write design for active non-PHI models — Phase 2 design item</td></tr> | |
| <tr><td>M1</td><td>User decomposition N+1 latency</td><td><span class="sev crit">OPEN</span></td><td>Server-side joins + documented eager-load — Phase 3</td></tr> | |
| <tr><td>M2</td><td>Cost model misses ingestion + security tools</td><td><span class="sev verified">CLOSED</span></td><td>cost correction applied (cost section since removed)</td></tr> | |
| <tr><td>M3</td><td>Field-encryption rotation story missing</td><td><span class="sev verified">CLOSED</span></td><td>ADR-009 — rotation = one-time migration script, not quarterly task</td></tr> | |
| <tr><td>M4</td><td>5 retained DDB tables wrong (outbox in Aurora)</td><td><span class="sev verified">CLOSED</span></td><td>§02 corrected to 4 + 9→4 refactor note</td></tr> | |
| <tr><td>M5</td><td>trustCompanyId immutability is a product constraint</td><td><span class="sev verified">CLOSED</span></td><td>E.2 C4 documents it</td></tr> | |
| <tr><td>M6</td><td>Step-up MFA ~50 lines underestimated</td><td><span class="sev verified">CLOSED</span></td><td>Re-estimated 200–500 lines in §49 / E.2</td></tr> | |
| <tr><td>L1</td><td>SequenceNumberHelper → SEQUENCE window</td><td><span class="sev crit">OPEN</span></td><td>Start SEQUENCE at max(existing)+1 — Phase 2 note</td></tr> | |
| <tr><td>L2</td><td>5-min onboarding is platform-only</td><td><span class="sev verified">CLOSED</span></td><td>D.3.1 already distinguishes platform vs business onboarding</td></tr> | |
| <tr><td>L3</td><td>PC list missing postSignUpTrigger</td><td><span class="sev verified">CLOSED</span></td><td>PC list now includes postSignUpTrigger (cost section since removed)</td></tr> | |
| <tr><td>Conflict-1</td><td>§02 outbox DDB/Aurora self-contradiction</td><td><span class="sev verified">CLOSED</span></td><td>§02 rewritten</td></tr> | |
| <tr><td>Conflict-3</td><td>B.3.2 74/75 off-by-one</td><td><span class="sev verified">CLOSED</span></td><td>§B.3.2 footnote reconciles defineBackend (74) vs filesystem (75)</td></tr> | |
| <tr><td>Conflict-4</td><td>Seed UUID hard-coded, no canonical tenant</td><td><span class="sev verified">CLOSED</span></td><td>E.2 C4 — Jon/product choose + document the legacy-tenant ID before backfill</td></tr> | |
| <tr><td>Conflict-5</td><td>backend.ts:175 always-truthy</td><td><span class="sev warn">GATED</span></td><td>§53 #18 — confirmed real; one-line fix in Phase 0</td></tr> | |
| <tr><td>Conflict-6</td><td>sendHospitalEmail residue (5 files)</td><td><span class="sev verified">CLOSED</span></td><td>Phase 1 cleanup deliverable (confirmed in repo)</td></tr> | |
| <tr><td>Conflict-7</td><td>9-table audit collapse is a real refactor</td><td><span class="sev verified">CLOSED</span></td><td>§02 note + Phase 2 deliverable</td></tr> | |
| </tbody> | |
| </table> | |
| <div class="callout ok"> | |
| <div class="callout-label">Tally</div> | |
| Of 6 criticals: 3 CLOSED, 3 GATED. Of 8 architecture gaps: all 8 CLOSED (2 also gated). Of 11 highs: 5 CLOSED, 2 GATED, | |
| 4 OPEN (staffing/design decisions for Jon). Of 7 mediums: 6 CLOSED, 1 OPEN. Of 7 code conflicts: 6 CLOSED, 1 GATED. | |
| H4/M7/Conflict-2 were ALREADY addressed before the review (stale snapshot). | |
| </div> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== E.2 · CRITICAL RESPONSES ================================ --> | |
| <!-- =============================================================== --> | |
| <section id="senior-review-criticals"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · E.2</span> | |
| <h2 style="border:0;margin:18px 0 0;">Critical-finding responses (C1–C6)</h2> | |
| </div> | |
| <h3>C1 — RDS Proxy connection pinning (ADR-013) <span class="sev verified">SUPERSEDED · June 2026</span></h3> | |
| <div class="callout ok"><div class="callout-label">Resolved by the Data API decision — pinning is no longer a concern</div> | |
| Mike's June 2026 decision replaces RDS Proxy with the Aurora Data API, which <strong>dissolves this entire risk class</strong>: there is no persistent pooled connection to pin, and each Data API transaction binds <code>SET LOCAL</code> to its own <code>transactionId</code> with the connection reset between transactions — so the catastrophic GUC-leak path below is structurally closed, not just guarded. The remaining Data API trade-offs are different in kind: each statement is a separate HTTPS call (a few more round-trips per request) and there is a result-set size cap (paginate large reads). The original analysis is retained below for the record.</div> | |
| <p><strong>Original analysis (RDS Proxy path — for the record).</strong> <code>SET LOCAL</code> session state causes RDS Proxy to <em>pin</em> the | |
| connection for the transaction's life — pinned connections are not returned to the pool. Two consequences:</p> | |
| <ul> | |
| <li><strong>Pooling tax:</strong> at dispatcher reserved concurrency 100, up to 100 pinned connections. Aurora <code>max_connections</code> ≈ 192 at 1 ACU — so the proxy degrades toward a direct-connection pool, paying proxy cost without the pooling benefit.</li> | |
| <li><strong>Correctness (catastrophic):</strong> a <code>SET</code> without <code>LOCAL</code>, or an exception path that skips COMMIT/ROLLBACK and returns the connection un-reset, leaves the GUC on a pooled connection → the next request inherits the wrong tenant. This is the single worst bug class in the architecture.</li> | |
| </ul> | |
| <p><strong>ADR-013 decision: accept pinning.</strong> Size Aurora <code>max_connections</code> to the dispatcher's reserved | |
| concurrency (+ headroom for chil_admin/reporter/migrator), keep IAM-auth, document the lost-pooling tradeoff. Alternatives | |
| rejected <em>at the time</em>: direct <code>pg.Pool</code> in the Lambda (loses IAM-auth centralization), and the RDS Data API. | |
| <strong>Update (June 2026):</strong> the Data API objection was reconsidered and reversed — <code>SET LOCAL</code> works via the Data API transaction API (bound to a <code>transactionId</code>), which is in fact safer than proxy pinning. The Data API is now the chosen path; this ADR-013 pinning stance is superseded. <strong>Belt-and-suspenders for correctness:</strong> the dispatcher | |
| always opens the transaction, always <code>SET LOCAL</code> (never bare <code>SET</code>), and a <code>finally</code> block | |
| runs <code>DISCARD ALL</code> before releasing — so even an escape path cannot leak a GUC. FF14 alarms on pinned-connection | |
| ratio + borrow latency.</p> | |
| <h3>C3 — Cutover window hardening</h3> | |
| <p><strong>Accepted.</strong> The 88-minute budget (T+0:02 export → T+1:30 ETL complete) assumes COPY throughput that | |
| 1-ACU Aurora cannot deliver cold. Changes:</p> | |
| <ul> | |
| <li><strong>Pre-warm Aurora to 4 ACU at T−30</strong> (not T+0) so COPY runs at full throughput from the first byte.</li> | |
| <li><strong>Load-test the COPY pipeline</strong> against a production-snapshot-restored Aurora at realistic per-table sizes; document per-table COPY duration at min and peak ACU. (§53 hard gate.)</li> | |
| <li><strong>Size the ETL Lambda fleet to per-table volume</strong>, not equal fan-out (Document table is 5–20 GB per D2; it gets more parallelism than VerificationTokens).</li> | |
| <li><strong>Hard abort-gate at T+1:30:</strong> if ETL is not complete, <em>abort</em> (drop the new Aurora data, leave DynamoDB hot, retry next weekend) rather than entering rollback. Rollback is for <em>after</em> a completed-but-failing cutover; a mid-ETL overrun is an abort, not a rollback. This distinction was missing and is now explicit in the cutover runbook.</li> | |
| </ul> | |
| <h3>C4 — Cognito backfill: irreversible + externally gated + product constraint</h3> | |
| <p><strong>Accepted.</strong> Three technical facts now documented:</p> | |
| <ul> | |
| <li><strong>One-way schema:</strong> Cognito custom attributes can never be removed or retyped. The seed tenant UUID must be chosen by Jon + product and documented as the legacy-tenant ID <em>before</em> the backfill — a wrong default is permanent without a second backfill. (Closes Conflict-4.)</li> | |
| <li><strong>Reassignment is a product feature (revised June 2026):</strong> <code>custom:trustCompanyId</code> is <strong>admin-mutable</strong>, so a member <em>can</em> be migrated between trusts (M&A, error correction, support reassignment, and the address-change reassignment flow in <a href="#tenancy-brands">§3b.3</a>). The <code>trust_company</code> DB row is the source of truth; the Cognito attribute is a mint-time cache. The HIPAA audit trail is preserved by the append-only <code>audit_event</code> history of the change — not by freezing the attribute. (Supersedes the earlier "immutable" stance; closes M5.)</li> | |
| <li><strong>Session re-auth:</strong> existing sessions don't pick up the new claim until token refresh. With Phase-3 fail-closed-on-missing-claim, every signed-in user must re-auth after the Phase-1 backfill — <em>unless</em> customAuth falls back to the seed tenant for old tokens during a documented grace period. The grace-period code is a Phase-1 deliverable.</li> | |
| </ul> | |
| <p>Gated: B.11 #12 (written SLA from the external pool owner) + FF15 (canary verifies the claim is in every minted JWT) + §53 #16.</p> | |
| <h3>C6 — Encryption SDK vs pgcrypto (ADR-009 alternatives-considered)</h3> | |
| <table> | |
| <thead><tr><th>Option</th><th>Cost at steady state</th><th>Key handling</th><th>Rotation</th><th>Verdict</th></tr></thead> | |
| <tbody> | |
| <tr><td><strong>AWS Encryption SDK envelope (chosen)</strong></td><td>Every read = 1 KMS Decrypt (data-key caching cuts this); ciphertext ~50% larger</td><td>Key never sits in the DB; encryption context binds to caller (userId+tenant)</td><td>One-time migration/batch script (NOT quarterly)</td><td><strong>Chosen</strong> — auditor-friendly; key never in DB</td></tr> | |
| <tr><td>pgcrypto column-level</td><td>Cheaper at steady state (decrypt in-DB)</td><td>Key fetched from KMS once per cold start; <em>sits in DB memory</em></td><td>Backfill DDL</td><td>Rejected — key in DB weakens the chil_admin-can't-decrypt property</td></tr> | |
| </tbody> | |
| </table> | |
| <p>The Encryption SDK remains the defensible choice precisely because the key never enters the database, so a chil_admin | |
| BYPASSRLS read still returns ciphertext (the Tier-2 guarantee). The per-decrypt cost is mitigated with the SDK's | |
| data-key caching (TTL < 5 min) and watched by FF16. Rotation is documented as its own one-time script.</p> | |
| </section> | |
| <hr> | |
| <!-- =============================================================== --> | |
| <!-- ===== E.3 · GAP CLOSURES ====================================== --> | |
| <!-- =============================================================== --> | |
| <section id="senior-review-gaps"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · E.3</span> | |
| <h2 style="border:0;margin:18px 0 0;">Architecture gap closures (AG1–AG8)</h2> | |
| </div> | |
| <h3>AG1 — Dispatcher failure-mode matrix (Phase 3 runbook)</h3> | |
| <table> | |
| <thead><tr><th>Failure</th><th>Response</th></tr></thead> | |
| <tbody> | |
| <tr><td>Cold start over P95 budget</td><td>PC keeps 5 warm; if exceeded, request proceeds (no fail); alarm on init_duration P95 > 2 s (FF10)</td></tr> | |
| <tr><td>Data API throttling / timeout</td><td><strong>Fail-closed: 503 to client</strong> + retry-after header; alarm on rds-data <code>ThrottlingException</code> / latency (§42). Never fail-open.</td></tr> | |
| <tr><td>KMS Decrypt rate limit</td><td>Exponential backoff (3 tries); if still failing, 503 for that field; alarm FF16. Data-key cache absorbs most.</td></tr> | |
| <tr><td>Aurora unavailable during failover</td><td>Maintenance page via AppSync error mapping; dispatcher retries writer endpoint for 30 s; on persist, 503</td></tr> | |
| <tr><td>JWT verify failure</td><td><strong>401</strong> (confirmed)</td></tr> | |
| <tr><td><ins class="rlsfix">Write-side</ins> RLS denial (SQLSTATE 42501<ins class="rlsfix">, WITH CHECK only</ins>)</td><td><strong>401 + alarm</strong> (confirmed, FF3)<ins class="rlsfix"> — read-side isolation failures raise no error; covered by the tenant-isolation canary (review F3)</ins></td></tr> | |
| <tr><td><ins class="rlsfix">GUC wiring error (SQLSTATE 42704 / 22P02)</ins></td><td><ins class="rlsfix"><strong>Fail closed: 500 + page</strong> — tenant context could not be established (review F3)</ins></td></tr> | |
| <tr><td><ins class="rlsfix">Single row > ~1 MB Data API result cap</ins></td><td><ins class="rlsfix">List queries paginate; single-row GET serves the body via S3 pointer + presigned URL; a legacy oversize inline row returns a typed error (never a generic 500) + alarm (review F6)</ins></td></tr> | |
| <tr><td>SQL statement_timeout (10 s)</td><td>504 to client; query logged with correlation_id; alarm if rate > 0.5%</td></tr> | |
| <tr><td>pg driver network partition</td><td>Retry once; on failure 503 + alarm; circuit-breaker opens after 5 consecutive</td></tr> | |
| </tbody> | |
| </table> | |
| <p>The dispatcher code is reviewed against this matrix as the rubric. <strong>Every unhandled path fails closed (deny), never open.</strong></p> | |
| <h3>AG2 — Multi-region decision (resolves §59 Q13)</h3> | |
| <p><strong>Decision for v3: single-region (us-east-1) with cross-region snapshot copy + 4-hour PITR RTO is the base case.</strong> | |
| Aurora Global Database is deferred, not adopted. Documented reopen triggers: (1) a customer SLA requiring < 1 h RTO; | |
| (2) a revenue threshold where a multi-hour us-east-1 outage is unacceptable (precedent: Dec 2021, Jun 2023); (3) a | |
| regulatory/residency requirement. If reopened: Aurora Global Database (see the multi-region note in §44/§45), S3 CRR for the audit | |
| archive, and the Cognito secondary-pool problem (Cognito is regional and does <em>not</em> replicate users — this is the | |
| hard part and must be scoped with the external pool owner).</p> | |
| <h3>AG3 — audit_event cross-tenant query pipeline (Phase 5)</h3> | |
| <p>audit_event lives in DynamoDB at millions/month; HIPAA evidence queries ("every PHI access for user X in trust Y between | |
| two dates") are awkward on DDB. Pipeline: <strong>DynamoDB Streams → Kinesis Firehose → S3 (Parquet, partitioned by | |
| trust_company_id + date) → Athena.</strong> Alarm on Firehose delivery failure. Query SQL templates live in the runbook. | |
| This is also the auditor evidence-pack query surface (§54).</p> | |
| <h3>AG4 — Scheduler execution context (the sharpest catch)</h3> | |
| <p><strong>Real gap.</strong> The 5 EventBridge schedulers (<code>assign-welons-scheduler</code> /6h, | |
| <code>wellness-check</code> hourly, <code>check-inactive-members</code> daily, <code>quarterly-review-scheduler</code> | |
| daily, <code>dynamic-waitlist</code>) call <code>client.models</code> but have <strong>no user JWT</strong> — and the | |
| post-migration dispatcher fail-closes on a missing tenant claim. Without a fix, every scheduled PHI job breaks at cutover.</p> | |
| <table> | |
| <thead><tr><th>Option</th><th>How</th><th>Audit / cost</th></tr></thead> | |
| <tbody> | |
| <tr><td><strong>A · <code>chil_system</code> role (recommended)</strong></td><td>A dedicated cross-tenant Postgres role; RLS bypassed for batch scope; <em>every</em> action pgaudit-logged with the system identity</td><td>Cleanest; one role; full audit; one extra compensating-control review</td></tr> | |
| <tr><td>B · per-tenant loop</td><td>Scheduler iterates <code>trust_company</code> rows, calls the dispatcher with each tenant context</td><td>Most faithful to RLS; N× invocations; slower</td></tr> | |
| <tr><td>C · pg_cron / Aurora event scheduling</td><td>Move scheduled work into Postgres-native scheduling</td><td>No Lambda; harder to share TS logic; different audit surface</td></tr> | |
| </tbody> | |
| </table> | |
| <p>Phase 2 deliverable: pick on purpose (lean A). Separately, resolve Jon's DR-09 (decommission <code>assign-welons-scheduler</code> | |
| in favor of deterministic assignment on creation) — not addressed in the base plan. §53 #21 gates this.</p> | |
| <h3>AG5 — S3 prefix inventory under multi-tenancy</h3> | |
| <p>§32 lists 6 prefixes needing <code>tenants/{trustCompanyId}/</code> wrapping but omits | |
| <code>public/support-requests/{identity}/*</code>, which the repo confirms. Today paths use IAM <code>entity('identity')</code> | |
| gating. Phase 4 deliverable: a full S3 prefix inventory + per-prefix migration plan (label-only rename vs full-key | |
| migration), with broker-Lambda key construction per the §32 pattern. Existing IAM-entity rules become tenant-prefix + IAM.</p> | |
| <h3>AG6 — Dispatcher DR (separate from Aurora DR)</h3> | |
| <p>§45 covers Aurora restoration but not dispatcher failure modes (bad deploy 500s every request, IAM regression, disabled | |
| KMS key, RDS Proxy ARN drift). Because the dispatcher concentrates all PHI traffic, its availability ≠ Aurora's. Phase 5 | |
| deliverable: a dispatcher rollback runbook with a <strong>one-command CFN parameter flip</strong> (like the §47 data-layer | |
| rollback but for the dispatcher version), plus <strong>canary deploy at 5% traffic</strong> before promote. §53 #20 gates the canary.</p> | |
| <h3>AG7 — "Add a new PHI model" checklist (Phase 5 steady-state)</h3> | |
| <ol> | |
| <li>Drizzle migration (DDL) for the new table.</li> | |
| <li>RLS policy file (FORCE RLS + tenant predicate).</li> | |
| <li>Tenant-isolation matrix (FF1) updated with the new table.</li> | |
| <li>Dispatcher op handler (get/list/create/update/delete).</li> | |
| <li>Field-auth projection map updated for the new columns.</li> | |
| <li><code>amplify/data/phi-fields.json</code> updated (the single source for scrubbers).</li> | |
| <li>Pino redaction entry for any PHI field.</li> | |
| <li>Sentry <code>beforeSend</code> scrubber entry.</li> | |
| <li>Encryption tier decision (Appendix C.1) — Tier 1 or Tier 2.</li> | |
| <li>FF6 coverage check passes in CI.</li> | |
| </ol> | |
| <h3>AG8 — Re-cutover window vs 24h Streams retention</h3> | |
| <p>If Phase 4 rolls back at T+4:00, ~20 h of DynamoDB Streams retention remain. Rule: <strong>a second cutover must start | |
| within 18 hours of the first rollback</strong>, or pause a week and enable DynamoDB Streams → Kinesis for longer retention | |
| before attempt three. Past the 24 h horizon, DDB↔Aurora divergence needs manual reconciliation. This guidance sits beside | |
| the rollback decision criteria in the cutover runbook; the CDC-lag < 5 min gate (§53) keeps catch-up well inside the window.</p> | |
| </section> | |
| <footer> | |
| Childfree Legacy — Aurora Migration v3: Complete Plan + DevOps Deep Review + Architecture Review + Security/DevOps Review + Tenant Deep-Dive + Senior-Review Closure · Internal · 2026-05-29<br> | |
| Companion to <code>CALL-PREP-UNIFIED.html</code> (diagnosis + Track A alternative + full decision context)<br> | |
| Code-validated against current main <code>fb4cf8b3e</code> · Reviews applied: arc42 · C4 · ATAM · AWS Well-Architected · Fitness functions · MADR · HIPAA/SOC 2 mapping · Security-architect · Platform-engineer · Independent senior-architect review (C1–C6 / AG1–AG8 / H1–H11)<br> | |
| Owner: Jon · Mike (DevOps) · Oleksii (Data Architect) | |
| </footer> | |
| <hr> | |
| <section id="review-history"> | |
| <div style="display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;"> | |
| <span class="section-num">Appendix · RH</span> | |
| <h2 style="border:0;margin:18px 0 0;">Review history & sources</h2> | |
| </div> | |
| <p>Six independent review rounds hardened this plan; every finding is folded into the body above. This is the compact provenance (the long per-round banners were removed in favour of this reference).</p> | |
| <table> | |
| <thead><tr><th>Round</th><th>Source</th><th>Net effect on the plan</th></tr></thead> | |
| <tbody> | |
| <tr><td>1 · PHI/RLS design review</td><td>external</td><td>Surfaced the permissive-RLS collapse, the GUC mismatch, the wrong rollback SQLSTATE, the latency contradiction, the 1 MB result cap, CDC decomposition, the operator-access gap, and non-queryable encryption.</td></tr> | |
| <tr><td>2 · auth-first decision</td><td>team + POC branch</td><td>Pivot to <code>connect-to-SQL</code> + <code>@auth</code> (model + field-level); RLS and the custom dispatcher deferred to a hardening track (Appendix · deferred track).</td></tr> | |
| <tr><td>3 · field-level @auth follow-up</td><td>external</td><td>The <code>.required()</code>+field-auth trap; verify Gen 2 redaction empirically; subscription leakage; FF1 must cover custom resolvers and be a pre-PHI merge gate.</td></tr> | |
| <tr><td>4 · team review + data audit</td><td>Mike + Vasyl</td><td>AWS Secrets Manager not used (Amplify <code>secret()</code>); Cognito verifies JWT natively; Mike provisions Aurora; plaintext SSN/credentials; Sentry leak; all-Welon over-exposure; FB Pixel on PHI routes.</td></tr> | |
| <tr><td>5 · 📄 data-architecture review</td><td>AWS console + schema audit (<em>Data_Architecture_Review</em>)</td><td><code>custom:trustCompanyId</code> missing (Phase-1 blocker); ~90 Lambdas at the 15-min max; medical-incident PHI logging; <code>birthdate</code> string→DATE; plaintext tokens; Notification subscriptions; structural gaps. <strong>📄 marks these updates in the body.</strong></td></tr> | |
| <tr><td>6 · PRD validation</td><td>live code + 2026-05-04 AWS snapshot</td><td>"Go with conditions": CFN ceiling refuted (largest stack 373/500); PHI split wrong by field content; event streams 11 not 5; volume table over-sized; 4 undocumented security issues; email in 14 tables; 77 functions; pull <code>tenant_id</code> RLS forward (with the minimal-dispatcher caveat).</td></tr> | |
| </tbody> | |
| </table> | |
| <p style="font-size:13px;color:var(--muted);"><strong>Source documents:</strong> Call-Prep gist <code>6c59c5aa…</code> · Data Architecture Review (AWS console + schema audit) · PRD <code>1Z4YhELRhJ1G…</code> · PHI/RLS Review Responses.</p> | |
| </section> | |
| </main> | |
| </div> | |
| </body> | |
| </html> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment