Mid-Course Check-In: Demo Day & Cert Challenge Readiness

session13-demo-day-cert-challenge.md

Mid-Course Check-In: Demo Day & Cert Challenge Readiness — Peer Supporter Facilitation Guide

Peer supporters only. This is a check-in room, not a notebook room — the deliverable is that every student leaves knowing (1) where their cert challenge stands two days before the deadline, and (2) one concrete scoping decision for Demo Day. Guide students TOWARD the concepts; consult answer keys privately, do not read them aloud.

Answer keys referenced (supporter-only): data/raw/v1.0/answer-keys/01_dense_vector_retrieval.md · 02_agentic_rag.md · 07_advanced_retrievers.md · 12_Production_Agent_Patterns_CHEATSHEET.md

Generated 2026-07-14 via the facilitator-quickstart skill (3-agent fan-out: retrieval decision, product precision, demo scope-cutting; grounded in course answer keys, LangChain docs, and Anthropic's Building Effective Agents guidance; diagram validated via Mermaid Chart MCP).

Friendly Reminders

• Within the first 2 minutes: as you read your script intro, have someone screen-share while you set expectations for the breakout room. Getting a volunteer sharer immediately makes the room participatory instead of a lecture.
• Always serve the majority of the room. Don't let one student derail the conversation. Keep the room focused on the check-in questions and Demo Day scoping — deep 1:1 debugging belongs after the room, not in it.
• Escalation path: if you're having an issue with a student, @Jacob on the live channel — he'll jump into your room to support you. Use it early; you can't serve the room and manage a disruption at the same time.
• This room is a pulse check, not a rescue mission. For students who are badly stuck on the cert challenge, your job is to deliver the remediation-loop message (below) and flag them same-day — not to solve their blocker live.
• Either answer can be right. On "keep dense vector retrieval?" and "what will you remove?", you're grading the reasoning, not the choice. Keep-with-evidence and swap-with-evidence are both wins; change-for-resume-value is the coaching moment.

Time-Boxed Agenda (25 min, per the session script)

Time	Segment
1 min	Intro — welcome, set the goal (check-in + Demo Day support)
~4 min	Demo Day expectations + cert challenge pulse check
~6 min	"Keep Dense Vector Retrieval for Demo Day? Why or why not?"
~5 min	"What do you plan to remove for Demo Day?"
~7 min	Dr. Greg's product-precision ladder (as time allows)
2 min	Wrap-up — clean stopping point

Timing inside the 22-minute discussion block is a suggestion — if the room is deep in good discussion on retrieval or scope-cutting, let the ladder be lighter. Never skip ladder rung 1 (ideal customer) or rung 7 (building your way out of feedback) if you run the ladder at all.

flowchart TD
    A["0-2 min: Intro + setup<br/>Read script intro, get a screen-sharer,<br/>set breakout expectations"] --> B["1 min: Welcome<br/>'Check in on how everybody's doing —<br/>second half + Demo Day'"]
    B --> C["22 min: Discussion"]
    C --> C1["Demo Day expectations<br/>+ cert challenge pulse check"]
    C1 --> C2["Keep Dense Vector Retrieval?<br/>Why / why not"]
    C2 --> C3["What will you REMOVE<br/>for Demo Day?"]
    C3 --> C4["Dr. Greg's product-precision<br/>questions (as time allows)"]
    C4 --> D["Last 2 min: Wrap-up<br/>'Clean stopping point that's<br/>useful for everyone'"]

    C -. "room goes quiet" .-> F["Fallbacks: your own demo plan,<br/>past-cohort presentations, pair-share"]
    F -.-> C
    C -. "student derails / disruptive" .-> E["@Jacob on the live channel —<br/>he'll jump into your room"]
    E -.-> C

    style A fill:#e8f0fe,stroke:#4285f4
    style D fill:#e6f4ea,stroke:#34a853
    style E fill:#fce8e6,stroke:#ea4335
    style F fill:#fef7e0,stroke:#f9ab00

Loading

Script Beats (verbatim)

1 min — Intro:

"Welcome in! Focus of this breakout room is just to check in on how everybody's doing, and see how I can best support you guys heading into the second half of the course and demo day."

22 min — Discussion openers (in order):

"Any initial questions about demo day expectations?"

"Where are you guys all at with your cert challenges? All wrapped up?"

"Do you plan to keep your RAG implementation via Dense Vector Retrieval for Demo Day? Why or why not?"

"What do you plan to remove for demo day?"

Last 2 min — Wrap-up:

"Hey everyone — it's time to start wrapping up, as breakout rooms close in just 2 minutes! Let's get a clean stopping point that's useful for everyone."

---

Segment 1: Demo Day Expectations + Cert Challenge Pulse Check (~4 min)

Demo Day expectations — what you can state confidently

Grounded in prior-cohort artifacts (data/demo_day.json, docs/lessons-learned/cohort-report.md); no formal Demo Day rubric doc exists in the repo, so treat these as strong patterns, not scored criteria:

• Lead with problem → solution, not tech stack. Every strong prior-cohort project entry opened with a crisp problem statement, then the solution — the AI came second.
• Short, structured, working demo beats production polish. Prior cohort data: 2–4 minute videos covering "working code → one key output → what I learned" consistently scored full marks; video was the highest-variance, most-skipped component. Record the demo before polishing.
• Honest evals and limitations read as strength. Naming your metrics, failure modes, and "what I'd do next" is what makes a demo credible — not a claim that it's finished.

Cert challenge pulse check

Deadline framing: the cert challenge is due Wednesday 7/16 at 7pm ET — two days out. Threshold is 85/100, and there is a remediation loop: a submission below threshold gets feedback and a chance to fix — but only if there's something gradable to remediate. The message is ship something, even imperfect: a blank submission gets no second chance; a rough one does.

• "Quick pulse: where is your cert challenge right now — submitted, building, or still picking a problem?" — good answers surface: a concrete artifact state (repo exists, demo video recorded, write-up drafted), not vibes like "pretty far along."
• "What's the ONE thing standing between you and hitting submit?" — good answers surface: a specific, finishable blocker (deploy issue, video not recorded, eval section unwritten). If the answer is "everything," use the behind-script below.
• "If you had to submit in 2 hours, what would you cut?" — good answers surface: students who know their rubric priorities — the write-up (use case, architecture, evals) and the demo video carry major weight; polish does not.

For students who are behind: don't panic, and don't go dark. Say plainly: "The bar for Wednesday is gradable, not perfect. Submit what you have — the remediation loop exists exactly for you. A 70 with feedback beats a zero." Scope down to: working core loop + short demo video + honest write-up.

Escalate (coordinator / cert-challenge channel, same-day) when a student: says they may not submit at all, has a life event eating the window, or has been stuck on one technical blocker for more than a day. Don't try to solve it in the room.

---

Segment 2: "Keep Dense Vector Retrieval for Demo Day?" (~6 min)

Key concepts

• Dense vector retrieval is the baseline, not the floor. It embeds query and chunks and ranks by cosine similarity — excellent at paraphrase and semantic intent ("get ready for" matches "prepare for"), and for many Demo Day scopes it's genuinely sufficient. "Still dense" is a defensible answer if the student can say why their eval shows it working.
• Dense has known, nameable failure modes. Rare exact tokens — acronyms, product codes, IDs, named entities — get "blurred" by embeddings; the query lands near generic neighbors instead of the defining passage. Current LangChain docs say this directly: dense embeddings don't handle exact-match queries as well as keyword indexes, which is exactly why hybrid (dense + BM25/sparse) exists. This is the S7 Q2 insight.
• The upgrade paths form a ladder, not a menu of "better." BM25 (exact-term recall) → parent-child (search small chunks, return large context — S7 Q3) → hybrid with rank fusion → retrieve-then-rerank → multi-query expansion → agentic retrieval, where retrieval becomes a tool the agent chooses to call zero/one/many times (S2 Q1). Each rung adds latency, cost, and a new failure mode.
• Eval decides, not vibes or resume value. The S7 framing: retrieval metrics (recall, MRR, latency) can move in opposite directions, and answer-level metrics (faithfulness, answer similarity) arbitrate whether an upgrade actually reached the user (S7 Q4). Swap only when a measured retrieval failure — or real user feedback — shows dense failing on your corpus and query types.
• Right-sized for Demo Day. A simple pipeline the student can explain, demo reliably, and defend with two metrics beats a six-stage pipeline they can't debug on stage. The strongest answers sound like a short product memo (the S7 Activity #2 competency), not a feature list.

Facilitation prompts

1. Main question: "Do you plan to keep dense vector retrieval for Demo Day? Why or why not?" — good answers surface: a decision tied to their corpus and query types, not fashion — e.g. "my users ask paraphrase-style questions, dense handles that" or "my domain is full of part numbers, I need hybrid/BM25." Either keep or swap can be full-credit reasoning; "I'm swapping because rerankers were in the course" is not. (Grounding: S1 Q3 similarity-score limits; S7 Q2.)
2. "What kinds of queries would your dense retriever fail on? Have you actually tried one?" — good answers surface: exact terms, acronyms, IDs, out-of-vocabulary jargon; ideally they've inspected retrieved chunks + scores before generation — the S1 vibe-check habit (S1 Q4) and the S7 Activity #1 method.
3. "If you did upgrade, what's the first rung you'd try, and what evidence would tell you it worked?" — good answers surface: a specific rung matched to a specific failure (exact-term miss → hybrid/BM25; lost surrounding context → parent-child; noisy top-k → reranker; vague multi-part questions → multi-query), plus ≥2 named metrics and one inspected ranking — the S7 Activity #2 checklist shape.
4. "What does each upgrade cost you?" — good answers surface: latency (rerank calls, multi-query fan-out), money, added moving parts to demo live, and new failure modes — e.g. an agentic retriever can wrongly skip retrieval it needed (S2 Q1); more recall can lower faithfulness by adding noise (S7 Q4).
5. "Would agentic retrieval change your answer — retrieval as a tool rather than a fixed step?" — good answers surface: it's orthogonal to dense-vs-sparse — the agent decides when/whether to retrieve, but the retriever underneath can still be dense; benefits (skip off-topic queries, multiple rewritten queries) come with the inspect-the-path burden (S2 Q1, Q3).

Common misconceptions

• "A more advanced retriever = a better demo." Redirect: "Which specific query in your demo does dense currently get wrong? Show me the retrieved chunks." (S7 Activity #2's whole point: a higher aggregate metric doesn't settle the product decision.)
• "Dense retrieval is outdated / naive." Redirect: "Outdated compared to what, on your queries? What did your eval say?" Every advanced rung in S7 is built on top of dense — hybrid and rerank pipelines still contain it.
• "The course covered rerankers/multi-query, so I should use them." Redirect: "What would faithfulness and answer similarity have to do for that swap to be worth its latency?" (S7 Q4's ship/don't-ship logic — and a faithfulness drop is a veto in a safety-adjacent domain.)
• "A high similarity score means my retrieval is fine." Redirect: "What does that score prove, and what doesn't it?" (S1 Q3: it's a ranking signal, not correctness, sufficiency, or answerability.)

If they're stuck

• S1 — dense vector retrieval key: Q3 (what a similarity score does/doesn't prove), Q4 (vibe-check retrieval before generation), and the Tune Retrieval activity — the cheapest "upgrade" is often chunk size/overlap/k tuning, no new retriever at all. Revisit 01_Dense_Vector_Retrieval/01_Cat_Health_Vector_RAG_LangChain_Qdrant.ipynb.
• S7 — advanced retrievers key: the retrieval-ladder overview, Q2 (when BM25 wins), Q3 (search small / return large), Q4 (ship/don't-ship via faithfulness + answer similarity), Activity #2 (defending a recommendation with real numbers). The compare_reports table in 07_Advanced_Retrievers/01_Cat_Health_Advanced_Retrieval.ipynb is the template for a Demo Day justification.
• S2 — agentic RAG key: Q1 (what changes when retrieval becomes a tool) if the conversation drifts to agents.
• External grounding for supporters: LangChain's current retrieval docs (docs.langchain.com, "Sparse and hybrid retrieval") state the dense-misses-exact-match / hybrid-covers-both-cases point directly — useful if a student wants a non-course citation.

---

Segment 3: "What Do You Plan to Remove for Demo Day?" (~5 min)

Key concepts

• A demo and a production system answer different questions. A 5-minute demo must prove one thing works end-to-end in front of people — the core user-visible loop, live, with a real input. Production must survive strangers, scale, and time (auth, multi-tenancy, observability, guardrail coverage, cost controls). Building the second when you're graded on the first is misallocated effort.
• "Start simple" is official guidance, not a cop-out. Anthropic's Building Effective Agents guide is explicit: find the simplest solution possible and only add complexity when it demonstrably improves outcomes — success isn't building the most sophisticated system, it's building the right system. Cutting an agent, a tool, or a retriever tier for Demo Day is that principle applied under a deadline.
• The course itself models the cut line. Session 12's framing (S12 cheatsheet "Anchor" + "Why this shape?" sections) is that guardrails/caching/A2A are hardening layered onto an already-working agent — the working agent came first. Students should demo the working agent; hardening is a slide, not a live path. Same with Session 11's "seam" idea: one swappable function is the demo; everything behind the seam is negotiable.
• Common safe cuts: the second/third agent, extra tools nobody calls, login/auth, multi-tenancy, the exotic retriever (fall back to plain vector search), streaming polish, half-working features (a feature that works 70% of the time is a 30% chance of dying on stage — cut it or pre-record it).
• Never-cut items: the core loop the user sees; error handling on the exact happy path you'll drive live; and at least one honest eval or metric ("faithfulness went from X to Y") — a demo with one real measurement beats a demo with five untested features.

Facilitation prompts

1. "What do you plan to remove for Demo Day?" — good answers surface: a specific named feature or component (not "nothing" or "I'll see"), plus a reason tied to demo risk or time; bonus if they distinguish "removed from the demo" from "removed from the repo."
2. "What does your demo have to PROVE in 5 minutes?" — good answers surface: one crisp claim ("my agent answers X-type questions grounded in Y data") — and the realization that everything not serving that claim is cuttable. Vague answers ("show my whole project") are the coaching moment.
3. "Which feature would you cut if you had to cut one tonight?" — good answers surface: an immediate candidate (they already know their shakiest part); follow with "so why is it still in the demo path?" If they genuinely can't pick one, ask which feature failed most recently.
4. "What's the one number you'll show?" — good answers surface: an eval result, a before/after metric, or a cost/latency measurement they actually ran; distinguishes "I built it" from "I know it works."

Common misconceptions

• "More features = a better demo." Redirect: "How many minutes does each feature get, and what happens to your core story when feature #4 breaks live?" Judges remember one thing working flawlessly, not six things gestured at.
• "Removing things means my project is weaker." Redirect: "What does Anthropic's own agent guidance say about the simplest solution that works?" Scope-cutting is the documented mark of engineering maturity — the strongest students can articulate what they cut and why.
• "Production patterns (observability, guardrails, auth) belong in the live demo." Redirect: "Do you want to spend 90 seconds logging in, or 90 seconds showing your agent think?" Production hardening is a great slide ("here's what I'd add before real users") — saying it beats showing it, and mentioning the roadmap earns credit without stage risk.

One-liner for supporters

"Your demo isn't a smaller version of your project — it's the proof of one claim. Everything that doesn't prove the claim is a liability with a progress bar."

---

Segment 4: Dr. Greg's Product-Precision Ladder (~7 min, as time allows)

Run in order — each rung is safe for more people than the last question would be. Skip rungs if time is short; never skip rung 1 or rung 7 if you run the ladder at all.

1. Ideal customer & pain — "Pick one person — real or imagined — who'd use your project. Who are they, and what hurts for them today?" Follow-up: "What do they do right now without your tool?" good answers surface: a specific role/context ("a compliance reviewer prepping docs for release") and a pain stated in the customer's words, not the tech's. Red flag: the "customer" is "anyone who works with documents," or the answer is a feature list.

2. Talked to a real user — "Who's actually shown this to one real human — or even just asked one about the problem? What surprised you?" Follow-up: "What did you change (or decide not to build) because of it?" good answers surface: one concrete learning that redirected the build. Red flag: "I am the user" with no second data point — fine to start, but name it as an assumption to test.

3. Business impact in time or dollars — "Rough math welcome: what does this save or earn, per user, per week?" Follow-up: "What number would you put on your Demo Day slide?" good answers surface: even a napkin estimate ("saves a reviewer ~2 hrs/doc") — the habit of quantifying, not precision. Red flag: "it's hard to quantify" as a full stop; help them pick any proxy metric.

4. A product people love — "Has anyone used yours twice — come back without you asking?" Follow-up: "What did they say, unprompted?" good answers surface: honest calibration — most will say "not yet," and naming what would count as love is a good answer. Red flag: conflating "my friend said it's cool" with love; love is repeated, unprompted use.

5. Enterprise-buyable — "If a company were going to pay for this, who signs, and what would they need to see first?" Follow-up: "What's the first objection procurement or security would raise?" good answers surface: awareness of a buyer distinct from the user, plus one real hurdle (data privacy, integration, auditability, eval evidence). Red flag: "enterprises would love this" with no named buyer or objection.

6. After Demo Day — "Who's still building this in August? What's your path to putting it in front of 10+ users?" Follow-up: "What's the very next experiment — not feature — you'd run?" good answers surface: a distribution idea (a community, a workplace pilot, a waitlist), not just a roadmap of features. Red flag: the plan to get users is "make it better first" — that's rung 7's disease.

7. Building your way out of feedback — the self-diagnosis rung; land it gently and include yourself. Suggested framing:

"Last one, and I'll confess I do this too. As engineers, when a user says something confusing or critical, our reflex is to go add a feature instead of asking a second question. Where in your project might you be building your way out of listening? No wrong answers — spotting it is the win."

Follow-up (only if it lands): "What's one question you could ask a user this week instead of one thing you could build?" good answers surface: self-recognition — "yeah, I added three features nobody asked for." Reward the honesty out loud; don't fix it. Red flag for the supporter, not to call out: a student who answers rungs 1–6 entirely in architecture terms. Follow up 1:1 after the room, not in front of it.

---

Fallbacks If the Room Is Quiet

1. Go first, imperfectly. Share your own Demo Day / project plan answered through ladder rungs 1–3, including a weak spot ("I haven't talked to a real user either — here's who I'd ask"). Modeling incompleteness lowers the bar to speak.
2. Tell a past-cohort story. E.g., a prior cohort's Context Challenger — an agentic RAG doc-validator — worked because the pitch opened with the reviewer's pain (catching contradictions before release) and showed evidence-backed findings, not a model diagram. Ask: "what made that pitch land?"
3. Pair-share, 2 minutes. Everyone pairs up (or trios); each answers rung 1 + rung 3 to their partner. Then ask two people to share what their partner said — it's easier to report someone else's answer than volunteer your own.
4. One-word status check. Round-robin: "one word for where your cert challenge is right now." Fast, zero-stakes, and it hands you the escalation list — anyone who says "stuck," "behind," or "help" gets the remediation-loop message and a follow-up.

---

Wrap-Up (last 2 min)

Read the script beat, then land one closing move:

• One takeaway each (fast rooms): "Before we close — one decision you're leaving with: a cut, a keep, or a person you'll talk to."
• Deadline echo (every room): "Cert challenge is Wednesday 7/16, 7pm ET. Gradable beats perfect — the remediation loop only works if you submit."
• Open door: name where they can find you (and the live channel) between now and Demo Day.

After the room: post your escalation list (anyone blocked, going dark, or at risk of not submitting) to the coordinator same-day.