Skip to content

Instantly share code, notes, and snippets.

@donbr

donbr/multi-model-rag-vlm.md

Created July 16, 2026 23:45
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save donbr/09f14723709114ef815f87a6cf43442e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save donbr/09f14723709114ef815f87a6cf43442e to your computer and use it in GitHub Desktop.
Download ZIP
Multi-Modal RAG with Vision-Language Models
Raw
multi-model-rag-vlm.md

Session 14 — Multi-Modal RAG with Vision-Language Models: Learning Journey (Student Version)

A companion to this session's notebook, multimodal_rag.ipynb — the one where you extend text RAG to images + text in one pipeline: a VLM that parses charts into searchable chunks, three cross-modal retrieval strategies on Qdrant, generation that reads exact numbers off chart pixels, and a video extension that answers with timestamp citations. It's a run log: the whole notebook was actually executed — uv sync, the ~600 MB CLIP download, every VLM call, the recall@3 eval, and the video answer — so you have a "known-good" reference for what a healthy run looks like and where people trip. Use it to tell a real problem apart from expected run-to-run noise.

This is the student version. The engineering findings and setup gotchas are all here, but the graded reasoning is yours to do. You demonstrate understanding through your executed work — the parsed captions, the three strategies' rankings, the answers the model read off the pixels, your recall@3 table, and the timestamped video answer — plus the four ❓ questions and two 🏗️ activities in the notebook. Prompts to check your own understanding are near the end, without answers. No API keys appear here.

A reference environment (yours may differ): WSL2/Linux, uv, Python 3.13, run inside the session's own .venv. LangChain v1 (init_chat_model

  • image content blocks), langchain-qdrant, sentence-transformers (CLIP), openai. An OPENAI_API_KEY available when you run the notebook. Default models: openai:gpt-4o (VLM) and text-embedding-3-small (text embedder).

Legend: ✅ verified by actually running it · ⚠️ watch this — the spot people trip · 💡 optional, do-it-better tip

The short version: does it all run?

Yes — the whole notebook runs clean, top to bottom, once your kernel is right. uv sync resolves without a fight, the VLM reads exact figures straight off chart pixels, all three retrieval strategies behave as described (including the "modality gap" you're asked about), and the video answer cites a real timestamp. There are no broken imports and no wrong APIs. The one thing that turns a healthy notebook into "nothing works" is which Python kernel is running it — that's #1 below, and it's the single most common ticket. Everything else is the CLIP download and your API key. None of it is a code edit.

Part-by-part: what a healthy run looks like

Breakout Room #1 — Parsing & Cross-Modal Retrieval (Sections 0–5) ✅ runs clean

When it's working you should see, in order:

  • Section 0 — setup: Environment OK, then VLM_MODEL=openai:gpt-4o EMBED_MODEL=text-embedding-3-small, then Vision-language model ready. If Cell 0.1 says packages are missing, don't re-run uv sync — jump to trip-wire #1, it's almost certainly your kernel.

  • Section 3 — the corpus: 6 charts + 6 docs load, and the data dictionary prints each chart's hidden fact (Q4 highest at $27M, APAC worst at 240ms, …). Those numbers live only in the chart pixels — that's the whole design.

  • Section 4 — the VLM as parser: the model reads a chart and returns strict JSON. For the revenue chart you should see a takeaway like "Q4 had the highest revenue at $27M" and data_points ["Q1: $12M", …, "Q4: $27M"]. It's reading numbers off an image. (Your exact wording will differ — that's fine.)

  • Section 5 — three strategies:

    • A (caption→text): "Which quarter had the highest revenue?" puts fig_revenue_quarterly.png first.
    • B (unified CLIP) — the important one: the query "a bar chart comparing API latency across regions" returns only text documents in the unified search (the images get crowded out!), but the images-only slice puts fig_latency_region.png first, matched by pixels alone. Seeing text outrank the obviously-correct chart is the modality gap — it's the setup for Question #2, not a bug.
    • C (separate stores + RRF): "why did on-call get paged about APAC?" brings the latency chart back into the top-3 alongside the infra doc.

  • ⚠️ Activity #1 ships with REPLACE ME: placeholder queries. Running it as-is prints meaningless, identical rankings — that's expected. The work is replacing them with two real ACME questions (try one visual phrasing and one factual phrasing) and explaining where A/B/C diverge. Different rankings across the strategies is the goal, not a mistake.

Breakout Room #2 — Generation, Evaluation & Video (Sections 6–10) ✅ runs clean · ⚠️ one score surprises people

A healthy run shows:

  • Section 7 — generation reads the pixels: the model answers with the exact numbers and cites the chunk ids — e.g. $27M (revenue), March … 6.2% (churn), Compute 55% / Storage 20% / Network 15% / Other 10% (the pie), NPS 22 → 41, dip to 28 in Q3. Every one of those came from the model looking at the retrieved image, not from a text file. That "retrieve to find the chart, look at the chart to answer" split is the core pattern.

  • Section 8 — recall@3: you should see something like A = 1.00, B = 0.25, C = 1.00 (see the ⚠️ below — B is supposed to be low).

  • Section 9 — video: 5 transcript segments, ~9 keyframes sampled, and the video question about APAC latency comes back citing a timestamp like [00:26-00:37] with the 240ms detail — a window that matches the transcript's APAC segment.

  • ⚠️ Strategy B's recall is low on purpose — do not "fix" it. On a real run, Strategy B scores far below A and C (we got 0.25). That's not a broken run: the recall harness runs B as one mixed ranking, so the modality gap buries the chart images and B only catches the questions whose answer is a text document. That low number is the exact weakness Question #2 is about, made numeric. Reason about why B is low (text crowds out images in one unified ranking); don't treat it as a score to rescue.

  • ⚠️ Activity #2 ships with my_gold = {} empty, so it re-prints the same n=8 numbers. The work is adding two real gold rows — one whose answer is a text doc, one whose answer is a chart image — then reporting whether any strategy's recall moved and why. "No movement" is a perfectly valid, well-reasoned answer on a corpus this small.

The trip-wires people actually hit

  1. ⚠️ Your kernel — the "I ran uv sync and it still says packages are missing" trap. This is the #1 issue by a mile. If Cell 0.1 fails with Missing ['sentence_transformers', 'imageio'] — run uv sync, the problem is almost never uv sync — it's that the notebook is running on the wrong Python (a global python3 kernel instead of this folder's .venv). Two reliable fixes:
    • Launch from the environment: uv run jupyter lab (the kernel is then guaranteed to be the .venv), or
    • In Cursor/VS Code, select this folder's .venv as the kernel (kernel picker, usually bottom-right).
    • Sanity check inside the notebook's kernel: import sys; print(sys.executable) — it must point inside 14_multimodal_rag/.venv. If it doesn't, that's the whole problem. Re-running uv sync won't help; switching the kernel will.
  2. ⚠️ The first Strategy-B cell downloads ~600 MB and looks like it hung — it didn't. The first time ClipEmbedder() runs (Section 5.2), it downloads the CLIP model (clip-ViT-B-32, ~600 MB) from Hugging Face. On a slow connection this takes a while and the cell just sits there. Let it finish — it's cached after the first run and instant thereafter.
  3. ⚠️ Your API key. Unlike some earlier sessions, this notebook does load .env for you (Cell 0.2 calls load_dotenv), so copying .env.example → .env and putting your OPENAI_API_KEY in it works exactly as the README says. (Exporting it in your shell before launching also works.) If the key is missing you get a clear error telling you what to do — no silent failure.
  4. ⚠️ Two shipped-doc mismatches, so you don't chase ghosts. The README mentions multimodal_rag_vlm.ipynb, but the real file is multimodal_rag.ipynb — open that one. And .env.example shows a commented example model openai:gpt-5.6-luna; that's just an illustration of how to override — the real, working default is openai:gpt-4o. Don't paste the example name in expecting it to resolve.
  5. 💡 What's random and what isn't. The retrieval results — the strategy rankings, the modality-gap demo, the recall@3 table — are stable run-to-run (fixed embedders). What does change each run is the VLM's caption wording and its answer prose. So if your $27M answer is phrased differently from a classmate's, or your B score is 0.12 vs 0.25, that's normal — the mechanism is what matters (right chunk found, number read off the image, timestamp cited).

Check your own understanding (no answers here)

This session is graded on your executed work plus your written answers to the four questions and two activities. If you can answer these from your own runs, you've got it. They're a method, not a key.

Parsing & the three strategies (Breakout Room #1)

  • LangChain's embedding interface only accepts text, but chat models can see images. Why does that one fact force the notebook to build three different retrieval strategies instead of one? For each of A, B, and C, say how it routes around the text-only embedder — not just its name.
  • In Strategy B, a query that literally describes a chart comes back as all text documents, yet the images-only search finds the right chart instantly. What is that failure called, and why does a text query land closer to text than to the correct image even though CLIP shares one space?
  • Strategy C merges two result lists with Reciprocal Rank Fusion instead of just keeping the highest similarity scores. Why would comparing the raw scores from the text store and the image store be a mistake — and what does RRF use instead of score magnitude?

Generation, evaluation & video (Breakout Room #2)

  • At answer time the notebook re-loads each retrieved image from disk and sends the actual pixels to the model, even though it already made a caption for that image during ingestion. Why not just reuse the caption? (Hint: what is a caption for versus what are pixels for — and where do the real numbers in this dataset live?)
  • Look at your recall@3 table. Strategy B scored much lower than A and C. Which kind of question does B miss, which kind does it catch, and how does that connect to the weakness you described for Strategy B earlier? Is B's low score a bug or the point?
  • In your Activity #1 output, did the three strategies rank your two queries differently? Tie each difference back to the tradeoffs table — why would a visual phrasing favor the CLIP-driven strategies and a factual one favor the caption/text ones?
  • Your video answer cited a timestamp like [00:26-00:37]. What two aligned streams make video RAG possible, and which Section-5 idea did the video pipeline reuse wholesale to get there?

References

The "known-good" sources when something disagrees with what you see.

Tools move fast. If a signature or a model name disagrees with these docs, the docs win — and remember the models are overridable via the VLM_MODEL / EMBED_MODEL env vars, so a model error is almost never a reason to edit the notebook.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment