Skip to content

Instantly share code, notes, and snippets.

@donbr

donbr/session14-mmrag-cheat-sheet.md

Last active July 17, 2026 02:17
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save donbr/75d15f86f86e3544aad620b7ec0409b9 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save donbr/75d15f86f86e3544aad620b7ec0409b9 to your computer and use it in GitHub Desktop.
Download ZIP
Session 14 Cheat Sheet — Multimodal RAG
Raw
session14-mmrag-cheat-sheet.md

Session 14 Cheat Sheet — Multimodal RAG

A frame to help you reason through the assignment — concepts, diagrams, and the API map. It deliberately does not contain the answers or filled-in activity code. Instead it gives you the questions to ask yourself and the cells to go inspect. The work — and the learning — is in running the cells, comparing what actually comes back, and writing your own conclusions.

Notebook: multimodal_rag.ipynb Corpus: ./data — six synthetic ACME Robotics FY2024 charts + six short Markdown docs + one ~45s narrated-slide video (all CC0, auto-generated if missing). The notebook tells you where the real numbers live and why that matters — pay attention to that design note in Section 3.

0. Why multi-modal RAG — and what you already carry into it

You already know text RAG: chunk, embed, retrieve, answer. Session 14 begins exactly where that instinct breaks. This corpus hides its real numbers — "$27M in Q4", "240ms in APAC" — only in the chart pixels, in no source text file anywhere (Section 3 says so out loud). Be precise about which text, though: the VLM parse step (Cell 4.2) later reads those numbers off the pixels and writes them into a generated caption — so once you've run Section 4, the numbers do live in some text (the Chunk.content). "Only in the pixels" is a claim about the shipped corpus, not about what your own pipeline produces — and the gap between those two is exactly what Question #3 turns on. So before you write a line of code, sit with the question the whole session answers:

If every fact you needed already lived in the text of your documents, would you need a vision model at all?

Three tempting shortcuts — feel the limit of each as you run the notebook:

  • "Just caption every image once, then do text RAG" (that's Strategy A). A caption is written once, before anyone asks a question. What happens when the question needs a number the caption never wrote down?
  • OCR / table-extraction. Great when a value is printed text. What does it do with a number that lives in a bar's height or a pie slice's angle — a shape, not a string?
  • Pure pixel matching (CLIP). It can find the right chart from a text query with no caption at all — but can it read the exact number off it?

Keep two verbs separate all session: find and read. For every component you build, ask which job it is doing — locate the right chart, or read the value off it. The whole pipeline is an answer to which tool does which. (Name the costs too: pixels are token-expensive, and CLIP has a "modality gap" — you'll meet both.)

This single split — find the source, then read the evidence — is the spine of the whole pipeline, and it's also how you diagnose a wrong answer: which half broke?

flowchart TD
    Q["Question"] --> F["FIND the source<br/>captions / CLIP / Qdrant / RRF"]
    F --> S["Selected chart"]
    S --> R["READ the evidence<br/>original pixels + VLM"]
    R --> A["Grounded answer"]
    F --- FF["retrieval failure:<br/>correct chart never selected"]
    R --- RF["interpretation failure:<br/>chart selected, value misread"]
    classDef fail fill:#fdecea,stroke:#c0392b,color:#611a15;
    class FF,RF fail;
Loading

ASCII fallback:

        Question
           │
           ▼
   ┌───────────────────┐   ✗ RETRIEVAL failure:
   │ FIND the source   │ ── the correct chart was never selected
   │ captions·CLIP·    │
   │ Qdrant·RRF        │
   └─────────┬─────────┘
             ▼
      Selected chart
             │
             ▼
   ┌───────────────────┐   ✗ INTERPRETATION failure:
   │ READ the evidence │ ── the chart was selected, but its
   │ original pixels + │    value was read incorrectly
   │ VLM               │
   └─────────┬─────────┘
             ▼
      Grounded answer

When an answer is wrong, locate the break: did retrieval hand over the wrong chart (a find failure, fixed by Strategy choice / filtering / fusion), or did the model misread a chart it was given (a read failure, fixed by sending real pixels to the VLM)? The two failures live in different halves and have different fixes — that's why the whole session keeps them apart.

You are not starting from zero — three pieces here have cohort history. Notice them as they reappear:

  • Qdrant has been your vector store since Session 1; here it just runs in :memory: mode.
  • Reciprocal Rank Fusion (Strategy C) is the same fusion you hand-wrote in Session 7 to combine BM25 + dense results — the formula didn't change, only why the two lists don't compare did. (More on this at Question #4 below.)
  • The recall@k-style + gold-set eval (Section 8) is the harness you built in Session 7 (retrieval_eval.py, which computed true recall@k and MRR) and the lightweight cousin of Session 6's RAGAS. (Heads-up: the S14 version actually computes hit-rate@k — §4 has the note.)

The rest of this sheet points you at where to look; the conclusions are yours to write.

What you already carry in — and the new wrinkle each piece grows

You are extending text RAG, not replacing it. Every row below is something you already built; the third column is the only thing Session 14 adds. Read it as "same muscle, one new joint."

Earlier learning Reused here (barely changed) New multimodal wrinkle
Text chunks (S1) one unified Chunk list some chunks are captions that point back to pixels via source_path
Text embeddings (S1) OpenAI text-embedding-3-small it can't embed an image — you need a caption (A) or CLIP (B/C)
Qdrant retrieval (S1) same QdrantVectorStore API, now :memory: one mixed collection + optional modality filter, or separate modality stores + RRF
BM25 + dense RRF (S7) the same 1/(k+rank) fusion, unchanged the two lists now come from different-modality embedders, not two scorers over one text
Gold-set eval (S6/S7) same question → correct-source-id mapping a correct source can now be a chart image; the same gold-set pattern could later be extended to time-windowed video chunks
Retrieve-then-answer (all RAG) same structure, one prompt generation is handed the original pixels, not just retrieved text

Keep this table open as you run: every time something feels new, find its row and notice that only the third column actually changed.

Same idea as a pipeline, so you can see where the additions land. The green lane is the text RAG you already built; the blue lane is Session 14. Only the CAPS stages are new or extended:

flowchart LR
    subgraph TEXT["Text RAG — what you already built"]
      direction LR
      T1["documents"] --> T2["chunk"] --> T3["text<br/>embedding"] --> T4["retrieve<br/>(cosine)"] --> T5["LLM<br/>answer"]
    end
    subgraph MM["Session 14 — multimodal RAG"]
      direction LR
      M1["text + IMAGES"] --> M2["chunk<br/>+ VLM CAPTION"] --> M3["text embed<br/>OR CLIP PIXELS"] --> M4["CROSS-MODAL<br/>retrieve"] --> M5["VLM answer<br/>+ ORIGINAL PIXELS"]
    end
    classDef same fill:#e8f5e9,stroke:#2e7d32,color:#1b3d1c;
    classDef ext  fill:#e3f2fd,stroke:#1565c0,color:#0d2b4e;
    class T1,T2,T3,T4,T5 same;
    class M1,M2,M3,M4,M5 ext;
Loading

ASCII fallback (CAPS = new or extended; everything else is your existing text-RAG stage):

stage:      source          chunk                embed                retrieve            answer
─────────────────────────────────────────────────────────────────────────────────────────────────
text RAG:   documents        split text           text embedding       cosine top-k        LLM (text)
Session 14: text + IMAGES    split + VLM CAPTION   text emb OR CLIP      CROSS-MODAL top-k   VLM + ORIGINAL PIXELS
            └ extended       └ new step           └ extended           └ extended          └ extended

The genuinely new roles are the VLM as ingestion parser (the CAPTION step), CLIP as the image-capable embedder (the PIXELS path), and the VLM as answer-time reader of the original pixels. Everything else is a familiar RAG stage extended to additional modalities.

Readiness check — can you answer these before you start?

If any answer is fuzzy, revisit the linked earlier session first; this session assumes all four.

  • Text RAG loop (S1): can you name the five stages — chunk → embed → store → retrieve → answer — and say what each produces?
  • Embeddings vs. similarity (S1): why does cosine over text vectors put a query near its answer, and what is an embedding of?
  • RRF (S7): can you write 1/(k+rank) from memory and say why you fuse by rank instead of raw score?
  • Retrieval eval (S6/S7): what does a gold set contain, and what does "a correct id landed in the top-k" measure?

1. Quick Reference

You want to… Reach for One-liner
Load key + build the VLM init_chat_model vlm = init_chat_model(VLM_MODEL, temperature=0) — one line swaps vision-capable providers (see §3 caveat)
Parse a chart into a searchable chunk VLM as ingestion parser vlm_parse_image(path) → strict JSON → folded into Chunk.content
Send an image to any chat call image_block / multimodal_message {"type":"image","source_type":"base64","data":...,"mime_type":...}
Strategy A — caption→text one Qdrant text collection store_A = QdrantVectorStore.from_documents(docs_A, text_embedder, ...)
Strategy B — unified CLIP shared text+image vector space clip.embed_images(paths) + qdrant_store_from_vectors(...)
Strategy B, image-only slice Qdrant metadata filter retrieve_B(q, modality="image")
Strategy C — separate stores, fused Reciprocal Rank Fusion (Session 7's fusion, hand-written) reciprocal_rank_fusion([text_hits, img_hits])
One retriever, strategy-swappable MultiModalRetriever MultiModalRetriever(strategy="C", k=3).retrieve(query)
Generate a grounded answer answer() answer(q, strategy="C")
Measure retrieval quality recall_at_k recall_at_k("A", k=3) → hit-rate@3 (see §4 note) against a hand-written gold dict
Video RAG transcript + keyframes → time-windowed chunks build_video_chunks(...) → answer_video(q) cites [MM:SS]

One sentence to keep in mind while you read the notebook: LangChain's Embeddings interface is text-only, but chat models are multimodal — that single fact is the reason Section 5 needs three different strategies at all. Everything else you're asked to explain traces back to it or to a consequence of it. Don't take that on faith — go find the evidence for it yourself in Sections 0 and 5.

2. The Big Picture — one pipeline, three retrieval strategies, two VLM jobs

flowchart TD
    IMG["6 chart PNGs"] -->|"VLM parses<br/>Cell 4.2"| CAP["JSON caption per chart"]
    TXT["6 text docs"] --> CHUNKS["unified Chunk list"]
    CAP --> CHUNKS
    CHUNKS --> A["Strategy A<br/>caption-to-text embed<br/>one Qdrant store"]
    CHUNKS --> B["Strategy B<br/>unified CLIP<br/>pixels + text, one space"]
    CHUNKS --> C["Strategy C<br/>text store + image store<br/>fused by RRF"]
    A --> RET["MultiModalRetriever"]
    B --> RET
    C --> RET
    RET --> GEN["answer()<br/>reloads image file<br/>calls the VLM"]
    RET --> EVAL["hit-rate@3 harness<br/>gold dict, A/B/C compared"]
    VID["video: transcript + keyframes"] -->|"reuses Strategy C pattern"| VRET["retrieve_video to RRF"]
    VRET --> VANS["answer_video()<br/>cites [MM:SS]"]
Loading

ASCII fallback:

6 charts ─VLM parse─► captions ─┐
6 docs ───────────────────────► ├─► chunks ─┬─► A (caption→text)  ─┐
                                             ├─► B (unified CLIP)   ├─► retriever ─► answer()
                                             └─► C (text+img, RRF) ─┘              └─► hit-rate@3 eval (A/B/C)
video (transcript+keyframes) ──reuses Strategy C + RRF──► answer_video() [timestamp citation]

Orientation, not answer: Section 1 of the notebook tells you the VLM does two different jobs — find the sentence that names them and notice which sections of the notebook correspond to each. The retrieval strategies (A/B/C) exist for a reason spelled out in Section 0 — that's Question #1's whole territory. The video extension in Section 9 isn't a fourth strategy; look closely at which earlier function it reuses before you assume it's new machinery.

3. Setup & the component roles

uv sync          # from 14_multimodal_rag, then select the uv kernel

Needs OPENAI_API_KEY (VLM parsing + generation + text embeddings). First CLIP call downloads clip-ViT-B-32 (~600MB) via sentence-transformers — a one-time cost. VLM_MODEL / EMBED_MODEL are the only two lines to change for a provider swap — with a caveat the "one-line" framing hides: the new provider must be vision-capable and must accept the same v1 image content-block shape ({"type":"image","source_type":"base64",...}). init_chat_model unifies the call, not each provider's multimodal support — a text-only or differently-shaped model will import fine and then fail at the first image. Build the env on Python 3.13 (this session pins >=3.13,<3.14 — the only one with an upper bound, to keep the CV/ML wheels resolvable).

Tooling you already have — and one telling absence. Everything here is a uv project (as since day one), and Qdrant — your vector store since Session 1 — is back, just in :memory: mode (a real Qdrant engine inside the notebook process, no server to stand up). Now look at what's not in pyproject.toml: no rerank/ensemble library — no langchain-cohere, no rank-bm25, the two retrieval libs you installed in Session 7. That's deliberate: the reciprocal_rank_fusion in Cell 5.4 is hand-written in about eight lines of plain Python, so you can see exactly how the two result lists combine before ever reaching for a library. (What is new: the CLIP + video stack — sentence-transformers, opencv-python-headless, imageio, pillow.)

Versions this session expects (validated). Python 3.13 (requires-python = ">=3.13,<3.14") and LangChain v1 — the notebook uses v1-only APIs (init_chat_model, v1 image content blocks, langchain-qdrant), so an older LangChain 0.x will ImportError. A clean uv sync resolves to langchain 1.3.14, langchain-openai 1.3.5, langchain-qdrant 1.1.0, qdrant-client 1.18.0, sentence-transformers 5.6.0. (Provenance for those pins is in Version and maintenance notes at the end of this sheet.)

Role Object What it does
VLM (both jobs) vlm (init_chat_model(VLM_MODEL)) used in two different cells for two different purposes — find both
Text embedder text_embedder (OpenAIEmbeddings) embeds text; used by which store(s)?
CLIP embedder clip (ClipEmbedder) has both an image side and a text side — check ClipEmbedder's methods
Vector store Qdrant :memory: (multiple named collections) dies with the kernel; same API as a hosted cluster

4. Core concepts — investigate, don't skip

These are the things Questions #1–4 ask you to explain in your own words. Below is where to look and what to compare — not the conclusion. Write your own description before you read the notebook's own callouts (Cell 34 and the Section 8 "Gotchas" list say a lot out loud — better to form your own read first, then check it against theirs).

The three strategies side by side (the overview shows that they branch; this shows how they differ). For each, watch four things: (1) what turns an image into a vector, (2) which model embeds the query, (3) how many stores exist, (4) where fusion or filtering happens.

A — Caption first  (Cell 5.1)
   image ──VLM caption──► text embed ┐
   text ────────────────► text embed ├──► ONE text store ──► similarity_search
   image→vector: VLM writes a caption, text model embeds it   query: OpenAI text
   stores: 1                                                  fuse/filter: none

B — Shared CLIP space  (Cells 5.2–5.3)
   image ──► CLIP image encoder ┐
   text ───► CLIP text encoder  ├──► ONE mixed store ──► search (optional modality filter)
   image→vector: CLIP encodes the PIXELS   query: CLIP text tower
   stores: 1 (mixed)                       fuse/filter: metadata filter per modality

C — Separate and fuse  (Cell 5.4)
   text ───► OpenAI embed ──► text store  ┐
   image ──► CLIP embed   ──► image store ├──► RRF fuse ──► top-k
   image→vector: CLIP encodes the PIXELS   query: OpenAI (text store) + CLIP (image store)
   stores: 2                               fuse/filter: RRF at query time

Read the panels together: A never embeds a pixel (it embeds words about the pixel); B and C both do, but B forces text and images into one space (hence the modality gap) while C keeps them apart and reconciles by rank. Every row in the tradeoffs table later is a consequence of these three shapes.

The one fact everything traces back to (→ Question #1)

Reread the callout at the end of Cell 0.3. Then, for Strategies A (Cell 5.1), B (Cells 5.2–5.3), and C (Cell 5.4): find the exact moment each strategy would normally need to turn an image into a vector, and note what actually happens at that moment instead. Do all three route around the same wall the same way, or differently from each other? What does each one give up or add to make its choice work?

Two rankings, one query (→ Question #2, weakness #1)

Run both lines in Cell 5.3 back to back: retrieve_B(q) and retrieve_B(q, modality="image"), same q. Same query, two different top-3 lists — what changed between them? Write your own one-sentence description of the pattern before you read the markdown cell immediately after (Cell 34) — it names the phenomenon and offers two fixes. Then ask: does Strategy C's design (Cell 5.4) run into the same problem? Why or why not?

The shape to predict before you run (schematic — your actual ids will vary, but the shape is the point):

Unified mixed search retrieve_B(q) Image-filtered retrieve_B(q, modality="image")
1. a text document 1. the matching chart
2. a text document 2. another chart
3. a text document 3. another chart

That top-heavy text column — even though the query literally describes a chart — is the modality gap made visible. Does your run match this shape? Only after you've written your own one-sentence why should you open Cell 34 to check it.

What CLIP retrieval alone can and can't do (→ Question #2, weakness #2)

Strategy B can find the right chart by matching pixels — you just proved that above. Now ask: if Strategy B's retriever were the whole pipeline (no further step), could it tell you the actual number on that chart? What kind of task is "read this exact figure off an image" versus "find the image whose content matches this description"? Are those the same skill?

What gets stored vs. what gets sent (→ Question #3)

Look at what's stored in a Chunk.content for an image (built in Cell 4.3) — it's text. Now look at what build_answer_message (Cell 7.1) actually attaches to the model's context for that same image. Compare the two. If the caption's data_points already list some numbers, why does the notebook still reload the original file from source_path and send it as image_block(...)? Section 3 has a design note about where the real numbers live in this dataset — reread it and connect it to what you just compared.

Comparing two lists that came from different places (→ Question #4)

In Cell 5.4, look at how store_C_text and store_C_img are each built — what embedding model produces the vectors for each one? Are the two the same model? Now open reciprocal_rank_fusion in that same cell and trace, line by line, what information it actually uses from each result — does the function ever touch a similarity score, or only a result's position in its list? Given your answer to the "are they the same model" question, what would go wrong if the merge instead just kept whichever candidates had the highest raw score across both lists?

You've fused lists like this before — go compare. In Session 7 you hand-wrote the same reciprocal_rank_fusion to merge a BM25 (keyword) list with a dense (embedding) list — two scorers over the same text. Here the two lists come from different embedders over different modalities (OpenAI text vs. CLIP images), and the formula (1/(k+rank), k=60) is identical. So ask yourself: the reason the two lists are incomparable changed (different scorers → different modalities), but the fix didn't — what is RRF actually protecting you from, and is it ever really about "modality" at all, or something more general? One more distinction to carry forward: Session 7 also reranked the survivors with Cohere Rerank — a model that rescores each candidate by relevance (a paid model call). RRF fuses by rank with no model call; a reranker rescores with one. Which is which — and which of the two is Section 10's optional "cross-encoder reranker" reaching back to?

Tradeoffs at a glance (from the notebook, Cell 37 — reread it after you've run Section 5, don't just skim it)

A. Caption→text B. Unified CLIP C. Separate+fuse
Cross-modal search via captions only native (pixels) native (image store)
Ingestion cost high (VLM call/image) low medium
Fine-grained numbers as good as caption weak text strong, image weak
Query latency low low higher (2 searches + fuse)
Main failure ? ? ?

The notebook fills in that last row (Cell 37) — go read it after you've run Section 5 and formed your own guess, not before. Before Activity #1, ask yourself: given this table, which strategy would you predict handles a query like "a pie chart of cloud spend" best, and which handles "what caused the March churn spike" best — before you run them? Then check your prediction against what actually comes back.

Measuring retrieval — recall_at_k

gold = {"some question": {"the_id_that_should_come_back"}, ...}
def recall_at_k(strategy, k=3): ...   # fraction of gold questions where a correct id lands in top-k

This is the mechanism Activity #2 extends — go read Cell 54 to see the existing eight gold questions before you write your own two. This gold-set + recall@k harness is not new to you: it's the same shape you built in Session 7 (lib/retrieval_eval.py, which also reported MRR) and the hand-rolled cousin of Session 6's RAGAS.

One precision point worth catching (the notebook doesn't flag it). Read Cell 54's function line by line: it scores a question as a hit when len(want & got) > 0 — i.e. any one gold id in the top-k — then divides by the number of questions. That is strictly hit-rate@k (a.k.a. success@k), not the |found ∩ relevant| / |relevant| recall that Session 7's retrieval_eval.py computed. For the six single-id questions the two are equal; but two gold entries here have two correct ids each ("When did NPS dip" and "What caused the March churn spike") — and for those, retrieving just one of the two still scores a full hit, where true recall would be 0.5. So ask: which of your two Activity-#2 questions should be single-id, and would picking a two-id one change what the printed number even means? Then ask what MRR would tell you that neither hit-rate nor recall@3 can — and whether it would change how you read Strategy A/B/C's scores.

Video extension — same pattern, new axis (time)

Video = two aligned streams (a timestamped transcript + sampled keyframes). Before you read Section 9's code closely, ask: given everything above about Strategy C, what would you expect the video retrieval function to reuse? Check your guess against retrieve_video in Cell 9.5.

Once you've made that guess, here's the shape to check it against. The ./data clip is 45s of narrated slides; each transcript segment is time-aligned with the keyframe(s) on screen in that window — a time-windowed chunk. The two streams become two stores, fused by RRF — which is exactly Strategy C, only the axis that aligns the modalities is now time:

time  0:00      0:06        0:16        0:26        0:37     0:45
      │ intro  │ revenue   │ churn     │ APAC lat. │ team   │
  transcript ──┴───────────┴───────────┴───────────┴────────┘ ─► TEXT store  (OpenAI embed)
  keyframes  ──┬───────────┬───────────┬───────────┬────────┐ ─► IMAGE store (CLIP pixels)
      │  kf    │  kf        │  kf       │  kf       │  kf    │
                                        ▲
        query: "when did the latency alert happen in the video?"
                                        │
        retrieve_video = RRF( TEXT store , IMAGE store )   ← identical to Strategy C
                                        │
                                        ▼
        answer_video() cites the winning window  →  [00:26–00:37]

So Section 9 is not a fourth strategy: it's Strategy C (store_C_text + store_C_img + the same reciprocal_rank_fusion) pointed at a transcript and its keyframes, with t_start/t_end carried on every chunk so the answer can cite [MM:SS–MM:SS]. Confirm each piece of that against Cells 9.4–9.6.

5. Questions — what to ask yourself (write your own answers in the notebook)

Section 4 was the investigation — where you ran cells, compared lists, and traced code. These four are the write-ups that investigation feeds. Each points back to the §4 subsection that set it up: the task here is to state your conclusion, not to re-run the mechanics. If a question feels unanswerable, go do the matching §4 subsection first, then come back and write.

Q1 — Text-only embeddings, three workarounds

What single property of LangChain's Embeddings interface (Cell 0.3), in tension with one property of chat models, forces the whole three-strategy design? You already traced how A, B, and C each route around that wall in §4's "The one fact everything traces back to" — so don't re-run them; settle the payoff it left open: are the three genuinely different answers, or variations on one trick?

Q2 — Strategy B's two weaknesses

Name Strategy B's two failures — and the pipeline part that compensates for each. §4's "Two rankings, one query" and "What CLIP retrieval alone can and can't do" already had you see both. Write them up as two distinct failures at two different moments: the modality gap (before any answer exists) and CLIP-can't-read-values (after retrieval has already succeeded). Is the fix for each the same mechanism, or two different ones?

Q3 — Pixels over the cached caption

If the caption were always perfectly complete, would resending the original pixels still matter? You compared Chunk.content (Cell 4.3) against what build_answer_message (Cell 7.1) sends in §4's "What gets stored vs. what gets sent." Answer that counterfactual, and tie it to Section 3's note on where the exact numbers live — and to the source-pixels-vs-generated-caption distinction from Section 0.

Q4 — Raw scores vs. Reciprocal Rank Fusion

Since store_C_text (OpenAI) and store_C_img (CLIP) score on different scales, what would "keep the higher raw score" actually be measuring — relevance, or scale? You already traced both stores and the reciprocal_rank_fusion body in §4's "Comparing two lists that came from different places." State the conclusion, and say why summing 1/(k + rank) sidesteps the scale problem entirely.

6. Activities — the deliverable + how to get there

Activity 1 — Probe the three strategies with your own queries

Deliverable: two of your own real questions about the ACME corpus (the data-dictionary printout in Section 3 is your menu) run through Strategies A, B, and C via the scaffold cell (Cell 41), plus a written observation on where the rankings differed and why, given the tradeoffs table. Method: try one visual phrasing (e.g., "a pie chart of…") and one more factual phrasing — the notebook itself nudges you this way. Run both through all three strategies and actually compare the printed id lists, not just skim them. For each place the rankings diverge, connect it back to a specific row of the tradeoffs table (ingestion cost? native pixel search? caption quality?) rather than just noting that they diverged.

Activity 2 — Extend the evaluation set

Deliverable: two new entries added to gold — one whose answer lives in a text doc, one whose answer lives in a chart image — plus a written observation on whether any strategy's hit-rate@3 moved, and why that might be. (Two cells, don't confuse them: Cell 54 defines the original eight-question gold; Cell 73 is the my_gold stub where you add yours and re-run — it does gold.update(my_gold), so leave Cell 54 alone.) Method: use the data dictionary (Section 3) and the doc bodies (Section 3) to pick two real questions with unambiguous correct ids — double-check your mapping before running. Then look at the actual before/after numbers per strategy, not just whether the overall average moved. If nothing changed, that's a real, reportable result too — ask yourself why nothing changed given what you now know about this corpus and these three strategies.

Advanced Build (optional, post-cohort)

Provider swap, BM25+dense hybrid on text (Session 7's hybrid retriever, brought to this corpus), Markdown-table extraction in the parse prompt, a cross-encoder reranker (the Cohere Rerank move from Session 7 — rescore the fused survivors), scene-change keyframe sampling, or a "the data can't answer this" decline test — and measure whether it actually helped (that's what the hit-rate@3 harness is for).

7. References & going deeper

Primary sources for the APIs and concepts this session leans on. Open them to verify what the notebook does and to check your own answers against the docs — not to shortcut the reasoning. (Links current as of 2026-07-16; APIs move fast, so confirm the version.)

The one fact everything traces back to — text-only embeddings vs. multimodal chat (Q1, Q3)

Provider-agnostic model construction (Section 0, the one-line provider swap)

The vector store — Qdrant in :memory: mode (Sections 3, 5)

Unified CLIP embeddings & the modality gap (Strategy B, Q2)

Fusing two rankings — Reciprocal Rank Fusion (Strategy C, Q4)

Cohort callbacks — go re-read your own earlier code

  • Session 1 — Qdrant first appears (01_Cat_Health_Vector_RAG_LangChain_Qdrant.ipynb).
  • Session 6 — RAGAS, the eval framework recall@k is the hand-rolled cousin of.
  • Session 7 — the same hand-written reciprocal_rank_fusion (1/(60+rank)), the BM25+dense hybrid (rank-bm25), Cohere Rerank (langchain-cohere, a rescoring model call — contrast with RRF's rank-only fusion), and lib/retrieval_eval.py, which reports both recall@k and MRR. Section 10's optional cross-encoder reranker reaches back to this.

Version and maintenance notes

Provenance for the version pins in Section 3 — kept here so it doesn't interrupt the learning sequence.

  • Resolved (clean uv sync, validated 2026-07-16): langchain 1.3.14, langchain-openai 1.3.5, langchain-qdrant 1.1.0, qdrant-client 1.18.0, sentence-transformers 5.6.0; Python >=3.13,<3.14.
  • 2026-07-16 — two fixes landed, no behavior change: (1) the pyproject.toml LangChain/Qdrant floors were tightened from >=0.3 to the cohort's v1 convention (langchain>=1.3.0,<2.0.0, langchain-openai>=1.2.0,<2.0.0, langchain-qdrant>=1.0.0,<2.0.0, qdrant-client>=1.15.0,<2.0.0) so a fresh install can't silently pull a pre-v1 LangChain; (2) the README's notebook filename was corrected to multimodal_rag.ipynb (it previously read multimodal_rag_vlm.ipynb, which doesn't exist).

The learning is in running the cells, comparing the actual printed rankings and recall numbers, and writing your own conclusions. The notebook's own callouts (Cell 34, Section 8's Gotchas, Section 10's recap) are there to check your reasoning against — read them after you've formed your own answer, not instead of forming one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment