Session 10 Cheat Sheet — LLM Servers

10_LLM_Servers_CHEATSHEET_STUDENT.md

Session 10 Cheat Sheet — LLM Servers

A frame to help you reason through the assignment — concepts, diagrams, and the API map. It deliberately does not contain the answers or filled-in activity code. Instead it gives you the questions to ask yourself and the method to get there. The work — and the learning — is in deploying the endpoint, running the cells, reading the output/traces, and writing your own conclusions.

Repo: 10_LLM_Servers/ — endpoint_slammer.ipynb (endpoint test) + an app/ Fireworks-powered RAG LangGraph agent. Q1/Q2 are answered in README.md. Corpus: data/cat-health-guide.pdf (2021 AAHA/AAFP Feline Life Stage Guidelines; the RAG tool indexes it in-memory).

Quick Reference

You want to…	Reach for	One-liner
Use an open-source model with no infra	Fireworks serverless endpoint	accounts/fireworks/models/gpt-oss-20b — pay per token, instant, shared
Get guaranteed capacity/latency	Fireworks dedicated deployment	firectl create deployment gpt-oss-20b … — pay per GPU-hour, shut it down
Hit Fireworks with LangChain	ChatOpenAI + base URL	ChatOpenAI(model="accounts/fireworks/…", openai_api_base="https://api.fireworks.ai/inference/v1", openai_api_key=FIREWORKS_API_KEY)
Hit Fireworks with the native client	langchain_fireworks.ChatFireworks	ChatFireworks(model=model_endpoint) (the notebook's way)
Embed with a Fireworks model	OpenAIEmbeddings + base URL	OpenAIEmbeddings(model="…qwen3-embedding-8b", openai_api_base=<fireworks>, check_embedding_ctx_length=False, dimensions=4096)
Probe throughput/latency under load	asyncio.gather over .ainvoke	endpoint_slammer.ipynb fires 24 concurrent requests
Build a RAG tool	in-memory Qdrant + a 2-node graph	retrieve → generate, exposed as @tool retrieve_information
Give an agent tools	bind_tools + ToolNode	[TavilySearch, ArxivQueryRun, retrieve_information]
Fix gpt-oss tool calls	fix_tool_calls()	strips a trailing <|call|> token so the JSON parses
Bound an agent loop	a message-count ceiling	helpfulness graph: len(messages) > 10 → END
Measure RAG quality	RAGAS	faithfulness, answer_relevancy, context_precision, context_recall
Measure token cost per query	LangSmith tracing / usage_metadata	LANGSMITH_TRACING=true + cost dashboard, or get_openai_callback()

Anchor: an "LLM server" is just an OpenAI-compatible HTTP endpoint — keep the LangChain client, change only openai_api_base + the model slug. Everything else (RAG, agents, eval) is provider-agnostic.

The Big Picture

flowchart LR
  subgraph BOR1[BOR#1 — Endpoints]
    FW["Fireworks: gpt-oss-20b<br/>serverless OR dedicated"]
    Slam["endpoint_slammer.ipynb<br/>1 invoke + 24-way async slam"]
    FW --> Slam
  end
  subgraph BOR2[BOR#2 — RAG app]
    PDF[cat-health PDF] --> Split[token-aware split 750] --> Q[(in-memory Qdrant)]
    Emb["Fireworks Qwen3 embeddings"] --> Q
    Q --> RAGtool["retrieve_information @tool"]
    RAGtool --> Agent["LangGraph tool-calling agent<br/>(+ fix_tool_calls, + helpfulness loop)"]
    FW --> Agent
  end
  subgraph EVAL[Activity 1 — eval + cost]
    Agent --> RAGAS[RAGAS: faithfulness / relevancy / precision / recall]
    Agent --> LS[LangSmith: tokens + $ / query]
    GPT41["OpenAI gpt-4.1-mini (same Qs, same retrieval)"] --> RAGAS
  end

Loading

ASCII fallback:

BOR1:  Fireworks gpt-oss-20b ─► endpoint_slammer (1 invoke, then 24× async .ainvoke)
BOR2:  cat PDF ─► split(750) ─► Qdrant(:memory:) ◄─ Fireworks Qwen embeddings
                                    │
                                    ▼
              retrieve_information @tool ─► LangGraph agent ─► answer
                                          (fix_tool_calls; helpfulness loop w/ safe limit)
EVAL:  same questions + same retrieval ─► {gpt-oss-20b vs gpt-4.1-mini} ─► RAGAS scores
                                                                        └─► LangSmith tokens/$ 

Why this shape? The session teaches serving: a deployed open-source model is reached exactly like OpenAI (same API surface), so the only real change from prior RAG sessions is the base URL + model id on both the chat and embedding clients. The slammer probes the serving property that matters in production (throughput/latency under concurrency); the RAG app proves you can build on the endpoint; the eval quantifies the open-source-vs-hosted cost/quality trade-off.

Setup & roles

uv sync                 # Python 3.13; langchain-fireworks/-openai, langgraph, qdrant, pymupdf
cp .env.example .env    # fill FIREWORKS_API_KEY (+ OPENAI_API_KEY & LANGSMITH_API_KEY for Activity 1)
# ⚠️ SHUT DOWN any dedicated Fireworks deployment when finished — it bills per GPU-hour.

Component	Role
endpoint_slammer.ipynb	endpoint smoke test: 1 ChatFireworks.invoke + 24 concurrent .ainvoke
app/models.py	get_chat_model() → ChatOpenAI @ Fireworks base URL; fix_tool_calls() shim
app/rag.py	PDF → split 750/0 → Fireworks Qwen embeddings → in-memory Qdrant → retrieve_information
app/tools.py	tool belt: TavilySearch, ArxivQueryRun, retrieve_information
app/graphs/simple_agent.py	tool-calling agent: agent → tools_condition → {action | END}
app/graphs/agent_with_helpfulness.py	adds a Y/N judge loop with a safe message-count limit
langgraph.json	manifest: simple_agent, agent_with_helpfulness
ENDPOINT_SETUP.md	how to deploy serverless vs dedicated gpt-oss-20b on Fireworks

Core concepts

1. Serverless vs dedicated endpoints

Serverless = shared multi-tenant GPUs, pay per token, zero ops, instant — but no capacity guarantee, rate-limited, variable latency. Dedicated = your own GPU replica(s), pay per GPU-hour, guaranteed/consistent latency + tunable (GPU, count, quantization, autoscaling, scale-to-zero) — but you own the lifecycle and must shut it down. Docs: https://docs.fireworks.ai/guides/ondemand-deployments

2. The OpenAI-compatible base-URL swap (the crux)

Any OpenAI-API-compatible server works with the OpenAI clients — you change two things and nothing else. This is why RAG/agents built on OpenAI "just work" on a Fireworks-hosted open-source model. Docs: https://docs.fireworks.ai/tools-sdks/openai-compatibility

ChatOpenAI(model="accounts/fireworks/models/gpt-oss-20b",
           openai_api_key=os.environ["FIREWORKS_API_KEY"],
           openai_api_base="https://api.fireworks.ai/inference/v1")

3. Two ways to reach Fireworks

The notebook uses the native langchain_fireworks.ChatFireworks(model=…); the app uses langchain_openai.ChatOpenAI pointed at the Fireworks base URL. Both hit the same endpoint — the OpenAI-compat path is the reusable pattern.

4. Throughput & latency under load

endpoint_slammer.ipynb fires 24 concurrent requests to observe how the endpoint holds up: latency = TTFT + inter-token (perceived UX); throughput = tokens/sec + requests/sec (how many users before queueing). Tail (p95/p99) matters.

tasks = [send_request(llm, i) for i in range(1, 25)]
await asyncio.gather(*tasks)   # all 24 should return; "Unclosed client session" = benign

5. RAG over the corpus — embeddings + Qdrant

app/rag.py loads the cat PDF, splits token-aware (chunk_size=750, tiktoken as the ruler), embeds with a Fireworks Qwen3 model, and stores in in-memory Qdrant. check_embedding_ctx_length=False disables LangChain's OpenAI tiktoken pre-flight (invalid for a non-OpenAI model); dimensions must match the deployed embedding model. Docs: https://python.langchain.com/docs/integrations/vectorstores/qdrant/

6. The tool-calling agent + the gpt-oss quirk

simple_agent.py is a ReAct loop (agent → tools_condition → {action | END}). gpt-oss sometimes appends a <|call|> control token to tool-call JSON, making it unparseable → LangChain drops it into invalid_tool_calls. fix_tool_calls() strips the token and promotes it back, so the tool actually fires.

response = fix_tool_calls(model.bind_tools(get_tool_belt()).invoke(messages))

7. The helpfulness loop (bounded)

agent_with_helpfulness.py adds a judge node that returns Y/N on the answer; N loops back to agent, Y ends — with a safe limit (len(messages) > 10 → HELPFULNESS:END) so it can't run away. The "agentic RAG with a self-check" pattern.

8. RAGAS — measuring RAG quality

Score generations against retrieved context (and optional ground truth): faithfulness (grounded, not hallucinated), answer_relevancy, context_precision, context_recall (needs ground truth). Docs: https://docs.ragas.io/

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

9. LangSmith — token usage + cost

Turn on tracing (LANGSMITH_TRACING=true + project) and read per-run tokens/cost from the cost dashboard, or capture usage_metadata in code via get_openai_callback(). Compare $/query and $/1k-queries across providers. Docs: https://docs.smith.langchain.com/

---

Questions — reason it through yourself (no answers here)

Q1 — What is the difference between serverless and dedicated endpoints?

Ask yourself:

• When you use the serverless id, whose GPUs are you running on, and what happens to your latency when other people are also hammering that model?
• How are you billed in each case — per token or per GPU-hour? Which one keeps charging you while your app sits idle overnight?
• Deploy a dedicated endpoint (or read ENDPOINT_SETUP.md): what can you tune that you can't on serverless (GPU type, replica count, quantization, scale-to-zero)?
• Why does every README banner scream SHUT IT DOWN — which endpoint type is that warning really about, and why?
• One-liner test: finish "Pick serverless when… / pick dedicated when…".

Q2 — Why consider token throughput and latency for user-facing apps?

Ask yourself:

• Run the 24-way slam. What's the first-token delay vs. how fast text streams after? Which one does a user notice most in a chat UI?
• If your average latency is great but 1-in-20 requests stalls for 10s, is your app "fast"? (Look up p95/p99 — the tail, not the mean.)
• How many simultaneous users can one endpoint serve before requests queue or time out — and how does that connect to your bill and your scaling plan?
• A bigger, smarter model is usually slower and costlier. What are you trading against when you pick an LLM for a user-facing product — and how does that tie back to Q1?

Activities — the deliverable + how to check yourself (no solutions)

endpoint_slammer.ipynb — endpoint test

Do: deploy or pick your gpt-oss endpoint, set model_endpoint, run all cells. Inspect: Did the single .invoke come back coherent (endpoint alive)? Did all 24 async requests print a Response N? Any Request N failed:? (The Unclosed client session lines are harmless — don't chase them.) Note: the README says "replace both model values," but you'll find only one model_endpoint — set that one. And it says the notebook tests "embeddings," but look closely: does it? Where do embeddings actually get used?

RAG app (app/) — point it at your endpoint and run it

Do: wire BOTH the chat and embedding clients to your Fireworks endpoint, build the index, and ask a feline-health question (try main.py's "recommended vaccinations for kittens?"). Inspect: Are you truly on Fireworks, or did something silently fall back to OpenAI? Does the retriever return cat-health chunks? Did the agent actually call the RAG tool, or answer from memory? What happens on a question the corpus can't answer — does it say "I don't know," or hallucinate? If the agent never calls the tool on gpt-oss, look hard at how tool-call output is parsed.

Activity 1 — RAGAS Evaluation with Cost Analysis

Do: evaluate your Fireworks gpt-oss-20b RAG against an OpenAI gpt-4.1-mini equivalent, and instrument both with LangSmith to capture tokens + cost per query. Design questions to get right first:

• What must stay identical between the two runs so the comparison is fair — the questions? the retrieval? the RAGAS judge? (What breaks if you change the retriever per provider?)
• Which RAGAS metrics need ground-truth answers and which don't?
• When you read cost, does your token-counter know Fireworks prices, or only OpenAI's? (Check what get_openai_callback().total_cost reports for a Fireworks run.)
• Build the table: quality per provider and $/query (or $/1k). Then write the takeaway: where does the cheaper open-source endpoint win, where does it cost you quality, and at what scale does the cost gap dominate? A number dump isn't the answer — the trade-off is.

Advanced Activity — Local Models (Ollama, optional)

Do: swap the endpoints for local Ollama models (same base-URL idea, pointed at localhost), rebuild the RAG, and compare quality + latency. Ask yourself: why must you re-embed the corpus when you change embedding models (hint: vector dimensions)? What do you gain locally (cost? privacy? offline?) and what do you give up (VRAM ceiling? latency? ops burden?) vs a managed Fireworks endpoint?