Session 10 · LLM Servers — A Learning Journey

A step-by-step companion to README.md and ENDPOINT_SETUP.md. Walk this alongside the assignment, not instead of it. Each step tells you what you're doing, why it matters, what will bite you, and the lesson to carry forward. The README asks you to answer the questions and build the app — this journey helps you understand what you built.

⚠️ Before anything else, internalize the one warning the README repeats twice: if you deploy a dedicated Fireworks endpoint, it bills per GPU‑hour whether or not you use it. Set auto‑shutdown / scale‑to‑zero, or delete it the moment you finish. The single most expensive mistake in this session is a forgotten GPU.

The one idea behind the whole session

Every prior RAG and agent session called OpenAI. This session swaps the model for an open‑source model you serve yourself (or that Fireworks serves for you) — and the punchline is that almost nothing in your code changes.

An "LLM server" is just an OpenAI‑compatible HTTP endpoint. Keep the same LangChain client, change only two things: the base URL and the model slug (plus the API key). RAG, tools, agents, and evaluation are all provider‑agnostic once you accept that.

That's the mental model. Everything below is a variation on it.

# The crux — this is the entire lesson in five lines:
ChatOpenAI(
    model="accounts/fireworks/models/gpt-oss-20b",         # a fully-qualified slug, not "gpt-4"
    openai_api_key=os.environ["FIREWORKS_API_KEY"],        # Fireworks key, not OpenAI
    openai_api_base="https://api.fireworks.ai/inference/v1" # <-- the whole trick
)

LangChain documents this pattern explicitly: "Many providers offer endpoints compatible with OpenAI's Chat Completions API. You can connect to these using ChatOpenAI with a custom base_url." (See the appendix.) Together AI, vLLM, Ollama, Fireworks — same move every time.

flowchart LR
  App["Your LangChain code<br/>ChatOpenAI / OpenAIEmbeddings<br/><b>(unchanged)</b>"]
  App -->|"set openai_api_base +<br/>model slug + api_key"| SW{{"the ONLY two<br/>things that change"}}
  SW -->|"api.openai.com/v1<br/>gpt-4.1-mini"| OAI[["OpenAI cloud"]]
  SW -->|"api.fireworks.ai/inference/v1<br/>accounts/fireworks/models/gpt-oss-20b"| FW[["Fireworks<br/>serverless OR dedicated"]]
  SW -->|"localhost:11434/v1<br/>gpt-oss:20b"| OLL[["Ollama<br/>on your machine"]]
  classDef srv fill:#e3f2fd,stroke:#1565c0,color:#0d47a1;
  classDef hub fill:#fff3e0,stroke:#e65100,color:#bf360c;
  class OAI,FW,OLL srv;
  class SW hub;

Same client, same RAG, same agent — only the arrow you follow changes. That's the whole session in one picture.

Map of the journey

flowchart TD
  A[Step 0 · Setup<br/>uv sync + .env] --> B[Step 1 · Deploy the endpoint<br/>serverless OR dedicated]
  B --> C[Step 2 · Slam it<br/>endpoint_slammer.ipynb<br/>1 invoke + 24 concurrent]
  C --> D[Step 3 · Build the RAG app<br/>base-URL swap on chat AND embeddings]
  D --> E[Step 4 · The agent<br/>tool-calling + the gpt-oss quirk]
  E --> F[Step 5 · Helpfulness loop<br/>self-check, bounded]
  F --> G[Activity 1 · RAGAS + cost<br/>gpt-oss-20b vs gpt-4.1-mini]
  G --> H[Advanced · Local models<br/>Ollama, same swap]
  H --> I[Shut down your endpoint 🔻]

Breakout Room #1 is Steps 1–2. Breakout Room #2 is Steps 3–5. Activity 1 and the Advanced Build are the graded extensions.

✅ This journey was run end-to-end on Fireworks serverless (Option A). Every step below was executed live against accounts/fireworks/models/gpt-oss-20b: the slammer returned 24/24 with zero failures; main.py's agent called the RAG tool and answered from the corpus; langgraph dev served both graphs; and Activity 1's RAGAS + cost run produced real numbers (see Activity 1). The concrete gotchas flagged here are the ones that actually bit during that run — especially the RAGAS install break in Activity 1.

Step 0 — Setup

cd 10_LLM_Servers
uv sync                 # Python 3.13; langchain-fireworks/-openai, langgraph, qdrant, pymupdf, ragas? (see below)
cp .env.example .env    # fill in your keys

Your .env needs FIREWORKS_API_KEY at minimum, plus TAVILY_API_KEY (the agent has a Tavily tool), and — for Activity 1 — OPENAI_API_KEY and LANGSMITH_API_KEY. There are also two optional model overrides: FIREWORKS_CHAT_MODEL and FIREWORKS_EMBEDDING_MODEL.

Lesson: each session folder is its own isolated uv project. uv sync builds a .venv from this pyproject.toml. Select that interpreter in your editor or the imports will resolve against the wrong environment.
Challenge you'll hit later: ragas is not in pyproject.toml. That's intentional — you'll add it in Activity 1 (uv add ragas, which pulls datasets). Don't treat its absence as a bug.

Step 1 — Deploy the endpoint (serverless vs dedicated)

This is Question #1 made concrete. You have two ways to get a gpt-oss-20b endpoint, and choosing between them is the learning.

Option A — Serverless (the easy path)

Use the shared endpoint directly, no setup:

accounts/fireworks/models/gpt-oss-20b

You're renting a slice of GPUs that everyone else is also using. You pay per token (for gpt-oss-20b: ~$0.07 / 1M input tokens, ~$0.30 / 1M output tokens), you manage nothing, and it's instantly available. The cost: no capacity guarantee, rate limits, and variable latency — cold starts and noisy neighbors mean your speed depends on aggregate demand you don't control.

Option B — Dedicated / on‑demand (the "real production" path)

You rent your own GPU replica(s) via firectl or the web console:

firectl create deployment gpt-oss-20b \
  --model accounts/fireworks/models/gpt-oss-20b \
  --min-replica 1 --max-replica 1

Now you pay per GPU‑hour (an H100 80GB is ~$7/hr; a B200 is ~$10/hr — you pay whether the GPU is busy or idle), and in return you get guaranteed, consistent latency and capacity plus knobs: GPU type, replica count, quantization (FP8), autoscaling, and scale‑to‑zero after N minutes idle (the cost‑control safety net — set it).

The lesson (this is your Q1 answer, in your own words)

Serverless optimizes for zero ops + pay‑only‑for‑what‑you‑use, at the price of predictability. Dedicated optimizes for predictable, controllable latency/throughput, at the price of ops overhead and idle‑capacity cost.

Pick serverless for prototyping, spiky or low volume, or when variable latency is acceptable. Pick dedicated for sustained high volume, strict latency SLAs, or when you need specific hardware/quantization. Dedicated becomes cheaper only past a break‑even utilization — an idle dedicated GPU is far more expensive than serverless.

Challenges & traps

❌ "Serverless is free." No — it's pay‑per‑token. It just has no idle cost.
❌ "Dedicated is always cheaper." Only at sustained high utilization. Idle GPU‑hours can dwarf a serverless bill.
❌ Confusing dedicated with a local model. Dedicated is still Fireworks‑managed cloud GPU. Running the model on your own laptop is the Advanced Activity, a different axis entirely.
🔻 The forgotten deployment. If you chose Option B, this is where you write yourself a reminder to shut it down.

Step 2 — Slam the endpoint (`endpoint_slammer.ipynb`)

This notebook is Question #2 made concrete. You'll do exactly two things:

One request to prove the endpoint is alive:

from langchain_fireworks import ChatFireworks
model_endpoint = "accounts/fireworks/models/gpt-oss-20b"  # or your dedicated id
llm = ChatFireworks(model=model_endpoint)
print(llm.invoke("How much wood could a woodchuck chuck...?").content)

24 concurrent requests to see how it holds up under load:

async def main():
    tasks = [send_request(llm, i) for i in range(1, 25)]
    await asyncio.gather(*tasks)
await main()

Why fire 24 at once?

A single .invoke measures baseline latency. Firing 24 .ainvoke calls through asyncio.gather probes throughput and concurrency — the properties that decide whether a user‑facing app survives real traffic.

Latency = perceived responsiveness. It's really two numbers: time‑to‑first‑ token (TTFT) — what streaming UIs optimize so the answer feels alive — and inter‑token latency — how fast text streams after that. What matters isn't the average but the tail (p95/p99): an endpoint that's usually fast but occasionally stalls 10s still loses users.
Throughput = tokens/sec and, critically, requests/sec under concurrency. It decides how many simultaneous users you serve before requests queue and time out, and it drives cost‑per‑request.

This is the whole point of choosing a server carefully: a dedicated deployment buys predictable numbers under load; a serverless endpoint's numbers drift with shared demand and rate limits. So Q1 and Q2 are the same decision viewed from two angles — the serverless‑vs‑dedicated choice is a throughput/latency choice.

Challenges & traps

🟡 Unclosed client session / aiohttp warnings are benign. They're cleanup noise from the async HTTP client, not errors. They appear even on a fully successful run. Don't chase them.
🟡 The README says "replace both model values" — there's only one. The notebook uses a single model_endpoint variable, reused by both the single invoke and the async slam. Set it once. (Leaving the default serverless id is a legitimate choice — Option A is allowed.)
🟡 The README says this cell tests "Endpoint and Embeddings" — it only tests chat. Embeddings are exercised downstream in the RAG app (app/rag.py), not here. Don't go looking for an embeddings cell.
🟢 Non‑deterministic order is expected. With gather, Response 7 may print before Response 2. That's concurrency working, not a bug.

Step 3 — Build the RAG app: the base‑URL swap, twice

Open the provided app/ package — it is a working reference. The insight of Step 3 is that you apply the base‑URL swap to both clients: the chat model and the embeddings model. Miss either and you've silently fallen back to OpenAI.

Chat model (app/models.py):

ChatOpenAI(
    model=os.environ.get("FIREWORKS_CHAT_MODEL", "accounts/fireworks/models/gpt-oss-20b"),
    openai_api_key=os.environ["FIREWORKS_API_KEY"],
    openai_api_base="https://api.fireworks.ai/inference/v1",
)

Embeddings (app/rag.py) — note the two extra arguments:

OpenAIEmbeddings(
    model=os.environ.get("FIREWORKS_EMBEDDING_MODEL", "accounts/fireworks/models/qwen3-embedding-8b"),
    openai_api_key=os.environ["FIREWORKS_API_KEY"],
    openai_api_base="https://api.fireworks.ai/inference/v1",
    check_embedding_ctx_length=False,   # <-- why this matters, below
    dimensions=4096,                    # <-- must match the deployed model
)

The rest of the pipeline is exactly what you built in earlier sessions: DirectoryLoader + PyMuPDFLoader loads data/cat-health-guide.pdf (the 2021 AAHA/AAFP Feline Life Stage Guidelines) → RecursiveCharacterTextSplitter with chunk_size=750, using tiktoken as a length ruler (measuring chunks in tokens, not characters — the gpt-4o encoding here is a ruler only, it makes no OpenAI call) → an in‑memory Qdrant vector store → a two‑node retrieve → generate subgraph exposed as the @tool retrieve_information.

Two subtle-but-critical arguments

check_embedding_ctx_length=False disables LangChain's OpenAI‑specific tiktoken pre‑flight. That pre‑flight assumes an OpenAI tokenizer; against a Qwen embedding model it's invalid and breaks the call. Turning it off is required whenever you point OpenAIEmbeddings at a non‑OpenAI server.
dimensions=4096 pins the vector width. Qdrant fixes a collection's dimensionality at creation time — every vector you later query with must match. This sets up the single biggest trap of the session ↓.

🔴 The #1 config trap: the 4B / 8B embedding mismatch

There is a genuine inconsistency baked into the materials, and it's the most common cause of "my RAG returns garbage / errors":

Source	Embedding model id
`ENDPOINT_SETUP.md`	`accounts/fireworks/models/qwen3-embedding-4b`
`app/rag.py` default	`accounts/fireworks/models/qwen3-embedding-8b` + `dimensions=4096`

If you provision the 4B model per the setup doc but leave the code at 8B (or vice versa), you get a model‑not‑found error, a dimension mismatch, or a silently wrong vector store. The lesson: the embedding model id, the dimensions value, and the Qdrant collection must be consistent end‑to‑end. Pick one — 4B or 8B — and make everything agree. The clean fix is to set FIREWORKS_EMBEDDING_MODEL in your .env to exactly what you deployed and match dimensions to that model's width.

Challenges & traps

🔴 Silent OpenAI fallback. If you forget openai_api_base on either client, LangChain quietly uses the default OpenAI URL and your OPENAI_API_KEY — and now you're not using the open‑source endpoint at all. Verify both clients point at Fireworks.
🔴 Embedding 4B/8B/dimensions mismatch (above) — the #1 retrieval killer.
🟢 Grounding guardrail. The generation prompt says "Only use the provided context… else respond 'I don't know'." That constraint is what keeps the model faithful — and it's exactly what RAGAS will measure in Activity 1.

Step 4 — The agent, and the gpt‑oss tool‑call quirk

app/graphs/simple_agent.py is a standard LangGraph tool‑calling loop: agent → tools_condition → {action | END}, with action → agent. The tool belt is [TavilySearch, ArxivQueryRun, retrieve_information]. Nothing new — until you actually run it on gpt-oss-20b and the agent never calls the RAG tool.

The challenge (and it's a real one, not boilerplate)

gpt-oss models are trained on OpenAI's Harmony response format, in which a tool call is emitted on a commentary channel and terminated by a special control token: <|call|>. When Fireworks serves the model through the OpenAI‑compatible shim, that terminator sometimes leaks into the tool‑call JSON arguments, making them unparseable. LangChain can't parse the args, so it drops the call into invalid_tool_calls — and tools_condition sees no valid tool call and routes straight to END. The agent silently answers from parametric memory and never touches your RAG.

The fix (app/models.py::fix_tool_calls) strips the trailing token, re‑parses, and promotes the call back to tool_calls:

cleaned = re.sub(r"\s*<\|call\|>\s*$", "", tc["args"])
parsed = json.loads(cleaned)          # now valid → promote back into tool_calls

Every model node wraps its response in this shim: response = fix_tool_calls(model.invoke(messages)).

sequenceDiagram
  participant A as Agent node
  participant M as gpt-oss-20b @ Fireworks
  participant F as fix_tool_calls()
  participant T as tools_condition
  A->>M: invoke(messages) with bound tools
  M-->>A: tool-call JSON + trailing CALL_TOKEN token
  Note over A: args unparseable →<br/>dropped into invalid_tool_calls
  A->>F: fix_tool_calls(response)
  F->>F: strip CALL_TOKEN, json.loads,<br/>promote back to tool_calls
  F-->>A: valid tool_calls
  A->>T: route on last message
  alt tool_calls present
    T-->>A: action → RAG fires
  else none
    T-->>A: END → agent skips RAG
  end

🔬 What actually happened on this run. Running python main.py against Fireworks serverless, gpt-oss-20b returned a valid tool call — the <|call|> token did not leak, so fix_tool_calls was a no‑op that pass‑through. That's exactly why the code guards on if not response.invalid_tool_calls: return response. The leak is intermittent (note the "sometimes" above): it depends on the model build and the serving layer's current behavior. Keep the shim — it costs nothing when clean and saves you when it isn't. Lesson within the lesson: defensive integration code should be a no‑op on the happy path, not a transform you have to reason about every call.

The lesson: "OpenAI‑compatible" is compatible at the API surface, not always at the content level. Open models have their own output conventions (Harmony's channels and control tokens here), and the serving layer's translation can be leaky. When an open model "doesn't call tools," suspect the tool‑call parsing, not the model's intelligence. This is the kind of integration wrinkle you only learn by serving open models yourself.

Step 5 — The helpfulness loop (agentic self‑check, bounded)

app/graphs/agent_with_helpfulness.py adds one idea: after the agent answers, a judge node asks the model "is this response extremely helpful? Y/N." Y ends the run; N loops back to agent to try again.

The important detail is the guardrail:

if len(state["messages"]) > 10:
    return {"messages": [AIMessage(content="HELPFULNESS:END")]}

Lesson: any self‑correcting loop needs a hard stop, or a stubborn "N" judge spins forever (burning tokens — and on a dedicated endpoint, GPU‑hours). A message‑count ceiling is the simplest safe bound. This is the "agentic RAG with a self‑check" pattern with the safety rail attached.

Both graphs are registered in langgraph.json (simple_agent, agent_with_helpfulness); run them with python main.py or langgraph dev. main.py asks "What are the recommended vaccinations for kittens?" — a grounded answer (kitten core vaccines: FVRCP + rabies) that comes back by calling the RAG tool is your proof the whole pipeline works end‑to‑end.

Activity 1 — RAGAS evaluation with cost analysis

Now you quantify the open‑source‑vs‑hosted trade‑off on two axes: quality (RAGAS) and cost (LangSmith). First, uv add ragas.

🔴 First real snag: RAGAS won't import on the LangChain v1 stack

uv add ragas installs the latest (ragas==0.4.3), and then import ragas crashes before you write a single line of eval:

ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai'

This is not your bug — it's a known ragas issue. ragas/llms/base.py eagerly imports ChatVertexAI (and VertexAI) from a langchain-community path that the v1 line this repo pins (langchain-community 0.4.x) deleted — those classes moved to langchain-google-vertexai. Vertex is never actually used here; the import just runs at module load and dies. Two clean fixes:

Shim it (no downgrade, recommended): register a stub module before importing ragas, so the dead import resolves to a harmless placeholder:

import sys, types
_vx = types.ModuleType("langchain_community.chat_models.vertexai")
_vx.ChatVertexAI = type("ChatVertexAI", (), {})       # never used
sys.modules["langchain_community.chat_models.vertexai"] = _vx
import langchain_community.llms as _llms
if not hasattr(_llms, "VertexAI"): _llms.VertexAI = type("VertexAI", (), {})
import ragas   # now imports cleanly

Pin it: uv add "ragas==0.3.9" (predates the eager Vertex import). Simpler, but 0.3.x predates LangChain v1 and can surface other conflicts in this stack — the shim keeps you on the current release.

Lesson: "add the library" is rarely the whole story at the bleeding edge. A fast-moving ecosystem (LangChain v1 splitting integrations into standalone packages) breaks downstream libraries that hard-code old import paths. Reading the traceback — which package, which removed symbol — is the skill; the fix follows from it.

The experimental discipline that makes the comparison valid

Evaluate your Fireworks gpt-oss-20b RAG against an OpenAI gpt-4.1-mini equivalent on the same questions with the same retrieval — hold the retriever constant and swap only the generator. If you change the retriever too, you've confounded the experiment and can't attribute any difference to the model.

fireworks_llm = ChatOpenAI(model="accounts/fireworks/models/gpt-oss-20b",
                           openai_api_base="https://api.fireworks.ai/inference/v1",
                           openai_api_key=os.environ["FIREWORKS_API_KEY"], temperature=0)
openai_llm    = ChatOpenAI(model="gpt-4.1-mini", temperature=0)   # default OpenAI base URL + OPENAI_API_KEY
# ... same retriever, same eval questions, only `generator_llm` changes.

flowchart TB
  Q["Same eval questions"] --> R["Same retriever<br/>Fireworks qwen3-embedding-8b<br/>(held constant)"]
  R --> CTX["Same retrieved context"]
  CTX --> G1["Generator A<br/>gpt-oss-20b"]
  CTX --> G2["Generator B<br/>gpt-4.1-mini"]
  G1 --> J["Identical RAGAS judge<br/>gpt-4.1-mini + same embeddings"]
  G2 --> J
  J --> OUT["faithfulness &amp; answer_relevancy MOVE<br/>context_precision/recall stay EQUAL<br/>cost/query differs ~3.5x"]
  classDef vary fill:#fff3e0,stroke:#e65100,color:#bf360c;
  classDef hold fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20;
  class G1,G2 vary;
  class Q,R,CTX,J hold;

The four RAGAS metrics (and what each tells you)

faithfulness — is the answer grounded in the retrieved context, or hallucinated? (This is what the "only use the context" prompt protects.)
answer_relevancy — does the answer actually address the question?
context_precision — are the relevant chunks ranked highly?
context_recall — did retrieval pull in enough of the ground truth? (Requires reference answers; the other three don't. Omitting ground truth → report 3 of 4, and say so.)

Because retrieval is shared, context_precision/context_recall should come out near‑identical across the two providers (a good sanity check) — the generator swap should move faithfulness/answer_relevancy and cost.

Judge discipline: the RAGAS judge LLM and its embeddings must be the same for both providers, or the scores aren't comparable.

🔴 The cost trap: Fireworks shows `$0.00`

You'll instrument tokens/cost via LangSmith (LANGSMITH_TRACING=true + LANGSMITH_PROJECT; read the cost dashboard) or in code via get_openai_callback() / usage_metadata. The trap: get_openai_callback().total_cost only knows OpenAI's prices. For Fireworks it captures the token counts correctly but reports $0.00 — because it has no Fireworks price table. You must price Fireworks manually (tokens × Fireworks' published gpt-oss-20b rate) or configure a custom price in LangSmith. A cost table that reports Fireworks at $0.00 is the tell that someone fell into this.

The takeaway you're actually after

Don't dump numbers — state the trade‑off. Here are the actual measured results from running this exact setup on Fireworks serverless (5 feline-health questions, shared retriever, identical gpt-4.1-mini judge — your numbers will vary):

Provider (generator)	faithfulness	answer_relevancy	context_precision	context_recall	tok/q	$/1k queries
Fireworks `gpt-oss-20b`	0.704	0.743	0.833	1.00	2,475	$0.27
OpenAI `gpt-4.1-mini`	0.856	0.744	0.833	1.00	2,077	$0.94

Read the table the way the experiment was designed: context_precision and context_recall are identical across rows (0.833 / 1.00) — proof the shared retriever worked and the comparison is clean. The generator swap moved faithfulness (0.856 vs 0.704 — the hosted model stayed closer to the context) while answer_relevancy was a wash. And the cost line is the headline: OpenAI cost ~3.5× per query ($0.94 vs $0.27 per 1k). Fireworks $/1k was priced manually ($0.07/$0.30 per 1M in/out) because get_openai_callback reported $0.00 for it — the trap above, live.

The open‑source endpoint is dramatically cheaper per token but traded ~15 points of faithfulness here. At high volume the ~3.5× per‑query cost gap can dwarf that quality gap — so the right answer is "it depends on your SLA and scale," and being able to say when each provider wins is the point.

Advanced Activity — go fully local with Ollama

The reward for understanding the crux: going fully local is the same one‑line swap, pointed at Ollama's OpenAI‑compatible server.

ChatOpenAI(model="gpt-oss:20b",           openai_api_base="http://localhost:11434/v1", openai_api_key="ollama")
OpenAIEmbeddings(model="nomic-embed-text", openai_api_base="http://localhost:11434/v1",
                 openai_api_key="ollama", check_embedding_ctx_length=False, dimensions=768)  # note: 768, not 4096

🔴 You must re‑embed the corpus. You can't mix embedding models in one Qdrant collection, and nomic-embed-text is 768‑dim, not 4096. Rebuild the vector store with the local embedder or retrieval breaks.
Ollama ignores the API key — pass any placeholder.
The reflection (this is the deliverable): weigh local vs managed. Local = zero per‑token cost, data stays on your machine, works offline — but you hit VRAM ceilings, cold‑start/throughput limits, and you own all the ops. Managed Fireworks = guaranteed capacity and effortless scaling — but per‑token or per‑GPU‑hour cost and your data leaves the building.

🔻 Close the loop: shut it down

If you deployed a dedicated endpoint, delete it or confirm scale‑to‑zero kicked in now. Serverless has no idle cost, so nothing to do there. This is the last step of every dedicated‑endpoint journey — make it a habit.

firectl list deployments
firectl delete deployment <DEPLOYMENT_ID>

The journey in one table: challenges → lessons

Where	The challenge	The lesson
Step 1	Serverless vs dedicated isn't obvious	Billing model (per‑token vs per‑GPU‑hour) + capacity guarantee are the real axes; idle dedicated GPUs are expensive
Step 2	`Unclosed client session` warnings	Benign aiohttp cleanup noise — ignore
Step 2	"Replace both model values" / "tests embeddings"	README quirks — one `model_endpoint`; chat‑only test
Step 2	Why 24 concurrent requests?	Latency (TTFT, tail p95/p99) = UX; throughput = concurrency + cost
Step 3	Silent fallback to OpenAI	The base‑URL swap must be on both chat and embeddings
Step 3	RAG returns garbage/errors	4B/8B + `dimensions` mismatch — keep id + dims + Qdrant collection consistent
Step 3	Embeddings call fails	`check_embedding_ctx_length=False` is required for non‑OpenAI embedders
Step 4	Agent never calls the RAG tool	gpt‑oss can leak the Harmony `<\|call\|>` token → `fix_tool_calls` strips it (defensive no‑op when clean — it didn't fire on this serverless run)
Activity 1	`import ragas` crashes on a fresh install	ragas 0.4.3 hard‑imports removed `langchain_community…vertexai` → shim it or pin `ragas==0.3.9`
Step 5	Self‑check loops forever	Bound every agentic loop (message‑count ceiling)
Activity 1	Confounded comparison	Same questions + same retrieval; swap only the generator; identical judge
Activity 1	Fireworks cost shows `$0.00`	`get_openai_callback` only prices OpenAI — price Fireworks manually
Advanced	Local retrieval breaks	Re‑embed; `dimensions` must match the local embedder (768 for nomic)
Everywhere	The forgotten GPU	Shut down dedicated endpoints

The through‑line: you spent nine sessions learning RAG and agents against OpenAI. This session proves that knowledge is portable — an LLM server is an OpenAI‑compatible HTTP endpoint, and the same pipeline runs on a model you (or Fireworks, or Ollama) serve. The only genuinely new things you had to learn were serving properties (throughput/latency, serverless vs dedicated), one open‑ model integration wrinkle (the Harmony tool‑call token), and how to quantify the cost/quality trade‑off you're now free to make.

Zoom out: where Session 10 sits in the LLM app stack

Everything you built maps cleanly onto a16z's Emerging Architectures for LLM Applications — the industry reference for how these systems are assembled. Session 10 isn't a toy; it is a complete, if minimal, instance of the canonical stack. The diagram places each component you touched into its a16z layer.

flowchart LR
  UQ(["User query"]) --> ORCH
  subgraph DATA["Contextual data and ingestion"]
    direction TB
    D["cat-health-guide.pdf"] --> PIPE["1 Data pipeline<br/>DirectoryLoader / PyMuPDFLoader<br/>split 750 tok"]
    PIPE --> EMB["2 Embeddings<br/>Fireworks Qwen3-8b"]
    EMB --> VDB[("3 Vector DB<br/>Qdrant in-memory")]
  end
  VDB -->|"top-k context"| ORCH
  subgraph ORCHL["4 Orchestration"]
    direction TB
    ORCH["LangChain + LangGraph<br/>tool-calling agent"]
    PLUG["6 APIs / Plugins<br/>Tavily / Arxiv"]
    ORCH <--> PLUG
  end
  ORCH -->|"prompt + context"| GEN
  subgraph LLMS["10-11 LLM APIs"]
    direction TB
    GEN["OPEN: Fireworks gpt-oss-20b<br/>serverless / dedicated"]
    BASE["PROPRIETARY: OpenAI gpt-4.1-mini<br/>eval baseline"]
  end
  GEN --> OUT(["Grounded answer"])
  GEN -.->|"RAGAS compares"| BASE
  subgraph OPS["Cross-cutting platform layers"]
    direction LR
    VAL["9 Validation<br/>RAGAS / grounding prompt / fix_tool_calls"]
    OBS["8 LLMOps<br/>LangSmith"]
    CACHE["7 LLM cache<br/>lru_cache RAG graph"]
    HOST["12 App hosting<br/>langgraph dev"]
    CLOUD["13 Cloud/compute<br/>Fireworks GPU / Ollama"]
  end
  ORCHL -.runs on.-> OPS
  classDef fw fill:#e3f2fd,stroke:#1565c0,color:#0d47a1;
  classDef oa fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c;
  classDef ops fill:#f1f8e9,stroke:#558b2f,color:#33691e;
  class EMB,GEN fw;
  class BASE oa;
  class VAL,OBS,CACHE,HOST,CLOUD ops;

Table 1 — The 14 a16z layers → what Session 10 uses

#	a16z layer	Canonical tools (a16z)	Session 10 instantiation
1	Data Pipelines	Databricks, Airflow, Unstructured, Airbyte	`DirectoryLoader` + `PyMuPDFLoader` → `RecursiveCharacterTextSplitter` (750 tok, tiktoken ruler)
2	Embedding Models	OpenAI Ada, Cohere, Sentence Transformers	Fireworks `qwen3-embedding-8b` via `OpenAIEmbeddings` + base‑URL swap (`dimensions=4096`)
3	Vector Databases	Pinecone, Weaviate, Chroma, Qdrant, pgvector	Qdrant (`location=":memory:"`)
4	Orchestrators	LangChain, LlamaIndex, Autogen, Haystack	LangChain + LangGraph tool‑calling agent (`simple_agent`, `agent_with_helpfulness`)
5	Playgrounds	OpenAI Playground, Humanloop, Parea	Fireworks model playground / `endpoint_slammer.ipynb` as a smoke test
6	APIs / Plugins	SerpAPI, Wolfram, Zapier	`TavilySearch`, `ArxivQueryRun` (+ the RAG `retrieve_information` tool)
7	LLM Caches	Redis, SQLite, GPTCache	`@lru_cache` on the compiled RAG graph (build‑once)
8	Logging / LLMOps / Eval	W&B, Arize, Helicone, PromptLayer, promptfoo	LangSmith tracing + cost dashboard (`LANGSMITH_TRACING`)
9	Validators	Guardrails, Rebuff, Guidance, Outlines	RAGAS metrics + the "only use the context / I don't know" grounding prompt + `fix_tool_calls`
10	LLM APIs (proprietary)	OpenAI, Anthropic, Cohere	OpenAI `gpt-4.1-mini` (Activity 1 eval baseline)
11	LLM APIs (open source)	Hugging Face, Replicate, Ollama, GPT4All	Fireworks `gpt-oss-20b` (the star) · Ollama (Advanced Activity)
12	App Hosting	Vercel, Netlify, Streamlit, Modal	LangGraph Server (`langgraph dev`, `langgraph.json`)
13	Cloud Providers	AWS, GCP, Azure, CoreWeave	Fireworks GPU cloud (dedicated H100/B200) · local machine (Ollama)
14	Opinionated Clouds	Databricks, Anyscale, Modal, Runpod	Fireworks is the opinionated inference cloud here (serverless + on‑demand)

What this session adds to the reference picture: the a16z diagram treats the LLM API as a single box. Session 10's whole lesson lives inside layers 10–14 — it splits "the LLM API" into open vs proprietary (10 vs 11) and, for the open model, how it's served (serverless vs dedicated, layers 13–14). The base‑URL swap is what makes layers 2, 10, and 11 hot‑swappable without touching layers 1, 3, 4, or 9.

Table 2 — The serving layer (10–14), decomposed (this session's real subject)

Serving option	a16z layer	Billing	Latency/capacity	You manage	Use when
Fireworks serverless `gpt-oss-20b`	11 open + 14 opinionated cloud	per token (~$0.07/$0.30 per 1M in/out)	shared, variable, rate‑limited	nothing	prototyping, spiky/low volume
Fireworks dedicated (on‑demand)	11 open + 13/14 cloud	per GPU‑hour (H100 ~$7/hr, B200 ~$10/hr)	your replica, guaranteed & consistent	lifecycle (shut it down)	sustained volume, strict SLA, HW/quant control
OpenAI `gpt-4.1-mini`	10 proprietary	per token (higher)	managed, consistent	nothing	max quality baseline; the eval comparator
Ollama `gpt-oss:20b`	11 open + 13 (local)	$0 per token	your hardware's ceiling	everything (VRAM, ops)	offline, private, dev — Advanced Activity

The point of the session, in stack terms: you can slide across the entire row of Table 2 without rewriting layers 1–9 — that is what "an LLM server is just an OpenAI‑compatible HTTP endpoint" buys you.

Appendix — References (grounded)

Fireworks AI

Pricing (serverless per‑token, on‑demand per‑GPU‑hour): https://fireworks.ai/pricing
On‑demand / dedicated deployments guide: https://docs.fireworks.ai/guides/ondemand-deployments
OpenAI compatibility: https://docs.fireworks.ai/tools-sdks/openai-compatibility
Models catalog: https://fireworks.ai/models
API keys: https://fireworks.ai/api-keys

LangChain — the OpenAI‑compatible base‑URL pattern (the crux)

OpenAI‑compatible endpoints (Python): https://docs.langchain.com/oss/python/concepts/providers-and-models#openai-compatible-endpoints
Custom base URL / proxy settings: https://docs.langchain.com/oss/python/langchain/models#base-url-and-proxy-settings
Chat Completions API compatibility (note: non‑standard fields not preserved): https://docs.langchain.com/oss/python/integrations/chat/index#chat-completions-api
Fireworks provider (ChatFireworks, FireworksEmbeddings): https://docs.langchain.com/oss/python/integrations/providers/fireworks
Qdrant vector store: https://python.langchain.com/docs/integrations/vectorstores/qdrant/

gpt‑oss & the Harmony response format (why fix_tool_calls exists)

OpenAI Harmony response format (channels, control tokens, tool calls): https://developers.openai.com/cookbook/articles/openai-harmony
Harmony renderer / spec: https://github.com/openai/harmony
openai/gpt-oss-20b model card: https://huggingface.co/openai/gpt-oss-20b

RAGAS — RAG evaluation

Docs home: https://docs.ragas.io/
Available metrics (faithfulness, answer_relevancy, context_precision, context_recall): https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/
Vertex import break on LangChain v1 (the install gotcha): vibrantlabsai/ragas#2745

LLM app stack (a16z reference architecture)

Emerging Architectures for LLM Applications (the diagram this session maps onto): https://a16z.com/emerging-architectures-for-llm-applications/
a16z-infra/llm-app-stack (curated tool list per layer): https://github.com/a16z-infra/llm-app-stack

LangSmith — token usage & cost

Cost tracking (auto token/cost capture + custom prices): https://docs.langchain.com/langsmith/cost-tracking
Observability concepts (projects, traces, runs): https://docs.langchain.com/langsmith/observability-concepts
Tracing OpenAI‑compatible providers: https://docs.langchain.com/langsmith/trace-with-openai-compatible

Local models

Ollama (OpenAI‑compatible server at http://localhost:11434/v1): https://ollama.com/

Companion to README.md, ENDPOINT_SETUP.md, endpoint_slammer.ipynb, and the app/ package in this folder. Reference material grounded via the LangChain docs MCP, Fireworks/RAGAS/Harmony documentation, and the Session 10 instructor answer key & cheat sheet.

donbr/llm-servers-learning-journey.md

Select an option

No results found

Select an option

No results found

Session 10 · LLM Servers — A Learning Journey

The one idea behind the whole session

Map of the journey

Step 0 — Setup

Step 1 — Deploy the endpoint (serverless vs dedicated)

Option A — Serverless (the easy path)

Option B — Dedicated / on‑demand (the "real production" path)

The lesson (this is your Q1 answer, in your own words)

Challenges & traps

Step 2 — Slam the endpoint (`endpoint_slammer.ipynb`)

Why fire 24 at once?

Challenges & traps

Step 3 — Build the RAG app: the base‑URL swap, twice

Two subtle-but-critical arguments

🔴 The #1 config trap: the 4B / 8B embedding mismatch

Challenges & traps

Step 4 — The agent, and the gpt‑oss tool‑call quirk

The challenge (and it's a real one, not boilerplate)

Step 5 — The helpfulness loop (agentic self‑check, bounded)

Activity 1 — RAGAS evaluation with cost analysis

🔴 First real snag: RAGAS won't import on the LangChain v1 stack

The experimental discipline that makes the comparison valid

The four RAGAS metrics (and what each tells you)

🔴 The cost trap: Fireworks shows `$0.00`

The takeaway you're actually after

Advanced Activity — go fully local with Ollama

🔻 Close the loop: shut it down

The journey in one table: challenges → lessons

Zoom out: where Session 10 sits in the LLM app stack

Table 1 — The 14 a16z layers → what Session 10 uses

Table 2 — The serving layer (10–14), decomposed (this session's real subject)

Appendix — References (grounded)

donbr/llm-servers-learning-journey.md

Session 10 · LLM Servers — A Learning Journey

The one idea behind the whole session

Map of the journey

Step 0 — Setup

Step 1 — Deploy the endpoint (serverless vs dedicated)

Option A — Serverless (the easy path)

Option B — Dedicated / on‑demand (the "real production" path)

The lesson (this is your Q1 answer, in your own words)

Challenges & traps

Step 2 — Slam the endpoint (endpoint_slammer.ipynb)

Why fire 24 at once?

Challenges & traps

Step 3 — Build the RAG app: the base‑URL swap, twice

Two subtle-but-critical arguments

🔴 The #1 config trap: the 4B / 8B embedding mismatch

Challenges & traps

Step 4 — The agent, and the gpt‑oss tool‑call quirk

The challenge (and it's a real one, not boilerplate)

Step 5 — The helpfulness loop (agentic self‑check, bounded)

Activity 1 — RAGAS evaluation with cost analysis

🔴 First real snag: RAGAS won't import on the LangChain v1 stack

The experimental discipline that makes the comparison valid

The four RAGAS metrics (and what each tells you)

🔴 The cost trap: Fireworks shows $0.00

The takeaway you're actually after

Advanced Activity — go fully local with Ollama

🔻 Close the loop: shut it down

The journey in one table: challenges → lessons

Zoom out: where Session 10 sits in the LLM app stack

Table 1 — The 14 a16z layers → what Session 10 uses

Table 2 — The serving layer (10–14), decomposed (this session's real subject)

Appendix — References (grounded)

Step 2 — Slam the endpoint (`endpoint_slammer.ipynb`)

🔴 The cost trap: Fireworks shows `$0.00`