This post covers the core concepts of AI engineering — not machine learning research, but the practical discipline of taking pre-built models and shipping them into real products. By the end, you'll have a working mental model of LLMs, RAG, MCP, agents, fine-tuning, and quantization, and understand how they fit together.
These two disciplines are often conflated, but they're distinct.
Machine learning is about building models. An ML engineer writes Python (or another language), designs algorithms, selects architectures, feeds training data, and produces a trained model. The output is typically two artifacts: the model code (model.py) and a weights file (parameters.bin).
AI engineering is about using those models. An AI engineer takes a foundation model created by ML engineers and integrates it into a product — handling real-time data retrieval, connecting to external services, managing latency and cost, and deploying it so users can actually benefit from it. Any developer — backend, Android, DevOps — can do AI engineering. The constraint is not ML expertise; it's systems thinking.
The distinction matters because the skills, costs, and challenges are completely different. Creating a large language model costs tens or hundreds of millions of dollars and requires massive GPU infrastructure. Using one well requires engineering judgment about system design.
A language model is a model that understands text — its grammar, structure, and meaning — and can predict or generate more text. Before large language models existed, we already had language models, but two things changed with the "large" era:
-
Training data scale. Modern LLMs are trained on essentially everything on the internet — a dataset so vast it encodes general knowledge about language, facts, reasoning patterns, and more.
-
Parameter count. After training, the model stores what it learned as a file of numbers — the weights (also called parameters). These numbers are the gold. When a company open-sources a model, they're releasing two things: the model code and the weights file. The weights represent billions of parameters — numbers that encode everything the model learned.
To understand weights, consider a real estate pricing model. You have historical deals: number of bedrooms, square footage, balcony presence, and price. From that data, you can derive a formula:
price = W1 × bedrooms + W2 × sq_ft + W3 × balcony
The values of W1, W2, W3 are determined by the training process. Feed enough data through the model, and it converges on coefficients that fit the data. Once training is complete, you save those numbers to a file. For any new property, you load the weights and run the formula.
An LLM works the same way, but instead of 3 weights, it has billions — W1 through W100,000,000,000 or beyond. The parameters.bin file stores all of them. This is the artifact that costs $100M+ to produce and is why model weights are so valuable.
Each parameter is a floating-point number. At 32-bit precision (FP32), each number takes 4 bytes. With 100 billion parameters:
100,000,000,000 × 4 bytes = 400 GB
Drop to 16-bit precision (FP16) and it halves to 200 GB. The tradeoff is that lower precision means slightly less accuracy, but also faster computation and lower memory requirements. This is the basis of quantization, covered below.
This is widely misunderstood. An LLM has no network connection. It's two files in a directory: model.py and parameters.bin. Once trained, it needs nothing else. You can download an open-source model, disconnect from the internet, and run it indefinitely.
Internally, when you send text to an LLM:
- Tokenization: The text is converted to numbers using a mapping file (e.g., "he" → 12, "is" → 7, "good" → 42).
- Embeddings: Those token IDs are converted into higher-dimensional vectors that the model can operate on mathematically.
- Forward pass: The model performs matrix operations to produce output vectors.
- Decoding: The output vectors are converted back through the mapping to produce tokens, which are assembled into text.
The model predicts tokens one at a time, always offline, always from its frozen weights.
This leads to a critical consequence: an LLM cannot make a network call. It cannot fetch live data. It cannot call an API. It can only transform the text it receives. A capable model might recognize that a question requires real-time data and say so, but it cannot retrieve that data itself.
Suppose you train a model in 2020 and ask it the current Bitcoin price. It can only give you what it learned during training — a stale number, or a refusal if it's smart enough to recognize the question requires live data.
You cannot solve this by retraining the model. Retraining is prohibitively expensive, and Bitcoin's price changes faster than any training cycle. The model needs to remain as-is (it's a brilliant language engine) while the surrounding system handles real-time information retrieval.
This is where RAG comes in.
RAG is a pattern for extending an LLM's knowledge with external data at inference time. The name describes the three steps:
- Retrieval: Fetch relevant information from an external source — an API, a database, a document, the web.
- Augmentation: Inject that retrieved information into the prompt sent to the LLM.
- Generation: The LLM generates a response grounded in both its language knowledge and the retrieved facts.
User asks: "What is the current Bitcoin price?"
The backend:
- Calls a crypto API → receives
65000 USD - Sends to the LLM: "The user asked for the current Bitcoin price. The crypto API returned: 65000 USD. Write a clear response for the user."
- LLM responds: "The current Bitcoin price is $65,000 USD, based on live market data."
The LLM contributes sentence structure, tone, and natural language fluency. The external API contributes the fact. Neither could do this alone.
RAG applies to any external source: databases, PDFs, search results, internal APIs. The pattern is always the same: retrieve, inject, generate.
As you add more RAG-based capabilities, each new data source requires new logic on the backend. Crypto prices → crypto API call. Top blog results → Google API call. YouTube video summaries → YouTube API call. PDF content → PDF parser call.
For each, the backend must:
- Recognize the intent
- Route to the right data source
- Handle the response
- Package it for the LLM
This grows into an unsustainable tangle of if-else chains. The backend is doing the cognitive work of deciding what to do — work that the LLM, with its language understanding, could do far better. The backend is becoming the brain. It shouldn't be.
MCP addresses this architectural problem by moving the routing decision from the backend to the LLM.
HTTP is the standard protocol for communication between web services. MCP is the analogous standard for connecting AI applications to external systems. When an AI system needs to interact with the world, MCP provides the interface.
An MCP server exposes tools — named functions the LLM can request. Each tool comes with machine-readable metadata:
{
"name": "fetch_crypto_price",
"description": "Fetches the current price of a cryptocurrency from live market data",
"input_schema": {
"symbol": { "type": "string", "description": "Ticker symbol, e.g. BTC" }
}
}The description field is critical. The LLM reads descriptions in natural language and uses them to decide which tool applies to a given user request.
You can find existing MCP servers at mcp.so — crypto prices, web search, YouTube transcripts, PDF extraction, and more.
Instead of hardcoded routing logic, the backend now does one thing: on startup, it fetches the metadata from every connected MCP server and loads all tool descriptions into memory.
When a user sends a request, the backend combines the user's query with all the tool metadata and sends the full payload to the LLM. The LLM reads the descriptions, identifies the relevant tool, and responds not with an answer — but with a tool call recommendation:
{
"tool": "fetch_youtube_results",
"arguments": {
"query": "reflection in programming",
"max_results": 5
}
}The backend executes this tool call against the MCP server, gets the results, sends them back to the LLM with the original question, and the LLM generates the final grounded response.
This loop — query → tool recommendation → tool execution → final response — can be repeated multiple times for complex queries requiring several data sources. The LLM drives every decision. The backend only executes.
The brain has moved from the backend to the LLM, where it belongs.
Feeding an entire PDF to the LLM creates two problems:
- Context window limits. Current models support up to roughly 1 million tokens — impressive, but a large PDF can exceed that.
- Speed. Processing an entire 300-page physics textbook to answer one question about reflection is wasteful and slow.
The solution is chunking combined with semantic search.
- Chunk: Split the PDF into paragraphs or fixed-size segments (e.g., 400 chunks from a 100-page document).
- Embed: Convert each chunk to a numerical vector using an embedding model. Similar text produces similar vectors.
- Store: Save all (chunk text, chunk vector) pairs in a vector database optimized for similarity search.
- Query: When a user asks about "reflection," convert the query to a vector and ask the vector DB for the closest-matching chunks.
The vector DB returns the 2–5 most semantically relevant chunks rather than the full document. Those chunks go to the LLM, which generates a precise, grounded answer without wading through irrelevant content.
This logic lives inside an MCP server — the backend and LLM don't need to know about vector DBs or chunking directly. The MCP server exposes a clean interface: "ingest a PDF" and "query a PDF." The LLM recommends when to call each one.
A foundation model is trained on general internet data. It's good at everything and great at nothing in particular. If you need a model to explain topics in the specific style and depth your organization uses, the general model will disappoint.
Fine-tuning is the process of further training a foundation model on domain-specific data. You take a pre-trained model with its existing weights (the result of a $100M+ training run) and run additional training on your curated dataset. The model's weights shift slightly — enough to reflect the style and knowledge in your data without losing its general language capabilities.
Fine-tuning is practical because:
- You start with a model that already understands language, grammar, and reasoning. You're only teaching it new preferences, not rebuilding from scratch.
- The compute cost is orders of magnitude lower than pre-training.
- You need far less data — even a few megabytes of domain content can meaningfully shift model behavior.
The fine-tuned model has the same number of parameters as the base model; the values are simply adjusted. You save the result as a new parameters.bin and use that instead of the original.
An AI agent is any piece of code that uses tools, makes decisions (typically via an LLM), and takes multiple actions to complete a complex task autonomously.
The system built throughout this post — the backend loop that sends queries to the LLM, receives tool recommendations, calls MCP servers, and iterates until it has a final answer — is an agent. It acts on behalf of the user. It uses tools. It chains decisions.
The same definition applies to a script that:
- Scans job postings in your domain
- Reads each job description
- Rewrites your resume to emphasize relevant experience
- Applies to matching positions
That's an agent. The language doesn't matter. The library doesn't matter. What matters is the pattern: tools + decision-making + multi-step execution.
Libraries like LangChain and LangGraph reduce boilerplate when building agents, but understanding the pattern is more valuable than knowing any specific library. Libraries come and go; the concept is stable.
A 100B-parameter model at FP32 requires 400 GB of memory just to load for inference. Very few machines have that. Quantization reduces the precision of the stored weights to shrink the model's footprint.
| Precision | Bytes per parameter | Size (100B params) |
|---|---|---|
| FP32 | 4 | 400 GB |
| FP16 | 2 | 200 GB |
| INT8 | 1 | 100 GB |
Quantization is simple in practice: read each weight, reduce its precision, write it to a new file. The result is a faster, smaller model that trades a small amount of accuracy for a significant reduction in memory and compute requirements.
When you see a model listed as llama3-7B-Q8, you're reading: Llama 3, 7 billion parameters, quantized to 8-bit. Compare it to llama3-7B-chat (same size, same parameter count, fine-tuned for chat, not quantized) and you can immediately infer the memory and speed tradeoffs.
Understanding how much memory different operations require helps you choose what's feasible on your hardware.
Inference (prediction only) requires loading the weights into memory: num_parameters × bytes_per_parameter.
Training from scratch requires 4–6× the inference memory, because the optimizer must store gradients, optimizer states, and intermediate activations alongside the weights.
Full fine-tuning requires the same memory as training from scratch, for the same reasons.
For a 7B-parameter FP32 model (28 GB for inference), the memory requirements look like this:
| Operation | Memory needed |
|---|---|
| Inference | ~28 GB |
| Training / full fine-tuning | ~112–168 GB |
On a 64 GB machine, you can run inference on this model but cannot full fine-tune it.
Parameter-Efficient Fine-Tuning (PEFT) is a family of techniques that fine-tune only a small subset of parameters, dramatically reducing memory requirements.
Instead of updating all weights, LoRA freezes the base model entirely and injects a small set of additional "adapter" weights into the architecture. Only these adapter weights are trained. The base model's 28 GB sits frozen in memory; the adapters add a fraction of that.
The result: fine-tuning that requires roughly 1.1× the inference memory instead of 4–6×.
On a 64 GB machine with a 7B FP32 model:
- Inference: 28 GB ✓
- Full fine-tuning: ~140 GB ✗
- LoRA fine-tuning: ~31 GB ✓
QLoRA combines quantization with LoRA to push fine-tuning onto even more constrained hardware. The steps:
- Quantize the base model (e.g., FP32 → INT8), reducing it from 28 GB to ~7 GB.
- Freeze the quantized model.
- Apply LoRA adapters on top.
- Train only the adapter weights.
Memory requirement: ~7.7 GB (7 GB quantized model + ~0.7 GB for adapters).
On a 16 GB machine, this is viable where neither full fine-tuning nor standard LoRA would be.
The tradeoff is accuracy: quantization introduces precision loss. But for many practical use cases, the difference is acceptable — and it makes fine-tuning accessible on consumer hardware.
The AI tutor built through this post illustrates how all these concepts compose into a production system:
- LLM provides language understanding, summarization, and grounded response generation.
- RAG grounds the LLM in real-time or document-specific data it wasn't trained on.
- MCP moves routing decisions from brittle backend logic to the LLM itself, using tool metadata.
- Vector DB makes PDF-scale retrieval efficient — feeding only the most relevant chunks to the LLM.
- Fine-tuning adapts the model's style and domain knowledge to your specific use case.
- Agents are the code that orchestrates all of the above — the loop that takes a user query and drives it through tool calls and LLM interactions to a final answer.
- Quantization / PEFT / QLoRA make it possible to deploy and fine-tune models on hardware that would otherwise be inadequate.
The key insight across all of this: the LLM is a language engine. It understands text, generates text, and reasons about text. Every other component in the system exists to either give the LLM better information (RAG, MCP, vector DB) or to make the LLM's capabilities accessible on real-world infrastructure (quantization, fine-tuning). Design the system around the LLM's strengths, and you'll build something genuinely useful.