QA List for Andrej Karpathy's Zero to Hero Series

QA - The spelled-out intro to language modeling: building makemore

# The spelled-out intro to language modeling: building makemore

**Source:** https://www.youtube.com/watch?v=PaCmpygFfXo&list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ&index=2
**Site:** www.youtube.com
**Exported:** 7/15/2026, 7:18:17 PM

---

## Q1: Write this in place operation for P down as note.

Here is the in-place operation shown at this point in the video:

```python
P = N.float()
P /= P.sum(1, keepdims=True)
```

---

### Note

| Line | What it does |
|------|-------------|
| `P = N.float()` | Create a floating-point copy of the count matrix `N`. The original `N` stores integers, but probabilities need floats. |
| `P /= P.sum(1, keepdims=True)` | **In-place** divide every row of `P` by its row sum, so each row becomes a valid probability distribution that sums to 1. |

---

### Why `P /= ...` instead of `P = P / ...`?

`P /= ...` is an **in-place operation**. It modifies the existing tensor `P` directly rather than creating a brand-new tensor and storing it back into `P`.

- `P = P / P.sum(1, keepdims=True)` creates a new tensor in memory and assigns the name `P` to it.
- `P /= P.sum(1, keepdims=True)` reuses the same memory location and updates `P` in place.

This can be faster and more memory-efficient, especially for large tensors.

---

### What the result looks like

After this operation:

```python
P[0].sum()   # returns 1.0  (first row is normalized)
P[1].sum()   # returns 1.0  (second row is normalized)
# etc.
```

Every row of `P` is now a probability distribution over the 27 characters, ready to be used for sampling names with `torch.multinomial`.

## Q2: Why is log probability the better choice to evaluate if a model is good enough?

Log probability is preferred because it turns an awkward **product** of many small numbers into a much easier **sum** — while still measuring the same thing.

### The problem with raw likelihood

The **likelihood** of the whole dataset is the product of every individual probability the model assigns:

\[
\text{likelihood} = p_1 \times p_2 \times p_3 \times \dots
\]

Because every probability is between 0 and 1, multiplying thousands of them gives a **tiny number** that is hard to represent accurately on a computer.

### Why log fixes this

Taking the log converts multiplication into addition:

\[
\log(p_1 \times p_2 \times p_3) = \log p_1 + \log p_2 + \log p_3
\]

So instead of multiplying a long chain of tiny probabilities, we just **sum their logs**.

### What the graph on screen means

The WolframAlpha plot shows `log(x)` from 0 to 1:
- At `x = 1`, `log(1) = 0`. A perfect model gives log-likelihood 0.
- As `x` gets smaller, `log(x)` becomes more negative.
- At `x = 0`, `log(x)` goes to negative infinity.

So:
- **Better predictions** → probabilities closer to 1 → log-likelihood closer to 0.
- **Worse predictions** → probabilities closer to 0 → log-likelihood more negative.

### Why this is better for training

1. **Numerical stability**: Summing logs avoids multiplying tiny numbers into something the computer rounds to 0.
2. **Easier math**: Gradients and optimization work better with sums than with products.
3. **Same goal**: Log is a monotonic function, so maximizing log-likelihood is exactly equivalent to maximizing likelihood.

In practice, we usually flip the sign and use **negative log-likelihood** as a loss function, where lower is better.

## Q3: Why do we take the mean negative log likelihood here by dividing n? Can't we just use the nll as the loss function?

You **can** use the raw sum of negative log likelihoods as the loss function. Both work fine. We take the mean (divide by `n`) mostly for **convenience and interpretability**.

### Why the mean is nicer

| | Sum NLL | Mean NLL |
|---|---|---|
| **Depends on dataset size?** | Yes. More examples → larger sum. | No. It is an average per example. |
| **Easy to compare across datasets?** | No. | Yes. A 2.4 loss means the same thing whether you trained on 10 examples or 1 million. |
| **Interpretation** | Total surprise across all examples. | Average surprise per bigram. |
| **Optimization result** | Same optimal parameters. | Same optimal parameters. |

### The key insight: sum and mean are the same optimization problem

Dividing by `n` is just multiplying the loss by a constant:

\[
\text{mean NLL} = \frac{1}{n} \sum -\log p_i
\]

When you compute gradients, that constant factor just scales the gradient:

\[
\frac{\partial}{\partial W} (\text{mean NLL}) = \frac{1}{n} \frac{\partial}{\partial W} (\text{sum NLL})
\]

So every weight gets nudged in the **exact same direction**. The optimal weights are identical. You might just need a larger learning rate if you use the sum, but the final model will be the same.

### Example from the video

On the first 3 words (`Emma`, `Olivia`, `Ava`):

- `log_likelihood = -38.79`
- `nll = 38.79`
- `nll / n = 38.79 / 20 ≈ 1.94`

If you added a fourth word, the sum `nll` would grow (e.g., to 45), but the average `nll / n` would stay in the same ballpark if the model quality is similar.

### Practical summary

Use the **mean NLL** as the loss because:
1. It is a stable, interpretable number.
2. It lets you compare models trained on different amounts of data.
3. It has the same minimum as the sum NLL, so you lose nothing.

In code, this is exactly what `loss.mean()` does in the neural-network version later in the video.

## Q4: Write smoothing down as note.

Here is a note on **model smoothing**:

---

### Problem

A bigram model assigns **zero probability** to any bigram that never appeared in the training set. When evaluating a word like `andrejq`, the bigram `jq` might have zero probability, causing:

```python
log_prob = torch.log(0.0)   # -inf
loss = inf
```

This is undesirable because the model becomes infinitely confident that a reasonable-looking string is impossible.

---

### Solution: Model Smoothing

Add a small **fake count** to every entry in the count matrix before normalizing:

```python
P = (N + 1).float()
P /= P.sum(1, keepdims=True)
```

Or equivalently:

```python
N += 1          # add one fake count to every bigram
P = N.float()
P /= P.sum(1, keepdims=True)
```

---

### Effect

| Before smoothing | After smoothing |
|---|---|
| Some bigrams have probability 0 | No bigram has probability 0 |
| `log(0) = -inf` | `log(tiny) = very negative, but finite` |
| Model is overconfident on training data | Model is more robust on unseen data |

---

### Tuning the smoothing strength

```python
N += 1    # light smoothing
N += 5    # stronger smoothing
N += 1_000_000   # extremely strong smoothing → uniform distribution
```

- **Smaller value** (e.g., `+1`): distribution stays close to the data, only slightly less peaked.
- **Larger value** (e.g., `+5`, `+100`): distribution becomes more uniform.
- **Huge value**: all probabilities become roughly equal to `1/27`.

---

### Intuition

Model smoothing is like saying: *"Even if I never saw this exact bigram in training, it should still be possible, just very unlikely."*

It is the explicit-counting equivalent of **regularization** in the gradient-based neural network approach, where adding a term like `0.01 * (W**2).mean()` to the loss also pushes the model toward more uniform predictions.

## Q5: why is it not feasible to feed neurons integer values as input?

You can't feed integer indices directly into a neural network because **integers are arbitrary labels, not meaningful numerical values**.

### The problem

The training examples are integers like `0`, `5`, `13`, where each number represents a character index:
- `0` = `.` (start/end token)
- `5` = `e`
- `13` = `m`

A neural network computes:

\[
\text{output} = W \cdot x + b
\]

If you plug in the integer `x = 13` for the character `m`, the network multiplies it by weights. But the number `13` has no special relationship to the character `m`. It was just assigned randomly. The character `m` isn't "more" than `e` just because `13 > 5`.

So multiplying `13` by a weight would be meaningless: the network would incorrectly treat the character index as a magnitude.

### Example

Suppose you have two input characters:
- `e` → index `5`
- `m` → index `13`

If you feed in the raw integer `5` or `13`, the network would learn weird relationships based on the accidental sizes of these numbers. For instance:
- `m` has twice the "input strength" of `e` because `13 > 5`
- `a` (index `1`) would be "smaller" than `e` (index `5`)

This is wrong because the letters themselves have no inherent ordering or magnitude.

### The solution: one-hot encoding

Instead, we turn each integer into a vector of all zeros except one position:

```python
# e = index 5 becomes a 27-dimensional vector
x = [0, 0, 0, 0, 0, 1, 0, 0, ..., 0]
#     0  1  2  3  4  5  6  7      26
```

Now each character has its own dedicated input dimension, and the weight matrix can learn how strongly that character predicts each possible next character. The matrix multiplication `W @ x` effectively just **selects the row of W** corresponding to the active character.

This makes the input meaningful for the neural network's `W x + b` computation.

## Q6: When we hot encode like this: x = [0, 0, 0, 0, 0, 1, 0, 0, ..., 0], is that another way of saying: in this round, we selected "e" as input and this character's index is 5, of course the index is figured out by the model itself and was irrelevant to the calculation.

Almost, but there's one important correction and one clarification.

### Correction: the index is not chosen by the model

The index `5` is **not** figured out by the model. It comes from the mapping we created earlier:

```python
s2i = {'.': 0, 'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, ...}
```

So `e` maps to `5` simply because we decided to sort the alphabet and put `.` at position `0`. A different ordering would give `e` a different index, but the model would still learn the same relationships.

### What is correct: the index is not used as a number

The raw value `5` is **not** multiplied or treated as a magnitude. Instead, the one-hot vector simply says: *"turn on the 5th input channel and leave the rest off."*

So:

```python
x = [0, 0, 0, 0, 0, 1, 0, 0, ..., 0]
#          0  1  2  3  4  5  6  7     26
```

means: **"this training example is the character `e`."**

### Why this matters for the matrix multiplication

When you multiply this one-hot vector by the weight matrix `W`:

```python
logits = x_enc @ W
```

the result is literally just the **5th row of W** (using 0-indexing). The 1 in position `5` selects that row, and the zeros everywhere else ignore all other rows.

As the presenter notes in the video, this is exactly equivalent to the earlier lookup-table approach:

```python
# Before: lookup row of counts directly
N[5]  # row for 'e'

# Now: one-hot encode and multiply
x_enc = F.one_hot(torch.tensor(5), num_classes=27)
logits = x_enc @ W   # gives the same row as W[5]
```

### Summary

| Statement | Verdict |
|---|---|
| "The one-hot vector means we selected `e` as input" | ✓ Correct |
| "The character's index is 5" | ✓ Correct |
| "The index is irrelevant to the calculation" | ✗ Partially wrong — it determines **which row of W** is selected |
| "The numerical value `5` is not used as a magnitude" | ✓ Correct |

So the index is used as a **selector**, not as a numerical value. That is why one-hot encoding is the right way to feed categorical inputs into a neural network.

## Q7: So in this case, when doing the xenc @ W, do selected row 0, 5, 13, 13 and 1 in the W matrix, output the w X x for this particular neuron in this round of input. is that correct?

Yes, that's correct — with one small clarification.

At this exact moment in the video, `W` has shape `(27, 1)`, so there is **only one neuron**. For each of the 5 input examples, `xenc @ W` selects the corresponding row of `W` and outputs the dot product, which is just a single scalar per example.

### Why rows 0, 5, 13, 13, 1?

Those are the indices of the input characters for the word **Emma** (with start/end tokens):

| Position | Character | Index | Selected row of W | Output |
|----------|-----------|-------|-------------------|--------|
| 0 | `.` | 0 | `W[0]` | `W[0, 0]` |
| 1 | `e` | 5 | `W[5]` | `W[5, 0]` |
| 2 | `m` | 13 | `W[13]` | `W[13, 0]` |
| 3 | `m` | 13 | `W[13]` | `W[13, 0]` |
| 4 | `a` | 1 | `W[1]` | `W[1, 0]` |

### Why this is exactly equivalent to `w · x`

For a one-hot vector, the dot product picks out just one element:

```python
x_enc = [0, 0, 0, 0, 0, 1, 0, ..., 0]   # index 5 is on
W = [w0, w1, w2, ..., w26]^T            # column vector

x_enc @ W = 0*w0 + 0*w1 + ... + 1*w5 + ... + 0*w26 = w5
```

So each output is simply the weight corresponding to the active character.

### The output shape

```python
xenc.shape  # torch.Size([5, 27])
W.shape     # torch.Size([27, 1])
(xenc @ W).shape  # torch.Size([5, 1])
```

The 5 outputs you see are the 5 activations of this single neuron on the 5 input examples.

---

### Note: this changes when W becomes (27, 27)

Right after this, the presenter expands `W` to shape `(27, 27)` to create **27 output neurons**. Then `xenc @ W` gives an output of shape `(5, 27)` — one logit per character for each of the 5 input examples.

## Q8: Is expanding W to shape (27,27) means there are 27 neurons?

Yes — **exactly**.

Expanding `W` from shape `(27, 1)` to `(27, 27)` means you now have **27 neurons** instead of just 1.

### Why?

Each neuron has 27 input weights, one for each of the 27 input dimensions. The shape of `W` is:

```python
W.shape  # (27, 27)
#         ↑   ↑
#       input output
#       dim   neurons
```

| Part | Meaning |
|------|---------|
| Rows (27) | One weight per input dimension |
| Columns (27) | One neuron per output dimension |

So each **column** of `W` is the weight vector of one neuron.

### What the output becomes

```python
xenc.shape  # (5, 27)   → 5 input examples
W.shape     # (27, 27)  → 27 neurons, each with 27 inputs
(xenc @ W).shape  # (5, 27)  → 5 examples × 27 neuron activations
```

So the output is a **5 × 27 matrix**: for each of the 5 input examples, you get the firing rate of all 27 neurons.

### What each neuron will later represent

In this model, each of the 27 output neurons corresponds to one possible **next character**. After applying softmax, the 27 outputs become the probability distribution over which character follows the input character.

So:
- `W[:, 0]` = weights for the neuron predicting `.`
- `W[:, 1]` = weights for the neuron predicting `a`
- `W[:, 5]` = weights for the neuron predicting `e`
- etc.

## Q9: Is matrix multiplication and dot product different in operation?

Not fundamentally different — **matrix multiplication is made up of many dot products**.

### Dot product
A **dot product** operates on two **vectors** (one-dimensional arrays) and returns a single number:

```python
a = [a1, a2, a3]
b = [b1, b2, b3]

a · b = a1*b1 + a2*b2 + a3*b3   # scalar
```

### Matrix multiplication
A **matrix multiplication** operates on two **matrices** (two-dimensional arrays) and returns a matrix. Each entry of the result is the dot product of one row from the first matrix and one column from the second matrix:

```python
C = A @ B
C[i, j] = dot(A[i, :], B[:, j])
```

So matrix multiplication is just a way of computing **many dot products in parallel**.

---

### In the video context

At this timestamp:

```python
W = torch.randn((27, 27))
xenc @ W        # shape (5, 27)
```

`xenc @ W` is a **matrix multiplication**. But the element at position `[3, 13]`:

```python
(xenc @ W)[3, 13]   # tensor(0.5008)
```

is the **dot product** between:
- the 3rd row of `xenc` (the one-hot vector for input example 3)
- the 13th column of `W` (the weights of the 13th neuron)

The presenter verifies this explicitly by computing:

```python
(xenc[3] * W[:, 13]).sum()   # also 0.5008
```

This is the same dot product, just written out element-wise.

---

### Summary

| Operation | Inputs | Output | Relationship |
|-----------|--------|--------|--------------|
| Dot product | two vectors | one scalar | One unit of computation |
| Matrix multiplication | two matrices | one matrix | Many dot products stacked together |

So: **matrix multiplication = a collection of dot products**.

## Q10: Is exp() function use the Euler's number as base and power that number by the element in the (xenc @ W) matrix?

Yes, exactly.

`torch.exp()` (or `.exp()`) computes **e^x element-wise**, where `e` is Euler's number (≈ 2.71828) and `x` is each element in the `(xenc @ W)` matrix.

So for every output logit:

\[
\text{exp}(x) = e^x
\]

This is why the result is always positive. For example:
- If a logit is `0.6927`, then `exp(0.6927) ≈ 1.99`
- If a logit is `-0.3012`, then `exp(-0.3012) ≈ 0.74`

These positive "fake counts" can then be normalized into a probability distribution.

## Q11: Write this part down as note on logits, counts and probs.

Here is a note on the **logits → counts → probabilities** pipeline:

---

## Logits, Counts, and Probabilities

### The three-step transformation

```python
logits = xenc @ W          # log-counts
counts = logits.exp()       # equivalent to N
probs = counts / counts.sum(1, keepdims=True)  # normalized probabilities
```

These three lines are the core of the forward pass. They convert the raw neural network outputs into a valid probability distribution.

---

### 1. Logits

```python
logits = xenc @ W
```

- **Shape:** `(5, 27)`
- **Interpretation:** log-counts
- Logits are the raw outputs of the linear layer. They can be positive or negative.
- We interpret them as **logarithms of counts**, not as probabilities directly.
- The term "logits" is standard in machine learning for the pre-softmax outputs of a neural network.

---

### 2. Counts

```python
counts = logits.exp()
```

- **Shape:** `(5, 27)`
- **Interpretation:** fake / predicted counts
- `exp()` is applied element-wise: `counts[i, j] = e^(logits[i, j])`
- This maps negative logits to values between 0 and 1, and positive logits to values greater than 1.
- The result is always positive, so it behaves like a count.
- This matrix is analogous to the count matrix `N` from the explicit bigram-counting approach.

---

### 3. Probabilities

```python
probs = counts / counts.sum(1, keepdims=True)
```

- **Shape:** `(5, 27)`
- **Interpretation:** probability distribution over next characters
- Each row is normalized to sum to 1.
- `probs[i, j]` tells us: given the i-th input example, how likely is character j to come next?

---

### Why this works

| Quantity | Properties | How obtained |
|---|---|---|
| **Logits** | Any real number, positive or negative | Linear layer `xenc @ W` |
| **Counts** | Always positive | `logits.exp()` |
| **Probs** | Positive and sum to 1 | `counts / counts.sum(...)` |

The neural network cannot directly output probabilities because:
- Probabilities must be positive and sum to 1.
- A linear layer can output any real number.

So we use:
- Exponentiation to make everything positive.
- Normalization to make every row sum to 1.

---

### Softmax

These last two operations together are called the **softmax**:

```python
softmax(logits) = exp(logits) / sum(exp(logits))
```

Softmax is a common function that converts arbitrary real-valued logits into a probability distribution.

---

### Key checks

```python
probs.shape       # torch.Size([5, 27])
probs[0].sum()    # tensor(1.)
```

- Every row has 27 entries, one for each possible next character.
- Every row sums to 1, confirming it is a valid probability distribution.

---

### Connection to the earlier counting model

| Explicit counting model | Neural network model |
|---|---|
| `N` = count matrix | `counts = logits.exp()` |
| `P = N / N.sum(1, keepdims=True)` | `probs = counts / counts.sum(...)` |
| Row indexed by input character | Row selected by `xenc @ W` |

The neural network learns the equivalent of `log(N)` in the weight matrix `W`. When optimized, `W.exp()` approximates the count matrix from the explicit approach.

## Q12: Explain this part: The neural network learns the equivalent of log(N) in the weight matrix W. When optimized, W.exp() approximates the count matrix from the explicit approach.

Here is the explanation of that equivalence:

---

## Why `W` is the Neural Network's Version of `log(N)`

### 1. In the explicit counting approach

We built a count matrix `N` where:

```python
N[i, j] = number of times character j followed character i
```

Then we got probabilities by normalizing:

```python
P = N / N.sum(1, keepdims=True)
```

So each row of `P` is the probability distribution for the next character.

### 2. In the neural network approach

We have:

```python
logits = xenc @ W
counts = logits.exp()
probs = counts / counts.sum(1, keepdims=True)
```

The structure is almost identical:

| Counting approach | Neural network approach |
|---|---|
| `N` | `counts = logits.exp()` |
| `P = N / N.sum(...)` | `probs = counts / counts.sum(...)` |

The only difference is: instead of directly storing the counts in `N`, we store **log-counts** in `W`.

### 3. Why `W` stores log-counts

Because:

```python
counts = logits.exp() = (xenc @ W).exp()
```

If we ignore the one-hot encoding and matrix multiplication for a moment, the core operation is:

```python
counts = W.exp()
```

So `W` is literally the **logarithm of the count matrix**. Therefore:

```python
W ≈ log(N)
W.exp() ≈ N
```

### 4. Why the rows match up

When the input character has index `i`, one-hot encoding gives a vector with a `1` at position `i` and `0` elsewhere. Multiplying by `W` selects the `i`-th row of `W`:

```python
xenc[i] @ W = W[i, :]
```

So the logits for input character `i` are the `i`-th row of `W`. After exponentiation:

```python
counts[i, :] = W[i, :].exp()
```

And after normalization:

```python
probs[i, :] = counts[i, :] / counts[i, :].sum()
```

This is exactly equivalent to:

```python
P[i, :] = N[i, :] / N[i, :].sum()
```

### 5. Why optimization makes them match

In the counting approach, we filled `N` by literally counting bigrams in the data.

In the neural network approach, we start with random `W` and use gradient descent to minimize the negative log-likelihood loss. The loss penalizes the model when it assigns low probability to the actual next character. As the model improves, the probabilities `probs` move closer to the empirical probabilities `P`.

Since the two models have the same expressiveness (both produce one probability distribution per input character), the optimal `W` will satisfy:

```python
W.exp() ≈ N
```

or equivalently:

```python
W ≈ log(N)
```

### 6. Numerical intuition

Suppose in the data, `a` follows `.` 1,000 times out of 10,000 total `.` bigrams. Then:

- Counting approach: `N[0, 1] = 1000`
- Neural network: `W[0, 1]` should be around `log(1000) ≈ 6.9`

After exponentiation:

```python
W[0, 1].exp() ≈ 1000
```

After normalization across the row, this gives roughly `1000 / 10000 = 0.10` probability.

### Summary

| Quantity | Explicit counting | Neural network |
|---|---|---|
| Raw counts | `N` | `W.exp()` |
| Log of counts | `log(N)` | `W` |
| Probabilities | `N / N.sum(...)` | `W.exp() / W.exp().sum(...)` |

The neural network discovers the same count statistics, but indirectly through gradient-based optimization rather than explicit counting.

## Q13: Explain why the lecturer is mentioning about backpropagatable operations here.

The lecturer is emphasizing that every operation in the forward pass is **backpropagatable** because that is the entire reason the neural network approach works.

Here is the key idea:

### We want to update W

`W` is the only thing we can change in this model. We start with random `W` and we want to adjust it so that the model assigns higher probability to the correct next character.

To do that, we need to know:

\[
\frac{\partial \text{loss}}{\partial W}
\]

That is, how does changing each element of `W` affect the final loss?

### Backpropagation requires every step to be differentiable

The loss is computed through a chain of operations:

```python
logits = xenc @ W
counts = logits.exp()
probs = counts / counts.sum(1, keepdims=True)
loss = -probs[torch.arange(5), ys].log().mean()
```

For PyTorch to compute `d(loss)/d(W)`, it must be able to propagate gradients backward through every single step:

| Forward operation | Backward operation needed |
|---|---|
| `xenc @ W` | gradients with respect to `W` |
| `logits.exp()` | gradient of `e^x` |
| `counts.sum()` | gradient of sum |
| division `/` | gradient of division |
| `torch.log()` | gradient of `log(x)` |
| `.mean()` | gradient of mean |

### Why this matters

Because all these operations are differentiable, PyTorch can build a **computational graph** during the forward pass. Then:

```python
loss.backward()
```

automatically fills in `W.grad` with the gradient of the loss with respect to every element of `W`.

Once we have `W.grad`, we can update `W`:

```python
W.data += -learning_rate * W.grad
```

This nudges `W` in the direction that reduces the loss.

### Summary

The lecturer is making the point that:

> "This isn't just a random transformation. Every step is differentiable, so we can use gradient descent to optimize W."

This is what separates the neural network approach from the explicit counting approach. In the counting approach, we manually set the probabilities. In the neural network approach, we let the gradients of a differentiable loss function find the right `W` automatically.

## Q14: Why is this classification not regression? Predicting probabilities are classification problems?

Yes, predicting a probability distribution over a fixed set of discrete categories is exactly what makes this a **classification** problem.

### Why this is classification

In this model:

- The input is a character (e.g., `.`).
- The label is one of **27 discrete categories** (the next character: `a`, `b`, `c`, ..., `.`).
- The model outputs **27 probabilities**, one for each possible category.

```python
probs[i] = probability that character i is the next character
```

The task is: *which of the 27 possible characters comes next?* That is a **classification** task.

---

### Why this is not regression

In regression, the target is a **continuous number** like height, temperature, or a coordinate.

The micrograd example used regression: the targets were continuous values like `1.0`, `-1.0`, `-1.0`, `1.0`, and the loss was:

```python
loss = sum((yout - ygt)**2 for ...)
```

This is **mean squared error (MSE)**, the standard regression loss.

But in makemore, the target is not a continuous number. It is a discrete category index:

```python
ys = [5, 13, 13, 1, 0]   # indices of characters
```

So we use **negative log likelihood**, the standard classification loss.

---

### Summary table

| | Regression | Classification |
|---|---|---|
| Target | Continuous number | Discrete category |
| Output | Single number | Probability distribution over classes |
| Loss | Mean squared error | Negative log likelihood / cross-entropy |
| Example | Predicting a value | Predicting the next character |

So the lecturer is saying: because we are predicting which character comes next out of 27 possible characters, and because we output a probability distribution over those 27 categories, this is classification — not regression.

## Q15: Explain why loss is the calculated that way in cell 570?

The loss in cell 570 is:

```python
loss = -probs[torch.arange(5), ys].log().mean()
```

This is the **negative log likelihood loss**. Let me break down why it is calculated this way.

---

### 1. `probs[torch.arange(5), ys]`

`probs` has shape `(5, 27)`: 5 examples, each with a probability distribution over 27 possible next characters.

`ys` is the true next-character label for each example:

```python
ys = [5, 13, 13, 1, 0]
```

So `ys` says:
- Example 0: correct next character is index 5
- Example 1: correct next character is index 13
- Example 2: correct next character is index 13
- Example 3: correct next character is index 1
- Example 4: correct next character is index 0

`torch.arange(5)` creates `[0, 1, 2, 3, 4]`, the row indices.

Together, `probs[torch.arange(5), ys]` selects the **probability assigned to the correct next character** for each example:

```python
[
  probs[0, 5],   # probability the model assigned to the correct answer for example 0
  probs[1, 13],  # probability the model assigned to the correct answer for example 1
  probs[2, 13],  # ...
  probs[3, 1],
  probs[4, 0]
]
```

This is a vector of 5 numbers, each between 0 and 1.

---

### 2. `.log()`

We take the natural logarithm of each probability.

Why? Because the product of probabilities is unwieldy:

```python
likelihood = p1 * p2 * p3 * p4 * p5
```

The log turns products into sums:

```python
log_likelihood = log(p1) + log(p2) + log(p3) + log(p4) + log(p5)
```

For probabilities between 0 and 1:
- `log(1.0) = 0`
- `log(0.5) ≈ -0.69`
- `log(0.1) ≈ -2.30`
- `log(0.01) ≈ -4.61`

The better the model, the closer the probabilities are to 1, so the log likelihood is closer to 0.

---

### 3. `.mean()`

We average the log probabilities over the 5 examples:

```python
average log likelihood = (log(p1) + ... + log(p5)) / 5
```

This gives a single number representing the average log likelihood across all examples.

---

### 4. The leading `-`

We want a **loss function**, where **lower is better**.

But log likelihood is higher when the model is better (closer to 0).

So we flip the sign:

```python
loss = - average log likelihood
```

Now:
- If probabilities are high → log likelihood is near 0 → loss is near 0 → **good**
- If probabilities are low → log likelihood is very negative → loss is large positive → **bad**

---

### Full summary

```python
loss = -probs[torch.arange(5), ys].log().mean()
#       ↑  ↑                      ↑     ↑
#       |  |                      |     average over examples
#       |  |                      |
#       |  |                      take log of probabilities
#       |  |
#       |  select probability of correct next character
#       |
#       flip sign so lower is better
```

This is the standard **cross-entropy / negative log likelihood** loss for classification. The model is trained to make this loss as small as possible, which means making the probabilities it assigns to the correct next characters as high as possible.

## Q16: Explain why this neuron approach achieves the same loss result as the previous bigram counting approach.

The neuron approach achieves the same loss because **it is fundamentally the same model, just optimized differently**.

Here is why:

### 1. Both models use the same information

Both models are **bigram models**. They only look at the previous character to predict the next one. They are trained on the exact same 228,146 bigrams from the names dataset.

### 2. Both models have the same capacity

The counting approach had a **27 × 27 table** of counts `N`. The neural network has a **27 × 27 weight matrix** `W`.

So both have the same number of degrees of freedom: one probability distribution per input character.

### 3. Both models compute probabilities the same way

| Counting approach | Neural network approach |
|---|---|
| `counts = N` | `counts = W.exp()` |
| `P = N / N.sum(1, keepdims=True)` | `probs = counts / counts.sum(...)` |

As the lecturer explained earlier, `W` stores **log-counts**. When optimized, `W.exp()` approximates the count matrix `N` from the explicit approach.

### 4. Both optimize the same loss function

The counting approach directly estimates the empirical probabilities by counting. The neural network approach starts with random `W` and uses gradient descent to minimize the **negative log likelihood loss** on the same training data.

Since the loss function is the same, and the model can represent the same probability distributions, the optimal solution is the same.

### 5. Why the loss is around 2.47

This is the best possible loss for a **bigram model**. You cannot do better than this because you are only allowed to use one previous character. Any more complex model would use more context, but the bigram model is fundamentally limited.

The counting approach reaches this optimum directly. The neural network reaches it through gradient descent, as you can see in the output:

```
2.4726...
2.4724...
2.4722...
```

Both converge to the same number because they are solving the same problem.

### Summary

> **Same task → same model capacity → same probability distributions → same loss.**

The only difference is *how* the probabilities are obtained:
- Counting: count and normalize explicitly.
- Neural network: learn log-counts through gradient descent.

The neural network approach is more flexible, but for this simple bigram case, both arrive at the same solution.

## Q17: In one hot encoding, does xenc @ W the same as W @ xenc?

No, **`xenc @ W`** and **`W @ xenc`** are **not the same**, and only `xenc @ W` works correctly here.

### Why?

Matrix multiplication requires the **inner dimensions to match**:

- `xenc` has shape **(5, 27)**  
  - 5 examples, each a 27-dimensional one-hot vector
- `W` has shape **(27, 27)**  
  - 27 input features, 27 neurons

```python
xenc @ W     # (5, 27) @ (27, 27)  →  (5, 27)  ✓
```

This gives 5 rows of output logits, one for each input example.

```python
W @ xenc     # (27, 27) @ (5, 27)  →  ERROR  ✗
```

This fails because the inner dimensions **27 ≠ 5**.

---

### What if you tried to transpose?

You could technically write:

```python
W @ xenc.T   # (27, 27) @ (27, 5)  →  (27, 5)
```

But the result would be **(27, 5)** instead of **(5, 27)**. The rows and columns would be swapped, so the outputs would no longer correspond to "one row per input example." It would be the wrong orientation.

---

### Semantics

`xenc @ W` means:
- For each input example (row of `xenc`)
- Compute the dot product with each neuron's weights (column of `W`)
- Produce one output per example, per neuron

So the lecturer uses:

```python
logits = xenc @ W
```

because the rows of `xenc` are the **examples**, and the rows of `logits` must also be the **examples**.

## Q18: Explain here in neuron case we randomly generated W and let the loss guide us to the same matrix. Especially on how neuron counts maps to bigram counts.

At this timestamp (01:50:03), the lecturer is pointing at the **bigram count matrix** `N` from the explicit counting approach. The key claim is:

> The neural network learns a weight matrix `W` such that `W.exp()` becomes essentially this same count matrix.

Here is how that works.

---

## 1. The count matrix `N`

The heatmap you see is the **27 × 27 bigram count matrix**. Each cell `N[i, j]` tells us:

> How many times character `j` followed character `i` in the training data.

For example:
- If `N[0, 5]` is large, that means the bigram `.e` (start followed by `e`) appears often.
- If `N[13, 1]` is small, that means the bigram `mb` is rare.

To get probabilities, we normalize each row:

```python
P = N / N.sum(1, keepdims=True)
```

So each row of `P` is a probability distribution over the next character.

---

## 2. The neural network matrix `W`

In the neural network approach, we start with random `W` of shape **27 × 27**. The forward pass is:

```python
logits = xenc @ W
counts = logits.exp()
probs = counts / counts.sum(1, keepdims=True)
```

Because `xenc` is a one-hot vector, `xenc @ W` just selects the corresponding row of `W`. So:

> Row `i` of `W` stores the **log-counts** for what character comes after character `i`.

After exponentiation:

> `W[i, :].exp()` is the neural network's version of the count row `N[i, :]`.

---

## 3. Why `W.exp()` matches `N`

Both models are trying to solve the same task with the same capacity:

- **Same input:** one previous character (27 possibilities).
- **Same output:** probability distribution over 27 next characters.
- **Same training data:** all 228,146 bigrams in the dataset.
- **Same loss function:** negative log likelihood.

In the counting approach, we set the probabilities directly to the empirical frequencies. In the neural network approach, we start with random `W` and use gradient descent to minimize the loss.

The loss tells `W` to make the correct next character more probable. Over many iterations, this forces the rows of `W.exp()` to grow in proportion to how often each bigram actually appears in the data.

So:

```python
W.exp() ≈ N
```

Or equivalently:

```python
W ≈ log(N)
```

---

## 4. Visual mapping

Think of the heatmap as `N`. The neural network learns a hidden version of this exact same heatmap:

| Bigram counting model | Neural network model |
|---|---|
| `N[i, j]` = raw count | `W[i, j].exp()` = predicted count |
| `N / N.sum(...)` = probabilities | `W.exp() / W.exp().sum(...)` = probabilities |
| Counts come from direct counting | Counts come from gradient descent |

The cell value `N[i, j]` in the heatmap corresponds to the exponentiated weight `W[i, j].exp()` after training.

---

## 5. Why the loss is identical

Because the two models are mathematically equivalent:

- They can represent the exact same set of probability distributions.
- They are trained on the same data.
- They optimize the same loss.

The explicit counting approach finds the optimal solution in one step. The neural network approach finds the same optimal solution through gradient descent. That is why the final loss is approximately the same (around 2.45–2.47).

---

## Summary

The lecturer is saying: **don't be fooled by how different the neural network looks**. Under the hood, it is learning the same thing as the bigram count matrix. The matrix `W` is just the **logarithm** of the bigram count matrix, and gradient descent is the mechanism that shapes it to match the empirical counts.

## Q19: Take notes of this regularization concept.

Here are notes on the **regularization** concept at this timestamp:

---

## Regularization as Model Smoothing

### 1. Background: model smoothing in the counting approach

In the explicit counting approach, we had a problem: some bigrams never appeared in the training data, so their probability was zero. This caused infinite loss on any name containing those bigrams.

The fix was **model smoothing**: add fake counts to every entry.

```python
N += 1  # add one to every count
P = N / N.sum(1, keepdims=True)
```

The effect:
- **More fake counts** → more uniform probabilities → smoother model.
- **Fewer fake counts** → more peaked probabilities → sharper model.

---

### 2. The equivalent in the neural network approach

In the neural network framework, the same smoothing effect is achieved through **regularization**.

The key observation is that:

- If `W = 0`, then `logits = 0`, so `counts = exp(0) = 1`, so all probabilities are **uniform**.
- The larger the values of `W`, the more **peaked** the probability distribution.

So if we encourage `W` to stay near zero, we get a smoother, more uniform distribution.

---

### 3. L2 regularization

We add a new term to the loss function:

```python
loss = -probs[torch.arange(num), ys].log().mean() + 0.01 * (W ** 2).mean()
```

This has two parts:

| Term | Meaning | Purpose |
|---|---|---|
| Data loss | `-probs[...].log().mean()` | Fit the training data |
| Regularization loss | `0.01 * (W ** 2).mean()` | Keep weights small |

This is called **L2 regularization** or **weight decay**.

---

### 4. Why it works

- Squaring `W` removes signs, so all weights contribute positively.
- The loss is zero when all `W = 0`, and grows as weights move away from zero.
- During optimization, gradients push `W` both:
  - **toward** fitting the data, and
  - **toward** zero.

This acts like a **spring force** or **gravity** pulling every weight toward zero.

---

### 5. Controlling the strength

The regularization coefficient controls how much smoothing happens:

- **Large coefficient** (e.g., `0.1`, `1.0`, `10.0`):
  - Regularization dominates the data loss.
  - Weights stay very small.
  - Probabilities become nearly uniform.
  - Equivalent to adding **many** fake counts in the counting approach.

- **Small coefficient** (e.g., `0.001`, `0.01`):
  - Data fitting dominates.
  - Weights can grow to fit the training data.
  - Probabilities are more peaked.
  - Equivalent to adding **few** fake counts.

- **Zero coefficient**:
  - No regularization.
  - Model is as sharp as the data allows.
  - Some probabilities can become very small or zero.

---

### 6. Connection to smoothing

| Counting approach | Neural network approach |
|---|---|
| `N += k` | `loss += λ * (W ** 2).mean()` |
| Add fake counts to the count matrix | Pull weights toward zero |
| More counts → more uniform | Larger λ → more uniform |
| Fewer counts → more peaked | Smaller λ → more peaked |

---

### 7. Intuition

> **Regularization is the neural network's way of doing model smoothing.**

Both techniques prevent the model from becoming overconfident. They ensure that even unseen bigrams still get some small probability, avoiding infinite loss and making the model more robust.

---

### 8. Code summary

```python
# forward pass
xenc = F.one_hot(xs, num_classes=27).float()
logits = xenc @ W
counts = logits.exp()
probs = counts / counts.sum(1, keepdims=True)

# loss with regularization
loss = -probs[torch.arange(num), ys].log().mean() + 0.01 * (W ** 2).mean()
```

---

### Key takeaways

1. Regularization penalizes large weights.
2. It pushes the model toward uniform predictions.
3. It is mathematically equivalent to adding fake counts in the counting approach.
4. The regularization coefficient `λ` controls the amount of smoothing.
5. It is a standard technique to prevent overfitting and improve generalization.

## Q20: Row i of W stores the log-counts for what character comes after character i. and you say W[i, :].exp() is the neural network's version of the count row N[i, :]., explain why do we have to do the exp() again? To just squeeze the value to a negative inf and 0?

No — it is not about squeezing to between negative infinity and 0. In fact, it is the opposite: **`exp()` turns any real number into a positive number**.

Here is why `exp()` is necessary.

---

## 1. What `W` stores

`W` stores **log-counts** (also called **logits**).

A log-count can be any real number:
- Positive if the bigram is common.
- Negative if the bigram is rare.
- Very negative if the bigram is almost never seen.

For example:
- If `W[i, j] = 6.9`, the actual count is roughly `e^6.9 ≈ 1000`.
- If `W[i, j] = -2.3`, the actual count is roughly `e^-2.3 ≈ 0.1`.

---

## 2. Why `exp()`?

**Counts cannot be negative.** A bigram either occurs 0 times, 1 time, 1000 times, etc. You cannot have a count of -5.

The linear layer `logits = xenc @ W` produces **any real number**. The output of a linear layer is not naturally positive.

So we need a transformation that:
- Accepts any real number as input.
- Always outputs a positive number.

That is exactly what the exponential function `exp()` does:

```
exp(x) > 0 for all x
```

So:

```python
counts = logits.exp()
```

This converts log-counts back into actual counts.

---

## 3. The whole picture

| Quantity | Meaning | Range | How obtained |
|---|---|---|---|
| `W[i, j]` | log-count of bigram `i → j` | any real number | learned by gradient descent |
| `logits = xenc @ W` | same as selecting a row of `W` | any real number | linear layer |
| `counts = logits.exp()` | unnormalized count | positive number | undo the log |
| `probs = counts / counts.sum()` | probability distribution | between 0 and 1, sums to 1 | normalize |

This is the **softmax** operation in disguise.

---

## 4. What if we skipped `exp()`?

If we tried to normalize `logits` directly:

```python
probs = logits / logits.sum(...)
```

Two problems arise:

1. **Negative logits** would create negative probabilities, which are invalid.
2. The relative differences between logits would not be preserved correctly for probability interpretation.

So `exp()` is the bridge that converts the linear layer's real-valued outputs into valid, positive counts.

---

## 5. The neural network count map

So the mapping from the neural network to the bigram count matrix is:

```python
N ≈ W.exp()
```

Or equivalently:

```python
W ≈ log(N)
```

That is why the lecturer says: **"`W` is the log-counts, not the counts. `W.exp()` is the array."**

---

## Summary

`exp()` is needed because:
1. `W` stores log-counts, which can be positive or negative.
2. To convert log-counts into actual counts, we undo the log.
3. The result is always positive, which is required before normalization into probabilities.

## Q21: In cell 715, why the calculation of logits = xenc @ W didn't involve a bias b?

The short answer is: **the bias was deliberately omitted to keep this the simplest possible neural network**.

At around 01:16 in the video, the speaker explicitly says: "These neurons perform W * X. They don't have a bias and they don't have a non-linearity like tanh. We're going to leave them to be a linear layer."

Here is the deeper reason for this choice:

1. **Demonstration of equivalence:** The goal is to show that this tiny neural network can learn the exact same bigram model as the counting-based approach. In that counting model, the first character directly selects a row of the count matrix `N`. With one-hot encoding, `xenc @ W` literally just "plucks out" the corresponding row of `W`. That row is the log-counts for the next character. Adding a bias would break this clean one-to-one correspondence.

2. **No need for extra flexibility:** A bias would add extra parameters, but for a bigram model the single `W` matrix is already sufficient to represent every possible conditional distribution. The model does not need a bias to be expressive enough.

3. **Regularization is used instead:** Later in the same cell, you can see the regularization term `+ 0.01*(W**2).mean()`. This acts like the "model smoothing" the speaker discussed earlier, pushing the probabilities toward uniform. It is not the same as a bias, but it serves a related purpose.

In short, `logits = xenc @ W` has no bias because the lecture is intentionally building the **simplest** bigram neural network possible before scaling up to more complex architectures.