Curriculum: Mathematical Foundations of AI

This is a personalized, self-paced roadmap for mastering the mathematical foundations of Machine Learning (ML) and Artificial Intelligence (AI). It is designed to build a strong "intuition-first" theoretical base, immediately backed by practical Python/NumPy implementations and empirical experiments.

Primary Theoretical Sources & Video Series

[MFML] Mathematical Foundations of Machine Learning by Prof. Prathosh AP (IISc)
[PF] Probability Foundations by Prof. Krishna Jagannathan (IIT Madras)
[Prathosh-Prob] Probability Theory by Prof. Prathosh (IISc)
[3B1B-LA] 3Blue1Brown: Essence of Linear Algebra
[3B1B-NN] 3Blue1Brown: Neural Networks

Primary Reading Resources

[PML-1] Probabilistic Machine Learning: An Introduction by Kevin Murphy - Primary bridge for classical ML, statistics, and neural nets.
[PML-2] Probabilistic Machine Learning: Advanced Topics by Kevin Murphy - Reserved for advanced generative and modern models.
[MML] Mathematics for Machine Learning by Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong

Module 1: Linear Algebra & Vector Spaces (The Workspace of Data)

Linear Algebra defines the "coordinate systems" and "shapes" where data lives.

1.1: Vectors and Spaces
- Topics: Vector spaces, subspaces, linear independence, span, and basis.
- Intuition: What does it mean to "span" a space? Why is a basis like a "minimal set of Lego bricks" for building any vector?
- Reading Ref: [MML] Chapter 2: Sections 2.1 - 2.5.
- Visual Ref: [3B1B-LA] Videos 1 - 4 (Vectors, Span, Linear combinations).
1.2: Linear Transformations & Matrices
- Topics: Matrices as functions, matrix multiplication, rank, null space.
- Intuition: Why is matrix multiplication actually a "space warp" (rotation, scaling, shearing)? What is the rank of a matrix (how many dimensions survive the warp)?
- Reading Ref: [MML] Chapter 2: Sections 2.6 - 2.8.
- Visual Ref: [3B1B-LA] Videos 3 - 8 (Transformations, Matrix multiplication, 3D transformations).
1.3: Inner Products and Orthogonality
- Topics: Inner products, norms, angles between vectors, projections, Gram-Schmidt orthogonalization.
- Intuition: Why is a projection an "optimal approximation" (finding the shadow of a vector)?
- Reading Ref: [MML] Chapter 3: Sections 3.1 - 3.8.
- Visual Ref: [3B1B-LA] Video 9 (Dot products) & Video 11 (Change of basis).
1.4: Spectral Theory (Eigenvalues & Eigenvectors)
- Topics: Characteristic equation, eigenvalues, eigenvectors, diagonalization.
- Intuition: What are the directions in space that only stretch and do not rotate under a transformation?
- Reading Ref: [MML] Chapter 4: Sections 4.1 - 4.4.
- Visual Ref: [3B1B-LA] Video 14 (Eigenvalues and Eigenvectors).
1.5: Singular Value Decomposition (SVD) & PCA
- Topics: Symmetric matrices, positive semidefinite (PSD) matrices, SVD derivation, low-rank approximation, Principal Component Analysis (PCA).
- Intuition: How SVD breaks down any transformation into "rotate-scale-rotate". How PCA finds the axes of maximum variance.
- Reading Ref: [MML] Chapter 4: Section 4.5 & Chapter 10 (Dimensionality Reduction via PCA).
1.6: Numerical Linear Algebra & Stability
- Topics: Solving systems without explicit matrix inverses (LU/QR decomposition), matrix conditioning (Condition Number), floating-point errors, and numerical stability.
- Intuition: Why do computers avoid calculating $\mathbf{A}^{-1}$? How do rounding errors blow up in unstable systems?
- Reading Ref: [MML] Chapter 2: Section 2.3 & Appendix methods.

Module 2: Probability, Statistics, and Information Theory

Probability is how AI handles noise, incomplete information, and predictions. Study the practical mathematics of events, inference, and distributions.

2.1: Probability Spaces & Conditioning
- Topics: Sample spaces, events, probability measures, conditional probability, Bayes' theorem, independence of events.
- Intuition: How does new information shrink our sample space? What does Bayes' theorem look like visually?
- Reading Ref: [MML] Chapter 6: Sections 6.1 - 6.3 & [PML-1] Chapter 2: Sections 2.1 - 2.2.
2.2: Random Variables & Canonical Distributions
- Topics: Random variables (discrete/continuous), CDFs, PDFs, joint distributions, marginals, conditional distributions, and Canonical Distributions (Bernoulli, Categorical, Poisson, Gaussian).
- Intuition: A random variable maps the sample space to the real line. Canonical distributions are the specific mathematical shapes we use to model real-world ML tasks (e.g., Bernoulli for binary classification, Gaussian for regression).
- Reading Ref: [MML] Chapter 6: Section 6.4 & [PML-1] Chapter 2: Sections 2.3 - 2.5.
2.3: Expectation, Moments, and Covariance
- Topics: Mathematical expectation, variance, covariance, correlation, covariance matrices, and the Multivariate Gaussian distribution.
- Intuition: Expectation as a center of mass; Covariance as how two variables "stretch" together. The multivariate Gaussian as a stretched, rotated probability dome.
- Reading Ref: [MML] Chapter 6: Section 6.5 & [PML-1] Chapter 3: Section 3.2.
2.4: Statistical Inference & Estimators
- Topics: Likelihood function, Maximum Likelihood Estimation (MLE), Maximum A Posteriori (MAP), Conjugate Priors (Beta-Binomial, Dirichlet-Multinomial), sampling distributions, Central Limit Theorem (CLT), bias/variance of estimators, and Bayesian updating.
- Intuition: MLE as "fitting the parameters to best explain the data we saw"; MAP as "balancing the data with our prior beliefs." Conjugate Priors act as mathematical cheat codes where the Posterior magically retains the exact same geometric shape as the Prior, allowing computers to actually calculate Bayesian updates.
- Reading Ref: [MML] Chapter 6: Section 6.6 & [PML-1] Chapter 4 (Statistics).
2.5: Information Theory
- Topics: Entropy, Joint/Conditional Entropy, Cross-Entropy, Kullback-Leibler (KL) Divergence, and Mutual Information.
- Intuition: Entropy as "measure of surprise"; KL Divergence as "the extra penalty of using the wrong codebook to describe our data".
- Reading Ref: [PML-1] Chapter 6.

Module 3: Calculus, Optimization, and Training Dynamics

Calculus, optimization, and training dynamics are the engines that power model training. We learn how mathematical curvature translates to training stability.

3.1: Calculus Refresh & Single-Variable Foundations
- Topics: Single-variable derivatives/integrals, chain rule, product rule, Taylor series approximation, and local curvature.
- Intuition: Taylor series as building a polynomial curve approximation around a single point.
- Reading Ref: [MML] Chapter 5: Sections 5.1 - 5.3.
3.2: Multivariable Calculus & Vector Gradients
- Topics: Partial derivatives, multivariable chain rule, gradients, Jacobians, and Hessians.
- Intuition: The gradient as the direction of steepest ascent; Jacobian as the best linear approximation of a vector function.
- Reading Ref: [MML] Chapter 5: Sections 5.4 - 5.7.
3.3: Optimization Basics & Gradient Descent
- Topics: Convex functions, local/global minima, smoothness, convexity, strong convexity intuition, and basic Gradient Descent.
- Intuition: Rolling down a hill; why learning rate acts as the step size.
- Reading Ref: [MML] Chapter 7: Sections 7.1 - 7.2.
3.4: Advanced Optimizers & Training Dynamics
- Topics: Batch vs. Stochastic (SGD) vs. Mini-batch gradient descent, Momentum, Nesterov momentum, adaptive methods (AdaGrad, RMSProp, Adam/AdamW), learning-rate schedules and warmup, saddle points, ill-conditioning, exploding/vanishing gradients, gradient clipping, and L1/L2 weight decay regularization.
- Intuition: Adam as gradient descent with a memory of previous directions (momentum) and a dynamic speed adjustment for each parameter.
- Code Experiment: Write a mini-batch solver. Implement and compare SGD, Momentum, and Adam on a toy optimization landscape (e.g. Rosenbrock function) and compare their trajectories, convergence speeds, and behaviors near saddle points.
- Reading Ref: [PML-1] Chapter 8 (Optimization).
3.5: Constrained Optimization (Intuition)
- Topics: Lagrange multipliers, basic duality intuition. (Exhaustive KKT theory is deferred to SVM study).
- Intuition: Finding the highest point on a mountain while staying on a designated path.
- Reading Ref: [MML] Chapter 7: Section 7.3.

Module 4: Core Classical ML (Theory & Practice)

Here we apply all mathematical pillars to study classical models, regularization, and model evaluation.

4.1: Model Evaluation & Regularization Theory
- Topics: Train/validation/test splits, data leakage, evaluation metrics (accuracy, precision, recall, F1, ROC-AUC), bias-variance tradeoff, overfitting, cross-validation, and L1/L2 regularization math.
- Intuition: Why does penalizing the size of the weight vector stabilize models?
- Reading Ref: [PML-1] Chapter 4 & 5.
4.2: Distance & Probability Models
- Topics: k-Nearest Neighbors (k-NN), Naive Bayes Classifier.
- Intuition: k-NN as local coordinate grouping; Naive Bayes as conditional independence assumptions.
- Reading Ref: [PML-1] Chapters 2 & 16.
4.3: Linear & Logistic Regression
- Math: Ordinary Least Squares (OLS) derivation via projections (Linear Algebra) + Maximum Likelihood Estimation (Probability).
- Reading Ref: [MML] Chapter 9 (Linear Regression) & [PML-1] Chapter 10 & 11.
4.4: Tree Models & Ensembles
- Topics: Decision Trees, Random Forests, Gradient Boosting (XGBoost intuition).
- Intuition: Trees as hierarchical axis-aligned space partitioning.
- Reading Ref: [PML-1] Chapter 18.
4.5: Clustering & Latent Variable Models
- Topics: K-Means clustering, Gaussian Mixture Models (GMMs), and the Expectation-Maximization (EM) algorithm.
- Intuition: EM as the chicken-and-egg dance: assign points to clusters (E-step), then re-calculate cluster centers (M-step).
- Reading Ref: [PML-1] Chapters 8 & 21.
4.6: Kernel Methods & Support Vector Machines (SVMs)
- Topics: SVM hyperplanes, constrained optimization (KKT/Duality), the "Kernel Trick", and Mercer's Theorem.
- Intuition: Projecting low-dimensional entangled data into a higher-dimensional space where a flat sheet can easily cut it in half.
- Reading Ref: [MML] Chapter 12 & [PML-1] Chapter 17.

Module 5: Neural Networks & Practical Training

Building upon multivariable calculus and matrix operations, we study the internals of neural networks and the practical math behind training them.

5.1: Deep Perceptrons & Backpropagation
- Topics: Multilayer Perceptrons (MLPs), activation functions, multivariable chain rule, computational graphs, and Autodiff (automatic differentiation).
- Intuition: Backpropagation as passing "blame" for the prediction error backward through the network layers.
- Reading Ref: [PML-1] Chapter 13.
5.2: Optimization & Training Internals
- Topics: Weight initialization (Xavier/He), minibatches, momentum, advanced optimizers (RMSprop, Adam), weight decay (L2), dropout, learning rate schedules, and gradient checking.
- Intuition: Adam as gradient descent with a memory of previous directions (momentum) and a dynamic speed adjustment for each parameter.
- Reading Ref: [PML-1] Chapter 8 & 13.
5.3: Normalization Techniques
- Topics: Batch Normalization, Layer Normalization.
- Intuition: Keeping the data distribution stable at every layer so subsequent layers don't have to constantly adapt to changing inputs (internal covariate shift).
- Reading Ref: [PML-1] Chapter 14.

Module 6: Generative Models & Modern Architectures

Combining all mathematical disciplines to study modern deep learning models.

6.1: Transformers & Self-Attention
- Topics: Token embeddings, positional information, linear projections to queries, keys, and values ($Q$, $K$, $V$), scaled dot-product attention formula: $$\mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$ Why the $\sqrt{d_k}$ scaling factor is critical, Multi-Head Attention, residual connections, layer normalization, feed-forward blocks, causal masking for autoregressive language models, cross-entropy training, next-token prediction loss, and attention's $O(n^2)$ sequence-length computational complexity.
- Intuition: Attention as a dynamic, coordinate-based database lookup.
- Code Experiment: Implement a single-head attention block from scratch in NumPy (handling projections, scores, masking, scaling, and output). Once complete, verify the dimensions of $Q, K, V$, masks, and outputs match expected shapes.
- Reading Ref: [PML-1] Chapter 15.
6.2: Generative Models (VAEs & GANs)
- Math: Latent variables, Evidence Lower Bound (ELBO) maximization, Jensen's inequality, minimax games, Nash equilibrium.
- Reading Ref: [PML-2] Chapters 16 & 18.
6.3: Diffusion Models (DDPM)
- Math: Forward stochastic diffusion, reverse score-matching, noise estimation.
- Reading Ref: [PML-2] Chapter 20.
6.4: State-Space Systems & Mamba
- Math: Linear time-invariant systems, discretization, parallel scans.
- Reading Ref: Modern state-space research literature.

vijayanant/curated_curriculum.md

Select an option

No results found