2017-10-20 DL Assignment: Generative Models

generative-models-assignment

# Generative Models

## Introduction
Generative models are models that learn the *distribution* of the data.

Suppose we have a collection of N D-Dimensional points: $\{x_1, ..., x_N\}$.  Each, $x_i$ might represent a vector of pixels in an image, or the words in a sentence.   

In generative modeling, we imagine that these points are samples from a D-dimensional probability distribution.  The distribution represents whatever real-world process was used to generate that data.  Our objective is to learn the parameters of this distribution.  This allows us to do things like 

- Generate *new* data samples, given our estimate of the data distribution.
- Estimate the probability that some new piece of data was generated by your model.

In this assignment, we will work with the MNIST dataset (get it at: http://www.deeplearning.net/tutorial/gettingstarted.html).  For the first part of this assignment, we will only use the images, and not their corresponding labels.

## Problem 1: Training a Naive Bayes model with Expectation Maximization

We now want to learn a *model* of the MNIST digits.

We are going to use Expectation Maximization to train a Naive Bayes model on the MNIST Digits.  For the first part, we will only train on digits corresponding to the labels $\{0, 1, 2, 3, 4\}$.  For simplicity we will consider the images to be binary variables (so we will discretize pixel values to 0 or 1)

Read the tutorial at (TODO: Get source: maybe http://www.cs.columbia.edu/~mcollins/em.pdf)?

1. From the MNIST dataset, select digits whose labels are in $\{0, 1, 2, 3, 4\}$, and binarize the images.  Keep the training and test digits separate for later.
2. Using the training digits, train a Naive Bayes model, as described in XXXX, on the selected digits with labels in $\{0, 1, 2, 3, 4\}$.
3. Evaluate the average log probability of your training and test data under the model.
4. Now, select digits wholse labels are in $\{5, 6, 7, 8, 9\}$, and binarize them as before.  Evaluate the average log-probabilty of these digits under your model.
5. Suppose you were to train another model of digits $\{5, 6, 7, 8, 9\}$.  Can you describe how you might use these two models to build a classifier, which tells you whether a digit has label in $\{0, 1, 2, 3, 4\}$ or $\{5, 6, 7, 8, 9\}$?



A Naive Bayes model is one of the simplest generative models.




## Problem 2: Training a Variational Autoencoder