Variational Autoencoders (VAE) are extremely appealing as they allow for learning complicated distributions taking advantage of recent progress in gradient descent algorithms and accelerated processing with GPUs. The latent space of regular autoencoders are typically very sparse and unrestricted, making it difficult to generate data robust to variations of the latent variables. Traditional VAEs use normal priors regularize the latent space however these assuptions are unsound when it comes to complex data such as images or text.
We propose Dirichlet priors for a multinomial latent space. This latent space allows us to explore the data by interpreting
We explore various methods for regularizing an Autoencoder so that the latent space is multinomial, and compare how these methods perform on a classification task using the latent space as the input to a k-nearest neighbours algorithm.