Generative Model

Variational Autoencoder (VAE)

Latent variable model trained with variational inference: $p(x)=\int p(x\mid z)p(z)dz$

This is a variant of the Autoencoder that is much more powerful, which uses distributions to represent features in its bottleneck. There are issues that arise with Backprop, but they overcome it with a reparametrization trick.

Resources

• https://lilianweng.github.io/posts/2018-08-12-vae/

“VAE is deeply rooted in the methods of variational bayesian and graphical model.”

• #todo I don’t fully understand this

It's basically an Autoencoder but we add gaussian noise to latent variable z?

Key difference:

• Regular Autoencoder
  • Input → Encoder → Fixed latent representation → Decoder → Reconstruction.
• VAE
  • Input → Encoder → Latent distribution → Sample from distribution (adds Gaussian noise via reparameterization trick) → Decoder → Reconstruction

Process

Forward Pass (Encoding → Sampling → Decoding)

1. Encoder:
   Input data $x$, outputs parameters (mean and variance) of latent distribution $z$:

$$q_{\varphi }(z|x)=\mathcal{N}(z;{\mu }_{\varphi }(x),{\sigma }_{\varphi }^2(x))$$

2. Reparameterization Trick:
   Differentiably sample latent variable $z$:

$$z={\mu }_{\varphi }(x)+{\sigma }_{\varphi }(x)\odot ϵ,ϵ\sim \mathcal{N}(0,I)$$

3. Decoder:
   Reconstruct data from sampled latent vector $z$:

$$p_{\theta }(x|z)$$

Loss Function (Negative ELBO):
Optimize encoder and decoder parameters by minimizing:

$$\mathcal{L}(\varphi ,\theta )=-{\mathbb{E}}_{q_{\varphi }(z|x)}[\log p_{\theta }(x|z)]+D_{\text{KL}}(q_{\varphi }(z|x)\Vert p(z))$$