Generative Model

Variational Autoencoder (VAE)

Latent variable model trained with variational inference: $p(x)=\int p(x\mid z)p(z)dz$

I got the intuition for VAEs here https://chatgpt.com/share/693a58cf-e2e4-8002-9fea-eb7fad7817b1

• g variational autoencoder: tries to map original distribution to a Gaussian, and also maps back to original distribution. Each is encoded in the loss function.

This is a variant of the Autoencoder that is much more powerful, which uses distributions to represent features in its bottleneck. There are issues that arise with Backprop, but they overcome it with a reparametrization trick.

Resources

• https://lilianweng.github.io/posts/2018-08-12-vae/
• https://arxiv.org/pdf/1906.02691

It's basically an Autoencoder but we add gaussian noise to latent variable z?

Key difference:

• Regular Autoencoder
  • Input → Encoder → Fixed latent representation → Decoder → Reconstruction.
• VAE
  • Input → Encoder → Latent distribution → Sample from distribution (adds Gaussian noise via reparameterization trick) → Decoder → Reconstruction

Variational autoencoders provide a principled framework for learning deep latent-variable models and corresponding inference models.

Process

Forward Pass (Encoding → Sampling → Decoding)

1. Encoder:
   Input data $x$, outputs parameters (mean and variance) of latent distribution $z$:

$$q_{\varphi }(z|x)=\mathcal{N}(z;{\mu }_{\varphi }(x),{\sigma }_{\varphi }^2(x))$$

2. Reparametrization Trick: Differentiably sample latent variable $z$:

$$z={\mu }_{\varphi }(x)+{\sigma }_{\varphi }(x)\odot ϵ,ϵ\sim \mathcal{N}(0,I)$$

3. Decoder:
   Reconstruct data from sampled latent vector $z$:

$$p_{\theta }(x|z)$$

Loss Function (Negative ELBO):
Optimize encoder and decoder parameters by minimizing:

$$\mathcal{L}(\varphi ,\theta )=-{\mathbb{E}}_{q_{\varphi }(z|x)}[\log p_{\theta }(x|z)]+D_{\text{KL}}(q_{\varphi }(z|x)\Vert p(z))$$

Notes from the guide

The VAE can be viewed as two coupled, but independently parameterized models:

1. encoder (recognition model)
2. decoder (generative model)

Motivation

We want to maximize the log likelihood ${\mathbb{E}}_{x\sim p_{unknwown}}[\log p_{\theta }(x)]$ To make this a generative process, we want it conditioned on some known probability distribution (so then it becomes mapping probability distribution $p(z)$ to $p(x)$) (else it’s just like an Autoencoder, always deterministic).

We expand out $p_{\theta }(x)$ $p_{\theta }(x)=\int p_{\theta }(x,z)dz=\int p_{\theta }(x|z)p_{\theta }(z)dz$

• However, this is NOT tractatable. It does not have a closed form solution. Trying every single $z$ value for $p(z)$ is not tractable, $p_{\theta }(x|z)$ is implemented as a neural net.

I'm confused on what is tractable and what is not tractable?

• $p(z)$ is tractable - Simple prior to generate $z$ (i.e. unit gaussian)
• $p(x|z)$ is tractable- simple neural net to general $x$ conditioned on $z$
• $p(x)$ is NOT tractable - need to integrate over all $z$
• $p(z|x)$ is NOT tractable because it needs $p(x)$ applying Bayes Rule

So what do we do? We approximate the intractable posterior $p(z|x)$ with $q_{\varphi }(z|x)$ and maximize a tractable lower bound (ELBO) on the true log-likelihood.

We approximate $q_{\varphi }(z|x)\approx p(z|x)$ and then optimize by maximizing

$p_{\theta }(x)\approx {\mathbb{E}}_{z\sim q_{\varphi }(z|x)}\log p_{\theta }(x|z)$

Variants

• cVAE