Evidence Lower Bound (ELBO)

Used in VAE.

$$\text{ELBO}(q)={\mathbb{E}}_{q(z)}[\log p_{\theta }(x\mid z)]-\text{KL}(q(z)\Vert p(z))$$

Uses multiple concepts for the derivation:

• KL Divergence
• Jensen’s Inequality
• Bayes Rule

Why ELBO?

ELBO is useful because it provides a guarantee on the worst-case for the log-likelihood of some distribution (e.g. $p(X)$) which models a set of data.

Deriving the ELBO

Start with the intractable log-likelihood:

$$\log p_{\theta }(x)=\log \int p_{\theta }(x,z)dz$$

Insert any distribution $q(z)$:

$$\log p_{\theta }(x)=\log \int q(z)\frac{p_{\theta }(x,z)}{q(z)}dz$$

Interpret the integral as an expectation:

$$\log p_{\theta }(x)=\log {\mathbb{E}}_{z\sim q(z)}\left[\frac{p_{\theta }(x,z)}{q(z)}\right]$$

Apply Jensen’s Inequality ($\log \mathbb{E}[Y]\ge \mathbb{E}[\log Y]$):

$$\log p_{\theta }(x)\ge {\mathbb{E}}_{z\sim q(z)}\left[\log \frac{p_{\theta }(x,z)}{q(z)}\right]$$

Define the ELBO:

$$\text{ELBO}(q)={\mathbb{E}}_{q(z)}[\log p_{\theta }(x,z)]-{\mathbb{E}}_{q(z)}[\log q(z)]$$

Expand $p_{\theta }(x,z)=p_{\theta }(x\mid z)p(z)$:

$$\text{ELBO}(q)={\mathbb{E}}_{q(z)}[\log p_{\theta }(x\mid z)]+{\mathbb{E}}_{q(z)}[\log p(z)-\log q(z)]$$

The last two terms form a KL divergence:

$${\mathbb{E}}_{q(z)}[\log p(z)-\log q(z)]=-\text{KL}(q(z)\Vert p(z))$$

Final ELBO expression:

$$\boxed{\text{ELBO}(q)={\mathbb{E}}_{q(z)}[\log p_{\theta }(x\mid z)]-\text{KL}(q(z)\Vert p(z))}$$

**In VAEs, we use $q(z)=q_{\varphi }(z\mid x)$.

Why is this a lower bound?

There is an exact identity:

$$\log p_{\theta }(x)=\text{ELBO}(q)+\text{KL}(q(z)\Vert p_{\theta }(z\mid x))$$

Since the KL divergence is always nonnegative:

$$\text{KL}(q(z)\Vert p_{\theta }(z\mid x))\ge 0$$

we get:

$$\boxed{\log p_{\theta }(x)\ge \text{ELBO}(q)}$$

The gap is exactly:

$$\log p_{\theta }(x)-\text{ELBO}(q)=\text{KL}(q(z)\Vert p_{\theta }(z\mid x))$$

Interpretation:

• ELBO is never larger than the true log-likelihood.
• Maximizing ELBO makes $q(z|x)$ approach the true posterior.
• As $q(z\mid x)$ improves, the KL gap shrinks.
• If $q(z\mid x)=p_{\theta }(z\mid x)$, the ELBO is exactly equal to $\log p_{\theta }(x)$.

Thus ELBO is the best computable surrogate for the true log-likelihood.