Evidence Lower Bound (ELBO)

Used in VAE.

$$\text{ELBO}(q)={\mathbb{E}}_{q(z)}[\log p_{\theta }(x\mid z)]-\text{KL}(q(z)\Vert p(z))$$

Uses multiple concepts for the derivation:

• KL Divergence
• Jensen’s Inequality
• Bayes Rule

Why ELBO?

ELBO is useful because it provides a guarantee on the worst-case for the log-likelihood of some distribution (e.g. $p(X)$) which models a set of data.

Deriving the ELBO

Start with the intractable log-likelihood:

$$\log p_{\theta }(x)=\log \int p_{\theta }(x,z)dz$$

Insert any distribution $q(z)$:

$$\log p_{\theta }(x)=\log \int q(z)\frac{p_{\theta }(x,z)}{q(z)}dz$$

Interpret the integral as an expectation:

$$\log p_{\theta }(x)=\log {\mathbb{E}}_{z\sim q(z)}\left[\frac{p_{\theta }(x,z)}{q(z)}\right]$$

Apply Jensen’s Inequality ($\log \mathbb{E}[Y]\ge \mathbb{E}[\log Y]$):

$$\log p_{\theta }(x)\ge {\mathbb{E}}_{z\sim q(z)}\left[\log \frac{p_{\theta }(x,z)}{q(z)}\right]$$

Define the ELBO:

$$\text{ELBO}(q)={\mathbb{E}}_{q(z)}[\log p_{\theta }(x,z)]-{\mathbb{E}}_{q(z)}[\log q(z)]$$

Expand $p_{\theta }(x,z)=p_{\theta }(x\mid z)p(z)$:

$$\text{ELBO}(q)={\mathbb{E}}_{q(z)}[\log p_{\theta }(x\mid z)]+{\mathbb{E}}_{q(z)}[\log p(z)-\log q(z)]$$

The last two terms form a KL divergence:

$${\mathbb{E}}_{q(z)}[\log p(z)-\log q(z)]=-\text{KL}(q(z)\Vert p(z))$$

Final ELBO expression:

$$\boxed{\text{ELBO}(q)={\mathbb{E}}_{q(z)}[\log p_{\theta }(x\mid z)]-\text{KL}(q(z)\Vert p(z))}$$

**In VAEs, we use $q(z)=q_{\varphi }(z\mid x)$.

Why is this a lower bound?

There is an exact identity:

$$\log p_{\theta }(x)=\text{ELBO}(q)+\text{KL}(q(z)\Vert p_{\theta }(z\mid x))$$

Since the KL divergence is always nonnegative:

$$\text{KL}(q(z)\Vert p_{\theta }(z\mid x))\ge 0$$

we get:

$$\boxed{\log p_{\theta }(x)\ge \text{ELBO}(q)}$$

The gap is exactly:

$$\log p_{\theta }(x)-\text{ELBO}(q)=\text{KL}(q(z)\Vert p_{\theta }(z\mid x))$$

Interpretation:

• ELBO is never larger than the true log-likelihood.
• Maximizing ELBO makes $q(z|x)$ approach the true posterior.
• As $q(z\mid x)$ improves, the KL gap shrinks.
• If $q(z\mid x)=p_{\theta }(z\mid x)$, the ELBO is exactly equal to $\log p_{\theta }(x)$.

Thus ELBO is the best computable surrogate for the true log-likelihood.

Alternative derivation (CS231n 2025 Lec 13)

CS231n derives ELBO without invoking Jensen — pure algebra on Bayes’ rule. Start from

$\log p_{\theta }(x)=\log \frac{p_{\theta }(x\mid z)p(z)}{p_{\theta }(z\mid x)}$

Multiply top and bottom by $q_{\varphi }(z\mid x)$:

$=\log \frac{p_{\theta }(x\mid z)p(z)q_{\varphi }(z\mid x)}{p_{\theta }(z\mid x)q_{\varphi }(z\mid x)}=\log p_{\theta }(x\mid z)-\log \frac{q_{\varphi }(z\mid x)}{p(z)}+\log \frac{q_{\varphi }(z\mid x)}{p_{\theta }(z\mid x)}$

$\log p_{\theta }(x)$ doesn’t depend on $z$, so we can wrap the whole RHS in ${\mathbb{E}}_{z\sim q_{\varphi }(z\mid x)}$ without changing anything:

$\log p_{\theta }(x)={\mathbb{E}}_{z\sim q_{\varphi }}[\log p_{\theta }(x\mid z)]-D_{\text{KL}}(q_{\varphi }(z\mid x)\Vert p(z))+D_{\text{KL}}(q_{\varphi }(z\mid x)\Vert p_{\theta }(z\mid x))$

The last term is a KL, so it’s $\ge 0$ — and we can’t compute it (it depends on the intractable true posterior $p_{\theta }(z\mid x)$). Drop it to get a lower bound:

$\log p_{\theta }(x)\ge \underset{\text{reconstruction}}{\underbrace{{\mathbb{E}}_{z\sim q_{\varphi }}[\log p_{\theta }(x\mid z)]}}-\underset{\text{prior matching}}{\underbrace{D_{\text{KL}}(q_{\varphi }(z\mid x)\Vert p(z))}}$

This is the VAE training objective — both terms are tractable (closed-form KL for Gaussians, Monte-Carlo reconstruction via the reparametrization trick).

Equivalent framing to the Jensen derivation above, but this one makes the “gap = KL to the true posterior” identity visible on a single line.

Source

CS231n 2025 Lec 13 slides ~92–102 (Bayes → multiply by $q_{\varphi }$ → split into three log terms → wrap in expectation → identify two KLs → drop the posterior KL to get ELBO).