Variational Inference

Resources

• The Variational Approximation for Bayesian Inference
• https://chrischoy.github.io/posts/Expectation-Maximization-and-Variational-Inference/
• https://matrixmaster.me/blog/2023/variational-inf-1/

Writing down the derivation as shown in the blog article

Variational Bayes

A specific application of variational inference to Bayesian inference problems.

I first learned about these reading lilian’s blog on VAEs.

Concepts

• Variational Distribution

In Bayesian statistics, the posterior is often intractable:

$$p(\theta \mid D)\propto p(D\mid \theta )p(\theta )$$

Variational Bayes uses VI to approximate this posterior $p(\theta \mid D)$.

So VB = VI applied to Bayesian inference.

You’ll see this pretty often $logp(x)=ELBO(q)+KL(q(z)\Vert p(z\mid x))$ The goal is to maximize ELBO.