Generative Model

You have GANs and Diffusion Model that can generate data. There’s also GPT-3.

Types of generative models (source):

1. Likelihood-based models: approximate the probability distribution $p(x)$. Ex:
   • Autoregressive Model: $p(x)={\prod }_{t=1}^{T}p(x_t\mid x_{<t})$
   • Variational Antoencoder (VAE): $p(x)=\int p(x\mid z)p(z)dz$
   • Flow-Based Models: $x=f(z),p(x)=p(z)\left|\det \left(\frac{\partial f^{-1}}{\partial x}\right)\right|$
2. Implicit / Score-Based Models: do not model $p(x)$ explicitly, or use alternative objectives to generate samples. Ex:
   • Generative Adversarial Network (GAN): ${\min }_G{\max }_D{\mathbb{E}}_{x\sim p_{\text{data}}}[\log D(x)]+{\mathbb{E}}_{z\sim p(z)}[\log (1-D(G(z)))]$
   • Diffusion Models: $x_{t-1}={\alpha }_t(x_t-\gamma {ϵ}_{\theta }(x_t,t))+\mathcal{N}(0,{\sigma }_t^2)$.
   • Energy-Based Model (EBM): $p(x)\propto \exp (-E(x))$

I still don't fully get the difference....?

It’s not about starting from noise and then denoisinig.

What is p(x)?

• $p(x)$ is the probability density (or mass) of a data point $x$ under your model.
• It tells you how likely your model thinks that $x$ is.

Does this matter anymore?

Like just slap a transformer and feed data, does this really matter? The architecture is becoming standardized (transformers), but the generative modeling paradigm — diffusion vs autoregressive vs GAN — still shapes what the model does and how it learns.

https://lilianweng.github.io/posts/2021-07-11-diffusion-models/ https://yang-song.net/blog/2021/score/

• From lilian wag’s blog, it seems that all of these are really similar.

Taxonomy (CS231n 2025 Lec 13)

CS231n uses the Goodfellow 2017 tree — splits by how the model relates to the density $p(x)$, with the normalization constraint $\int p(x)dx=1$ implying different values of $x$ compete for probability mass:

                  Generative models
                  /               \
         Explicit density     Implicit density
         (model computes      (can only sample
             p(x))             from p(x))
          /        \              /        \
    Tractable   Approximate    Direct     Indirect
      ↓             ↓            ↓           ↓
Autoregressive     VAE          GAN      Diffusion

• Tractable — actually evaluate $p(x)$ (e.g. Autoregressive via chain rule).
• Approximate — can’t evaluate $p(x)$ exactly but bound/approximate it (VAE: maximize ELBO instead of $\log p(x)$).
• Direct implicit — single-shot sample from a noise vector (GAN generator).
• Indirect implicit — iterative sampling procedure (diffusion: denoise $T$ times).

Discriminative vs generative vs conditional-generative:

	Models	Used for
Discriminative	$p(y\mid x)$	classification
Generative	$p(x)$	density / sampling / anomaly detection
Conditional generative	$p(x\mid y)$	class-conditional / text-to-image generation

Bayes ties them: $p(x\mid y)=p(y\mid x)p(x)/p(y)$, so a generative model + a class prior gives a discriminative one.

Source

CS231n 2025 Lec 13 slides ~35–47, 113–115 (discriminative/generative/conditional split, density normalization, taxonomy tree, Goodfellow attribution).

Generative Video Models

Do generative video models understand physics?

• https://arxiv.org/pdf/2501.09038 this paper proves NO

It’s just learning to correlate frames, but it has no understanding of the world’s physics.