Generative Model

Diffusion Model

Add noise gradually and learn to reverse the process: $x_{t-1}={\alpha }_t(x_t-\gamma {ϵ}_{\theta }(x_t,t))+\mathcal{N}(0,{\sigma }_t^2)$. Can be interpreted as: $x^{\prime }=x-\gamma \nabla E(x)$

See DDPM for actual implementation detail notes. There’s also

Models

• DALL-E 2
• Stable Diffusion
• Imagen

• https://medium.com/data-science/diffusion-models-made-easy-8414298ce4da

Diffusion Models in Vision: https://arxiv.org/pdf/2006.11239.pdf

I worked on one for HackWestern.

• Stable Diffusion Animation: https://replicate.com/andreasjansson/stable-diffusion-animation/examples#nby3ut2bljhixb2gejkljmcib4

Walkthrough (CS231n 2025 Lec 14)

Diffusion terminology is famously a mess (DDPM, score matching, SDEs, flow matching, rectified flow, ε-prediction, v-prediction, …). CS231n 2025 introduces the modern Rectified Flow formulation first because it’s the cleanest, then shows the rest as a generalized diffusion template.

Intuition

• Pick a simple noise distribution $z\sim p_{\text{noise}}$ (usually $\mathcal{N}(0,I)$).
• Corrupt data $x$ at varying noise levels $t\in [0,1]$ to get $x_t$ ($t=0$ no noise, $t=1$ full noise).
• Train a network $f_{\theta }(x_t,t)$ to remove a little noise.
• At inference, sample $x_1\sim p_{\text{noise}}$ and apply $f_{\theta }$ many times in sequence to land at a clean $x_0$.

Rectified Flow training

The cleanest concrete instance — straight-line interpolation between data and noise (same idea as Flow Matching). Each iteration: $z\sim p_{\text{noise}},x\sim p_{\text{data}},t\sim \text{Uniform}[0,1]$ $x_t=(1-t)x+tz,v_{\text{gt}}=z-x$

Train the network to predict the constant velocity $v$: $L=\Vert f_{\theta }(x_t,t)-v{\Vert }_2^2$

Whole training loop is a few lines:

for x in dataset:
    z = torch.randn_like(x)
    t = random.uniform(0, 1)
    xt = (1 - t) * x + t * z
    v = model(xt, t)
    loss = (z - x - v).square().sum()

Rectified Flow sampling

Pick number of steps $T$ (often $T=50$). Start from noise and Euler-step backward:

sample = torch.randn(x_shape)
for t in torch.linspace(1, 0, num_steps):
    v = model(sample, t)
    sample = sample - v / num_steps

Conditional Rectified Flow

Same loop, model takes a condition $y$ (class label, text embedding, etc.):

for (x, y) in dataset:
    ...
    v = model(xt, y, t)
    loss = (z - x - v).square().sum()

Classifier-Free Guidance (CFG)

Question: how much should we emphasize conditioning $y$? CFG (Ho & Salimans 2022) trick: during training, randomly drop $y$ (replace with $y_{\varnothing }$) ~50% of the time. Now the same network is both conditional and unconditional.

At sampling, evaluate two velocities at each step and combine:

• $v^{\varnothing }=f_{\theta }(x_t,y_{\varnothing },t)$ — points toward $p(x)$
• $v^y=f_{\theta }(x_t,y,t)$ — points toward $p(x\mid y)$
• $v^{\text{cfg}}=(1+w)v^y-w{v}^{\varnothing }$ — extrapolates more toward $p(x\mid y)$

Step according to $v^{\text{cfg}}$. “Classifier-free” because earlier methods used a separately-trained discriminative classifier $p(y\mid x)$ to compute step direction ${\partial }_x\log p(y\mid x)$ (Dhariwal & Nichol 2021). CFG is used everywhere in practice — critical for high-quality outputs — but doubles sampling cost (two model evals per step).

Optimal prediction & noise schedules

What is the network actually trying to predict?

• Many $(x,z)$ pairs can produce the same $x_t$ — the network must average over all of them.
• $t=1$ (full noise): optimal $v$ is the mean of $p_{\text{data}}$ — easy.
• $t=0$ (no noise): optimal $v$ is the mean of $p_{\text{noise}}$ — easy.
• Middle noise is hardest, most ambiguous — but uniform $t$ sampling gives equal weight to all noise levels.

Solution: use a non-uniform noise schedule that emphasizes middle $t$. Common choice — logit-normal sampling (Esser et al. 2024 “Scaling Rectified Flow Transformers”):

t = torch.randn(()).sigmoid()

For high-resolution data, also shift to higher noise to account for pixel correlations.

Latent diffusion (LDMs) — see Latent Diffusion Model

Naive diffusion doesn’t scale to high-resolution pixels. Modern pipelines diffuse in a compressed latent (typically a VAE + GAN-trained autoencoder, $D=8$, $C=16$, so $256\times 256\times 3$ → $32\times 32\times 16$ latents). LDMs are the standard form today.

Diffusion Transformer (DiT) — Peebles & Xie ICCV 2023

Diffusion uses standard Transformer blocks; the design question is how to inject conditioning (timestep $t$, text, class label):

• Predict scale/shift (adaLN-Zero) — most common for the diffusion timestep $t$.
• Cross-attention / joint attention — common for text and image conditioning.
• Concatenate on sequence dimension — alternative.

Text-to-Image example: FLUX.1 [dev]

• Text encoder: T5 + CLIP
• VAE encoder/decoder: 8× downsampling
• Diffusion: 12B-parameter DiT, $2\times 2$ patchify → $64\times 64=1024$ image tokens
• $1024\times 1024\times 3$ image ↔ $128\times 128\times 16$ latents

Text-to-Video — same recipe, video latents

Add a temporal axis: latents are $t\times h\times w\times c$. Meta MovieGen (2024): 30B-param DiT, $1\times 2\times 2$ patchify → 76K tokens, outputs $257\times 1024\times 576\times 3$ video. The era of large video diffusion models exploded in 2024 (Sora, Gen3, Dream Machine, MovieGen, CogVideoX, Mochi, Veo 2, Wan, Cosmos, Kling 2.0 …).

Diffusion distillation

Sampling needs ~30–50 model evals per image — slow. Distillation algorithms (Progressive Distillation 2022, Consistency Models 2023, Adversarial Diffusion Distillation 2024, Multistep Distillation via Moment Matching 2025) collapse this down to a few steps or even 1 step, and can bake CFG into the student.

Generalized Diffusion

All diffusion variants share a template — only the coefficient functions differ: $x_t=a(t)x+b(t)z,y_{\text{gt}}=c(t)x+d(t)z$ $y_{\text{pred}}=f_{\theta }(x_t,t),L=\Vert y_{\text{gt}}-y_{\text{pred}}{\Vert }_2^2$

Variant	$a(t)$	$b(t)$	$c(t)$	$d(t)$
Rectified Flow	$1-t$	$t$	$-1$	$1$
Variance Preserving (VP) (Song et al. ICLR 2021)	$\sqrt{\sigma (t)}$	$\sqrt{1-\sigma (t)}$	varies	varies
Variance Exploding (VE) (Karras et al. NeurIPS 2022)	$1$	$\sigma (t)$	varies	varies

Prediction targets:

• x-prediction: $y_{\text{gt}}=x$ ($c=1,d=0$)
• ε-prediction (DDPM): $y_{\text{gt}}=z$ ($c=0,d=1$)
• v-prediction (Salimans & Ho 2022): $y_{\text{gt}}=b(t)z-a(t)x$ ($c=b(t),d=-a(t)$)

Coefficients are usually picked through some mathematical formalism (SDEs, ODEs, score matching).

Other perspectives on diffusion

Sander Dieleman’s blog post (https://sander.ai/2023/07/20/perspectives.html) lists 8 equivalent views:

1. Diffusion = autoencoders
2. Diffusion = deep latent variable models (variational lower bound, same as VAE — see Deep Latent Variable Model)
3. Diffusion learns the score function $s(x)={\nabla }_x\log p(x)$ — a vector field pointing toward high-density regions; train a net to approximate the score of $p_{\text{data}}$. See Score Matching.
4. Diffusion solves reverse SDEs $dx=f(x,t)dt+g(t)dw$
5. Diffusion = flow-based models — see Flow Matching, Flow-Based Model
6. Diffusion = recurrent neural networks (iterated function)
7. Diffusion = autoregressive models (over noise levels)
8. Diffusion estimates expectations

Autoregressive models strike back

Pixel-AR was abandoned because raw pixels are too long a sequence (3M for 1024² RGB). But autoregressive models work great on (discrete) latents: train a VQ-VAE-style encoder/decoder with discrete latents (van den Oord 2017, Razavi VQ-VAE-2, ESSER VQGAN, Yu 2022 Parti), then run an AR Transformer over the latent token sequence.

Source

CS231n 2025 Lec 14 slides ~36–123 (intuition slide, Rectified Flow training+sampling+conditional, CFG, optimal prediction + noise schedules + logit-normal, LDMs end-to-end including VAE+GAN+diffusion combo, DiT conditioning variants, text-to-image FLUX, text-to-video MovieGen + 2024 video era timeline, distillation, generalized diffusion template + VP/VE/x/ε/v variants, latent-variable / score / SDE perspectives, AR-strikes-back via discrete latents).

Related

• Latent Diffusion Model
• Flow Matching
• Score Matching
• Generative Adversarial Network
• Variational Autoencoder
• Deep Latent Variable Model
• DDPM