Generative Model

Generative Adversarial Network (GAN)

A GAN trains a generator and discriminator in a minimax game so the generator learns to produce samples indistinguishable from real data.

${\min }_G{\max }_D{\mathbb{E}}_{x\sim p_{\text{data}}}[\log D(x)]+{\mathbb{E}}_{z\sim p(z)}[\log (1-D(G(z)))]$

where:

• $G$ is the generator mapping noise $z$ to a sample
• $D$ is the discriminator estimating the probability a sample is real

Intuition

A two-player game between a forger and a detective. The generator forges samples from noise, the discriminator tries to tell forgeries from real data, and both improve by playing each other. At Nash equilibrium the forgeries are indistinguishable, i.e. $p_G=p_{\text{data}}$. Notice there’s no explicit density to write down: we never compute $p(x)$, we just need to sample from it, which is exactly what $G$ does.

To steven: you’re probably looking for A Neural Representation of Sketch Drawings and https://quickdraw.withgoogle.com/.

Resources:

• https://developers.google.com/machine-learning/gan
• Intro video by Computerphile
• Training tips: https://github.com/soumith/ganhacks
• Applications: https://github.com/nashory/gans-awesome-applications#3d-object-generation
• Tutorial: https://www.youtube.com/watch?v=Mng57Tj18pc&ab_channel=DigitalSreeni
• Course: https://github.com/johnowhitaker/aiaiart

GANs are composed of two Neural Nets:

• The generator network creates new synthetic images trying to fool the discriminator
• The discriminator network learns to tell a fake, synthesized image from a real one

The competition between both networks allows them to improve, until the generator becomes so good that fake samples cannot be distinguished from real ones.

Examples: DCGAN, StyleGAN, CycleGAN

Mathematical Formalization

Formally, the generative model is pitted against an adversary: a discriminative model that learns to determine whether a sample is from the model distribution or the data distribution.

$D$ and $G$ play the following two-player minimax game with value function $V(G,D)$: ${\min }_G{\max }_{D}V(D,G)={\mathbb{E}}_{x\sim p_{data}(x)}[\log D(x)]+E_{z\sim p_z(z)}[\log (1-D(G(z)))]$

where:

• $D$ and $G$ are differentiable functions represented by an MLP
• $D(x)$ is the probability that $x$ came from the data rather than $p_g$
• $D$ is trained to maximize the probability of assigning the correct label to both training examples and samples from $G$
• $G$ takes $z$ as input and outputs an image, trained to minimize $\log (1-D(G(z)))$

With a GAN, given enough iterations, $p_z$ converges to $p_{data}$. In other words, from a random vector $z$, the network $G$ can synthesize an image $G(z)$ that resembles one drawn from the true distribution $p_{data}$.

The discriminator is a traditional CNN. The generator takes in random noise and produces an output.

Wasserstein GAN

Wasserstein Metric GAN: https://www.youtube.com/watch?v=xs9uibPODGk&ab_channel=Oneworldtheoreticalmachinelearning

Wasserstein GAN Paper: https://arxiv.org/pdf/1701.07875.pdf

Walkthrough (CS231n 2025 Lec 14)

Setup

GAN is the implicit-direct branch of the Goodfellow taxonomy: no explicit density $p(x)$, just sample. Sample $z\sim p(z)$ (e.g. $\mathcal{N}(0,I)$), pass through generator $G$, get $x=G(z)$ from implicit distribution $p_G$. The goal is $p_G=p_{\text{data}}$, but we have no way to compute $p_G$, so we can’t do MLE. Trick: train a discriminator $D$ to classify real vs fake, then use it as the loss for $G$.

Minimax objective

${\min }_G{\max }_{D}V(G,D)={\mathbb{E}}_{x\sim p_{\text{data}}}[\log D(x)]+{\mathbb{E}}_{z\sim p(z)}[\log (1-D(G(z)))]$

• Inner max: $D$ wants to push real samples toward 1 and fake toward 0
• Outer min: $G$ wants $D(G(z))\to 1$ (fool the discriminator)

Trained via alternating gradient updates: $D+={\alpha }_D{\nabla }_{D}V$, then $G-={\alpha }_G{\nabla }_{G}V$.

No single loss curve to monitor

You’re chasing a saddle point, not minimizing a function. Loss values on their own don’t tell you whether training is working

Saturation problem & non-saturating fix

At the start of training, $G$ is bad, so $D(G(z))\approx 0$ and $\log (1-D(G(z)))$ is flat with near-zero gradient. $G$ can’t learn from a vanishing signal.

Fix (Goodfellow 2014): instead of minimizing $\log (1-D(G(z)))$, train $G$ to maximize $\log D(G(z))$. Same direction (fool $D$), but the gradient is large precisely when $D$ is winning. Now standard.

Optimal discriminator

For fixed $G$, the optimal $D$ has a closed form: $D_G^{\ast }(x)=\frac{p_{\text{data}}(x)}{p_{\text{data}}(x)+p_G(x)}$

Plugging back into the outer min, the global optimum is $p_G=p_{\text{data}}$. If $D$ is optimal at every step, $G$ converges to the data distribution. Caveats in practice: finite-capacity nets, no convergence guarantee from alternating SGD, mode collapse.

Read $D^{\ast }$ as a likelihood ratio: at points where real data is more likely than fake, $D^{\ast }>1/2$; where fake is more likely, $D^{\ast }<1/2$. The only way for $D^{\ast }$ to be exactly $1/2$ everywhere (maximally confused detective) is $p_G=p_{\text{data}}$.

DC-GAN (Radford et al. ICLR 2016)

First architecture to make GANs work on non-toy data: strided/transposed convolutions, BatchNorm, no fully-connected layers, ReLU/LeakyReLU. Aside: Alec Radford later did GPT-1 and GPT-2.

StyleGAN (Karras et al. CVPR 2019)

Two-stage generator: a mapping network $f:z\to w$ produces a style vector, and a synthesis network $g$ generates the image, conditioning every layer on $w$ via AdaIN (Adaptive Instance Norm): $\text{AdaIN}(x,w,b{)}_i=w_i\cdot \frac{x_i-\mu (x)}{\sigma (x)}+b_i$

Per-layer style injection lets you control coarse vs fine attributes independently.

Latent-space interpolation

Linearly interpolate $z_t=t{z}_0+(1-t)z_1$ and decode $x_t=G(z_t)$. The output sweeps smoothly between the two endpoints. StyleGAN3 cat morphs are the canonical demo.

Summary: pros & cons

• Pros: simple objective, very high sample quality, fast single-step generation
• Cons: no loss curve to monitor, training is unstable (mode collapse, vanishing $D$ gradients), hard to scale, no explicit $p(x)$ for likelihood evaluation
• Era: GANs were the go-to image generator from ~2016-2021, then displaced by diffusion

From CS231n 2025 Lec 14 slides ~14-35 (minimax setup, alternating updates, saturation problem and non-saturating fix, optimal discriminator derivation, DC-GAN, StyleGAN AdaIN, latent interpolation, GAN summary pros/cons).

Related

• CoGAN
• Diffusion Model
• Generative Model
• Variational Autoencoder