Vanishing & Exploding Gradients

The vanishing gradient problem is when gradients shrink toward zero as they’re backpropagated through a deep network, so early layers stop learning. The exploding gradient problem is the mirror image: gradients grow without bound, causing wild weight updates and NaNs.

Why does this happen?

Backprop through $L$ layers multiplies $L$ Jacobians together. If each one has typical singular value $s$, the gradient at layer 1 scales like $s^L$.

• $s=0.5,L=20$: $\approx 10^{-6}$ (vanish)
• $s=1.5,L=20$: $\approx 3300$ (explode)

There is no stable middle, you sit on a knife’s edge unless you actively engineer $s\approx 1$.

The chain-rule intuition

A loss gradient w.r.t. an early-layer weight is a product:

$\frac{\partial \mathcal{L}}{\partial W_1}=\frac{\partial \mathcal{L}}{\partial h_L}\cdot \frac{\partial h_L}{\partial h_{L-1}}\cdot \frac{\partial h_{L-1}}{\partial h_{L-2}}\cdots \frac{\partial h_2}{\partial h_1}\cdot \frac{\partial h_1}{\partial W_1}$

where:

• each $\partial h_k/\partial h_{k-1}=W_k^{\top }\cdot \text{diag}({\sigma }^{\prime }(z_k))$ is a weight matrix times the activation derivative
• the final $\partial h_1/\partial W_1$ is just the input to layer 1

Multiplying $L$ such factors is a geometric process: small numbers shrink exponentially, large numbers grow exponentially. Linear depth, exponential consequences.

Common confusion: "isn't the weight gradient just the input?"

For a layer $z_l=W_l{x}_{l-1}$, the gradient of the loss w.r.t. that weight factors as:

$\frac{\partial \mathcal{L}}{\partial W_l}=\underset{\text{upstream}}{\underbrace{{\delta }^{(l)}}}\cdot \underset{\text{local}}{\underbrace{x_{l-1}^{\top }}},{\delta }^{(l)}=\frac{\partial \mathcal{L}}{\partial z_l}$

where:

• the local term $\partial z_l/\partial W_l=x_{l-1}$ really is “just the input”, if activations are bounded, so is this factor
• the upstream error ${\delta }^{(l)}$ is where the explosion lives

The upstream error itself recurses backward through every later layer:

${\delta }^{(l)}=(W^{(l+1)}{)}^{\top }{\delta }^{(l+1)}\odot {\varphi }^{\prime }(z^{(l)})$

So the weight matrices compounding into $\partial \mathcal{L}/\partial W^{(l)}$ aren’t $W^{(l)}$ itself, they’re all the upstream weights $W^{(l+1)},\dots ,W^{(L)}$ baked into $\delta$ by the chain rule. Even if every $\Vert W{\Vert }_2\approx 1.5$ and every ${\varphi }^{\prime }\le 1$, a depth-$L$ chain gives $1.5^{L-l}$ growth. One weight being “a bit larger than 1” compounds across depth.

Large inputs $x$ can inflate the local term (and in ReLU nets forward activations themselves grow with depth when weights are big), but the dominant mechanism is the Jacobian product on the backward pass.

From Jacobians to singular values

“Small numbers” and “large numbers” is loose talk, these factors are matrices, not scalars. So what does it mean for a matrix to be “small” or “large”?

The chain rule produces a Jacobian $J_l=\partial h_{l+1}/\partial h_l$ at every layer (literally $W_l$ for a linear layer, $W_l\cdot \text{diag}({\sigma }^{\prime }(z_l))$ with a nonlinearity). The gradient flowing backwards is:

$\frac{\partial \mathcal{L}}{\partial h_l}=J_l^{\top }\cdot \frac{\partial \mathcal{L}}{\partial h_{l+1}}$

At every layer the gradient gets multiplied by a matrix. The relevant question is: how much can multiplying by $J_l$ stretch a vector? That maximum stretching factor is exactly $\Vert J_l{\Vert }_2$, the spectral norm of $J_l$, equal to its largest singular value. We get the bound:

${\Vert \frac{\partial \mathcal{L}}{\partial h_1}\Vert }_2\le {\prod }_{l=1}^L\Vert J_l{\Vert }_2\cdot {\Vert \frac{\partial \mathcal{L}}{\partial h_L}\Vert }_2$

where:

• $\Vert J_l{\Vert }_2$ is the largest singular value of the layer-$l$ Jacobian
• if the typical $\Vert J_l{\Vert }_2$ is $s$, the worst-case gradient at layer 1 scales like $s^L$

Singular values aren’t a new concept beyond derivatives, they’re just the right way to measure the size of the Jacobian matrices the chain rule already gives you.

Concrete example: sigmoid

The sigmoid derivative ${\sigma }^{\prime }(x)=\sigma (x)(1-\sigma (x))$ peaks at 0.25 (when $x=0$) and is much smaller elsewhere. In a 10-layer net using sigmoids, even in the best case the activation-derivative product is at most $0.25^{10}\approx 10^{-6}$. The gradient reaching layer 1 is roughly a million times smaller than the gradient at layer 10, so layer 1 effectively does not train.

This is the historical reason deep nets were considered “untrainable” before ~2010. It wasn’t compute, it was sigmoid + bad init.

Exploding gradients

Same mechanism, opposite direction. If weight matrices have spectral norm $>1$, the product blows up:

• activations like ReLU don’t cap the forward pass (no saturation on positives), so signals can grow
• a weight matrix with largest singular value $1.5$ gives gradient growth $1.5^L$
• symptom: loss suddenly becomes NaN, or you see gradient norms in the millions in your logs

RNNs are the classic offender

RNNs apply the same weight matrix $W$ at every timestep, so backprop-through-time multiplies $W^{\top }$ by itself $T$ times. The largest eigenvalue of $W$ controls everything:

• $|{\lambda }_{\max }|>1$ → explode
• $|{\lambda }_{\max }|<1$ → vanish
• you essentially never land on $|{\lambda }_{\max }|=1$ exactly

This is why vanilla RNNs can’t learn long-range dependencies and why LSTM’s additive gated cell state was such a breakthrough: the gradient flows through addition, not repeated multiplication.

Saturated neurons (the activation side of vanishing)

A neuron is saturated when its activation derivative is ~0, so no gradient flows through it.

• sigmoid saturates when $|x|$ is large, output flattens at 0 or 1, derivative → 0
• tanh saturates the same way (and is zero-centered, so slightly better, but the gradient still dies)
• ReLU “dead neuron”: once a ReLU is pushed into the negative region by a bad weight update, it outputs 0 forever, gradient is 0 forever, and that unit never recovers. Often caused by a too-large learning rate or bad init

Dead ReLUs are permanent

There’s no gradient to push a dead unit back out of the zero region, so it stays silent for the rest of training. Permanent brain damage.

See Activation Function for the full menu.

How to diagnose

• log per-layer gradient norms during training, vanishing: norms decay geometrically toward the input, exploding: norms spike or NaN
• watch early-layer weight histograms over epochs, if they don’t move, gradients aren’t reaching them
• loss plateaus immediately at a high value → likely vanishing
• loss is NaN after a few steps → almost certainly exploding

How modern nets solve it

Each fix targets a different factor in the product:

Fix	What it does
Kaiming init	Scales $W$ so each layer’s Jacobian has singular values $\approx 1$ at initialization
ReLU / GELU / SiLU	Activation derivative is $1$ (not $0.25$) on the active region, no per-layer shrinkage
BatchNorm / LayerNorm	Re-centers activations each layer, preventing drift into saturated regimes
Residual connections ($h_{l+1}=h_l+f(h_l)$)	Gradient flows through the identity path: $\partial h_{l+1}/\partial h_l=I+\partial f/\partial h_l$, so the worst case is “gradient passes through unchanged” instead of shrinking
LSTM gating	Cell state $c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$ has an additive path, forget gate $f_t\approx 1$ keeps gradient alive across timesteps
Gradient clipping	Bandage for exploding gradients only, caps $\Vert g\Vert$ at a threshold, doesn’t help vanishing

The pattern

Prevent geometric decay/growth at the source (init, normalization, activation choice, residual paths). Clipping is the only “cleanup at the end” fix and only helps the exploding side.

Related

• Backpropagation
• Kaiming Initialization
• Batch Normalization
• Activation Function
• Gradient Clipping
• LSTM
• RNN

References

• How does LSTM prevent the vanishing gradient problem? (CrossValidated)