Recurrent Neural Network

Long Short-Term Memory (LSTM)

LSTM (Hochreiter & Schmidhuber, 1997) is the standard fix for vanilla RNN’s vanishing-gradient problem. It adds a cell state $c_t$ alongside the hidden state $h_t$, with multiplicative gates that control reads, writes, and erases. Legendary Colah explainer.

Why?

Vanilla RNN backprop multiplies by $W_{hh}$ at every timestep — gradients vanish or explode geometrically. LSTM gives the cell state an additive update path with no per-step matrix multiply: $c_t=f\odot c_{t-1}+i\odot g$. Backprop from $c_t$ to $c_{t-1}$ is just elementwise multiply by $f$. If $f\approx 1$, gradient flows uninterrupted across many timesteps — the same trick as ResNet’s identity skip.

Equations (CS231n 2024 Lec 7)

Stack the previous hidden state and current input, multiply by one big weight matrix of shape $4h\times 2h$, then split the $4h$-dim output into four chunks and pass each through its own nonlinearity:

$\left(\begin{array}{c}i\\ f\\ o\\ g\end{array}\right)=\left(\begin{array}{c}\sigma \\ \sigma \\ \sigma \\ \tanh \end{array}\right)W\left(\begin{array}{c}{h}_{t-1}\\ x_t\end{array}\right)$

$c_t=f\odot c_{t-1}+i\odot g$ $h_t=o\odot \tanh (c_t)$

gate	nonlinearity	role
$i$ — input gate	sigmoid	whether to write to cell
$f$ — forget gate	sigmoid	whether to erase cell
$o$ — output gate	sigmoid	how much of cell to reveal as $h_t$
$g$ — gate gate	tanh	how much to write to cell

Why gradient flow works

Backprop from $c_t$ to $c_{t-1}$ through $c_t=f\odot c_{t-1}+i\odot g$ is just elementwise multiplication by $f$ — no matrix multiply by $W$. Across the whole unrolled chain, $\partial c_T/\partial c_0={\prod }_t{f}_t$ (elementwise). If the forget gate stays near 1, gradient is preserved.

This doesn’t guarantee no vanishing/exploding, but it makes long-distance dependencies much easier to learn than in a vanilla RNN. The structural analogue to ResNet — additive skip path that bypasses the nonlinear transformation — is not a coincidence.

Highway Networks

In between vanilla RNN and LSTM lies the Highway Network (Srivastava et al. ICML 2015): $y=g\odot H(x,W_H)+(1-g)\odot x$ where $g=T(x,W_T)$ is a learned gate. Same gating idea applied to feedforward depth.

Source

CS231n 2024 Lec 7 slides 96, 111–121 (LSTM equations, gate roles, gradient flow, ResNet analogy, Highway Networks).

Related

• RNN
• GRU
• Vanishing Gradient Problem
• ResNet