Neural Network

Recurrent Neural Network (RNN)

“Learning from the past”. Good for times series data.

Resources

• https://karpathy.github.io/2015/05/21/rnn-effectiveness/
• https://www.youtube.com/watch?v=TQQlZhbC5ps&ab_channel=CodeEmporium
• https://cs231n.github.io/rnn/
• https://www.deeplearningbook.org/contents/rnn.html

Three types of models:

1. Vector-Sequence Models (ex: captioning an image)
2. Sequence-Vector Models (ex: Sentiment Analysis, converting a sentence into a number)
3. Sequence-sequence models (ex: Language translation)

Disadvantages:

1. Slow to train
2. Long sequences lead to vanishing / exploding gradients → Solution: Long Short-Term Memory

Output $y_t$ vs hidden state $h_t$

Had this confused when I was reading World Models paper. In an RNN, the hidden state is the internal representation that carries information forward in time. It’s designed to be rich enough to summarize the relevant history of the sequence.

In my words

The way you can think of $h_t$ (the hidden state) is that it’s right before some transformation $W$ to get the output $y_t$ ($y_t=W{h}_t$) . But $h_t$ is propagated through time. It doesn’t make sense to propagate $y_t$, but that’s generally lossy, less information than what the hidden state has.

The output $y_t$ is typically computed as a simple transformation of $h_t$, e.g.

$$y_t=W{h}_t+b$$

or through a softmax layer for classification.

Propagating $y_t$ instead of $h_t$ wouldn’t make sense because $y_t$ is usually a lower-dimensional, task-specific (and often lossy) projection of $h_t$. It doesn’t preserve enough information for the next step in the sequence to condition on.

That’s why the recurrence is defined in terms of $h_t$ and not in terms of $y_{t-1}$:

$$h_t=f(h_{t-1},x_t)$$