Attention

Bahdanau Attention

Seq2Seq originally compressed the full source sequence into one vector $c=h_T$. Bahdanau attention replaces that bottleneck by letting the decoder look back at all encoder states at each step.

For decoder timestep $t$:

1. Score each encoder state: $e_{t,i}=f_{att}(s_{t-1},h_i)$
2. Normalize: $a_{t,\ast }=\text{softmax}(e_{t,\ast })$
3. Build context: $c_t={\sum }_i{a}_{t,i}{h}_i$
4. Decode using $c_t$

This is usually called additive attention because the score comes from a learned MLP instead of a dot product. Modern Transformer attention keeps the same idea, but swaps the scoring function for scaled dot products and cleanly separates keys from values.