Attention

Multi-Head Attention

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.

$MultiHead(Q,K,V)=\text{Concat}({\text{head}}_1,\dots ,{\text{head}}_h)W^O$ where ${\text{head}}_i=\text{Attention}(Q{W}_i^Q,K{W}_i^K,V{W}_i^V)$

Where the projections are parameter matrices

$$W_i^Q\in {\mathbb{R}}^{d_{\text{model}}\times d_k},W_i^K\in {\mathbb{R}}^{d_{\text{model}}\times d_k},W_i^V\in {\mathbb{R}}^{d_{\text{model}}\times d_v},W^O\in {\mathbb{R}}^{h{d}_v\times d_{\text{model}}}$$

In the original paper, they employ h = 8 parallel attention layers, or heads. For each of these we use $d_k=d_v=d_{model}/h=64$ .

Due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality

• from https://www.youtube.com/watch?v=bCz4OMemCcA

I don't understand, don't the heads just end up doing the same things?

We want each head to watch different aspects of the same word.

Intuitively, it could happen. But here’s why it usually doesn’t:

• Each head has its own $W_q$, $W_k$, $W_v$ matrices, all initialized differently.
• During training, if two heads start doing the same thing, they don’t both get rewarded equally — gradients nudge them to specialize and reduce redundancy.
• Why? Because doing the same thing doesn’t reduce the loss as effectively as learning different complementary patterns.

• Source: https://www.youtube.com/watch?v=0VLAoVGf_74