Transformer

Attention (Transformer)

Terms:

• Self-Attention
• Masked Attention
• Sparse Attention
• Flash Attention
• Paged Attention (used for faster compute)
• Multi-Head Latent Attention
• Multi-Head Attention
• Multi-Query Attention (modern)
• Grouped Query Attention

Confusion:

• https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms

$Attention(Q,K,V)=softmax(\frac{Q{K}^T}{\sqrt{d_k}})V$ In matrix form

$$\left[\begin{array}{c}\text{softmax}\left(\frac{⟨{\mathbf{q}}_1,{\mathbf{k}}_1⟩}{\sqrt{d}}\right){\mathbf{v}}_1^T+\cdots +\text{softmax}\left(\frac{⟨{\mathbf{q}}_1,{\mathbf{k}}_n⟩}{\sqrt{d}}\right){\mathbf{v}}_n^T\\ ⋮\\ \text{softmax}\left(\frac{⟨{\mathbf{q}}_m,{\mathbf{k}}_1⟩}{\sqrt{d}}\right){\mathbf{v}}_1^T+\cdots +\text{softmax}\left(\frac{⟨{\mathbf{q}}_m,{\mathbf{k}}_n⟩}{\sqrt{d}}\right){\mathbf{v}}_n^T\end{array}\right]\in {\mathbb{R}}^{m\times d}$$

Attention mechanism

• Query roughly speaking is what am I looking for.

• Key is what I represent

• Value is what I actually contain

• Query (Q): “What am I looking for?”

• Key (K): “What do I have?”

• Value (V): “What do I give you if you choose me?”

• https://www.youtube.com/watch?v=bCz4OMemCcA

Another vid

• https://www.youtube.com/watch?v=QCJQG4DuHT0

I was really confused about QKV. https://www.reddit.com/r/MachineLearning/comments/19ewfm9/d_attention_mystery_which_is_which_q_k_or_v/ https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms

Like conceptually, there are no physical properties that allow us to distinguish Q from K. Like for all we known, Q is K and K is Q.

• Query (Q): “What am I looking for?”
• Key (K): “What do I have?”
• Value (V): “What do I give you if you choose me?”

Think of a library:

• Query = what you want to read about
• Keys = the summary on the card catalog for each book
• Values = the full content of each book

Question

in attention, is W_Q, W_K and W_V really necessary? Just use the raw embedding as your Q, K and V. the embedding is going to change too anyways

$Q=X{W}_Q,K=X{W}_K,V=X{W}_V$

• Inputs:
  • Query $Q\in {\mathbb{R}}^{m\times d}$
  • Value $V\in {\mathbb{R}}^{n\times d}$
  • Key $K\in {\mathbb{R}}^{L\times C}$

$\mathbf{V}=\left[\begin{array}{c}{\mathbf{v}}_1^T\\ ⋮\\ {\mathbf{v}}_n^T\end{array}\right]\in {\mathbb{R}}^{n\times d},\mathbf{K}=\left[\begin{array}{c}{\mathbf{k}}_1^T\\ ⋮\\ {\mathbf{k}}_n^T\end{array}\right]\in {\mathbb{R}}^{n\times d},\mathbf{Q}=\left[\begin{array}{c}{\mathbf{q}}_1^T\\ ⋮\\ {\mathbf{q}}_m^T\end{array}\right]\in {\mathbb{R}}^{m\times d}$

Output: an $m\times d$ matrix

The dot product of the query and key tells you how well the key and query are aligned.

Then (Softmax operation is row-wise, i.e., $\text{softmax}(\mathbf{z}{)}_i=\frac{e^{z_i}}{{\sum }_{j=1}^n{e}^{z_j}}$):

What is $d_k$? I think that is the number of dimensions

• It’s just a scaling factor
• I asked the professor and he said empirically, it gives the best performance

Attention Layer

This is cross-attention

Related

• KV Cache