Attention

Masked Attention

https://www.youtube.com/watch?v=bCz4OMemCcA

How is masked attention implemented? Just use a lower triangular matrix right.

• Andrej Karpathy shows how this is implemented

$\text{MaskedAttention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}+M\right)V$

• You just add the Causal Attention Mask

The causal mask looks like this

$$M=\left[\begin{array}{cccc}0& -\infty & -\infty & -\infty \\ 0& 0& -\infty & -\infty \\ 0& 0& 0& -\infty \\ 0& 0& 0& 0\end{array}\right]$$