Triangular Matrix
Saw Karpathy use this in https://www.youtube.com/watch?v=kCc8FmEb1nY&t=529s&ab_channel=AndrejKarpathy his build GPT from scratch, which is useful for adding up things up to a certain thing?
Yup, for the attention mechanism!
From Andrej Karpathy: Query roughly speaking is what am I looking for. Key will tell you what do I contain.
The dot product of the query and key tells you how well the key and query are aligned.