Rotary Positional Encoding (RoPE)
It applies a rotation to query and key vectors in multi-head attention, based on their position index.
The rotation is computed using predefined sin and cos functions, very similar in spirit to sinusoidal encoding — but instead of adding or concatenating, RoPE encodes position by rotating vectors in a complex plane.
Citation
(Su et al., 2021)
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. https://arxiv.org/pdf/2104.09864