Root Mean Square Normalization (RMSNorm)

They do this in Llama.

Resources

• https://pytorch.org/docs/stable/generated/torch.nn.modules.normalization.RMSNorm.html