Regularization

Weight Decay

Weight decay is a regularization technique that shrinks the weights toward zero by a multiplicative factor at each gradient step, equivalent to L2 regularization under plain SGD.

Decay as a general idea

There is also learning rate decay for Neural Network training, see Learning Rate.

L2 Equivalence Derivation

Start with an L2-regularized loss:

${\tilde{L}}_{\theta }(x,y)=L_{\theta }(x,y)+\frac{\lambda }{2}\Vert \theta {\Vert }_2^2$

Take the gradient with respect to $\theta$:

$\nabla {\tilde{L}}_{\theta }=\nabla L_{\theta }+\lambda \theta$

One gradient step:

${\theta }_t\leftarrow {\theta }_{t-1}-\eta (\nabla L_{{\theta }_{t-1}}+\lambda {\theta }_{t-1})$

Rearrange:

${\theta }_t\leftarrow (1-\eta \lambda ){\theta }_{t-1}-\eta \nabla L_{{\theta }_{t-1}}$

where:

• $\eta$ is the learning rate
• $\lambda$ is the regularization strength
• $(1-\eta \lambda )<1$ is the shrinkage factor applied each step

Intuition

Every gradient step, first shrink every weight toward zero by a fixed fraction, then apply the usual gradient step. Weights only grow if the loss gradient is strong enough to outrun the steady pull back to zero. That’s why weight decay is the MAP equivalent of a Gaussian prior on weights: small weights are assumed more plausible, and the data has to work to justify any weight being large. It stops any single weight from explaining too much on its own.

Hence “weight decay”: L2 regularization reinterpreted as a shrinkage step.

Equivalence breaks under adaptive optimizers

For Adam / RMSProp, L2 regularization and weight decay are not the same. The adaptive step rescales the gradient by $1/\sqrt{{\hat{v}}_t}$, so the L2 term gets rescaled too, dampening regularization on frequently-updated parameters. The weights you most want to regularize (big, fast-moving ones) end up regularized the least, which is the opposite of what you want.

AdamW fixes this by applying decay directly to the weights instead of through the gradient:

${\theta }_t\leftarrow (1-\eta \lambda ){\theta }_{t-1}-\eta \cdot \text{adam-step}$

This is why modern training uses AdamW, not Adam + L2.

Effects

• Prevents weights from growing unboundedly
• Keeps the model in a low-norm region of parameter space, giving a smaller function class and better generalization
• Acts as a Gaussian prior on weights in the MAP interpretation

Hyperparameters:

• $\lambda$: decay strength, typically $10^{-4}$ to $10^{-2}$
• Often excluded from biases and layer-norm parameters

Slides: http://www.gautamkamath.com/courses/CS480-fa2025-files/lec10.pdf

Related

• Regularization
• L2 Regularization
• Gradient Descent
• Neural Network