Regularization

Regularization is any technique that discourages model complexity to prevent overfitting, related to Occam’s Razor. Used widely in Machine Learning.

The general form adds a penalty $\lambda R(W)$ to the loss:

$L=\frac{1}{N}\sum _i{L}_i(f(x_i,W),y_i)+\lambda R(W)$

where:

• $\lambda$ is the regularization strength
• $R(W)$ penalizes model complexity (e.g. weight norm)

Intuition

The data term wants the model to fit the training set. The penalty term wants the model to stay simple. $\lambda$ is the exchange rate. A flexible model usually has many weights that fit the training data equally well; the penalty picks the simplest one from that bunch, which is the one most likely to generalize. Equivalently (Bayesian view), $R(W)$ is a prior over weights, and minimizing loss plus penalty is MAP estimation.

Parameter penalties (reduce effective model capacity):

• L2 Regularization: $R(W)={\sum }_k{\sum }_l{W}_{k,l}^2$
• L1 Regularization (Lasso): $R(W)={\sum }_k{\sum }_l|W_{k,l}|$
• Elastic Net (L1 + L2): $R(W)={\sum }_k{\sum }_l\beta W_{k,l}^2+|W_{k,l}|$
• Max-norm regularization

Training-time regularization:

• Early Stopping
• Dropout
• Data Augmentation
• Stochastic depth

Model/structure choices (capacity control):

• Simpler model / fewer parameters
• Feature selection / dimensionality reduction (PCA)
• Ensembling (bagging, random forest): reduces variance

Linear models:

• SVM margin (hinge loss + $C$) acts like regularization via margin/penalty tradeoff

Normalization (not strictly regularization but stabilizes / prevents overfitting):

• BatchNorm / LayerNorm
• Gradient Clipping
• Weight constraints: max-norm, spectral norm

L1 vs L2 preference

From the CS231n Lec 3 slides. Given $x=[1,1,1,1]$, two weight vectors with the same dot product $w^{\top }x=1$:

• $w_1=[1,0,0,0]$: L1 picks this (“sparse”)
• $w_2=[0.25,0.25,0.25,0.25]$: L2 picks this (“spread out”)

Both produce identical predictions on this $x$; the regularizer is what breaks the tie. Pick the penalty that matches your prior over what a “good” classifier looks like.

Why regularize?

1. Express preferences over weights (which solution among the equivalent ones?)
2. Make the model simple so it generalizes to test data
3. Improve optimization by adding curvature (the loss landscape becomes more bowl-shaped)

From MATH213

Definition 2: Regularization

A function $f$ is the regularization of a function $g$ if

1. For all $x$ in the domain of $g$, $f(x)=g(x)$
2. For all $x$ such that ${\lim }_{t\to x}g(x)$ exists but $f(x)$ is undefined, $f(x)={\lim }_{t\to x}g(x)$

Theorem 1

If $f(x)$ is the regularization of a function $g$ that has a finite number of discontinuities then $\int f(x)dx=\int g(x)dx$

Also see Finite Zeros and Finite Poles.

Related

• Entropic Regularization