Regularization

Regularization prevents overfitting. This is used widely in Machine Learning. Related to the idea of Occam’s Razor.

Regularization Methods:

Parameter Penalties

• L1 Regularization or L2 Regularization or elastic net or max norm regularization

This reduces the effective capacity of the model.

Training-time regularization:

• Early Stopping
• Dropout
• Data Augmentation

Model/structure choices (capacity control)

• Simpler model / fewer parameters
• Feature selection / dimensionality reduction (PCA)
• Ensembling (bagging, random forest): reduces variance

For linear models specifically

• SVM margin (hinge loss + C) acts like regularization via margin/penalty tradeoff

Normalization

• BatchNorm / LayerNorm: not regularization strictly, but stabilizes and prevents overfitting
• Gradient Clipping
• Weight constraints: max-norm, spectral norm (controls capacity)

Regularization penalizes/punishes the complexity of the model. $\lambda$ is the regularization parameter, so we usually have $\lambda R(W)$.

We have a general formula $L=\frac{1}{N}\sum _i{L}_i(f(x_i,W),y_i)+\lambda R(W)$

Types

• L2 Regularization $R(W)=\sum _k\sum _l{W}_{k,l}^2$
• L1 Regularization (Lasso) $R(W)=\sum _k\sum _l|W_{k,l}|$
• Elastic Net (L1 + L2) $R(W)=\sum _k\sum _l\beta W_{k,l}^2+|W_{k,l}|$
• Max norm regularization

Neural Network Specific:

• Dropout
• Batch Normalization
• Stochastic depth

From MATH213

Definition 2: Regularization

A function $f$ is the regularization of a function $g$ if

1. For all $x$ in the domain of $g,f(x)=g(x)$.
2. For all $x$ such that ${\lim }_{t\to x}g(x)$ exists but $f(x)$ is undefined, $f(x)={\lim }_{t\to x}g(x)$.

Theorem 1

If $f(x)$ is the regularization of a function $g$ that has a finite number of discontinuities then $\int f(x)dx=\int g(x)dx$

Also see Finite Zeros and Finite Poles.

Related

• Entropic Regularization