Empirical Risk Minimization (ERM)

I think the idea is actually really simple, replace the theoretical with empirical values, and it works by Law of Large Numbers.

Goal (ideal): find parameters $w$ that minimize the expected risk

$$w^{\star }=\arg \underset{w}{\min }{\mathbb{E}}_{(x,y)\sim P}[{\ell }_w(x,y)].$$

• Problem: the true distribution $P$ is unknown; we only have samples $\{(x_i,y_i){\}}_{i=1}^n\sim P$.

What we do instead

• Use the empirical distribution defined by the dataset and minimize the empirical risk:

$${\hat{R}}_n(w)=\frac{1}{n}\sum _{i=1}^n{\ell }_w(x_i,y_i),\hat{w}=\arg \underset{w}{\min }{\hat{R}}_n(w).$$

---

Why this makes sense

• By the Law of Large Numbers

$${\hat{R}}_n(w)\underset{n\to \infty }{\to }R(w):={\mathbb{E}}_{(x,y)\sim P}[{\ell }_w(x,y)]$$

meaning that as we collect more samples, the empirical risk approaches the true expected risk.

• Intuition: with enough data and a well-chosen hypothesis class, minimizing empirical risk approximates minimizing true risk.

Notes

• Hypothesis class: $\mathcal{H}=\{f_w\}$ restricts model complexity and helps generalization.
• Regularized ERM (Structural Risk Minimization):

$$\hat{w}=\arg \underset{w}{\min }[{\hat{R}}_n(w)+\lambda \Omega (w)],$$

where $\Omega (w)$ penalizes complexity (e.g., $\Vert w{\Vert }_2^2$).

• Common losses: MSE for regression, cross-entropy for classification, hinge loss for SVMs.

---

Example

• Linear regression: ${\ell }_w(x_i,y_i)=(w^{\top }{x}_i-y_i{)}^2$
• ERM → minimize $\frac{1}{n}{\sum }_i(w^{\top }{x}_i-y_i{)}^2$