Label Smoothing

This is a trick to make your model less confident. Article here

Idea to me introduced by Andrej Karpathy through his lecture.

Basically, you add 1, so that when you take the Negative Log Likelihood, you guarantee that the value of your $x$ is never 0, so then the negative log likelihood would not return $\infty$.

Hard one-hot labels

$$q(k)=\{\begin{array}{ll}1& \text{if }k=y\\ 0& \text{otherwise}\end{array}$$

Label smoothing (ε ∈ [0,1], K classes)

$$q_{\text{LS}}(k)=\{\begin{array}{ll}1-ϵ& \text{if }k=y\\ \frac{ϵ}{K-1}& \text{if }k\ne y\end{array}$$

Cross-entropy loss with label smoothing

$${\mathcal{L}}_{\text{LS}}=-\sum _{k=1}^K{q}_{\text{LS}}(k)\log p_{\theta }(k\mid x)$$

Expanded form

$${\mathcal{L}}_{\text{LS}}=-(1-ϵ)\log p_{\theta }(y\mid x)-\frac{ϵ}{K-1}\sum _{k\ne y}\log p_{\theta }(k\mid x)$$

OMG isn’t this just Laplace Smoothing? Yea basically doing the same thing