Softmax Classifier

The Softmax Classifier is a linear classifier whose loss interprets class scores as unnormalized log-probabilities. It’s the multi-class generalization of logistic regression and the standard loss for modern classification networks.

Given linear scores $s=Wx+b\in {\mathbb{R}}^C$, the softmax turns them into a probability distribution over classes: $P(Y=k\mid X=x_i)=\frac{e^{s_k}}{{\sum }_j{e}^{s_j}}$

The loss is the negative log probability of the true class — the cross-entropy loss: $L_i=-\log P(Y=y_i\mid X=x_i)=-\log \left(\frac{e^{s_{y_i}}}{{\sum }_j{e}^{s_j}}\right)$

Equivalently, $L_i=-s_{y_i}+\log {\sum }_j{e}^{s_j}$.

Information-theoretic interpretation

Cross-entropy minimizes the KL divergence between the predicted distribution $P$ and the target one-hot distribution $Q=1[y=y_i]$: $H(Q,P)=-{\sum }_k{Q}_k\log P_k=-\log P_{y_i}$ Minimizing cross-entropy = pushing the model’s predicted distribution toward the (degenerate) target distribution.

Sanity checks at initialization

If $W$ is initialized small ($s_j\approx 0$ for all $j$), then $P_j\approx 1/C$, so: $L_i\approx -\log (1/C)=\log C$ For CIFAR-10 ($C=10$), the loss should start near $\log 10\approx 2.3$. If your softmax loss isn’t near $\log C$ on iteration 0, something is wrong.

• Min loss: $0$ (achieved only if $P_{y_i}=1$, requiring $s_{y_i}=+\infty$ and others $=-\infty$ — never reached)
• Max loss: $+\infty$

Numerical stability trick

$e^{s_j}$ overflows for large scores. Subtract the max score before exponentiating — the softmax is shift-invariant: $\frac{e^{s_k}}{{\sum }_j{e}^{s_j}}=\frac{e^{s_k-{\max }_j{s}_j}}{{\sum }_j{e}^{s_j-{\max }_j{s}_j}}$ This is what every framework does internally.

Difference with SVM / Hinge Loss

	SVM (multi-class hinge)	Softmax (cross-entropy)
What scores mean	Margins	Unnormalized log-probabilities
Loss = 0 condition	Correct score > all others by margin (achievable)	Never (requires $-\infty /+\infty$)
Behavior near correct	Stops caring once margin met	Always pushing $P_{y_i}\to 1$
Output interpretation	Just rankings	Calibrated-ish probabilities

The “probabilities” softmax outputs depend on the regularization strength: stronger $\lambda$ → smaller weights → softer (more uniform) distribution. So treat them as relative confidences, not true Bayesian probabilities.

Related

• Softmax Function
• Cross-Entropy Loss
• Logistic Regression
• Linear Classification
• Support Vector Machine
• Hinge Loss
• CS231n

Source

CS231n Lec 2 slides 70–83 (softmax, cross-entropy, log C sanity check, numerical stability, comparison with SVM).