Focal Loss

Focal loss (what’s “automatic” vs “manual”)

• The per-example “weight” is computed from the model’s current confidence.
• It down-weights easy examples automatically via the factor $(1-p_y{)}^{\gamma }$:
  • If the model is confident on the true class ($p_y$ is high), then $(1-p_y{)}^{\gamma }$ is small → loss shrinks.
  • If the model is not confident ($p_y$ is low), then $(1-p_y{)}^{\gamma }$ is larger → loss stays big.

$$\mathrm{FL}=-{\alpha }_y(1-p_y{)}^{\gamma }\log (p_y)$$

Important nuance

• The focusing term $(1-p_y{)}^{\gamma }$ is the “automatic” part (depends on the model’s prediction).
• But focal loss often includes ${\alpha }_y$, which is a manually chosen class weight (similar idea to weighted cross-entropy).