Gradient Clipping

Gradient clipping is an optimization technique used primarily to address the issue of Exploding Gradients.

The process involves checking the magnitude of the gradient vector and, if it exceeds a predetermined threshold or value (v), scaling the gradient down. The mathematical operation for gradient clipping is:

If ∣∣g∣∣_2 ≥v, then g←g/∣∣g∣∣ 2 ⋅v.