Gradient Vector

The gradient vector of is given by

The gradient vector points in the ascending slope position.


The gradient of the function took me a bit to understand, it is used to compute the gradient of ReLU and SVM multiclass loss. Think about how the function changes as you change a particular value. If is less than , than no matter what value, it’s a fixed value, i.e. .