Gradient Vector

The gradient vector of $f(x,y)$ is given by $\nabla f=(\frac{\partial f}{\partial x},\frac{\partial f}{\partial y})$

The gradient vector points in the ascending slope position.

Examples

$f(x,y)=x+y\to \frac{\partial f}{\partial x}=1\frac{\partial f}{\partial y}=1$

The gradient of the $\max$ function took me a bit to understand, it is used to compute the gradient of ReLU and SVM multiclass loss. Think about how the function changes as you change a particular value. If $x$ is less than $y$, than no matter what value, it’s a fixed value, i.e. $y$. $f(x,y)=\max (x,y)\to \frac{\partial f}{\partial x}=1(x\ge y)\frac{\partial f}{\partial y}=1(y\ge x)$