Matrix Norm

In mathematics, a matrix norm is a vector norm in a vector space whose elements (vectors) are matrices (of given dimensions).

Just like Vector Norm, matrix norm are used to measure the size of a matrix.

Frobenius Norm / Hibert-Schmidt Norm

The Frobenius Norm is given by $\Vert A{\Vert }_F=\sqrt{{\sum }_i^n{\sum }_j^n|a_{ij}{|}^2}$

It treats the matrix as a flat vector — sum of squared entries, then square root. Easy to compute, but doesn’t tell you anything about directional stretching.

Spectral Norm (Operator 2-Norm)

The spectral norm of $W$ is its largest singular value (see SVD):

$\Vert W{\Vert }_2={\sigma }_{\max }(W)={\max }_{x\ne 0}\frac{\Vert Wx{\Vert }_2}{\Vert x{\Vert }_2}$

Why?

Of all the matrix norms, spectral norm answers the question: “what’s the worst-case amount this matrix can stretch any input vector?” That makes it the natural tool for bounding gradient growth, Lipschitz constants, and stability of repeated matrix application.

Geometric intuition. A matrix maps the unit sphere to an ellipsoid. The semi-axes of that ellipsoid are the singular values ${\sigma }_1\ge {\sigma }_2\ge \dots$. The spectral norm is the longest semi-axis — the maximum stretch in any direction. The Frobenius norm, by contrast, is $\sqrt{{\sum }_i{\sigma }_i^2}$ — total “energy” across all directions, not the worst case.

For symmetric / normal matrices the spectral norm equals $|{\lambda }_{\max }|$, the largest eigenvalue in absolute value. For general matrices, use singular values of $W$ (equivalently, eigenvalues of $W^{\top }W$ then square-root).

Why it shows up in deep learning

Backprop multiplies the gradient by a Jacobian matrix at each layer, so the worst-case per-layer amplification is $\Vert J_l{\Vert }_2$. Stack $L$ layers and you get ${\prod }_l\Vert J_l{\Vert }_2$ — exponential in depth. See Vanishing Gradient Problem for the full Jacobian → singular value walkthrough, and Kaiming init for keeping that product $\approx 1$.

In RNNs the same $W$ is applied $T$ times, so the spectral radius (largest eigenvalue magnitude) of $W$ governs whether backprop-through-time explodes or vanishes.

Related

• Vector Norm
• Singular Value Decomposition
• Vanishing Gradient Problem
• Kaiming Initialization