Matrix Decomposition

Singular Value Decomposition (SVD)

SVD provides us a way to decompose a matrix into singular vectors and singular values (seems like cyclical definition, singular vectors and singular values are defined from the SVD decomposition)

First heard from f1tenth.

Resources

• Deep Learning Textbook
• Singular Value Decomposition playlist by Steve Brunton
• Also mentioned from this video https://www.youtube.com/watch?v=jBnCcr-3bXc 10:30 for Zero-shot Learning.

Be familiar with your eigenvalues

This is a prerequisite. I still forget the point of eigenvalues.

The singular value decomposition (SVD) generalizes the Eigendecomposition of a square normal matrix with an orthonormal eigenbasis to any $m\times n$ matrix.

Resources:

• https://www.youtube.com/watch?v=mBcLRGuAFUk&ab_channel=MITOpenCourseWare
• 5 minute explanation https://www.youtube.com/watch?v=giOpcCPHitY&ab_channel=CyrillStachniss by Cyrill Stachniss

$A=UD{V}^T$

• $U$ is $m\times m$, and a Orthogonal Matrix
• $D$ is $m\times n$, and a Diagonal Matrix
• $V$ is $n\times n$, and an Orthogonal Matrix as well

Notation:

• The columns of $U$ are called left-singular vectors
  • Left-singular vectors of $A$ are the eigenvectors of $A{A}^T$
• The columns of $V$ are called right-singular vectors
  • right-singular vectors of $A$ are the eigenvectors of $A^{T}A$
• The elements along diagonal of D are called singular values
  • non-zero singular values of $A$ are the square roots of eigenvalues of $A^{T}A$

Why SVD?

“Perhaps the most useful feature of the SVD is that we can use it to partially generalize matrix inversion to non-square matrices!” - DL textbook.

After we compute the SVD, we can easily compute the Pseudoinverse.

Related

• Pseudoinverse