机器学习学习笔记 PRML Chapter 2.0 - Prerequisite 2 -Singular Value Decomposition (SVD)

Chapter 2.0 : Prerequisite 2 -Singular Value Decomposition (SVD)

PRML, OXford University Deep Learning Course, Machine Learning, Pattern Recognition
Christopher M. Bishop, PRML, Chapter 2 Probability Distributions

1. Vector Terminology

• Orthogonality
  Two vectors [Math Processing Error]$\vec{i}$ and [Math Processing Error]$\vec{j}$ are said to be orthogonal to each other if their inner product equals zero, i.e., [Math Processing Error] $$\vec{i}\cdot \vec{j} = 0$$

• Normal Vector
  A normal vector (or unit vector ) [Math Processing Error]$\vec{u}$ is a vector of length 1, i.e., [Math Processing Error] $$|\vec{u}| = 1$$

• Orthonormal Vectors
  Vectors of unit length that are orthogonal to each other are said to be orthonormal.

2. Matrix Terminology

2.1 Orthogonal Matrix

A matrix [Math Processing Error]$A$ is orthogonal if [Math Processing Error] $$AA^T = A^TA = I,$$ where [Math Processing Error]$I$ is the identity matrix.

2.2 Eigenvectors and Eigenvalues

An eigenvector is a nonzero vector that satisfies the equation [Math Processing Error] $$A \vec{v} = \lambda \vec{v}$$
where [Math Processing Error]$A_{n \times n}$ is a square matrix,
- the scalar [Math Processing Error]$\lambda$ is an eigenvalue, and
- [Math Processing Error]$\vec{v}$ is the eigenvector.

Eigenvalues and eigenvectors are also known as, respectively, characteristic roots(特征值) and characteristic vectors(特征向量), or latent roots and latent vectors.

THE KEY IDEAS [see Ref-7]:

• [Math Processing Error]$A \vec{v} = \lambda \vec{v}$ says that eigenvectors [Math Processing Error]$\vec{v}$ keep the same direction when multiplied by [Math Processing Error]$A$.
• [Math Processing Error]$A \vec{v} = \lambda \vec{v}$ also says that [Math Processing Error]$det(A - \lambda I) = 0$. This determines [Math Processing Error]$n$ eigenvalues.
• The eigenvalues of [Math Processing Error]$A^2$ and [Math Processing Error]$A^{-1}$ are [Math Processing Error]$\lambda^2$ and [Math Processing Error]$\lambda^{-1}$, respectively, with the same eigenvectors.
• The sum of the [Math Processing Error]$\lambda$’s equals the sum down the main diagonal of [Math Processing Error]$A$ (the trace), i.e., [Math Processing Error] $$\lambda_1 + \lambda_2 + \dots + \lambda_n = \text{ trace } = a_{11} + a_{22} + \dots + a_{nn}$$
• The product of the [Math Processing Error]$\lambda$’s equals the determinant, i.e., [Math Processing Error] $$\lambda_1 \lambda_2 \dots \lambda_n = det(A)$$

2.3 Understanding eigenvectors and eigenvalues in terms of transformation and the corresponding matrix [see Ref-9]

In linear algebra, an eigenvector or characteristic vector of a linear transformation [Math Processing Error]$T$ from a vector space [Math Processing Error]$V$ over a field [Math Processing Error]$F$ into itself is a non-zero vector that does not change its direction when that linear transformation is applied to it. In other words, if [Math Processing Error]$v$ is a vector that is not the zero vector, then it is an eigenvector of a linear transformation [Math Processing Error]$T$ if [Math Processing Error]$T(v)$ is a scalar multiple of [Math Processing Error]$v$. This condition can be written as the mapping [Math Processing Error] $$T : v \mapsto \lambda v,$$ where [Math Processing Error]$\lambda$ is a scalar in the field [Math Processing Error]$F$, known as the eigenvalue or characteristic value associated with the eigenvector [Math Processing Error]$v$.

If the vector space [Math Processing Error]$V$ is finite-dimensional, then the linear transformation [Math Processing Error]$T$ can be represented as a square matrix [Math Processing Error]$A$, and the vector [Math Processing Error]$v$ by a column vector, rendering the above mapping as a matrix multiplication on the left hand side and a scaling of the column vector on the right hand side in the equation [Math Processing Error] $$A \vec{v} = \lambda \vec{v}.$$

There is a correspondence between [Math Processing Error]$n$ by [Math Processing Error]$n$ square matrices and linear transformations from an n-dimensional vector space to itself. For this reason, it is equivalent to define eigenvalues and eigenvectors using either the language of matrices or the language of linear transformations.

Geometrically, an eigenvector corresponding to a real, nonzero eigenvalue points in a direction that is stretched by the transformation and the eigenvalue is the factor by which it is stretched. If the eigenvalue is negative, the direction is reversed.

It can be shown in the following figure, where matrix [Math Processing Error]$A$ acts by stretching the vector [Math Processing Error]$x$, not changing its direction, so [Math Processing Error]$x$ is an eigenvector of [Math Processing Error]$A$.

3. Singular Value Decomposition

3.1 Understanding of SVD

Singular value decomposition (SVD) can be looked at from three mutually compatible points of view.
- 1) a method for transforming correlated variables into a set of uncorrelated ones that better expose the various relationships among the original data items.
- 2) a method for identifying and ordering the dimensions along which data points exhibit the most variation.
- 3) a method for data reduction, since once we have identified where the most variation is, it’s possible to find the best approximation of the original data points using fewer dimensions.

3.2 Statement of the SVD Theorem

SVD is based on a theorem from linear algebra which says that a rectangular matrix [Math Processing Error]$A_{m \times n}$ can be broken down into the product of three matrices:
- an orthogonal matrix [Math Processing Error]$U$(i.e., [Math Processing Error]$U U^T = U^T U = I$);
- a diagonal matrix [Math Processing Error]$S$;
- the transpose of an orthogonal matrix [Math Processing Error]$V$ (i.e, [Math Processing Error]$V V^T = V^T V = I$).

The theorem is usually presented something like this:
[Math Processing Error] $$A_{m \times n} = U_{m \times m} S_{m \times n} V_{n \times n}^T$$

• assuming [Math Processing Error]$m \geq n$ [see Ref-4 for this figure]:

  

• assuming [Math Processing Error]$m \leq n$ [see Ref-4 for this figure]:

  

• The columns of [Math Processing Error]$\mathbf {U}$ and the columns of [Math Processing Error]$\mathbf {V}$ are called the left-singular vectors and right-singular vectors of [Math Processing Error]$\mathbf {A}$ , respectively.

• The columns of [Math Processing Error]$\mathbf {U}$ are orthonormal eigenvectors of [Math Processing Error]$AA^T$.
  There is a brief proof. Let [Math Processing Error]$U_{m\times m} = [\vec{u_1}, \dots, \vec{u_m}]$, where the column vector [Math Processing Error]$\vec{u_i} = [u_{i1}, \dots, u_{im}]^T$, for [Math Processing Error]$i = 1, \dots, m$, with [Math Processing Error]$p = rank(A) \leq min(m, n) = n$.
  [Math Processing Error] $$\begin{split} &\because AA^T = (U S V^T)(U S V^T)^T = U S V^T V S^T U^T = U S \mathbf{I} S^T U^T = U S S^T U^T \\ \therefore &\Longleftrightarrow AA^TU= U S S^T U^T U =U S S^T \mathbf{I} = U S S^T \qquad (3.1) \end{split}$$ Firstly, to calculate the product of [Math Processing Error]$SS^T$:
  [Math Processing Error] $$\begin{split} S S^T &= \quad \begin{bmatrix} \sigma_1 & 0 & \dots\\ & \sigma_2 & 0 & ... \\ & & ... & \\ & & & \sigma_p & 0 & \dots\\ & & & 0 & ... \\ \end{bmatrix} _{m \times n} \times \quad \begin{bmatrix} \sigma_1 & 0 & \dots\\ & \sigma_2 & 0 & ... \\ & & ... & \\ & & & \sigma_p & 0 & \dots\\ & & & 0 & ... \\ \end{bmatrix} _{n \times m} \\ &= \quad \begin{bmatrix} \sigma_1^2 & 0 & \dots\\ & \sigma_2^2 & 0 & ... \\ & & ... & \\ & & & \sigma_p^2 & 0 & \dots\\ & & & 0 & ... \\ \end{bmatrix} _{m \times m} = \quad \begin{bmatrix} \mathbf{I}_{p \times p} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{bmatrix} _{m \times m} \qquad (3.2)\\ \end{split}$$
  The LHS of (3.1) equals:
  [Math Processing Error] $$\begin{split} \text{LHS of (3.1)} = AA^T \times [\vec{u_1}, \dots, \vec{u_m}] = [AA^T\vec{u_1}, \dots, AA^T\vec{u_m}] \qquad (3.3) \end{split}$$
  Substitute (3.2) into (3.1) to generate the RHS of (3.1): [Math Processing Error] $$\begin{split} \text{RHS of (3.1)} = U S S^T &= [\vec{u_1}, \dots, \vec{u_m}] \quad \begin{bmatrix} \sigma_1^2 & 0 & \dots\\ & \sigma_2^2 & 0 & ... \\ & & ... & \\ & & & \sigma_p^2 & 0 & \dots\\ & & & 0 & ... \\ \end{bmatrix} _{m \times m} \\ &= [\vec{u_1} \sigma_1^2, \dots, \vec{u_p}\sigma_1^2, \dots , \vec{0}] \qquad (3.4) \end{split}$$ You can testify the second line of (3.4) by listing all the elements of the column vectors, and doing matrix production based on the matrix product rule. Therefore (3.3) and (3.4) give us the following euqation
  [Math Processing Error] $$\begin{split} &\because [AA^T\vec{u_1}, \dots, AA^T\vec{u_m}] = [\vec{u_1} \sigma_1^2, \dots, \vec{u_p}\sigma_1^2, \dots , \vec{0}] \\ &\Longleftrightarrow AA^T\vec{u_i}= \vec{u_i} \sigma_i^2 = \sigma_i^2 \vec{u_i} \qquad \text{(due to the scalar }\sigma_i)\\ &\Longleftrightarrow \vec{u_i} \text{ is the eigenvector of matrix } AA^T, \text{ with the corresponding eigenvalue } \sigma_i^2. \\ &\Longleftrightarrow \qquad \qquad (3.5- a) \end{split}$$

• Similarly, we can prove that the columns of [Math Processing Error]$V$ are orthonormal eigenvectors of [Math Processing Error]$A^TA$,
  [Math Processing Error] $$\begin{split} &\because A^T A = (U S V^T)^T (U S V^T) = V S^T U^T U S V^T = V S^T \mathbf{I} S V^T = V S^T SV^T \\ \therefore &\Longleftrightarrow A^TAV= V S^T S V^T V =V S^T S \mathbf{I} = V S^T S \\ &\because [ A^T A \vec{v_1}, \dots, A^T A \vec{v_m}] = [\vec{v_1} \sigma_1^2, \dots, \vec{v_p}\sigma_1^2, \dots , \vec{0}] \\ &\Longleftrightarrow A^T A \vec{v_i}= \vec{v_i} \sigma_i^2 = \sigma_i^2 \vec{v_i} \qquad \text{(due to the scalar }\sigma_i)\\ &\Longleftrightarrow \vec{v_i} \text{ is the eigenvector of matrix } A^T A , \text{ with the corresponding eigenvalue } \sigma_i^2. \\ &\Longleftrightarrow \qquad \qquad (3.5- b) \end{split}$$

• [Math Processing Error]$S$ is a diagonal matrix containing the square roots of non-zero eigenvalues of both [Math Processing Error]$AA^T$ and [Math Processing Error]$A^TA$. A common convention is to list the singular values in descending order. In this case, the diagonal matrix [Math Processing Error]$S$ is uniquely determined by [Math Processing Error]$A$ (though not the matrices [Math Processing Error]$U$ and [Math Processing Error]$V$).
  [Math Processing Error] $$S = \begin{bmatrix} \sigma_1 & ... \\ & \sigma_2 & ... \\ & & ... & \\ & & & \sigma_n \\ & & & ... \\ \end{bmatrix} _{m \times n}$$
  assuming [Math Processing Error]$m \geq n$, with [Math Processing Error]$\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_p \geq 0, \sigma_{p+1} = \sigma_{p+2} = \dots \sigma_{n} = 0$, where [Math Processing Error]$\sigma_i$ is called the singular values of the matrix [Math Processing Error]$A$.

• [Math Processing Error]$p$ is the rank of matrix [Math Processing Error]$A$, i.e., [Math Processing Error]$p = rank(A) = dim(R(A))$, where [Math Processing Error]$R(A) \subseteq R^m$, [Math Processing Error]$R(A)$ means the range of [Math Processing Error]$A$, that is the set of possible linear combinations of the columns of [Math Processing Error]$A$.

Some Conclusion and Simple Proof:

Let [Math Processing Error]$U_{m\times m} = [\vec{u_1}, \dots, \vec{u_m}]$, where [Math Processing Error]$\vec{u_i} = [u_{i1}, \dots, u_{im}]^T$, for [Math Processing Error]$i = 1, \dots, m$; and [Math Processing Error]$V_{n\times n} = [\vec{v_1}, \dots, \vec{v_m}]$, where [Math Processing Error]$\vec{v_j} = [v_{j1}, \dots, v_{jn}]^T$, for [Math Processing Error]$j = 1, \dots, n$.
[Math Processing Error] $$\because U^TU = \begin{bmatrix} \vec{u_1}^T \\ \dots \\ \vec{u_m}^T \end{bmatrix} \begin{bmatrix} \vec{u_1}, \dots, \vec{u_m} \\ \end{bmatrix} = \begin{bmatrix} \vec{u_1} \cdot \vec{u_1} & \dots & \vec{u_1} \cdot \vec{u_m} \\ \dots & \dots & \dots \\ \vec{u_m} \cdot \vec{u_1} & \dots & \vec{u_m}\cdot \vec{u_m} \\ \end{bmatrix} = \begin{bmatrix} 1 & \dots & 0 \\ \dots & \dots & \dots \\ 0 & \dots & 1 \\ \end{bmatrix} = \mathbf{I}$$
[Math Processing Error] $$\therefore \vec{u_i}\cdot \vec{u_j} = \begin{cases} 1 & i = j \\ 0 & i \neq j \end{cases} \qquad (3.6)$$ where [Math Processing Error]$i, j = 1, 2, \dots, m$.
Similarly, we have [Math Processing Error] $$\vec{v_i} \cdot \vec{v_j} = \begin{cases} 1 & i = j \\ 0 & i \neq j \end{cases} \qquad (3.7)$$ where [Math Processing Error]$i, j = 1, 2, \dots, n$.
That is, the columns of [Math Processing Error]$\mathbf {U}$ and [Math Processing Error]$\mathbf {V}$ are orthonormal vectors, respectively.

3.3 An example of SVD:

[Math Processing Error] $$A = \begin{bmatrix} 3 & 1 & 1 \\ -1 & 3 & 1 \\ \end{bmatrix}$$
- To calculate [Math Processing Error]$U$ via finding the eigenvalues and corresponding eigenvectors of [Math Processing Error]$AA^T$, to give [Math Processing Error] $$U = \begin{bmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & - \frac{1}{\sqrt{2}} \\ \end{bmatrix}$$
- To calculate [Math Processing Error]$V$ via finding the eigenvalues and corresponding eigenvectors of [Math Processing Error]$A^TA$, to give [Math Processing Error] $$V = \begin{bmatrix} \frac{1}{\sqrt{6}} & \frac{2}{\sqrt{5}} & \frac{1}{\sqrt{30}}\\ \frac{2}{\sqrt{6}} & \frac{-1}{\sqrt{5}} & \frac{2}{\sqrt{30}}\\ \frac{1}{\sqrt{6}} & 0 & \frac{-5}{\sqrt{30}}\\ \end{bmatrix}$$
- S = [Math Processing Error] $$\begin{bmatrix} \sqrt{12} & 0 & 0 \\ 0 & \sqrt{10} & 0 \\ \end{bmatrix}$$
- SVD result is [Math Processing Error] $$A_{m \times n} = U_{m \times m} S_{m \times n} V_{n \times n}^T = \begin{bmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & - \frac{1}{\sqrt{2}} \\ \end{bmatrix} \begin{bmatrix} \sqrt{12} & 0 & 0 \\ 0 & \sqrt{10} & 0 \\ \end{bmatrix} \begin{bmatrix} \frac{1}{\sqrt{6}} & \frac{2}{\sqrt{6}} & \frac{1}{\sqrt{6}}\\ \frac{2}{\sqrt{5}} & \frac{-1}{\sqrt{5}} & 0\\ \frac{1}{\sqrt{30}} & \frac{2}{\sqrt{30}} & \frac{-5}{\sqrt{30}}\\ \end{bmatrix} = \begin{bmatrix} 3 & 1 & 1 \\ -1 & 3 & 1 \\ \end{bmatrix}$$

3.4 Intuitive Interpretations of SVD [see Ref-5]

1) Points in d-dimension Space:

To gain insight into the SVD, treat the rows of an [Math Processing Error]$n \times d$ (here we use [Math Processing Error]$n \times d$ instead of [Math Processing Error]$m \times n$, since it is common to be used to represent those n points of d-dimension) matrix [Math Processing Error]$A$ as [Math Processing Error]$n$ points in a d-dimensional space.

[Math Processing Error] $$A_{n \times d} \vec{u} = \lambda \vec{u}$$ is equivalent to
[Math Processing Error] $$\begin{bmatrix} \vec{a_1}^T \\ \dots \\ \vec{a_m}^T \end{bmatrix} \vec{u} = \begin{bmatrix} \vec{a_1} \cdot \vec{u} \\ \dots \\ \vec{a_m} \cdot \vec{u} \end{bmatrix} = \lambda \vec{u} \qquad (3.8)$$
where the inner product [Math Processing Error]$\vec{a_i} \cdot \vec{u}$ means the projection of point [Math Processing Error]$x_i$ (represented by column vector [Math Processing Error]$\vec{a_i}$, i.e., the [Math Processing Error]$i^{th}$ row of matrix A) onto the line along which [Math Processing Error]$\vec{v}$ is a unit vector.

2) The Best Least Squares Fit Problem:

Consider the problem of finding the best k-dimensional subspace with respect to the set of points. Here “best” means minimize the sum of the squares of the perpendicular distances of the points to the subspace. We begin with a special case of the problem where the subspace is 1-dimensional, a line through the origin. We will see later that the best-fitting k-dimensional subspace can be found by k applications of the best fitting line algorithm (i.e., 应用k次1-dim直线fitting即可得到the fitting k-dim subspace). Finding the best fitting line through the origin with respect to a set of points [Math Processing Error]$\{x_i \mid 1 \leq i \leq n \}$ in the plane means minimizing the sum of the squared distances of the points to the line. Here distance is measured perpendicular to the line (the corresponding problem is called the best least squares fit), or more often measured vertical in the y direction, to the subspace of [Math Processing Error]$x_i$ (with the corresponding problem - least squares fit).

Returning to the best least squares fit problem, consider projecting a point [Math Processing Error]$x_i$ onto a line through the origin. Then based on the following figure

we can get [Math Processing Error] $$\begin{split} &\because x_{i1}^2 + x_{i2}^2 + \dots + x_{id}^2 = \text{(length of projection)}^2 + \text{(distance of point to line)}^2 = \beta_i^2 + \alpha_i^2 \\ &\Longleftrightarrow \text{(distance of point to line)}^2 = (x_{i1}^2 + x_{i2}^2 + \dots + x_{id}^2) - \text{(length of projection)}^2 \\ &\Longleftrightarrow \alpha_i^2 = (x_{i1}^2 + x_{i2}^2 + \dots + x_{id}^2) - \beta_i^2 \qquad (3.9)\\ \end{split}$$

From (3.9) and the observation that [Math Processing Error]$\sum_{i = 1} ^n (x_{i1}^2 + x_{i2}^2 + \dots + x_{id}^2)$ is a constant ( i.e., independent of the line), we get the equivalence

[Math Processing Error] $$\begin{split} &\underset{v}{\operatorname{argmin}} \sum_{i = 1} ^n \alpha_i^2 \Longleftrightarrow \underset{v}{\operatorname{argmin}} \sum_{i = 1} ^n \left \{ (x_{i1}^2 + x_{i2}^2 + \dots + x_{id}^2) - \beta_i^2 \right\}\\ & \Longleftrightarrow \underset{v}{\operatorname{argmax}} \sum_{i = 1} ^n \beta_i^2 \qquad (3.10)\\ \end{split}$$
So minimizing the sum of the squares of the distances is equivalent to maximizing the sum of the squares of the lengths of the projections onto the line. This conclusion helps to introduce the subsequent definition of singular vectors.

3) Singular Vectors and Singular Values:

• Singular Vectors: Consider the rows of [Math Processing Error]$A$ as [Math Processing Error]$n$ points in a d-dimensional space. Consider the best fit line through the origin. Let [Math Processing Error]$\vec{v}$ be a unit vector along this line. The length of the projection of [Math Processing Error]$\vec{a_i}$ (i.e., the [Math Processing Error]$i^{th}$ row of [Math Processing Error]$A$) onto [Math Processing Error]$\vec{v}$ is [Math Processing Error]$\vec{a_i} \cdot \vec{v}$. From this we see that the sum of length squared of the projections is [Math Processing Error]$|A\vec{v}|^2$. The best fit line is the one maximizing [Math Processing Error]$|A\vec{v}|^2$ and hence minimizing the sum of the squared distances of the points to the line.

• The First Singular Vector: With this in mind, define the first singular vector, [Math Processing Error]$\vec{v_1}$ of [Math Processing Error]$A$, which is a column vector, as the best fit line through the origin for the [Math Processing Error]$n$ points in d-space that are the rows of [Math Processing Error]$A$. Thus
  [Math Processing Error] $$\vec{v_1} = \underset{|\vec{v}| = 1}{\operatorname{argmax}} |A\vec{v}| \qquad (3.11)$$

• The First Singular Value: The value [Math Processing Error]$\sigma_1 (A) = |A\vec{v_1}|$ is called the first singular value of [Math Processing Error]$A$. Note that [Math Processing Error]$\sigma_1^2$ is the sum of the squares of the projections of the points to the line determined by [Math Processing Error]$\vec{v_1}$.

• The Second Singular Vector: The second singular vector [Math Processing Error]$\vec{v_2}$, is defined by the best fit line perpendicular to [Math Processing Error]$\vec{v_1}$
  [Math Processing Error] $$\vec{v_2} = \underset{\vec{v} \perp \vec{v_1}, |\vec{v}| = 1}{\operatorname{argmax}} |A\vec{v}| \qquad (3.12)$$

• The Second Singular Value: The value [Math Processing Error]$\sigma_2 (A) = |A\vec{v_2}|$ is called the second singular value of [Math Processing Error]$A$. Note that [Math Processing Error]$\sigma_2^2$ is the sum of the squares of the projections of the points to the line determined by [Math Processing Error]$\vec{v_2}$.

• The Third Singular Vector: The third singular vector [Math Processing Error]$\vec{v_3}$ is defined similarly by
  [Math Processing Error] $$\vec{v_3} = \underset{\vec{v} \perp \vec{v_1}, \vec{v_2}; |\vec{v}| = 1}{\operatorname{argmax}} |A\vec{v}| \qquad (3.13)$$
• The process stops when we have found [Math Processing Error]$\vec{v_1}, \vec{v_2}, \dots, \vec{v_p}$ as singular vectors and [Math Processing Error] $$\underset{\vec{v} \perp \vec{v_1}, \vec{v_2}, \dots \vec{v_p} ; |\vec{v}| = 1}{\operatorname{argmax}} |A\vec{v}| \quad = \quad 0 \text{ (i.e., the projection along } \vec {v_i} \text{ for i > p = rank(A), is zero)} \qquad (3.14)$$ where [Math Processing Error]$p = rank(A)$, i.e, there exist at most [Math Processing Error]$p$ linearly independent eigenvectors.

4) The Frobenius norm of A:

Consider one row, say [Math Processing Error]$\vec{a_j}$ of matrix [Math Processing Error]$A$. Since [Math Processing Error]$\vec{v_1}, \vec{v_2}, \dots, \vec{v_p}$ span the space of all rows of [Math Processing Error]$A$, [Math Processing Error]$\vec{a_j} \cdot \vec{v}$ 0 for all [Math Processing Error]$\vec{v}$ perpendicular to [Math Processing Error]$\vec{v_1}, \vec{v_2}, \dots, \vec{v_p}$. Thus, for each row [Math Processing Error]$\vec{a_j}$, [Math Processing Error]$\sum_{i=1}^p (\vec{a_j} \cdot \vec{v_i})^2 = |\vec{a_j}|^2$. Summing over all rows,
[Math Processing Error] $$\sum_{j=1}^n |\vec{a_j}|^2 = \sum_{j=1}^n \sum_{i=1}^p (\vec{a_j} \cdot \vec{v_i})^2 = \sum_{i=1}^p \sum_{j=1}^n (\vec{a_j} \cdot \vec{v_i})^2 = \sum_{i=1}^p |A \vec{v_i}|^2 = \sum_{i=1}^p \sigma_i^2(A). \qquad (3.15)$$
But [Math Processing Error] $$\sum_{j=1}^n |\vec{a_j}|^2 =\sum_{j=1}^n \vec{a_j} \cdot \vec{a_j} =\sum_{j=1}^n \vec{a_j}^T \vec{a_j} = \sum_{j=1}^n \sum_{k=1}^d a_{jk}^2 \qquad (3.16)$$ that is the sum of squares of all the entries of [Math Processing Error]$A$. Thus, the sum of squares of the singular values of [Math Processing Error]$A$ is indeed the square of the “whole content of [Math Processing Error]$A$”, i.e., the sum of squares of all the entries. There is an important norm associated with this quantity, the Frobenius norm of [Math Processing Error]$A$, denoted by [Math Processing Error]$\|A\|_F$, defined as
[Math Processing Error] $$\text{Frobenius Norm : } \quad \|A\|_F \equiv \sqrt{\sum_{j,k}a_{jk}^2} \qquad (3.17)$$
It is shown is the following lemma:

[Math Processing Error]$\mbox{Lemma 3-1: For any matrix $A$, the sum of squares of the singular values equals the Frobenius norm, i.e., }$
[Math Processing Error] $$\sum_{j=1}^n \sigma_i^2(A) = \|A\|_F^2$$

3.5 Intuitive Interpretations of SVD [see Ref-6]

1) The image shows:

• Upper Left: The unit disc with the two canonical unit vectors.
• Upper Right: Unit disc transformed with M and singular Values [Math Processing Error]$\sigma_1$ and [Math Processing Error]$\sigma_2$ indicated.
• Lower Left: The action of [Math Processing Error]$\mathbf{V}^*$ on the unit disc. This is just a rotation. Here [Math Processing Error]$*$ means conjugate transpose.
• Lower Right: The action of [Math Processing Error]$\Sigma \mathbf{V}^*$ on the unit disc. [Math Processing Error]$\Sigma$ scales in vertically and horizontally.
  In this special case, the singular values are [Math Processing Error]$\phi$ and [Math Processing Error]$1 / \phi$ where [Math Processing Error]$\phi$ is the Golden ratio, i.e., [Math Processing Error] $$\phi = \frac{1 + \sqrt{5}}{2} = 1.618. \qquad (3.18)$$
  [Math Processing Error]$\mathbf{V}^∗$ is a (counter clockwise) rotation by an angle [Math Processing Error]$\alpha$ where [Math Processing Error]$\alpha$ satisfies [Math Processing Error]$tan(\alpha) = -\phi$. [Math Processing Error]$\mathbf{U}$ is a rotation by an angle [Math Processing Error]$\beta$ with [Math Processing Error]$tan(\beta) = \phi-1$.

2) Singular values as semiaxes of an ellipse or ellipsoid:

As shown in the figure, the singular values can be interpreted as the semiaxes of an ellipse in 2D. This concept can be generalized to n-dimensional Euclidean space, with the singular values of any [Math Processing Error]$n \times n$ square matrix being viewed as the semiaxes of an n-dimensional ellipsoid. See below for further details.

3) The columns of U and V are orthonormal bases:

Since [Math Processing Error]$\mathbf{U}$ and [Math Processing Error]$\mathbf{V}^*$ are unitary, the columns of each of them form a set of orthonormal vectors, which can be regarded as basis vectors. The matrix [Math Processing Error]$M$ maps the basis vector [Math Processing Error]$\vec{v_i}$ to the stretched unit vector [Math Processing Error]$\sigma_i \vec{u_i}$ . By the definition of a unitary matrix, the same is true for their conjugate transposes [Math Processing Error]$\mathbf{U}^*$ and [Math Processing Error]$\mathbf{V}$, except the geometric interpretation of the singular values as stretches is lost. In short, the columns of [Math Processing Error]$\mathbf{U}$, [Math Processing Error]$\mathbf{U}^*$, [Math Processing Error]$\mathbf{V}$ and [Math Processing Error]$\mathbf{V}^*$ are orthonormal bases.

4. Expansion of eigenvalues and eigenvectors [see Ref-8]

Problem - PRML Exercise 2.19:

Show that a real, symmetric matrix [Math Processing Error]$A_n$ satisfying the eigenvector equation [Math Processing Error]$A \vec{u_i}=\lambda_i \vec{u_i}$ cam be expressed as an expansion of its eigenvalues and eigenvectors of the following form
[Math Processing Error] $$A=\sum_{i =1}^n \lambda _i \vec{u_i} \vec{u_i}^T \qquad (4.1)$$
and similarly, the inverse [Math Processing Error]$A^{−1}$ can be expressed as
[Math Processing Error] $$A^{-1}=\sum_{i =1}^n \frac{1}{\lambda _i} \vec{u_i} \vec{u_i}^T \qquad (4.2)$$

Solution:

1) Lemma 4-1: 实对称矩阵正交相似于对角矩阵。即： [Math Processing Error]$A_n$为实对称方阵 [Math Processing Error]$\Longrightarrow$ [Math Processing Error]$\exists$正交矩阵[Math Processing Error]$U$, such that

[Math Processing Error] $$U^{-1}AU = U^T A U = \Lambda = \begin{bmatrix} \lambda_1 \\ & \lambda_2 \\ & & \ddots \\ &&& \lambda_n\end{bmatrix} \qquad (4.3)$$
or due to [Math Processing Error]$UU^T = I$, equivalent equations include [Math Processing Error] $$AU = U \Lambda \qquad (4.4)$$ and [Math Processing Error] $$A = U \Lambda U^T \qquad (4.5)$$

2) Lemma 4-2: Matrix [Math Processing Error]$A$ and [Math Processing Error]$B$ are identical if and only if for all vectors [Math Processing Error]$\vec{v}$, [Math Processing Error]$A\vec{v} = B\vec{v}$. That is,

[Math Processing Error] $$A = B \Longleftrightarrow A\vec{v} = B\vec{v}, \text{ for any vector } \vec{v} \qquad (4.6)$$

3) Proof:

The proof of (4.1) and (4.2) use (4.5) and (4.6). For any column vector [Math Processing Error]$\vec{x}$,
we have
[Math Processing Error] $$\begin{split} A\vec{x} = U \Lambda U^T \vec{x} &= U \Lambda \begin{bmatrix} \vec{u_1}^T \vec{x} \\ \vdots \\ \vec{u_n}^T \vec{x} \end{bmatrix} = U \begin{bmatrix} \lambda_1 \vec{u_1}^T \vec{x} \\ \vdots \\ \lambda_n \vec{u_n}^T \vec{x} \end{bmatrix} \\ &= \begin{bmatrix} \vec{u_1}, \dots, \vec{u_n} \\ \end{bmatrix} \begin{bmatrix} \lambda_1 \vec{u_1}^T \vec{x} \\ \vdots \\ \lambda_n \vec{u_n}^T \vec{x} \end{bmatrix} = \sum_{k= 1}^n \lambda_k (\vec{u_k}^T \vec{x} ) \vec{u_k} \qquad(4.7) \\ \end{split}$$

Since the inner product [Math Processing Error]$\vec{u_k} \cdot \vec{x} = \vec{u_i}^T \vec{x}$ in (4.7) is a scalar, and [Math Processing Error]$\lambda_k$ is also a scalar, therefore we can change the order of the terms,

[Math Processing Error] $$\begin{split} A\vec{x} &= \sum_{k= 1}^n \lambda_k \left(\vec{u_k}^T \vec{x} \right) \vec{u_k} \\ &= \sum_{k= 1}^n \lambda_k \vec{u_k} \left(\vec{u_k}^T \vec{x} \right) \quad \text{(since inner product is scalar)}\\ &= \left(\sum_{k= 1}^n \lambda_k \vec{u_k} \vec{u_k}^T \right) \vec{x} \quad \text{(due to associative property of multiplication)} \\ &= \qquad \qquad (4.8)\end{split}$$
Thus applying the Lemma 2 shown in (4.6) to (4.8), we can prove (4.1).

Since [Math Processing Error]$AU=U\Lambda$, inverting both sides gives [Math Processing Error]$U^TA^{−1}=\Lambda^{−1}U^T$, and hence [Math Processing Error]$A^{−1}=U \Lambda^{−1}U^T$. Applying the above result to [Math Processing Error]$A^{-1}$, noting that [Math Processing Error]$\Lambda^{−1}$ is just the diagonal matrix of the inverses of the diagonal elements of [Math Processing Error]$\Lambda$, we have proved (4.2).

5. Best Rank k Approximation using SVD [see Ref-5]

Let [Math Processing Error]$A$ be an [Math Processing Error]$n \times d$ matrix and think of the rows of [Math Processing Error]$A$ as [Math Processing Error]$n$ points in d-dimensional space. There are two important matrix norms, the Frobenius norm denoted [Math Processing Error]$\|A\|_F$ and the 2-norm denoted [Math Processing Error]$\|A\|_2$.
- The 2-norm of the matrix A is given by [Math Processing Error] $$\underset{|\vec{v}| = 1}{\operatorname{argmax}} |A \vec{v} |$$ and thus equals the largest singular value of the matrix. That is, the 2-norm is the square root of the sum of squared distances to the origin along the direction that maximizes this quantity.
- The Frobenius norm of [Math Processing Error]$A$ is the square root of the sum of the squared distance of the points to the origin, shown in (3.17).

Let [Math Processing Error]$r = rank(A)$ and [Math Processing Error] $$A=\sum_{i =1}^r \lambda_i \vec{u_i} \vec{u_i}^T \qquad (5.1)$$ be the SVD of [Math Processing Error]$A$. For [Math Processing Error]$k \in \{ 1, 2, \dots, r \}$, let [Math Processing Error] $$A_k=\sum_{i =1}^k \lambda_i \vec{u_i} \vec{u_i}^T \qquad (5.2)$$ be the sum truncated after [Math Processing Error]$k$ terms. It is clear that [Math Processing Error]$A_k$ has rank [Math Processing Error]$k$. Furthermore, [Math Processing Error]$A_k$ is the best rank [Math Processing Error]$k$ approximation to [Math Processing Error]$A$ when the error is measured in either the 2-norm or the Frobenius norm (see Theorem 5.2 and Theorem 5.3).
Without proof, we give the following theorems (if interested, please check Lemma 1.6, Theorem 1.7, Theorem 1.8, and Theorem 1.9 in page 9-10 of Ref-5).

Theorem 5.1:

The rows of matrix [Math Processing Error]$A_k$ are the projections of the rows of [Math Processing Error]$A$ onto the subspace [Math Processing Error]$V_k$ spanned by the first [Math Processing Error]$k$ singular vectors of [Math Processing Error]$A$.

Theorem 5.2:

Let [Math Processing Error]$A$ be an [Math Processing Error]$n \times d$ matrix, for any matrix [Math Processing Error]$B$ of rank at most [Math Processing Error]$k$, it holds that [Math Processing Error] $$\|A − A_k\|_F ≤ \|A − B\|_F$$

Theorem 5.3:

Let [Math Processing Error]$A$ be an [Math Processing Error]$n \times d$ matrix, for any matrix [Math Processing Error]$B$ of rank at most [Math Processing Error]$k$, it holds that [Math Processing Error] $$\|A − A_k\|_2 ≤ \|A − B\|_2$$

Theorem 5.4:

Let [Math Processing Error]$A$ be an [Math Processing Error]$n \times d$ matrix, for [Math Processing Error]$A_k$ in (5.2) it holds that [Math Processing Error] $$\|A − A_k\|_2 = \lambda_{k+1}^2$$

6. The Geometry of Linear Transformations [see Ref-3]

6.1 Matrix and Transformation

Let us begin by looking at some simple matrices, namely those with two rows and two columns. Our first example is the diagonal matrix [Math Processing Error] $$M_1 = \begin{bmatrix} 3 & 0 \\ 0 & 1\end{bmatrix}$$

Geometrically, we may think of a matrix like this as taking a point [Math Processing Error]$\vec{p_1} = (x, y)$ in the plane and transforming it into another point [Math Processing Error]$\vec{p_2}$using matrix multiplication: [Math Processing Error] $$\vec{p_2} = M_1 \vec{p_1} = \begin{bmatrix} 3 & 0 \\ 0 & 1\end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 3x \\ y \end{bmatrix}$$

The effect of this transformation is shown below : the plane is horizontally stretched by a factor of [Math Processing Error]$3$, while there is no vertical change.

Now let’s look at [Math Processing Error] $$M_2 = \begin{bmatrix} 2 & 1 \\ 1 & 2\end{bmatrix}$$ . [Math Processing Error] $$\vec{p_3} = M_2 \vec{p_1} = \begin{bmatrix} 2 & 1 \\ 1 & 2\end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 2x+y \\ x + 2y \end{bmatrix}$$
The four vertices of the red square shown in the following figure, [Math Processing Error]$(0, 0), (0, 1), (1, 1), \text{and }(1, 0)$ are transformed into [Math Processing Error]$(0, 0), (1, 2), (3, 3), \text{and }(2, 1)$, respectively, which produces this effect

It is not so clear how to describe simply the geometric effect of the transformation. However, let’s rotate our grid through a [Math Processing Error]$45 ^{\circ}$ angle and see what happens. The four vertices of the red square, [Math Processing Error]$(0, 0), (-\frac{\sqrt{2}}{2}, \frac{\sqrt{2}}{2}), (0, 1), \text{and }(\frac{\sqrt{2}}{2}, \frac{\sqrt{2}}{2})$ are transformed into [Math Processing Error]$(0, 0), (-\frac{\sqrt{2}}{2}, \frac{\sqrt{2}}{2}), (1, 2), \text{and }(\frac{3\sqrt{2}}{2}, \frac{3\sqrt{2}}{2})$, respectively, which produces this effect

We see now that this new grid is transformed in the same way that the original grid was transformed by the diagonal matrix: the grid is stretched by a factor of [Math Processing Error]$3$ in one direction.

This is a very special situation due to the fact that the matrix [Math Processing Error]$M_2$ is symmetric, i.e., [Math Processing Error]$M_2^T = M_2$. If we have a symmetric [Math Processing Error]$2 \times 2$ matrix, it turns out that

we may always rotate the grid in the domain so that the matrix acts by stretching and perhaps reflecting in the two directions. In other words, symmetric matrices behave like diagonal matrices.

结论：

以上的几张图，就是为了讨论given a [Math Processing Error]$2 \times 2$ symmetric matrix [Math Processing Error]$M_2$, 即 [Math Processing Error] $$M_2 ^T = M_2 = \begin{bmatrix} 2 & 1 \\ 1 & 2\end{bmatrix}.$$
- 如何放置坐标grid（或者说如何确定一个单位长度的正方形在坐标系中的位置和方向， 要知道这个正方形可以用两个彼此互相垂直的单位向量[Math Processing Error]$v_1$ 和[Math Processing Error]$v_2$ 来表示），使得当该正方形被施加transformation（represented by a symmetric matrix [Math Processing Error]$M_2$）时，这个正方形的形变发生沿着[Math Processing Error]$v_1$ 和 [Math Processing Error]$v_2$ 方向的单纯的拉伸或压缩。这就与后面即将讨论的矩阵的特征向量和特征值联系起来。 即：
[Math Processing Error] $$\begin{split} M_2 v_1 &= \lambda_1 v_1 \\ M_2 v_2 &= \lambda_2 v_2 \end{split}$$
表示特征向量 [Math Processing Error]$v_1, v_2$ 被矩阵 [Math Processing Error]$M_2$ 变换之后，新的向量与原来向量平行（包括同向和反向），只是模长发生了改变而已。
- 如何求得这样的[Math Processing Error]$v_1$ 和 [Math Processing Error]$v_2$呢？ 答案就是当[Math Processing Error]$M$为对称矩阵时（当然，对称矩阵是一种特殊情况，接下来我们会讨论更为一般的矩阵），这样的[Math Processing Error]$v_1$ 和 [Math Processing Error]$v_2$就是对称矩阵[Math Processing Error]$M$的两个特征向量。即由[Math Processing Error]$det(M_2 - \lambda I) = 0$，求得特征向量和特征值为：
[Math Processing Error] $$\begin{split} v_1 &= \left( \frac{\sqrt{2}}{2}, \frac{\sqrt{2}}{2}\right)^T \quad \lambda_1 = 3 \\ v_2 &= \left( \frac{\sqrt{2}}{2}, -\frac{\sqrt{2}}{2}\right)^T \quad \lambda_2 = 1 \end{split} \qquad (6.1)$$
which accords with the [Math Processing Error]$45^\circ$ rotation of the red sqaure shown above.
- 对于这种特殊的对称矩阵[Math Processing Error]$M$, 它的SVD就演变成了 Lemma 4-1: 实对称矩阵正交相似于对角矩阵，正如（4.5）所示。可以把它看成是SVD的一种特殊情况， 即：对于矩阵[Math Processing Error]$M$, 有如下SVD:
- 对于一般的矩阵[Math Processing Error]$M$, 存在正交矩阵[Math Processing Error]$U$和[Math Processing Error]$V$（即[Math Processing Error]$UU^T = I, VV^T = I$）, 使得 [Math Processing Error] $$M = U \Lambda V^T \qquad (6.2)$$ 对于(6.2)， [Math Processing Error]$U$ 即为[Math Processing Error]$MM^T$的特征向量组成，[Math Processing Error]$V$ 即为[Math Processing Error]$M^TM$的特征向量组成，对角矩阵[Math Processing Error]$\Lambda$由[Math Processing Error]$MM^T$（或者[Math Processing Error]$M^TM$）的特征值的正平方根构成 。
- 当[Math Processing Error]$M$是实对称矩阵时， 存在正交矩阵[Math Processing Error]$U$（即[Math Processing Error]$UU^T = I$）, 使得 [Math Processing Error] $$M = U \Lambda U^T \qquad (6.3)$$
对于(6.3)， [Math Processing Error]$U$ 即为对称矩阵[Math Processing Error]$M$的特征向量组成，对角矩阵[Math Processing Error]$\Lambda$为对称矩阵[Math Processing Error]$M$的特征值构成 。当然也可以通过上面介绍的方法求解，即[Math Processing Error]$U$ 是由[Math Processing Error]$MM^T$的特征向量组成，对角矩阵[Math Processing Error]$\Lambda$由[Math Processing Error]$MM^T$的特征值的正平方根构成[Math Processing Error]$MM^T$。两种方法是等价的、是一致的。

6.2 The Geometry of Eigenvectors and Eigenvalues

Said with more mathematical precision, given a symmetric matrix [Math Processing Error]$M$, we may find a set of orthogonal vectors [Math Processing Error]$v_i$ so that [Math Processing Error]$M v_i$ is a scalar multiple of [Math Processing Error]$v_i$; that is [Math Processing Error] $$M v_i = \lambda_i v_i \qquad (6.4)$$
where [Math Processing Error]$\lambda_i$ is a scalar.

Geometrically, this means that the vectors [Math Processing Error]$v_i$ are simply stretched and/or reflected(即方向改变了180°) when multiplied by [Math Processing Error]$M$. Because of this property, we call
- Eigenvectors: the vectors [Math Processing Error]$v_i$ eigenvectors of [Math Processing Error]$M$;
- Eigenvalues: the scalars [Math Processing Error]$\lambda_i$ are called eigenvalues.

An important fact, which is easily verified, is that eigenvectors of a symmetric matrix corresponding to different eigenvalues are orthogonal. If we use the eigenvectors of a symmetric matrix to align the grid, the matrix stretches and/or reflects the grid in the same way that it does the eigenvectors.

The geometric description we gave for this linear transformation is a simple one: the grid is simply stretched in one direction. For more general matrices, we will ask if we can find an orthogonal grid that is transformed into another orthogonal grid. Let’s consider a final example using a matrix that is not symmetric: [Math Processing Error] $$M = \begin{bmatrix} 1 & 1 \\ 0 & 1\end{bmatrix}$$

This matrix produces the geometric effect known as a shear, shown as

It’s easy to find one family of eigenvectors along the horizontal axis. However, our figure above shows that these eigenvectors cannot be used to create an orthogonal grid that is transformed into another orthogonal grid.
- Nonetheless, let’s see what happens when we rotate the grid first by [Math Processing Error]$30 ^\circ$, shown as

Notice that the angle at the origin formed by the red parallelogram on the right has increased.
- Let’s next rotate the grid by [Math Processing Error]$60 ^\circ$.

It appears that the grid on the right is now almost orthogonal.
- In fact, by rotating the grid in the domain by an angle of roughly [Math Processing Error]$58.28^\circ$, both grids are now orthogonal.

How to calculate this angle of roughly [Math Processing Error]$58.28^\circ$?

Solution:

Based on the discussion in (6.2), The columns of [Math Processing Error]$V$ are the eigenvectors of [Math Processing Error]$M^TM$, results in [Math Processing Error]$(M^T M- \lambda I ) x = \vec{0}$, and
[Math Processing Error] $$det(M^TM - \lambda I) = \begin{vmatrix} 1- \lambda & 1 \\ 1 & 2 - \lambda\end{vmatrix} =(2 - \lambda) (1 - \lambda) - 1 = \lambda^2 - 3 \lambda + 1 \equiv 0,$$ where [Math Processing Error] $$M = \begin{bmatrix} 1 & 1 \\ 0 & 1\end{bmatrix}.$$
We can get [Math Processing Error]$\lambda_1 = \frac{3+\sqrt{5}}{2} = 2.6180 , \lambda_2 = \frac{3 - \sqrt{5}}{2} = 0.3820$, the corresponding eigenvectors are [Math Processing Error] $$v_1 = [-0.5257, -0.8507 ]^T , v_2 = [-0.8507, 0.5257]^T.$$
where the directions of [Math Processing Error]$v_1$ and [Math Processing Error]$v_2$ are ( You can run Matlab function [Math Processing Error]$[U, S, V] = svd(M' * M)$ to get the result as follows)
[Math Processing Error] $$\begin{split} \tan(\alpha) &= v_{12} / v_{11} = \lambda_1 -1 = \frac{1 + \sqrt{5}}{2} \\ \tan(\beta) &= v_{22} / v_{21} = \lambda_2 -1 = \frac{1 - \sqrt{5}}{2} \\ \Longleftrightarrow \alpha &= \arctan \left( \frac{1 + \sqrt{5}}{2} \right) = 58.2825^\circ \quad \text{(see (3.18) Golden Ratio)}\\ \beta &= \arctan\left( \frac{1 - \sqrt{5}}{2} \right) = -31.7175 ^\circ \end{split}$$

6.3 The singular value decomposition

This is the geometric essence of the singular value decomposition for [Math Processing Error]$2 \times 2$ matrices:

for any [Math Processing Error]$2 \times 2$ matrix, we may find an orthogonal grid that is transformed into another orthogonal grid. We will express this fact using vectors:
- with an appropriate choice of orthogonal unit vectors [Math Processing Error]$v_1$ and [Math Processing Error]$v_2$, the vectors [Math Processing Error]$Mv_1$ and [Math Processing Error]$Mv_2$ are orthogonal.

We will use [Math Processing Error]$u_1$ and [Math Processing Error]$u_2$ to denote unit vectors in the direction of [Math Processing Error]$Mv_1$ and [Math Processing Error]$Mv_2$. The lengths of [Math Processing Error]$Mv_1$ and [Math Processing Error]$Mv_2$ – denoted by [Math Processing Error]$\sigma_1$ and [Math Processing Error]$\sigma_2$ – describe the amount that the grid is stretched in those particular directions. These numbers are called the singular values of [Math Processing Error]$M$. (In this case, the singular values are the golden ratio and its reciprocal, but that is not so important here.)

We therefore have
[Math Processing Error] $$Mv_1 = \sigma_1 u_1 \qquad (6.5)$$
[Math Processing Error] $$Mv_2 = \sigma_2 u_2 \qquad (6.6)$$

We may now give a simple description for how the matrix [Math Processing Error]$M$ treats a general vector [Math Processing Error]$x$. Since the vectors [Math Processing Error]$v_1$ and [Math Processing Error]$v_2$ are orthogonal unit vectors, we have [Math Processing Error] $$x = \left(v_1 \cdot x \right) v_1 + \left(v_2 \cdot x \right) v_2 \qquad (6.7)$$

This means that [Math Processing Error] $$\begin{split} Mx &= M \left[\left(v_1 \cdot x \right) v_1 + \left(v_2 \cdot x \right) v_2 \right] \\ &\text{(changing the order of terms, since inner product is scalar value.)} \\ &= \left(v_1 \cdot x \right) M v_1 + (v_2 \cdot x) M v_2 \quad \text{(applying (6.5 & 6.6))}\\ &= \left(v_1 \cdot x \right) \sigma_1 u_1 + \left(v_2 \cdot x \right) \sigma_2 u_2 \end{split} \qquad (6.8)$$

Remember that the inner dot product may be computed using the vector transpose [Math Processing Error] $$\left(v \cdot x \right) = v^T x \qquad (6.9)$$
which leads to
[Math Processing Error] $$\begin{split} Mx &= \left(v_1 \cdot x \right) \sigma_1 u_1 + \left(v_2 \cdot x \right) \sigma_2 u_2 \\ &= \left( v_1^T x \right) \sigma_1 u_1 + \left(v_2^T x \right) \sigma_2 u_2 \\ &\text{(changing the order of terms, since inner product is scalar value.)} \\ &= u_1 \sigma_1 \left( v_1^T x \right) + u_2 \sigma_2 \left(v_2^T x \right) \\ &\text{(due to associative rule of production)} \\ &= \left( u_1 \sigma_1 v_1^T \right) x + \left( u_2 \sigma_2 v_2^T \right) x \quad \text{, for } \forall x \\ & \Longleftrightarrow M = \left( u_1 \sigma_1 v_1^T \right) + \left( u_2 \sigma_2 v_2^T \right) \quad\text{(applying equation (4.6))} \end{split} \qquad (6.10)$$

This is usually expressed by writing [Math Processing Error] $$M = U \Lambda V^T \qquad (6.11)$$
where [Math Processing Error]$U$ is a matrix whose columns are the vectors [Math Processing Error]$u_1$ and [Math Processing Error]$u_2$, [Math Processing Error]$\Lambda$ is a diagonal matrix whose entries are [Math Processing Error]$\sigma_1$ and [Math Processing Error]$\sigma_2$, and [Math Processing Error]$V$ is a matrix whose columns are [Math Processing Error]$v_1$ and [Math Processing Error]$v_2$.

This shows how to decompose the matrix [Math Processing Error]$M$ into the product of three matrices:
- [Math Processing Error]$V$ describes an orthonormal basis in the domain (定义域), and
- [Math Processing Error]$U$ describes an orthonormal basis in the co-domain (值域), and
- [Math Processing Error]$\Lambda$ describes how much the vectors in [Math Processing Error]$V$ are stretched to give the vectors in [Math Processing Error]$U$.

6.4 How do we find the singular decomposition?

The power of the singular value decomposition lies in the fact that we may find it for any matrix. How do we do it? Let’s look at our earlier example and add the unit circle in the domain (定义域). Its image will be an ellipse whose major and minor axes define the orthogonal grid in the co-domain (值域).

Notice that the major and minor axes are defined by [Math Processing Error]$Mv_1$ and [Math Processing Error]$Mv_2$. These vectors therefore are the longest and shortest vectors among all the images of vectors on the unit circle.

In other words, the function [Math Processing Error]$|Mx|$ on the unit circle has a maximum at [Math Processing Error]$v_1$ and a minimum at [Math Processing Error]$v_2$. This reduces the problem to a rather standard calculus problem in which we wish to optimize a function over the unit circle. It turns out that the critical points of this function occur at the eigenvectors of the matrix [Math Processing Error]$M^TM$. Since this matrix is symmetric (since it is obvious that [Math Processing Error]$(M^TM)^T = M^TM$), eigenvectors corresponding to different eigenvalues will be orthogonal. This gives the family of vectors [Math Processing Error]$v_i$.

The singular values are then given by [Math Processing Error]$\sigma_i = | M v_i |$, and the vectors [Math Processing Error]$u_i$ are obtained as unit vectors in the direction of [Math Processing Error]$M v_i$.

But why are the vectors [Math Processing Error]$u_i$ orthogonal? To explain this, we will assume that [Math Processing Error]$\sigma_i$ and [Math Processing Error]$\sigma_j$ are distinct singular values. We have [Math Processing Error] $$\begin{split} Mv_i &= \sigma_i u_i \\ Mv_j &= \sigma_j u_j\end{split} \qquad (6.12)$$
Let’s begin by looking at the expression [Math Processing Error]$(Mv_i \cdot Mv_j)$ and assuming, for convenience, that the singular values are non-zero.
- On one hand, this expression is zero due to the orthogonal-to-one-another vectors [Math Processing Error]$v_i$s’ and [Math Processing Error]$v_j$s’, which are required to be eigenvectors of the symmetric matrix [Math Processing Error]$M^TM$, i.e., [Math Processing Error] $$M^T M v_j = \lambda_j v_j\qquad (6.13)$$
Therefore,
[Math Processing Error] $$\begin{split} (Mv_i) \cdot (Mv_j) &= (Mv_i)^T (Mv_j) = v_i^T (M^T M v_j) \\ &= v_i^T (\lambda_j v_j) \quad\text{(applying equation (6.13))} \\ &= \lambda_j v_i^T v_j = \lambda_j (v_i \cdot v_j) = 0 \end{split} \qquad(6.14)$$
- On the other hand, we have
[Math Processing Error] $$(Mv_i) \cdot (Mv_j) = \sigma_i \sigma_j (u_i \cdot u_j) = 0 \quad\text{(applying equation (6.12))}$$ Therefore, [Math Processing Error]$u_i$ and [Math Processing Error]$u_j$ are orthogonal, so we have found an orthogonal set of vectors [Math Processing Error]$v_i$ that is transformed into another orthogonal set [Math Processing Error]$u_i$. The singular values describe the amount of stretching in the different directions.

In practice, this is not the procedure used to find the singular value decomposition of a matrix since it is not particularly efficient or well-behaved numerically.

6.5 Another example

Let’s now look at the singular matrix [Math Processing Error] $$M = \begin{bmatrix} 1 & 1 \\ 2 & 2\end{bmatrix}$$

We can get [Math Processing Error]$\lambda_1 = 10 , \lambda_2 = 0$, the corresponding eigenvectors are [Math Processing Error] $$v_1 = [-\frac{\sqrt{2}}{2}, -\frac{\sqrt{2}}{2} ]^T , v_2 = [ -\frac{\sqrt{2}}{2}, \frac{\sqrt{2}}{2} ]^T.$$
where the directions of [Math Processing Error]$v_1$ and [Math Processing Error]$v_2$ are ( You can run Matlab function [Math Processing Error]$[U, S, V] = svd(M' * M)$ to get the result as follows)
[Math Processing Error] $$\begin{split} \alpha &= arctan2 \left( -\frac{\sqrt{2}}{2}, -\frac{\sqrt{2}}{2} \right) = -135 ^\circ \\ \beta &= \arctan\left(\frac{\sqrt{2}}{2}, -\frac{\sqrt{2}}{2} \right) = 135 ^\circ \end{split} \qquad (6.15)$$

The geometric effect of this matrix is the following:

In this case, the second singular value is zero so that we may write:
[Math Processing Error] $$M = u_1 \sigma_1 v_1^T \qquad (6.16)$$

In other words, if some of the singular values are zero, the corresponding terms do not appear in the decomposition for [Math Processing Error]$M$. In this way, we see that the rank of [Math Processing Error]$M$, which is the dimension of the image of the linear transformation, is equal to the number of non-zero singular values.

6.6 SVD Application 1 – Data compression

Singular value decompositions can be used to represent data efficiently. Suppose, for instance, that we wish to transmit the following image, which consists of an array of [Math Processing Error]$15 \times 25$ black or white pixels.

Since there are only three types of columns in this image, as shown below, it should be possible to represent the data in a more compact form.

We will represent the image as a [Math Processing Error]$15 \times 25$ matrix [Math Processing Error]$M$ in which each entry is either a 0, representing a black pixel, or 1, representing white. As such, there are [Math Processing Error]$375$ entries in the matrix. If we perform a singular value decomposition on [Math Processing Error]$M$, we find there are only three non-zero singular values [Math Processing Error]$\sigma_1 = 14.72, \sigma_2 = 5.22, \sigma_3 = 3.31.$
Therefore, the matrix [Math Processing Error]$M$ may be represented as
[Math Processing Error] $$M=u_1 \sigma_1 v_1^T + u_2 \sigma_2 v_2^T + u_3 \sigma_3 v_3^T \qquad (6.17)$$

This means that we have three vectors [Math Processing Error]$v_i$, each of which has [Math Processing Error]$15$ entries, three vectors [Math Processing Error]$u_i$, each of which has [Math Processing Error]$25$ entries, and three singular values [Math Processing Error]$\sigma_i$. This implies that we may represent the matrix using only [Math Processing Error]$123$ numbers rather than the [Math Processing Error]$375$ that appear in the matrix. In this way, the singular value decomposition discovers the redundancy in the matrix and provides a format for eliminating it.

Why are there only three non-zero singular values? Remember that the number of non-zero singular values equals the rank of the matrix. In this case, we see that there are three linearly independent columns in the matrix, which means that [Math Processing Error]$rank(M) = 3$.

6.7 SVD Application 2 – Noise reduction

The previous example showed how we can exploit a situation where many singular values are zero. Typically speaking, the large singular values point to where the interesting information is. For example, imagine we have used a scanner to enter this image into our computer. However, our scanner introduces some imperfections (usually called “noise“) in the image.

We may proceed in the same way: represent the data using a [Math Processing Error]$15 \times 25$ matrix and perform a singular value decomposition. We find the following singular values:
[Math Processing Error] $$\begin{split}\sigma_1 &= 14.15 \\ \sigma_1 & = 4.67 \\ \sigma_2 &= 3.00 \\ \sigma_3 & = 0.21 \\ \sigma_4 &= 0.19 \\ \dots \\ \sigma_{15} &= 0.05 \end{split}$$

Clearly, the first three singular values are the most important so we will assume that the others are due to the noise in the image and make the approximation
[Math Processing Error] $$M \simeq u_1 \sigma_1 v_1^T + u_2 \sigma_2 v_2^T + u_3 \sigma_3 v_3^T \qquad (6.18)$$
This leads to the following improved image.

6.8 SVD Application 3 – Data analysis

Noise also arises anytime we collect data: no matter how good the instruments are, measurements will always have some error in them. If we remember the theme that large singular values point to important features in a matrix, it seems natural to use a singular value decomposition to study data once it is collected. As an example, suppose that we collect some data as shown below:

We may take the data and put it into a matrix:
[Math Processing Error] $$\begin{bmatrix} -1.03 & 0.74 & -0.02 & 0.51 & -1.31 & 0.99 & 0.69 & -0.12 & -0.72 & 1.11\\ -2.23& 1.61 & -0.02 & 0.88 & -2.39 & 2.02 & 1.62 & -0.35 & -1.67 & 2.46 \end{bmatrix}$$
and perform a singular value decomposition. We find the singular values [Math Processing Error] $$\sigma_1 = 6.04, \sigma_2 = 0.22$$

With one singular value so much larger than the other, it may be safe to assume that the small value of [Math Processing Error]$\sigma_2$ is due to noise in the data and that this singular value would ideally be [Math Processing Error]$0$. In that case, the matrix would have rank one meaning that all the data lies on the line defined by [Math Processing Error]$u_1$.

This brief example points to the beginnings of a field known as principal component analysis (PCA), a set of techniques that uses singular values to detect dependencies and redundancies in data.

In a similar way, singular value decompositions can be used to detect groupings in data, which explains why singular value decompositions are being used in attempts to improve Netflix’s movie recommendation system. Ratings of movies you have watched allow a program to sort you into a group of others whose ratings are similar to yours. Recommendations may be made by choosing movies that others in your group have rated highly.

7. Summary

The logic relationship of those concepts is shown in the following figure:

8. Reference

[1]: Kirk Baker, Singular Value Decomposition Tutorial,
https://www.ling.ohio-state.edu/~kbaker/pubs/Singular_Value_Decomposition_Tutorial.pdf;
[2]: Singular Value Decomposition (SVD) tutorial,
http://web.mit.edu/be.400/www/SVD/Singular_Value_Decomposition.htm;
[3]: We Recommend a Singular Value Decomposition,
http://www.ams.org/samplings/feature-column/fcarc-svd;
[4]: Computation of the Singular Value Decomposition,
http://www.cs.utexas.edu/users/inderjit/public_papers/HLA_SVD.pdf;
[5]: CMU, SVD Tutorial,
https://www.cs.cmu.edu/~venkatg/teaching/CStheory-infoage/book-chapter-4.pdf.
[6]: Wiki: Singular value decomposition,
https://en.wikipedia.org/wiki/Singular_value_decomposition.
[7]: Chapter 6 Eigenvalues and Eigenvectors,
http://math.mit.edu/~gs/linearalgebra/ila0601.pdf.
[8]: Expressing a matrix as an expansion of its eigenvalues,
http://math.stackexchange.com/questions/331826/expressing-a-matrix-as-an-expansion-of-its-eigenvalues.
[9]: Wiki: Eigenvalues and eigenvectors,
https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors.