机器学习中的数学——奇异值和奇异值分解

前言

我们知道,只有方阵才有特征值和特征向量,因此只有方阵才能特征值分解, 那么非方阵怎么办?

奇异值和奇异值分解

特征值分解只适用于方阵, 奇异值分解适用于任意的矩阵.

奇异值分解 𝑆𝑖𝑛𝑔𝑢𝑙𝑎𝑟 𝑣𝑎𝑙𝑢𝑒 𝑑𝑒𝑐𝑜𝑚𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛，𝑆𝑉𝐷 ，一种重要的矩阵分解方法以一种方便快捷的方式将我们感兴趣的矩阵分解成更简单且有直观意义的矩阵的乘积
$$X\text{X}X=U\text{U}U\Sigma {V\text{V}V}^{-1}$$

𝑈, 𝑉是两个正交阵¹，𝛴是对角阵，对角元是𝑋的奇异值

---

例：
$$A\text{A}A=\left[\begin{array}{cc}4& 0\\ 3& -5\end{array}\right]$$想分解的形式:$A=U\Sigma V^T\text{A=UΣVT}A=U\Sigma V^T$.

分解步骤:

1. 求转置矩阵

${A\text{A}A}^T=\left[\begin{array}{cc}4& 3\\ 0& -5\end{array}\right]$

${A\text{A}A}^{T}A\text{A}A=\left[\begin{array}{cc}4& 3\\ 0& -5\end{array}\right]\left[\begin{array}{cc}4& 0\\ 3& -5\end{array}\right]=\left[\begin{array}{cc}25& -15\\ -15& 25\end{array}\right]$

2. 求特征值:

${A\text{A}A}^{T}A\text{A}A-\lambda E\text{E}E=\left[\begin{array}{cc}25-\lambda & -15\\ -15& 25-\lambda \end{array}\right]$

${\lambda }_1=40{\lambda }_2=10$

3. 求奇异值:$s_1=\sqrt{{\lambda }_1}=6.3245s_2=\sqrt{{\lambda }_2}=3.1622$

$S\text{S}S=\left[\begin{array}{cc}6.3245& 0\\ 0& 3.1622\end{array}\right];{S\text{S}S}^{-1}=\left[\begin{array}{cc}0.1581& 0\\ 0& 0.3164\end{array}\right]$

针对特征值${\lambda }_1$,计算特征向量:
${A\text{A}A}^{T}A\text{A}A-\lambda E\text{E}E=\left[\begin{array}{cc}-15& -15\\ -15& -15\end{array}\right]$

$({A\text{A}A}^{T}A\text{A}A-\lambda E\text{E}E){X\text{X}X}_1=0$

$\left[\begin{array}{cc}-15& -15\\ -15& -15\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]=0$

$x_1=-x_2;L=\sqrt{x_1^2-x_2^2}=x_1\sqrt{2}$

得:${X\text{X}X}_1=\left[\begin{array}{c}{x}_1/L\\ x_2/L\end{array}\right]=\left[\begin{array}{c}1/\sqrt{2}\\ -1/\sqrt{2}\end{array}\right]=\left[\begin{array}{c}0.7071\\ -0.7071\end{array}\right]$

同理可得${X\text{X}X}_2=\left[\begin{array}{c}0.7071\\ 0.7071\end{array}\right]$

我们想分解的形式为:$A\text{A}A=U\text{U}US\text{S}S{V\text{V}V}^T$

$V\text{V}V=\left[\begin{array}{cc}{X\text{X}X}_1& {X\text{X}X}_2\end{array}\right]=\left[\begin{array}{cc}0.7071& 0.7071\\ -0.7071& 0.7071\end{array}\right]$

${V\text{V}V}^T=\left[\begin{array}{cc}0.7071& -0.7071\\ 0.7071& 0.7071\end{array}\right]$

$U\text{U}U=AV{S}^{-1}\text{AVS−1}AV{S}^{-1}=\left[\begin{array}{cc}0.4472& 0.8944\\ 0.8944& -0.4472\end{array}\right]$

$A=US{V}^T\text{A=USVT}A=US{V}^T=\left[\begin{array}{cc}3.9998& 0\\ 2.9999& -4.9997\end{array}\right]\approx \left[\begin{array}{cc}4& 0\\ 3& -5\end{array}\right]$

代码实例:

import numpy as np
x2 = np.array([[4, 0], [3, 5]])
u, sigma, vt = np.linalg.svd(x2)
print('u', u)
print('sigma', sigma)
print('vt', vt)

u [[-0.4472136  -0.89442719]
 [-0.89442719  0.4472136 ]]
sigma [6.32455532 3.16227766]
vt [[-0.70710678 -0.70710678]
 [-0.70710678  0.70710678]]

---

寻找有效特征,数据压缩

---

1. 如果：$A\text{A}A{A\text{A}A}^T=E\text{E}E$（$E\text{E}E$为单位矩阵，${A\text{A}A}^t$表示“矩阵A的转置矩阵”。）或${A\text{A}A}^{T}A\text{A}A=E\text{E}E$，则n阶实矩阵A称为正交矩阵。 ↩︎