矩阵低秩近似Eckart-Young Theorem

引理1

设$\mathbf{U},\mathbf{V}$是酉矩阵
$$\Vert \mathbf{UA}{\Vert }_2=\Vert \mathbf{AV}{\Vert }_2=\Vert \mathbf{UAV}{\Vert }_2=\Vert \mathbf{A}{\Vert }_2$$
证明：
利用$\Vert \mathbf{A}{\Vert }_2=\sqrt{{\lambda }_{max}\left({\mathbf{A}}^H\mathbf{A}\right)}$就很显然了

引理2

设$\mathbf{A},\mathbf{B}\in {\mathbb{C}}^{m\times n}$, q = min ⁡ { m , n } q=\min\left\{m,n\right\} q=min{m,n}
${\sigma }_1\ge {\sigma }_2\ge \cdots \ge {\sigma }_q$代表奇异值
则
$${\sigma }_{i+j-1}\left(\mathbf{A}+\mathbf{B}\right)\le {\sigma }_i\left(\mathbf{A}\right)+{\sigma }_j\left(\mathbf{B}\right)$$
其中$1\le i,j\le q,i+j\le q+1$
证明：
对$\mathbf{A},\mathbf{B}$做SVD分解
$\mathbf{A}=\mathbf{V}{\mathbf{\Sigma }}_{\mathbf{A}}{\mathbf{W}}^H$
$\mathbf{B}=\mathbf{X}{\mathbf{\Sigma }}_{\mathbf{B}}{\mathbf{Y}}^H$
设
$\mathbf{W}=\left({\mathbf{w}}_1,\cdots ,{\mathbf{w}}_n\right),\mathbf{Y}=\left({\mathbf{y}}_1,\cdots ,{\mathbf{y}}_n\right)$
$\mathbf{V}=\left({\mathbf{w}}_1,\cdots ,{\mathbf{w}}_m\right),\mathbf{X}=\left({\mathbf{y}}_1,\cdots ,{\mathbf{y}}_m\right)$

设 S ′ = span ⁡ { w i , ⋯   , w n } \mathbf{S}'=\operatorname{span}\left\{\mathbf{w}_i,\cdots,\mathbf{w}_n\right\} S′=span{wi​,⋯,wn​}
S ′ ′ = span ⁡ { y i , ⋯   , y n } \mathbf{S}''=\operatorname{span}\left\{\mathbf{y}_i,\cdots,\mathbf{y}_n\right\} S′′=span{yi​,⋯,yn​}
注意到
$$\begin{array}{rl}v& =\dim \left({\mathbf{S}}^{\prime }\cap {\mathbf{S}}^{\prime \prime }\right)\\ & =\dim \left({\mathbf{S}}^{\prime }\right)+\dim \left({\mathbf{S}}^{\prime \prime }\right)-\dim \left({\mathbf{S}}^{\prime }\cup {\mathbf{S}}^{\prime \prime }\right)\\ & =n-i+1+n-j+1-\dim \left({\mathbf{S}}^{\prime }\cup {\mathbf{S}}^{\prime \prime }\right)\\ & \ge n-i+1+n-j+1-n\\ & =n-i+1-j+1\\ & \ge 1\end{array}$$
利用Min-max theorem
$$n-i+1+n-j+1-v=\dim \left({\mathbf{S}}^{\prime }\cup {\mathbf{S}}^{\prime \prime }\right)<=n\Rightarrow n-v+1\le i+j-1$$
所以
$$\begin{array}{rl}{\sigma }_{i+j-1}\left(\mathbf{A}+\mathbf{B}\right)& \le {\sigma }_{n-v+1}\left(\mathbf{A}+\mathbf{B}\right)\\ & =\underset{\genfrac{}{}{0ex}{}{\mathbf{S}\in \mathbb{C}}{\dim \left(\mathbf{S}\right)=v}}{\min }\underset{\genfrac{}{}{0ex}{}{\mathbf{x}\in \mathbf{S}}{\Vert \mathbf{x}\Vert =1}}{\max }\Vert \left(\mathbf{A}+\mathbf{B}\right)\mathbf{x}\Vert \\ & \le \underset{\genfrac{}{}{0ex}{}{\mathbf{x}\in {\mathbf{S}}^{\prime }\cap {\mathbf{S}}^{\prime \prime }}{\Vert \mathbf{x}\Vert =1}}{\max }\Vert \left(\mathbf{A}+\mathbf{B}\right)\mathbf{x}\Vert \\ & \le \underset{\genfrac{}{}{0ex}{}{\mathbf{x}\in {\mathbf{S}}^{\prime }\cap {\mathbf{S}}^{\prime \prime }}{\Vert \mathbf{x}\Vert =1}}{\max }\Vert \mathbf{Ax}\Vert +\underset{\genfrac{}{}{0ex}{}{\mathbf{x}\in {\mathbf{S}}^{\prime }\cap {\mathbf{S}}^{\prime \prime }}{\Vert \mathbf{x}\Vert =1}}{\max }\Vert \mathbf{Bx}\Vert \\ & \le \underset{\genfrac{}{}{0ex}{}{\mathbf{x}\in {\mathbf{S}}^{\prime }}{\Vert \mathbf{x}\Vert =1}}{\max }\Vert \mathbf{Ax}\Vert +\underset{\genfrac{}{}{0ex}{}{\mathbf{x}\in {\mathbf{S}}^{\prime \prime }}{\Vert \mathbf{x}\Vert =1}}{\max }\Vert \mathbf{Bx}\Vert \\ & ={\sigma }_i\left(\mathbf{A}\right)+{\sigma }_j\left(\mathbf{B}\right)\end{array}$$

Eckart-Young Theorem

设矩阵$\mathbf{A}$有SVD分解$\mathbf{A}=\mathbf{U\Sigma }{\mathbf{V}}^T$,其中$\mathbf{U},\mathbf{V}$为正交矩阵
设$k<r=\mathrm{rank}\left(\mathbf{A}\right)$
${\mathbf{A}}_k={\sum }_{i=1}^k{\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^T$
其中${\sigma }_i$为$\mathbf{A}$的奇异值，设$\mathbf{A}$有$p$个奇异值
${\sigma }_1\ge {\sigma }_2\ge \cdots \ge {\sigma }_r>{\sigma }_{r+1}=\cdots ={\sigma }_p=0$
则
$$\underset{\mathrm{rank}\left(\mathbf{B}\right)=k}{\min }\Vert \mathbf{A}-\mathbf{B}{\Vert }_2=\Vert \mathbf{A}-{\mathbf{A}}_k{\Vert }_2={\sigma }_{k+1}$$
$$\underset{\mathrm{rank}\left(\mathbf{B}\right)=k}{\min }\Vert \mathbf{A}-\mathbf{B}{\Vert }_F=\Vert \mathbf{A}-{\mathbf{A}}_k{\Vert }_F=\sqrt{\sum _{i=k+1}^p{\sigma }_i^2}$$

证明

二范数形式

$\mathbf{A}={\sum }_{i=1}^p{\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^T$
${\mathbf{A}}_k={\sum }_{i=1}^k{\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^T$
$\mathbf{A}-{\mathbf{A}}_k$的前$k$个奇异值为0，剩下的$p-k$个奇异值为${\sigma }_{k+1},\cdots ,{\sigma }_p$
因为$\Vert \mathbf{A}{\Vert }_2={\sigma }_1={\sigma }_{max}$
于是$\Vert \mathbf{A}-{\mathbf{A}}_{k+1}{\Vert }_2={\sigma }_{k+1}$

接着证明其他的解>=最优解
因为$\mathrm{rank}\left(\mathbf{B}\right)=k$
$\dim N\left(\mathbf{B}\right)=n-k$
于是存在$n-k$个标准正交向量${\mathbf{x}}_1,\cdots ,{\mathbf{x}}_{n-k}$,使得
N ( B ) = s p a n { x 1 , ⋯   , x n − k } N\left(\mathbf{B}\right)=span\left\{\mathbf{x}_1,\cdots,\mathbf{x}_{n-k}\right\} N(B)=span{x1​,⋯,xn−k​}
又因为
dim ⁡ s p a n { x 1 , ⋯   , x n − k } + dim ⁡ s p a n { v 1 , ⋯   , v k + 1 } = n + 1 > n \operatorname{dim}span\left\{\mathbf{x}_1,\cdots,\mathbf{x}_{n-k}\right\}+\operatorname{dim}span\left\{\mathbf{v}_1,\cdots,\mathbf{v}_{k+1}\right\}=n+1>n dimspan{x1​,⋯,xn−k​}+dimspan{v1​,⋯,vk+1​}=n+1>n
所以
s p a n { x 1 , ⋯   , x n − k } ∩ s p a n { v 1 , ⋯   , v k + 1 } ≠ { 0 } span\left\{\mathbf{x}_1,\cdots,\mathbf{x}_{n-k}\right\}\cap span\left\{\mathbf{v}_1,\cdots,\mathbf{v}_{k+1}\right\}\neq\left\{0\right\} span{x1​,⋯,xn−k​}∩span{v1​,⋯,vk+1​}={0}
存在$\mathbf{z}\left(\Vert \mathbf{z}\Vert =1\right)$,使得$\mathbf{Bz}=0$且 z ∈ s p a n { v 1 , ⋯   , v k + 1 } \mathbf{z}\in span\left\{\mathbf{v}_1,\cdots,\mathbf{v}_{k+1}\right\} z∈span{v1​,⋯,vk+1​}
$\mathbf{z}={\sum }_{i=1}^{k+1}{k}_i{\mathbf{v}}_i$并且${\sum }_{i=1}^{k+1}{k}_i^2=1$

$$\begin{array}{rl}& \Vert \mathbf{A}-\mathbf{B}{\Vert }_2\\ =& \Vert \mathbf{A}-\mathbf{B}{\Vert }_2\Vert \mathbf{z}\Vert \\ \ge & \Vert \left(\mathbf{A}-\mathbf{B}\right)\mathbf{z}\Vert \\ =& \Vert \mathbf{Az}\Vert \\ =& \Vert \mathbf{U\Sigma }{\mathbf{V}}^T\mathbf{z}\Vert \\ =& \Vert \mathbf{\Sigma }{\mathbf{V}}^T\mathbf{z}\Vert \\ =& \sum _{i=1}^p{\left({\sigma }_i{\mathbf{v}}_i^T\mathbf{z}\right)}^2\\ =& \sum _{i=1}^{k+1}{\left({\sigma }_i{\mathbf{v}}_i^T\mathbf{z}\right)}^2\\ \ge & {\sigma }_{k+1}\sum _{i=1}^{k+1}{\left({\mathbf{v}}_i^T\mathbf{z}\right)}^2\\ =& {\sigma }_{k+1}\sum _{i=1}^{k+1}{k}_i^2\\ =& {\sigma }_{k+1}\end{array}$$

F范数形式

利用引理2
$$\begin{array}{rl}& \Vert \mathbf{A}-{\mathbf{A}}_k{\Vert }_F^2\\ =& \sum _{i=k+1}^r{\sigma }_i\left(\mathbf{A}\right)\\ =& \sum _{i=k+1}^r{\sigma }_i\left(\mathbf{A}-\mathbf{B}+\mathbf{B}\right)\\ \le & \sum _{i=k+1}^r\left({\sigma }_{i-k}\left(\mathbf{A}-\mathbf{B}\right)+{\sigma }_{k+1}\left(\mathbf{B}\right)\right)\\ =& \sum _{i=k+1}^r{\sigma }_{i-k}\left(\mathbf{A}-\mathbf{B}\right)\\ \le & \sum _{i=1}^{r-k}{\sigma }_i\left(\mathbf{A}-\mathbf{B}\right)\\ \le & \Vert \mathbf{A}-\mathbf{B}{\Vert }_F^2\end{array}$$
所以成立

参考：https://zhuanlan.zhihu.com/p/361938622