奇异值分解之 Frobenious-范数下低秩近似的证明

Title: 奇异值分解之 Frobenious-范数下低秩近似的证明

本篇博客接着之前博客 “ 奇异值分解的常用结论” 留下的问题 (Frobenious-范数下低秩近似) 继续进行.

相关博文介绍

- 奇异值分解之常用结论

- 奇异值分解之 Courant-Fischer 定理及其变体

- 奇异值分解之 Weyl 不等式及其变体

- 奇异值分解之 Frobenious-范数下低秩近似

1. Low-Rank Approximation

Low-Rank Approximation

(Eckart-Young-Mirsky) For either the 2-norm $\Vert \cdot {\Vert }_2$ or the Frobenious norm $\Vert \cdot {\Vert }_F$
$$\Vert \mathbf{A}-{\mathbf{A}}_k\Vert \le \Vert \mathbf{A}-\mathbf{B}\Vert$$
for all rank-$k$ matrices $\mathbf{B}$. Furthermore,
$$\{\begin{array}{l}\Vert \mathbf{A}-{\mathbf{A}}_k{\Vert }_2={\sigma }_{k+1}\\ \Vert \mathbf{A}-{\mathbf{A}}_k{\Vert }_F={\left({\sum }_{i=k+1}^r{\sigma }_i^2\right)}^{\frac{1}{2}}\end{array}$$

其中 2-范数情况已经在 “奇异值分解之常用结论” 中完成证明, 这里开始 Frobenious-范数情况的证明.

---

2. Weyl’s inequality

[Weyl’s inequality] Let $\mathbf{X},\mathbf{Y}\in {\mathbf{M}}_{m,n}$ be given and let q = min ⁡ { m , n } q = \min\{m,n\} q=min{m,n}. The following inequality holds for the decreasingly ordered singular values of $\mathbf{X}$, $\mathbf{Y}$, and $\mathbf{X}+\mathbf{Y}$:
$${\sigma }_{i+j-1}(\mathbf{X}+\mathbf{Y})\le {\sigma }_i(\mathbf{X})+{\sigma }_j(\mathbf{Y})$$
for $1\le i,j\le q$ and $i+j\le q+1$.

上一篇博客 “奇异值分解之 Weyl 不等式及其变体” 中已经整理并证明了两种形式的 Weyl 不等式, 包括上述奇异值形式.

3. Frobenious-范数下低秩近似的证明

Proof^[1]

对 Weyl 不等式进行变量替换
$$\begin{array}{rl}\mathbf{X}& =\mathbf{A}-\mathbf{B}\\ \mathbf{Y}& =\mathbf{B}\\ j-1& =k\end{array}\text{\hspace{1em}(1)}$$
已知 KaTeX parse error: Undefined control sequence: \rank at position 1: \̲r̲a̲n̲k̲(\mathbf{B}) = …, 故有
$${\sigma }_i(\mathbf{B})=0,\text{for}i\ge k+1\text{\hspace{1em}(2)}$$
代入 Weyl 不等式得到
$${\sigma }_{i+k}(\mathbf{A})\le {\sigma }_i(\mathbf{A}-\mathbf{B})+{\sigma }_{k+1}(\mathbf{B})={\sigma }_i(\mathbf{A}-\mathbf{B})\text{\hspace{1em}(3)}$$

截断的奇异值分解 (truncated SVD)
A k ≜ ∑ i = 1 r a n k ( A ) σ i u i v i T = ∑ i = 1 min ⁡ { m , n } σ i u i v i T (4) {\mathbf{A}}_k \triangleq \sum_{i=1}^{{\rm rank}({\mathbf{A}})} {\sigma}_i \mathbf{u}_i \mathbf{v}_i^{\small\rm T} = \sum_{i=1}^{\min\{m,n\}} {\sigma}_i \mathbf{u}_i \mathbf{v}_i^{\small\rm T} \tag{4} Ak​≜i=1∑rank(A)​σi​ui​viT​=i=1∑min{m,n}​σi​ui​viT​(4)
其中 ${\sigma }_p=0$, 当 KaTeX parse error: Undefined control sequence: \rank at position 1: \̲r̲a̲n̲k̲({\mathbf{A}}) ….

由奇异值分解的范数性质
∥ A − A k ∥ F 2 = ∑ i = k + 1 r a n k ( A ) σ i ( A ) 2 = ∑ i = 1 r a n k ( A ) − k σ i + k ( A ) 2 ≤ ∑ i = 1 r a n k ( A ) − k σ i ( A − B ) 2 ≤ ∑ i = 1 min ⁡ { m , n } σ i ( A − B ) 2 = ∥ A − B ∥ F 2 (5) \begin{aligned} \| \mathbf{A} - {\mathbf{A}}_k \|_{F}^2 & = \sum_{i=k+1}^{{\rm rank}({\mathbf{A}})} \sigma_i(\mathbf{A})^2\\ & = \sum_{i=1}^{{\rm rank}({\mathbf{A}})-k} \sigma_{i+k}(\mathbf{A})^2\\ & \leq \sum_{i=1}^{{\rm rank}({\mathbf{A}})-k} \sigma_{i}(\mathbf{A}-\mathbf{B})^2\\ & \leq \sum_{i=1}^{\min\{m,n\}} \sigma_{i}(\mathbf{A}-\mathbf{B})^2\\ &= \|\mathbf{A}-\mathbf{B}\|_{F}^2 \end{aligned} \tag{5} ∥A−Ak​∥F2​​=i=k+1∑rank(A)​σi​(A)2=i=1∑rank(A)−k​σi+k​(A)2≤i=1∑rank(A)−k​σi​(A−B)2≤i=1∑min{m,n}​σi​(A−B)2=∥A−B∥F2​​(5)
证明完毕.

这样就补全了 奇异值分解之常用结论 中留下的问题.

参考文献

[1] Scribe(s): Joss Rakotobe, Haoyu Zhang, Instructor: Guillaume Rabusseau, “IFT 6760A - Lecture 5 SVD, Matrix Norms and Low Rank Approximation Theorem”, https://www-labs.iro.umontreal.ca/~grabus/courses/ift6760_W20_files/lecture-5.pdf