主成分分析(Principal Components Analysis)的数学原理

已知：

1. 有 $m$ 个点组成的集合 $S = \left \{ \boldsymbol {x} ^{(1)}, \cdots, \boldsymbol {x} ^{(m)} \right \} \subseteq \mathbb {R} ^{n}$
2. $n > l$

目标：

寻找一个编码函数

$$f: \mathbb {R} ^{n} \to \mathbb {R} ^{l}$$
和一个解码函数
$$g: \mathbb {R} ^{l} \to \mathbb {R} ^{n}$$

使得目标函数：

$$\sum _{i = 1} ^{m} \left \lVert \boldsymbol {x}^{(i)} - g \left (f \left ( \boldsymbol {x}^{(i)} \right ) \right ) \right \rVert ^{2} \tag {1}$$
最小。

PCA 约束：

$$\forall \boldsymbol {c} \in \mathbb {R} ^{l}, g(\boldsymbol {c}) = \boldsymbol {D} \boldsymbol {c} \tag {2}$$
其中 $\boldsymbol {D} = \begin {pmatrix} { \boldsymbol {d} _{1} , \cdots, \boldsymbol {d} _{l}} \end {pmatrix}$ 是 $n \times l$ 维矩阵，且
$$\boldsymbol {D} ^{\intercal} \boldsymbol {D} = \boldsymbol {I} _{l} \tag {3}$$

分析

本质上是要求解两个问题：
1. $\forall \boldsymbol {D}$,求函数 $f$ 使得目标函数最小。即 $\forall \boldsymbol {x} \in S, \forall \boldsymbol {D},$ 若存在一个 $\boldsymbol {c} \text{*} \in \mathbb {R} ^{l},$ 使得 $\left \lVert \boldsymbol {x} - g \left ( \boldsymbol {c} \text{*} \right ) \right \rVert$ 最小，则 $f \left ( \boldsymbol {x} \right ) = \boldsymbol {c} \text{*}$ 。因此

$$f \left ( \boldsymbol {x} \right ) = \underset {\boldsymbol {c}} {\operatorname {argmin}} \left \lVert \boldsymbol {x} - g \left ( \boldsymbol {c} \right ) \right \rVert \tag {4}$$
2. 矩阵 $\boldsymbol {D} \text{*}$ 使得目标函数最小 。

命题 1

$\forall \boldsymbol {x} \in S, \forall \boldsymbol {D}, f(\boldsymbol {x}) = \boldsymbol {D} ^{\intercal} \boldsymbol {x}$

证明

$\forall \boldsymbol {x} \in S, \forall \boldsymbol {c} \in \mathbb {R} ^{l},$
$\left \lVert \boldsymbol {x} - g \left (\boldsymbol {c} \right ) \right \rVert ^2$
$= \left [\boldsymbol {x} - g \left (\boldsymbol {c} \right ) \right ] ^{\intercal} \left [\boldsymbol {x} - g \left (\boldsymbol {c} \right ) \right ]$
$= \left [\boldsymbol {x} ^{\intercal} - g \left (\boldsymbol {c} \right ) ^{\intercal} \right ] \left [\boldsymbol {x} - g \left (\boldsymbol {c} \right ) \right ]$
$= \boldsymbol {x} ^{\intercal} \boldsymbol {x} - g \left (\boldsymbol {c} \right ) ^{\intercal} \boldsymbol {x} - \boldsymbol {x} ^{\intercal} g \left (\boldsymbol {c} \right ) + g \left (\boldsymbol {c} \right ) ^{\intercal} g \left (\boldsymbol {c} \right )$
由于$\boldsymbol {x} ^{\intercal} g \left (\boldsymbol {c} \right ) \in \mathbb R,$ 因此 $g \left (\boldsymbol {c} \right ) ^{\intercal} \boldsymbol {x} = \boldsymbol {x} ^{\intercal} g \left (\boldsymbol {c} \right ) ,$ 因此
上式 $= \boldsymbol {x} ^{\intercal} \boldsymbol {x} - 2 \boldsymbol {x} ^{\intercal} g \left (\boldsymbol {c} \right ) + g \left (\boldsymbol {c} \right ) ^{\intercal} g \left (\boldsymbol {c} \right )$
由于第一项 $\boldsymbol {x} ^{\intercal} \boldsymbol {x}$ 不依赖于 $\boldsymbol {c}$ ，且由 (4)，
$f \left ( \boldsymbol {x} \right ) = \underset {\boldsymbol {c}} {\operatorname {argmin}} - 2 \boldsymbol {x} ^{\intercal} g \left (\boldsymbol {c} \right ) + g \left (\boldsymbol {c} \right ) ^{\intercal} g \left (\boldsymbol {c} \right )$
将 (2) 代入，得
$f \left ( \boldsymbol {x} \right ) = \underset {\boldsymbol {c}} {\operatorname {argmin}} - 2 \boldsymbol {x} ^{\intercal} \boldsymbol {D} \boldsymbol {c} + \left ( \boldsymbol {D} \boldsymbol {c} \right ) ^{\intercal} \left ( \boldsymbol {D} \boldsymbol {c} \right )$
$= \underset {\boldsymbol {c}} {\operatorname {argmin}} - 2 \boldsymbol {x} ^{\intercal} \boldsymbol {D} \boldsymbol {c} + { \boldsymbol {c} } ^{\intercal} { \boldsymbol {D} } ^{\intercal} \boldsymbol {D} \boldsymbol {c}$
将 (3)代入，得
$f \left ( \boldsymbol {x} \right ) = \underset {\boldsymbol {c}} {\operatorname {argmin}} - 2 \boldsymbol {x} ^{\intercal} \boldsymbol {D} \boldsymbol {c} + { \boldsymbol {c} } ^{\intercal} \boldsymbol {I} _{l} \boldsymbol {c}$
$= \underset {\boldsymbol {c}} {\operatorname {argmin}} - 2 \boldsymbol {x} ^{\intercal} \boldsymbol {D} \boldsymbol {c} + { \boldsymbol {c} } ^{\intercal} \boldsymbol {c}$
令梯度为 $\vec {0}$ ，则
$\nabla _{ \boldsymbol {c}} \left ( - 2 \boldsymbol {x} ^{\intercal} \boldsymbol {D} \boldsymbol {c} + { \boldsymbol {c} } ^{\intercal} \boldsymbol {c} \right ) = -2 \boldsymbol {D} ^{\intercal} \boldsymbol {x} + 2 \boldsymbol {c} = \vec {0}$
则 $\boldsymbol {c} = \boldsymbol {D} ^{\intercal} \boldsymbol {x}$
于是 $f \left ( \boldsymbol {x} \right ) = \boldsymbol {D} ^{\intercal} \boldsymbol {x}$

命题 2

$\forall \boldsymbol {x} \in S, \forall \boldsymbol {D}, g \left (f \left ( \boldsymbol {x} \right ) \right ) = \boldsymbol {D} \boldsymbol {D} ^{\intercal} \boldsymbol {x}$

命题 3

令 $\boldsymbol {X} = { \begin{pmatrix} {\boldsymbol {x} ^{(1)} , \cdots , \boldsymbol {x} ^{(m)}} \end{pmatrix} }^{\intercal},$ 则
$\boldsymbol {D} \text{*} = \underset {\boldsymbol {D}} {\operatorname {argmax}} \operatorname {Tr} \left ( \boldsymbol {D} ^{\intercal} {\boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {D} \right )$

证明

$\forall \boldsymbol {x} \in S, \forall \boldsymbol {D},$
$\left \lVert \boldsymbol {x} - g \left (f \left ( \boldsymbol {x} \right ) \right ) \right \rVert ^{2}$
$= \left \lVert \boldsymbol {x} - \boldsymbol {D} \boldsymbol {D} ^{\intercal} \boldsymbol {x} \right \rVert ^{2}$
$= \left \lVert \left (\boldsymbol {I} _{n} - \boldsymbol {D} \boldsymbol {D} ^{\intercal} \right ) \boldsymbol {x} \right \rVert ^{2}$
$= {\left [\left (\boldsymbol {I} _{n} - \boldsymbol {D} \boldsymbol {D} ^{\intercal} \right ) \boldsymbol {x} \right ]} ^{\intercal} \left [\left (\boldsymbol {I} _{n} - \boldsymbol {D} \boldsymbol {D} ^{\intercal} \right ) \boldsymbol {x} \right ]$
$= \boldsymbol {x} ^{\intercal} {\left (\boldsymbol {I} _{n} - \boldsymbol {D} \boldsymbol {D} ^{\intercal} \right )} ^{\intercal} \left (\boldsymbol {I} _{n} - \boldsymbol {D} \boldsymbol {D} ^{\intercal} \right ) \boldsymbol {x}$
$= \boldsymbol {x} ^{\intercal} \left (\boldsymbol {I} _{n} - \boldsymbol {D} \boldsymbol {D} ^{\intercal} \right ) \left (\boldsymbol {I} _{n} - \boldsymbol {D} \boldsymbol {D} ^{\intercal} \right ) \boldsymbol {x}$
$= \boldsymbol {x} ^{\intercal} \left (\boldsymbol {I} _{n} - 2 \boldsymbol {D} \boldsymbol {D} ^{\intercal} + \boldsymbol {D} \boldsymbol {D} ^{\intercal} \boldsymbol {D} \boldsymbol {D} ^{\intercal} \right ) \boldsymbol {x}$
$= \boldsymbol {x} ^{\intercal} \left (\boldsymbol {I} _{n} - 2 \boldsymbol {D} \boldsymbol {D} ^{\intercal} + \boldsymbol {D} \boldsymbol {D} ^{\intercal} \right ) \boldsymbol {x}$
$= \boldsymbol {x} ^{\intercal} \left (\boldsymbol {I} _{n} - \boldsymbol {D} \boldsymbol {D} ^{\intercal} \right ) \boldsymbol {x}$
$= \boldsymbol {x} ^{\intercal} \boldsymbol {x} - \boldsymbol {x} ^{\intercal} \boldsymbol {D} \boldsymbol {D} ^{\intercal} \boldsymbol {x}$
因此目标函数
$\sum _{i = 1} ^{m} \left \lVert \boldsymbol {x}^{(i)} - g \left (f \left ( \boldsymbol {x}^{(i)} \right ) \right ) \right \rVert ^{2}$
$= \sum _{i = 1} ^{m} \left [ {\left ( \boldsymbol {x}^{(i)} \right )} ^{\intercal} \boldsymbol {x} ^{(i)} - {\left ( \boldsymbol {x}^{(i)} \right )} ^{\intercal} \boldsymbol {D} \boldsymbol {D} ^{\intercal} \boldsymbol {x} ^{(i)} \right ]$
$= \sum _{i = 1} ^{m} {\left ( \boldsymbol {x}^{(i)} \right )} ^{\intercal} \boldsymbol {x} ^{(i)} - \sum _{i = 1} ^{m} {\left ( \boldsymbol {x}^{(i)} \right )} ^{\intercal} \boldsymbol {D} \boldsymbol {D} ^{\intercal} \boldsymbol {x} ^{(i)}$
由于第一项 $\sum _{i = 1} ^{m} {\left ( \boldsymbol {x}^{(i)} \right )} ^{\intercal} \boldsymbol {x} ^{(i)}$ 不依赖于 $\boldsymbol {D}$ ，因此
$\boldsymbol {D} \text{*} = \underset {\boldsymbol {D}} {\operatorname {argmax}} \sum _{i = 1} ^{m} {\left ( \boldsymbol {x}^{(i)} \right )} ^{\intercal} \boldsymbol {D} \boldsymbol {D} ^{\intercal} \boldsymbol {x} ^{(i)}$
由于
$\sum _{i = 1} ^{m} {\left ( \boldsymbol {x}^{(i)} \right )} ^{\intercal} \boldsymbol {D} \boldsymbol {D} ^{\intercal} \boldsymbol {x} ^{(i)}$
$= \operatorname {Tr} \left ( {\boldsymbol {X}} \boldsymbol {D} \boldsymbol {D} ^{\intercal} { \boldsymbol {X} } ^{\intercal} \right )$
$= \operatorname {Tr} \left ( {\boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {D} \boldsymbol {D} ^{\intercal} \right )$
$= \operatorname {Tr} \left ( \boldsymbol {D} ^{\intercal} {\boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {D} \right )$
因此
$\boldsymbol {D} \text{*} = \underset {\boldsymbol {D}} {\operatorname {argmax}} \operatorname {Tr} \left ( \boldsymbol {D} ^{\intercal} {\boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {D} \right )$

命题 4

设半正定矩阵 ${\boldsymbol {X}} ^{\intercal} \boldsymbol {X}$ 的前 $l$ 个最大的特征向量（从大到小排列）为 $\lambda _{1}, \cdots, \lambda _{l}$ ，则
1. $\operatorname {Tr} \left ( \boldsymbol {D} ^{\intercal} {\boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {D} \right ) \le \sum _{i = 1} ^{l} \lambda _{i}$
2. $\boldsymbol {d} _{i}$ 为属于 $\lambda _{i}$ 的特征向量时等号成立。

证明

由于 ${\boldsymbol {X}} ^{\intercal} \boldsymbol {X}$ 是半正定的实对称矩阵，因此存在对角线矩阵
$\Lambda = \begin{pmatrix} \lambda _{1}, & & \\ & \ddots & \\ & & \lambda _{n} \end{pmatrix} ,$ 其中 $\{ \lambda _{i} \}$ 单调不增，即 $\lambda _{i} \ge \lambda _{j}, 1 \le i \le j \le n$ 。
存在正交矩阵 $\boldsymbol {P} _{n \times n}$, 使得
${\boldsymbol {X}} ^{\intercal} \boldsymbol {X} = \boldsymbol {P} \Lambda {\boldsymbol {P}} ^{\intercal}$
于是
$\boldsymbol {D} ^{\intercal} {\boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {D} = \boldsymbol {D} ^{\intercal} \boldsymbol {P} \Lambda {\boldsymbol {P}} ^{\intercal} \boldsymbol {D} = \boldsymbol {Q} ^{\intercal} \Lambda \boldsymbol {Q}$
其中 $\boldsymbol {Q} = {\begin {pmatrix} {q _{ij}} \end {pmatrix}}_{n \times l} = {\boldsymbol {P}} ^{\intercal} \boldsymbol {D}$
于是 $\boldsymbol {Q} ^{\intercal} \boldsymbol {Q} = {\left ({\boldsymbol {P}} ^{\intercal} \boldsymbol {D} \right )} ^{\intercal} {\boldsymbol {P}} ^{\intercal} \boldsymbol {D} = {\boldsymbol {D}} ^{\intercal} \boldsymbol {P} {\boldsymbol {P}} ^{\intercal} \boldsymbol {D} = \boldsymbol {I} _{l}$
且 $\operatorname {Tr} \left ( \boldsymbol {D} ^{\intercal} {\boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {D} \right ) = \operatorname {Tr} \left ( \boldsymbol {Q} ^{\intercal} \Lambda \boldsymbol {Q} \right )$
$= \sum _{i = 1} ^{n} \sum _{j = 1} ^{l} \lambda _{i} {q _{ij}} ^{2}$
$= \sum _{i = 1} ^{n} \lambda _{i} \left (\sum _{j = 1} ^{l} {q _{ij}} ^{2} \right )$
由于 $\boldsymbol {Q} ^{\intercal} \boldsymbol {Q} = \boldsymbol {I} _{l}$ ，因此
$\sum _{i = 1} ^{n} {q _{ij}} ^{2} = 1, 1 \le j \le n$
因此 $\sum _{i = 1} ^{n} \left (\sum _{j = 1} ^{l} {q _{ij}} ^{2} \right ) = \sum _{j = 1} ^{l} \sum _{i = 1} ^{n} \lambda _{i} {q _{ij}} ^{2}= l$
由于 $\boldsymbol {Q}$ 的 $l$ 个列向量可以扩展成 $n$ 组 $n$ 维的标准正交基，因此 $\boldsymbol {Q}$ 的行向量长度小于等于 $1$ ，即 $\sum _{j = 1} ^{l} {q _{ij}} ^{2} \le 1, 1 \le i \le n$
又由于 $\{ \lambda _{i} \}$ 单调不增，
因此 $\sum _{i = 1} ^{n} \lambda _{i} \left (\sum _{j = 1} ^{l} {q _{ij}} ^{2} \right ) \le \sum _{i = 1} ^{l} \lambda _{i}$
因此 $\operatorname {Tr} \left ( \boldsymbol {D} ^{\intercal} {\boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {D} \right ) \le \sum _{i = 1} ^{l} \lambda _{i}$
特别的，当 $q_{ij} = {\delta} _{ij}$ ，即 $\boldsymbol {Q} = \begin {pmatrix} {\boldsymbol {I} _{l} \\ \boldsymbol {0}} \end {pmatrix}$ 时，
$\sum _{i = 1} ^{n} \lambda _{i} \left (\sum _{j = 1} ^{l} {q _{ij}} ^{2} \right ) = \sum _{i = 1} ^{l} \lambda _{i}$
此时 $\boldsymbol {D} = \boldsymbol {P} \boldsymbol {Q} = \begin {pmatrix} {\boldsymbol {p} _{1}, \cdots, \boldsymbol {p} _{l}} \end {pmatrix}$ 即为 $\boldsymbol {P}$ 的前 $l$ 列组成的矩阵，即 $\boldsymbol {d} _{i}$ 为属于 $\lambda _{i}$ 的特征向量。

定义

添加约束：

$$\forall \boldsymbol {x} \in \mathbb {R} ^{n}, g(\boldsymbol {x}) = \boldsymbol {D} ^{\intercal}\boldsymbol {x} \tag {5}$$
令 $\boldsymbol {Y} = \begin {pmatrix} {\boldsymbol {y} _{1} , \cdots, \boldsymbol {y} _{l}} \end {pmatrix} = \boldsymbol {X} \boldsymbol {D} = \begin {pmatrix} {{f \left ( \boldsymbol {x}^{(1)} \right )} ^{\intercal} \\ \vdots \\ {f \left ( \boldsymbol {x}^{(m)} \right )} ^{\intercal} } \end {pmatrix},$
称 $\boldsymbol {y} _{i}$ 为 $\boldsymbol {Y}$ 的第 $i$ 个主成分。
称：
Percentage of total variation retained = $\dfrac {\operatorname{Var} \mathbf {Y } } { \operatorname{Var} {\mathbf {X} }}$

命题 5

$\boldsymbol {D} \text{*} = \underset {\boldsymbol {D}} {\operatorname {argmax}} \operatorname {Var} {\boldsymbol {Y} }$

证明

$\operatorname {Var} {\boldsymbol {Y} } = \operatorname {Tr} \left ( \boldsymbol {Y} ^{\intercal} \boldsymbol {Y} \right ) = \operatorname {Tr} \left ( \boldsymbol {D} ^{\intercal} {\boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {D} \right )$
因此 $\underset {\boldsymbol {D}} {\operatorname {argmax}} \operatorname {Var} {\boldsymbol {Y} }= \underset {\boldsymbol {D}} {\operatorname {argmax}} \operatorname {Tr} \left ( \boldsymbol {D} ^{\intercal} {\boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {D} \right )$
由命题3， $\boldsymbol {D} \text{*} = \underset {\boldsymbol {D}} {\operatorname {argmax}} \operatorname {Var} {\boldsymbol {Y} }$

命题 6

$\dfrac {\operatorname{Var} \mathbf {Y } } { \operatorname{Var} {\mathbf {X} }} \le \dfrac {\sum \limits_{i = 1} ^{l} \lambda_{i}} {\sum \limits_{i = 1} ^{n} \lambda_{i}}$

命题 7

当 $\boldsymbol {d} _{i}$ 为属于 $\lambda _{i}$ 的特征向量时， ${ \boldsymbol {y} _{i} } ^{\intercal} \boldsymbol {y} _{j} = \lambda _{j} \delta _{ij}$

证明

$\boldsymbol {y} _{i} = \boldsymbol {X} \boldsymbol {d} _{i}, 1 \le i \le l$
于是 ${ \boldsymbol {y} _{i} } ^{\intercal} \boldsymbol {y} _{j} = { \left (\boldsymbol {X} \boldsymbol {d} _{i} \right )} ^{\intercal} \boldsymbol {X} \boldsymbol {d} _{j} = { \boldsymbol {d} _{i} } ^{\intercal} { \boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {d} _{j}$
当 $\boldsymbol {d} _{i}$ 为属于 $\lambda _{i}$ 的特征向量时，
${ \boldsymbol {d} _{i} } ^{\intercal} { \boldsymbol {X}} ^{\intercal} \boldsymbol {X} \boldsymbol {d} _{j} = \lambda _{j} { \boldsymbol {d} _{i} } ^{\intercal} \boldsymbol {d} _{j} = \lambda _{j} \delta _{ij}$
因此 ${ \boldsymbol {y} _{i} } ^{\intercal} \boldsymbol {y} _{j} = \lambda _{j} \delta _{ij}$