机器学习-白板推导 P5_4 (PCA最小重构代价)

PCA最小重构代价

$$X={\left[\begin{array}{cccc}{x}_1& x_2& ...& x_N\end{array}\right]}^T=\left[\begin{array}{c}{x}_1^T\\ x_2^T\\ ⋮\\ x_N^T\end{array}\right]={\left[\begin{array}{cccc}{x}_{11}& x_{12}& ...& x_{1p}\\ x_{21}& x_{22}& ...& x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{N1}& x_{N2}& ...& x_{Np}\end{array}\right]}_{N\ast p}$$
$$1_N=\left[\begin{array}{c}1\\ 1\\ ⋮\\ 1\end{array}\right]$$
$Mean:\overline{X}=\frac{1}{N}{\sum }_{i=1}^N{x}_i=\frac{1}{N}{X}^T{1}_N$

$Covariance:S=\frac{1}{N}{\sum }_{i=1}^N(x_i-\overline{X})(x_i-\overline{X}{)}^T=\frac{1}{N}{X}^{T}HX$

$x_i$在$u_k$方向的投影值为$x_i^T{u}_k$,投影值和单位向量的乘机为$(x_i^T{u}_k)u_k$
所以

$x_i={\sum }_{k=1}^p(x_i^T{u}_k)u_k$

用PCA进行降维，降维后的维度为$q<p$

$\widehat{x_i}={\sum }_{k=1}^q(x_i^T{u}_k)u_k$

重构代价为：

$$\begin{array}{rl}J& =\frac{1}{N}\sum _{i=1}^N||x_i-\widehat{x_i}|{|}^2\\ & =\frac{1}{N}\sum _{i=1}^N||\sum _{k=q+1}^p(x_i^T{u}_k)u_k|{|}^2\\ & =\frac{1}{N}\sum _{i=1}^N\sum _{k=q+1}^p(x_i^T{u}_k{)}^2\end{array}$$
$x_i$进行中心化：
$$\begin{array}{rl}J& =\frac{1}{N}\sum _{i=1}^N\sum _{k=q+1}^p((x_i-\overline{X}{)}^T{u}_k{)}^2\\ & =\sum _{k=q+1}^p\sum _{i=1}^N\frac{1}{N}((x_i-\overline{X}{)}^T{u}_k{)}^2\\ & =\sum _{k=q+1}^p{u}_k^{T}S{u}_k\\ & s.t.u_k^T{u}_k=1\end{array}$$

最小重构代价：
$J=\frac{1}{N}{\sum }_{i=1}^N||x_i-\widehat{x_i}|{|}^2={\sum }_{k=q+1}^p{u}_k^{T}S{u}_{k}s.t.u_k^T{u}_k=1$

优化问题：
$u_k=arg\min {\sum }_{k=q+1}^p{u}_k^{T}S{u}_{k}s.t.u_k^T{u}_k=1$

最大投影方差与最小重构代价的区别
原先由$p$维取前$q$维
最大投影方差目标前$q$最大
最小重构代价目标后$p-q$维方差最小