C2eg1-Principal Components Analysis

Suppose

• $m$ points { x ( 1 ) , ⋯   , x ( m ) } ∈ R n \{\mathbf{x}^{(1)},\cdots,\mathbf{x}^{(m)}\} \in \mathbb{R}^n {x(1),⋯,x(m)}∈Rn
• each point ${\mathbf{x}}^{(i)}\in {\mathbb{R}}^n$ corresponding to code vector ${\mathbf{c}}^{(i)}\in {\mathbb{R}}^l$
• encode function: $f(\mathbf{x})=\mathbf{c}$
• decode function: $x\approx g(f(\mathbf{x}))$

Definition

• PCA is defined by our choice of the decoding function
• decoding: $\mathbf{D}\in {\mathbb{R}}^{n\times l}$, where $g(\mathbf{c})=\mathbf{D}\mathbf{c}$.
• constraints for simplify encoding problem: the columns of $\mathbf{D}$ must be orthogonal to each other.
• In PCA, $L^2$ norm use to optimal code point ${\mathbf{c}}^{\ast }$ and the squaring operation for monotonically increasing
  $${\mathbf{c}}^{\ast }=\arg \underset{\mathbf{c}}{\min }\Vert \mathbf{x}-g(\mathbf{c}){\Vert }_2^2$$

Solving

$$\begin{array}{ccc}\Vert \mathbf{x}-g(\mathbf{c}){\Vert }_2^2& =& (\mathbf{x}-g(\mathbf{c}){)}^T(\mathbf{x}-g(\mathbf{c}))\\ & =& {\mathbf{x}}^T\mathbf{x}-{\mathbf{x}}^{T}g(\mathbf{c})-g(\mathbf{c}{)}^T\mathbf{x}+g(\mathbf{c}{)}^{T}g(\mathbf{c})\\ & =& {\mathbf{x}}^T\mathbf{x}-2{\mathbf{x}}^{T}g(\mathbf{c})+g(\mathbf{c}{)}^{T}g(\mathbf{c})\end{array}$$
Simplify the problem with the orthogonality and unit norm constraints on $\mathbf{D}$:
$$\begin{array}{ccc}{\mathbf{c}}^{\ast }& =& \arg \underset{\mathbf{c}}{\min }{\mathbf{x}}^T\mathbf{x}-2{\mathbf{x}}^{T}g(\mathbf{c})+g(\mathbf{c}{)}^{T}g(\mathbf{c})\\ & =& \arg \underset{\mathbf{c}}{\min }-2{\mathbf{x}}^{T}g(\mathbf{c})+g(\mathbf{c}{)}^{T}g(\mathbf{c})\\ & =& \arg \underset{\mathbf{c}}{\min }-2{\mathbf{x}}^T\mathbf{D}\mathbf{c}+{\mathbf{c}}^T{\mathbf{D}}^T\mathbf{D}\mathbf{c}\\ & =& \arg \underset{\mathbf{c}}{\min }-2{\mathbf{x}}^T\mathbf{D}\mathbf{c}+{\mathbf{c}}^T{\mathbf{I}}_l\mathbf{c}\\ & =& \arg \underset{\mathbf{c}}{\min }-2{\mathbf{x}}^T\mathbf{D}\mathbf{c}+{\mathbf{c}}^T\mathbf{c}\end{array}$$
Solve the optimization problem
$$\begin{array}{c}{\nabla }_{\mathbf{c}}(-2{\mathbf{x}}^T\mathbf{D}\mathbf{c}+{\mathbf{c}}^T\mathbf{c})=0\\ -2{\mathbf{D}}^T\mathbf{x}+2\mathbf{c}=0\\ \mathbf{c}={\mathbf{D}}^T\mathbf{x}\end{array}$$
Get encode function
$$f(x)=\mathbf{x}{\mathbf{D}}^T\mathbf{x}$$
Get decode function
$$r(\mathbf{x})=g(f(x))=\mathbf{D}{\mathbf{D}}^T\mathbf{x}$$
Choose the encoding matrix $\mathbf{D}$
$$\begin{array}{ccc}{\mathbf{D}}^{\ast }=\arg \underset{\mathbf{D}}{\min }\sqrt{\sum _{i,j}(x_j^{(i)}-r({\mathbf{x}}^{(i)}{)}_j{)}^2}& \text{subject to}& {\mathbf{D}}^T\mathbf{D}={\mathbf{I}}_l\end{array}$$
Set $l=1$, $\mathbf{D}$ is a single vector $\mathbf{d}$
$$\begin{array}{ccc}{\mathbf{d}}^{\ast }=\arg \underset{\mathbf{d}}{\min }\sum _i\Vert {\mathbf{x}}^{(i)}-{\mathbf{d}\mathbf{d}}^T{\mathbf{x}}^{(i)}{\Vert }_2^2& \text{subject to}& \Vert \mathbf{d}{\Vert }_2=1.\end{array}$$
${\mathbf{d}}^T{\mathbf{x}}^{(i)}$ is a value, namely ${\mathbf{d}}^T{\mathbf{x}}^{(i)}={\mathbf{x}}^{(i)}{\mathbf{d}}^T$
$$\begin{array}{ccccc}{\mathbf{d}}^{\ast }& =& \arg \underset{\mathbf{d}}{\min }\sum _i\Vert {\mathbf{x}}^{(i)}-{\mathbf{d}\mathbf{d}}^T{\mathbf{x}}^{(i)}{\Vert }_2^2& & \\ & =& \arg \underset{\mathbf{d}}{\min }\sum _i\Vert {\mathbf{x}}^{(i)}-{\mathbf{d}}^T{\mathbf{x}}^{(i)}\mathbf{d}{\Vert }_2^2& & \\ & =& \arg \underset{\mathbf{d}}{\min }\sum _i\Vert {\mathbf{x}}^{(i)}-{\mathbf{x}}^{(i)T}\mathbf{d}\mathbf{d}{\Vert }_2^2& \text{subject to}& \Vert \mathbf{d}{\Vert }_2=1\\ & =& \arg \underset{\mathbf{d}}{\min }\Vert \mathbf{X}-{\mathbf{X}\mathbf{d}\mathbf{d}}^T{\Vert }_F^2& \text{subject to}& {\mathbf{d}}^T\mathbf{d}=1\end{array}$$
Solving the optimization problem of $\mathbf{d}$
$$\begin{array}{ccc}\arg \underset{\mathbf{d}}{\min }\Vert \mathbf{X}-{\mathbf{X}\mathbf{d}\mathbf{d}}^T{\Vert }_F^2& =& \arg \underset{\mathbf{d}}{\min }\text{Tr}((\mathbf{X}-{\mathbf{X}\mathbf{d}\mathbf{d}}^T{)}^T(\mathbf{X}-{\mathbf{X}\mathbf{d}\mathbf{d}}^T))\\ & =& \arg \underset{\mathbf{d}}{\min }\text{Tr}({\mathbf{X}}^T\mathbf{X}-{\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T-{\mathbf{d}\mathbf{d}}^T{\mathbf{X}}^T\mathbf{X}+{\mathbf{d}\mathbf{d}}^T{\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)\\ & =& \arg \underset{\mathbf{d}}{\min }\text{Tr}({\mathbf{X}}^T\mathbf{X})-\text{Tr}({\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)-\text{Tr}({\mathbf{d}\mathbf{d}}^T{\mathbf{X}}^T\mathbf{X})+\text{Tr}({\mathbf{d}\mathbf{d}}^T{\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)\\ & =& \arg \underset{\mathbf{d}}{\min }-\text{Tr}({\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)-\text{Tr}({\mathbf{d}\mathbf{d}}^T{\mathbf{X}}^T\mathbf{X})+\text{Tr}({\mathbf{d}\mathbf{d}}^T{\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)\\ & =& \arg \underset{\mathbf{d}}{\min }-2\text{Tr}({\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)+\text{Tr}({\mathbf{d}\mathbf{d}}^T{\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)\\ & =& \arg \underset{\mathbf{d}}{\min }-2\text{Tr}({\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)+\text{Tr}({\mathbf{X}}^T\mathbf{X}\mathbf{d}\mathbf{d}{\mathbf{d}\mathbf{d}}^T)\end{array}$$
Compute with constraint condition
$$\begin{array}{cccc}& \arg \underset{\mathbf{d}}{\min }-2\text{Tr}({\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)+\text{Tr}({\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T{\mathbf{d}\mathbf{d}}^T)& \text{subject to}& {\mathbf{d}}^T\mathbf{d}=1\\ =& \arg \underset{\mathbf{d}}{\min }-2\text{Tr}({\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)+\text{Tr}({\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)& \text{subject to}& {\mathbf{d}}^T\mathbf{d}=1\\ =& \arg \underset{\mathbf{d}}{\min }-\text{Tr}({\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)& \text{subject to}& {\mathbf{d}}^T\mathbf{d}=1\\ =& \arg \underset{\mathbf{d}}{\max }\text{Tr}({\mathbf{X}}^T{\mathbf{X}\mathbf{d}\mathbf{d}}^T)& \text{subject to}& {\mathbf{d}}^T\mathbf{d}=1\\ =& \arg \underset{\mathbf{d}}{\max }\text{Tr}({\mathbf{d}}^T{\mathbf{X}}^T\mathbf{X}\mathbf{d})& \text{subject to}& {\mathbf{d}}^T\mathbf{d}=1\end{array}$$
Set ${\mathbf{X}}^T\mathbf{X}$ as $\mathbf{A}$, then the optimal $\mathbf{d}$ is given by the eigenvector of $\mathbf{A}$ corresponding to the largest eigenvalue