PCA：Principle Component Analysis [3]

wikipedia上关于PCA的解释：http://en.wikipedia.org/wiki/Principal_component_analysis，这里写的都是基于对wikipedia上的一点总结，加一些自己对PCA特性的理解。

【1】.PCA是干什么的

PCA can be done by eigenvalue decomposition of a data covariance (orcorrelation) matrix or singular value decomposition of a data matrix, usually aftermean centering (and normalizing or using Z-scores) the data matrix for each attribute.Its operation can be thought of as revealing the internal structure of the data in a way that best explains the variance in the data. PCA defines a new orthogonal coordinate system that optimally describes variance in a single dataset.

在进行PCA之前，需要对数据进行预处理，将数据的质心移到原点处，也就是减去数据集的平均值，为什么呢？因为PCA是用来刻画方差的，在其寻找的主分量上，数据的方差最大，其公式定义和文字解释如下。

公式定义：

文字定义：

Given a set of points in Euclidean space, the first principal component corresponds to a line that passes through the multidimensional mean and minimizes the sum of squares of the distances of the points from the line

如果不将质心移到原点，第一主分量会或多或少跟收到质心位置的影响，wikipedia的解释如下：

Mean subtraction (a.k.a. "mean centering") is necessary for performing PCA to ensure that the first principal component describes the direction of maximum variance. If mean subtraction is not performed, the first principal component might instead correspond more or less to the mean of the data. A mean of zero is needed for finding a basis that minimizes the mean square error of the approximation of the data.

PCA就是要找到一个矩阵，将原来的数据X变换到新的空间，得到新的数据集T

X是（n，p）的矩阵，每行代表一个样本（向量），W是（p，p）的矩阵，每列是矩阵 X‘ * X的一个（单位）特征向量。得到了W矩阵，就能根据每个特征向量对应的特征值大小，合理的去掉一些特征值小的维度，将W变成一个（n，k）的矩阵，这里 { k <= n }，这样就达到了压缩数据，或者是去噪的目的。这里每个特征值对应的含义是：

The singular values (in Σ) are the square roots of the eigenvalues of the matrix X^TX. Each eigenvalue is proportional to the portion of the "variance" (more correctly of the sum of the squared distances of the points from their multidimensional mean) that is correlated with each eigenvector. The sum of all the eigenvalues is equal to the sum of the squared distances of the points from their multidimensional mean. PCA essentially rotates the set of points around their mean in order to align with the principal components. This moves as much of the variance as possible (using an orthogonal transformation) into the first few dimensions. The values in the remaining dimensions, therefore, tend to be small and may be dropped with minimal loss of information (see below).The eigenvalues represent the distribution of the source data's energy^{[clarification needed]} among each of the eigenvectors, where the eigenvectors form a basis for the data

【2】.关于W怎么求

两种方式：

(a)

where Λ is the diagonal matrix of eigenvalues λ_(k) of X^TX, (λ_(k) being equal to the sum of the squares over the dataset associated with each component k: λ_(k) = Σ_i t_k²_(i) = Σ_i (x_(i) ⋅ w_(k))²)

(b)

Comparison with the eigenvector factorisation of X^TX establishes that the right singular vectors W of X are equivalent to the eigenvectors of X^TX, while the singular values σ_(k) of X are equal to the square roots of the eigenvaluesλ_(k) of X^TX.

由方法(b)可以得出如下公式

所以如果通过 [u, s, v] = svd(X) 分解，再进行变换时，应该是用 X * W，因为之前一直用的是协方差矩阵 X' * X 进行的 svd() 分解，得出的 u 和 v 是相同的，所以用 X * u 做的，但其实这样是不对的。

两种方法相比，通过svd()直接分解的方法目前存在高效的算法，所以不需要像方法(a)一样，先计算协方差矩阵，所以方法(b)的效率更高。 最后通过得到的W，可以进行维度压缩：