Big Data Analytics 笔记整理 3

1 WHY Principal Component Analysis ?

PCA is useful to:

• reduce number of features (might reduce overfiting)
• reduce memory or disk storage required for features
• speed up execution of subsequent modelling step(s)
• visualization of features for higher order model
• find unknown structure in features via subsequent clustering
• detect outliers
• Usually in data analysis scheme:
  • Scale the covariates
  • Split the data into training and test set
  • Apply PCA to training and test separately
  • Build a model using features generated by PCA on training set
  • Assess prediction accuracy using features generated by PCA on test set

2 principle component

https://blog.csdn.net/MINGRAN_JIA/article/details/122464746

• by setting a threshold of variance to maintain, like 80%
• Cattell’s or Kaiser’s methods
• cross-validation

2.1 Concept of PCA

Suppose matrix $X:n\times p$, $p$ is the number of features, $\Sigma =Var(X)=X^{\mathsf{T}}X$ is the $p\times p$ covariance matrix of $X$.

For principle component $\overrightarrow{a_i}$, we want to project data in the direction of $\overrightarrow{a_i}$ (maximise the variance) $⟹$ $\underset{\overrightarrow{a_i}}{\max }Var(X\overrightarrow{a_i})=\underset{\overrightarrow{a_i}}{\max }{\overrightarrow{a_i}}^{\mathsf{T}}\overrightarrow{a_i}$, since this does not have upper bound, so:

Constraint $||\overrightarrow{a_i}||={\overrightarrow{a_i}}^{\mathsf{T}}\overrightarrow{a_i}=1$, use Lagrange fomula and Spectral decomposition of $\Sigma$

2.2 Useful result of PCA

Collossary:

$\Delta =diag({\lambda }_1,...,\lambda$