吴恩达·Machine Learning || chap14 Dimensionality Reduction简记

14-1 Motivation I:Data Compression

Data Compression

Reduce data from 2D to 1D:project line $x_1,x_2⟶z_1$

Reduce data from 3D to 2D:project plane $x_1,x_2,x_3⟶z_1,z_2$

14-2 Motivation I:Data Visualization

Data Visualization

14-3 Principal Component Analysis problem formulation

PCA 主成分分析

Principal Component Analysis (PCA) problem formulation

Reduce from 2-dimension to 1-dimension: Find a direction (a vector $u^{(i)}\in {\mathbb{R}}^n$) onto which to project the data so as to minimize the projection error

Reduce from n-dimension to k-dimension: Find k vectors $u^{(1)},u^{(2)},\cdots ,u^{(k)}$ onto which to project the data, so as to minimize the projection error.

PCA is not linear regression

linear regression: distance y x$⟶$y

PCA: distance $x_1,x_2,\cdots$

14-4 Principal Component Analysis algorithm

Data preprocessing

Training set: $x^{(i)},x^{(2)},\cdots ,x^{(m)}$

Preprocessing (feature scaling/mean normalization):

​ ${\mu }_j=\frac{1}{m}{\sum }_{i=1}^m{x}_j^{(i)}$

​ Replace each $x_j^{(i)}$ with $x_j-{\mu }_j$.

If different features on different scales(e.g.,$x_1$=size of house,$x_2$ =number of bedrooms), scale features to have comparable range of values.

Principal Component Analysis (PCA) algorithm

Reduce data from n-dimensions to k-dimensions
Compute “covariance matrix”
$\sum =\frac{1}{m}{\sum }_{i=1}^n(x^{(i)})(x^{(i)}{)}^T$
Compute "eigenvectors"of matrix $\sum$:

[U,S,V]=svd(Sigma); %or eig(sigma)

Sigma: $n\times n$ matrix

From [U,S,V]=svd(Sigma),we get:

$U_{reduce}$ ：$U=[u^{(1)},u^{(2)},u^{(3)},\cdots ,u^{(n)}]\in {\mathbb{R}}^{n\times n}$

---

After mean normalization(ensure every feature has zero mean )and optionally feature scaling:

$Sigma=\frac{1}{m}{\sum }_{i=1}^n(x^{(i)})(x^{(i)}{)}^T$

$[U,S,V]=svd(Sigma);$

$Ureduce=U(:,1:k);$

$z=Ureduc{e}^{\prime }\ast x$

14-5 Choosing the number of principal components

Choosing k (number of principal components)

14-6 Reconstruction from compressed representation

Reconstruction from compressed representation

$z=U_{reduce}^{T}x{x}_{approx}=U_{reduce}.z$

14-7 Advice for applying PCA

Supervise learning speedup

$$(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),\cdots ,(x^{(m)},y^{(m)})$$

Extract inputs:

​ Unlabeled dataset: $$x^{(1)},x^{(2)},\cdots ,x^{(m)}\in {\mathbb{R}}^{10000}⟶z^{(1)},z^{(2)},\cdots ,z^{(m)}\in {\mathbb{R}}^{1000}$$

​ New training set:

$$(z^{(1)},y^{(1)}),(z^{(2)},y^{(2)}),\cdots ,(z^{(m)},y^{(m)})$$

Note: Mapping $x^{(i)}\to z^{(i)}$ should be defined by running PCA only on the training set. This mapping can be applied as well to the examples $x_{cv}^{(i)}$ and $x_{test}^{(i)}$ in the cross validation and test sets.

Application of PCA

• Compression
  • Reduce memory/disk needed to store data
  • Speed up learning algorithm
• Visualization

Bad use of PCA: To prevent overfitting

Use :$z^{(i)}$ instead of $x^{(i)}$ to reduce the number of features to $k<n$
Thus, fewer features, less likely to overfit

$\times$

This might work OK, but isnt a good way to address overfitting. Use regularization instead

$$\underset{\theta }{\min }\frac{1}{2m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2+\frac{\lambda }{2m}\sum _{j=1}^n{\theta }_j^2$$

PCA is sometimes used where it shouldn’t be

Design of ML system:

• Get training set { ( x ( 1 ) , y ( 1 ) ) , ( x ( 2 ) , y ( 2 ) ) , ⋯   , ( x ( m ) , y ( m ) ) } \{ ( x ^ { ( 1 ) } , y ^ { ( 1 ) } ) , ( x ^ { ( 2 ) } , y ^ { ( 2 ) } ) , \cdots , ( x ^ { ( m ) } , y ^ { ( m ) } ) \} {(x(1),y(1)),(x(2),y(2)),⋯,(x(m),y(m))}

• Run PCA to reduce $x^{(i)}$ in dimension to get $z^{(i)}$

• Train logistic regression on { ( z ( 1 ) , y ( 1 ) ) , ⋯   , ( z ( m ) , y ( m ) ) } \{ ( z ^ { ( 1 ) } , y ^ { ( 1 ) } ) , \cdots , ( z ^ { ( m ) } , y ^ { ( m ) } ) \} {(z(1),y(1)),⋯,(z(m),y(m))}

• Test on test set:Map $x_{test}^{(i)}$ to $z_{test}^{(i)}$.Run $h_{\theta }(z)$ on { ( z t e s t ( 1 ) , y t e s t ( 1 ) , ⋯   , ( z t e s t ( m ) , y t e s t ( m ) ) } \{(z_{test}^{(1)},y_{test}^{(1)},\cdots,(z_{test}^{(m)},y_{test}^{(m)})\} {(ztest(1)​,ytest(1)​,⋯,(ztest(m)​,ytest(m)​)}

How about doing the whole thing without using PCA?

Before implementing PCA, first try running whatever you want to do with the original /raw data $x^{(i)}$. Only if that doesn’t do what you want, then implement PCA and consider using $z^{(i)}$