Introduction
We have:
-
a dataset X X X in R D R^D RD consisting of n vectors (n training examples)
-
n vectors X 1 X_1 X1, …, X n X_n Xn where X i X_i Xi are D-dimensional vectors
Objective:
- find a low dimensional representation of the data that is as similar to X X X as possible.
Three important concepts
1. linear combination of the basis vectors
The first one is that every vector in R D R^D RD can be represented as a linear combination of the basis vectors.
X n X_n Xn can be written as the sum of i = 1 to D of β i n \beta_{in} βin * b i b_i bi.
- D个basis vectors b i b_i bi的线性加和
- b i b_i bi are an orthonormal bases of R D R^D RD
- orthonormal: 正交(各自垂直)
2. orthogonal projection onto 1-D subspace
We can interpret
β
i
n
\beta_{in}
βin as the orthogonal projection of
X
n
X_n
Xn onto the one dimensional subspace spanned by the
i
t
h
i_{th}
ith basis vector.
3. orthogonal projection of X onto the subspace spanned by the M basis vectors
If we have an orthonormal basis
b
1
b_1
b1 to
b
m
b_m
bm of
R
D
R^D
RD and we define
B
B
B to be the matrix that consists of these orthonormal basis vectors.
Then the projection of X X X onto the subspace, we can write as X ~ \widetilde X X is B ∗ B . T ∗ X B * B.T * X B∗B.T∗X.
That means X ~ \widetilde X X is the orthogonal projection of X onto the subspace spanned by the M basis vectors.
And B . T ∗ X B.T * X B.T∗X are the coordinates of X ~ \widetilde X X with respect of the basis vectors collected in the matrix B B B. This is also called the code, so coordinates or code.
PCA
The key idea in PCA
To find a lower dimensional representation X ~ n \widetilde X_n X n that can be expressed using fewer basis vectors, let’s say M.
Assumptions:
- The data is centered, that means the dataset has mean zero.
- b 1 b_1 b1 to b D b_D bD are an orthonormal bases of R D R^D RD.
Generally, we can write any
X
~
n
\widetilde X_n
X
n in the following way:
This entire thing is still living in R D R^D RD. So, we took our general way of writing any vector in R D R^D RD which comes from property one, and we split the sum in property one into two sums. One is living in an M-dimensional subspace and the other one is living in a D minus M-dimensional subspace which is an orthogonal complement to this particular subspace.
In PCA, we ignore the second term, so we get rid of this part.
the principal subspace
And then we call the subspace that is spanned by the basis vectors b 1 b_1 b1 to b M b_M bM the principal subspace. So b 1 b_1 b1 to b M b_M bM span the principal subspace.
Although X ~ n \widetilde X_n X n is still a D-dimensional vector, it lives in an M-dimensional subspace of R D R^D RD and only M coordinates; β n 1 \beta_{n1} βn1 to β n M \beta_{nM} βnM are necessary to represent it.
So, these ones are the coordinate of this, X ~ n \widetilde X_n X n vector. The betas of n also called the code of the coordinates of X ~ n \widetilde X_n X n with respect to the basis vectors b 1 b_1 b1 to b M b_M bM.
Objective
And the setting now is as follows. Assuming we have data X 1 X_1 X1 to X n X_n Xn, we want to find parameters β i n \beta_{in} βin and orthonormal basis vectors b i b_i bi, such that the average squared reconstruction era is minimised.
the average squared reconstruction error
J: the average squared reconstruction error
example
Let’s have a look at an example. We have data living in two dimensions and now we want to find a good one dimensional subspace such that the squared or average squared reconstruction error of the original data points and their corresponding projection is minimised.
Here I’m plotting the original dataset with their corresponding projections onto one dimensional subspaces and I’m cycling through a couple of options of subspaces and you can see that some of these projections are significantly more informative than others and in PCA we are going to find the best one. Our approach is to compute the partial derivatives of J J J with respect to the parameters.
The parameters are the
β
i
n
\beta_{in}
βin and the
b
i
b_i
bi.
We set the partial derivatives of J with respect to these parameters to zero and solve for the optimal parameters. But one observation we can already make. And that observation is that the parameters only enter this loss function through X ~ n \widetilde X_n X n.
This means that in order to get our partial derivatives, we need to apply the chain rule. So, we can write d J d_J dJ by d either β i n \beta_{in} βin or b i b_i bi can be written as d J d_J dJ by d X ~ n d_{\widetilde X_n} dX n tilde times d X ~ n d_{\widetilde X_n} dX n by d either β i n \beta_{in} βin or b i b_i bi.
We can already compute the first part.