Manifold Interpretation of PCA and Linear Auto-Encoders
目标是寻找 x 在子空间的一个映射,并保存尽量多的信息
令编码器为
-
h
是
x 的一个低维模拟
解码器为
因为解码器和编码器都是线性的,那么最小化重构误差就是
也就是
W 的行形成协方差矩阵主要特征向量空间一组正交基
对于PCA, W 的行就是这些特征向量,以对应特征值的重要性排序
最优的重构误差
其中
- x∈RD
- h∈Rd
- λi 是协方差矩阵的特征值.
如果协方差的秩是 d , 那么重构误差就是 0.
ICA
Independent Component Analysis
独立成分分析
Herault and Ans, 1984; Jutten and Herault, 1991; Comon, 1994; Hyv¨arinen,1999; Hyv¨arinen et al., 2001
和概率PCA,特征分析相似,它也满足线性特征模型的条件
- sample real-valued factors
h∼P(h) - sample the real-valued observable variables
x=Wh+b+mnoise
其中不同的一点是它不假设先验分布是高斯分布,它只假设是参数化的,例如
P(h)=∏iP(hi)
假如假定隐藏变量是非高斯分布的,那么就可以重现他们。这也是ICA的目的。
Sparse Coding as a Generative Model
一个比较有趣的非高斯分布模型-分布是稀疏的
P(h) 在0附近值很大也就是 h 是0附近的概率很高 例如 参数化的拉普拉斯密度先验分布
P(h)=∏iP(hi)=∏iλ2e−λ|hi|
Student_t prior is
P(h)=∏iP(hi)∝∏i11+h2ivv+12
Greedy Layerwise Unsupervised Pre-Training
Greedy
- 不同层没有一起统合起来训练,可能会得到局部最优解Layerwise
- 每次只训练一层,训练第K层的时候保持前面的层保持不变Unsupervised
- 每一层都是无监督学习Pre-Training
- 它只是算法的第一步
Transfer Learning and Domain Adaptation
目标是抽取和利用数据集A的信息来应用到数据集B
譬如,不同的领域的具体评价不同(电影,音乐,书籍的评价),但有些地方是相同的。所以叫 Domain Adaptation两个例子
Extreme forms of transfer learning
- one-shot learning
- zero-shot learning
- zero-data learning
Manifold Interpretation of PCA and Linear Auto-Encoders
标签(空格分隔): 深度学习 个人兴趣
Look for projections of x into a subspace that preserves as much as information as possible about
x Let the encoder be
h=f(x)=WT(x−μ)-
h
is a low-dimensional representation of
x
Decoder
x^=g(h)=b+VhWith liner encoder and decoder, minimizing reconstruction error
E[||x−x^||2]
means thatV=W,μ=b=E[x]
and the rows of W form an orthonormal basis which spans the same subspace as the principal eigenvectors of the covariance matrix
C=E[(x−μ)(x−μ)T] In the case of PCA, the rows of W are these eigenvectors, ordered by the magnitude of the corresponding eigenvalues.
the optimal reconstruction error
minE[||x−x^||2]=∑i=d+1Dλi
Where-
D
is the dimension of
x -
d
is the dimension of
h - λi are the eigenvalues of the convariance.
If the covariance has rank d , the reconstrcution error is 0.
ICA
Independent Component Analysis
Herault and Ans, 1984; Jutten and Herault, 1991; Comon, 1994; Hyv¨arinen,1999; Hyv¨arinen et al., 2001Like probabilistic PCA and factor analysis, it also fits the linear factor model of Eqs.
- sample real-valued factors
h∼P(h) - sample the real-valued observable variables
x=Wh+b+mnoise
What is particular about ICA is that unlike PCA and factor analysis it does not assume that the prior is Gaussian. It only assumes that it is factorized, i.e.
P(h)=∏iP(hi)
In this case, if we assume that the latent variables arenon-Gaussian
, then we can recover them, and this is whatICA
is trying to achieve.
Sparse Coding as a Generative Model
A particularly interesting form of non-Gaussianity arises with distributions that are sparse.
P(h) puts high probability at or around 0. For instance, the factorized Laplace density prior is
P(h)=∏iP(hi)=∏iλ2e−λ|hi|
Student_t prior is
P(h)=∏iP(hi)∝∏i11+h2ivv+12
Greedy Layerwise Unsupervised Pre-Training
Greedy
- the different layers are not jointly trained with respect to a global training objective, which could make the procedure sub-optimalLayerwise
- it proceeds one layer at a time, training the k-layer while keeping the previous ones fixed.Unsupervised
- each layer is trained with an unsupervised representation learning algorithm.Pre-Training
- it should be only a first step before a joint training algorithm is applied to fine-tune all the layers together with respect to a criterion of interest
Transfer Learning and Domain Adaptation
The objective is to take advantage of data from a first setting to extract information that may be useful when learning or even directly making predictions in the second setting.
Two examples
Extreme forms of transfer learning
- one-shot learning
- zero-shot learning
- zero-data learning
- sample the real-valued observable variables