Autoencoders

Autoencoders

编码器：$h = f(x)$
解码器：$r = g(h)$

autoencoders are designed to be unable to learn to copy perfectly
Traditionally, autoencoders were used for dimensionality reduction or feature learning. Recently, theoretical connections between autoencoders and latent variable models have brought autoencoders to the forefront(前沿) of generative modeling(生成模型)

正则化编码机

An autoencoder whose code dimension is less than the input dimension is called undercomplete.
Helmholtz machine
generative stochastic networks
autoencoders fail to learn anything useful if the encoder and decoder are given too much capacity

编码的维度、编码器与解码器的能力应该取决于模型分布的复杂度

正则化得编码器并不是限制编码解码器的能力和编码的大小，而是利用损失函数使得自动编码机拥有一些其他的特性，例如编码的稀疏性，smallness，导数的表示，以及对噪音或错误输入的鲁棒性

Sparse Autoencoders

$$L(x,g(f(x))) + \Omega(h)$$
用于特征学习
违反了贝叶斯推理的解释

Sparse autoencoders are typically used to learn features for another task such as classification.

可以把稀疏惩罚想象为正则项
MAP

$$log\ p(\theta|x) = log p(x|\theta)p(\theta)$$

把稀疏惩罚想象为训练的生成模型包含潜在变量
假设模型包含一个显性变量x和一个潜在变量y

$$p_{model}(x,h) = p_{model}(h) p_{model}(x|h)$$

We refer to $p_{model}(h)$ as the model’s prior distribution over the latent variables, representing the model’s beliefs prior to seeing $x$.

Denoising Autoencoders

$$L(x,g(f(\hat x)))$$
$\hat x$是$x$的添加了噪音

理解

假设只有无标记的训练样本$\{x^{(1)},x^{(2)},x^{(3)}...\}$
自动编码机就是一个无监督的算法，利用后向传播，设定目标值和输入的值相等，$y^{(i)} = x^{(i)}$

autoencoder试图学习一个函数$h_{W,b}(x)$

it is trying to learn an approximation to the identity function, so as to output $\textstyle \hat{x}$ that is similar to $\textstyle x$

在学习的过程中，最终得到的恒等函数并不重要，而是通过设置隐藏层的大小和激活函数的能力，我们能够发现数据中有趣的结构

this simple autoencoder often ends up learning a low-dimensional representation very similar to PCAs
简单的自动编码机相当于PCA的效果
Informally, we will think of a neuron as being “active” (or as “firing”) if its output value is close to 1, or as being “inactive” if its output value is close to 0.