深度自解码器(Deep Auto-encoder)

版权声明:本文为博主原创文章,遵循 CC 4.0 by-sa 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/qq_32690999/article/details/78759435

本博客是针对李宏毅教授在youtube上发布的课程视频的学习笔记。

视频地址:ML Lecture 16: Unsupervised Learning - Auto-encoder

Auto-encoder

Encoder and Decoder

Each one of them can’t be trained respectively(no goal or no input), but they can be linked and trained together.

1

Starting from PCA

2

PCA only has one hidden layer, so we can deepen it to Deep Auto-encoder.

3

4

Above is Hinton(2006)’s design of deep auto-encoder, it achieves good result.

picture

Application: Text Retrieval

To compress an article into a CODE.

Vector Space Model and Bag-of-Word Model

picture

In Bag-of-Word the shortcoming is that semantic factor is not considered in model.

picture

picture

Focusing on pixel-wise similarity may not induce good result~(MJ similar to Magnet….)

picture

Use Deep auto-encoder to preprocess picture.

picture

Focusing on CODE similarity induce better result.

Pre-training DNN

Use Auto-encoder to do pre-training.

Learn a auto-encoder first (lower right, apply L1 regularization to avoid auto-encoder’s ‘remembering’ input), then learn another auto-encoder(middle right), ….., at last, 500-10 layer’s weight can be learned using backpropagation.

picture

pre-training is necessary before in training DNN, but now with development of training technology, we can get good training result without pre-training. But When we have many unlabeled data, we could still use these data to pre-training to make final training better.

De-noising auto-encoder (contractive auto-encoder)

picture

Restricted Boltzmann Machine (RBM) -different from DNN, just look similar

picture

Deep Belief Network (DBN) -different from DNN, just look similar

picture

Auto-encoder for CNN

picture

Unpooling

picture

Deconvolution

Padding+Convolution=Deconvolution

picture

Suquence-to-Sequence Auto-encoder

picture

Some data is not ‘good’ to be represented in vector(like voice, article[lose semantic meaning]), it’s better to represent them in sequence.

picture

After impact L2 regularization to training process, we can get below:

picture

展开阅读全文

没有更多推荐了,返回首页