《Reducing the Dimensionality of Data with Neural Network》阅读心得

      本论文是2006发表在《SCIENCE》上的一篇关于使用神经网络进行数据降维的文章，作者为G. E. Hinton and R. R. Salakhutdinov。

        论文的思想是使用一个多层的神经网络对数据进行处理，中间层神经元个数少，实现从高维数据到低维数据的‘编码’，然后再通过低维到高维的‘解码’进行数据的重构，目标是减少元数据与重构数据的交叉熵。但是如果权重的初始化不合适时结果会表现不够好例如陷于局部最小，因此在前期先对权重进行初始的训练使得初始值接近最优值，整个项目叫做‘自动编码器’。如图

整个过程分两个阶段：pretraining和unrolling

第一阶段：构造神经元依次减少的受限波兹曼机，前一个rbm输出是后一个rbm的输入，然后根据能量最小得到权重和偏置的迭代公式，其中用到了cd-1方法，在收敛时得到初始训练的权重和偏置。

能量函数：

权重更新：

cd-1可以理解为使用重构的数据再次得到隐藏层偏置

第二阶段：将训练出来的受限波兹曼机依次连接起来进行训练，通过编码解码得到重构数据，目标函数是减少原始数据与重构数据之间的差距，方法使用反向传播调整参数。

注：受限波兹曼机RBM知识：点击打开链接

cd-1是一种采样方法，可参见：随机采样方法

    对于作者的代码的我的注释如下：(注：只是本人的理解可能会有错误，欢迎大家指出来)源代码出处

1.mnistdeepauto。m

% Version 1.000
%
% Code provided by Ruslan Salakhutdinov and Geoff Hinton  
%
% Permission is granted for anyone to copy, use, modify, or distribute this
% program and accompanying programs and documents for any purpose, provided
% this copyright notice is retained and prominently displayed, along with
% a note saying that the original programs are available from our 
% web page. 
% The programs and documents are distributed without any warranty, express or
% implied.  As the programs were written for research purposes only, they have
% not been tested to the degree that would be advisable in any important
% application.  All use of these programs is entirely at the user's own risk.


% This program pretrains a deep autoencoder for MNIST dataset
% You can set the maximum number of epochs for pretraining each layer
% and you can set the architecture of the multilayer net.

clear all
close all

maxepoch=10; %In the Science paper we use maxepoch=50, but it works just fine. 最大迭代数
numhid=1000; numpen=500; numpen2=250; numopen=30;%rbm每层神经元个数1000-500-250-30

fprintf(1,'Converting Raw files into Matlab format \n');
converter; 

fprintf(1,'Pretraining a deep autoencoder. \n');
fprintf(1,'The Science paper used 50 epochs. This uses %3i \n', maxepoch);

makebatches;
[numcases numdims numbatches]=size(batchdata);%每批样本数、维度、批数

fprintf(1,'Pretraining Layer 1 with RBM: %d-%d \n',numdims,numhid);
restart=1;
rbm;
hidrecbiases=hidbiases; 
save mnistvh vishid hidrecbiases visbiases;%保存第1个rbm的权值、隐含层偏置项、可视化层偏置项，为mnistvh.mat

fprintf(1,'\nPretraining Layer 2 with RBM: %d-%d \n',numhid,numpen);
batchdata=batchposhidprobs;
numhid=numpen;
restart=1;
rbm;
hidpen=vishid; penrecbiases=hidbiases; hidgenbiases=visbiases;
save mnisthp hidpen penrecbiases hidgenbiases;

fprintf(1,'\nPretraining Layer 3 with RBM: %d-%d \n',numpen,numpen2);
batchdata=batchposhidprobs;
numhid=numpen2;
restart=1;
rbm;
hidpen2=vishid; penrecbiases2=hidbiases; hidgenbiases2=visbiases;
save mnisthp2 hidpen2 penrecbiases2 hidgenbiases2;

fprintf(1,'\nPretraining Layer 4 with RBM: %d-%d \n',numpen2,numopen);
batchdata=batchposhidprobs;
numhid=numopen; 
restart=1;
rbmhidlinear;
hidtop=vishid; toprecbiases=hidbiases; topgenbiases=visbiases;
save mnistpo hidtop toprecbiases topgenbiases;

backprop; 

converter.m

% Version 1.000
%
% Code provided by Ruslan Salakhutdinov and Geoff Hinton
%
% Permission is granted for anyone to copy, use, modify, or distribute this
% program and accompanying programs and documents for any purpose, provided
% this copyright notice is retained and prominently displayed, along with
% a note saying that the original programs are available from our
% web page.
% The programs and documents are distributed without any warranty, express or
% implied.  As the programs were writ