ufldl 深度学习入门 第三发： 自我学习与无监督特征学习

目的：使用稀疏自编码器提取特征，使用softmax做分类器，实现手写字符识别分类。

好处：相比较前面直接使用原始像素的softmax分类器（92%识别率），能够提升分类器的识别率，达到98%以上。

体会：通过自学习提取特征，能够模仿人类的大脑完成特征的抽象提取。而且是自动提取过程，对于海量数据，将有非常大的优势。

UFLDL学习链接：UFLDL，感谢吴恩达同学

说明：在实现这一节之前一定要先完成稀疏自编码器，softmax回归这两节，因为这一节的许多代码都是借用这两节的。

这里给出Andrew ng的代码以及自己的代码（YOUR CODE HERE 部分），并附上详细的说明分析

%% CS294A/CS294W Self-taught Learning Exercise

%  Instructions
%  ------------
% 
%  This file contains code that helps you get started on the
%  self-taught learning. You will need to complete code in feedForwardAutoencoder.m
%  You will also need to have implemented sparseAutoencoderCost.m and 
%  softmaxCost.m from previous exercises.
%
%% ======================================================================
%  STEP 0: Here we provide the relevant parameters values that will
%  allow your sparse autoencoder to get good filters; you do not need to 
%  change the parameters below.

inputSize  = 28 * 28;
numLabels  = 5;
hiddenSize = 200;
sparsityParam = 0.1; % desired average activation of the hidden units.
                     % (This was denoted by the Greek alphabet rho, which looks like a lower-case "p",
		             %  in the lecture notes). 
lambda = 3e-3;       % weight decay parameter       
beta = 3;            % weight of sparsity penalty term   
maxIter = 400;   % 我的电脑内存在4G的情况下，设为400会内存溢出，于是我设为80

%% ======================================================================
%  STEP 1: Load data from the MNIST database
%
%  This loads our training and test data from the MNIST database files.
%  We have sorted the data for you in this so that you will not have to
%  change it.

% Load MNIST database files
mnistData   = loadMNISTImages('mnist/train-images-idx3-ubyte');  %这里得到784×60000 的matrix
mnistLabels = loadMNISTLabels('mnist/train-labels-idx1-ubyte');     % 这里得到 60000×1的matrix

% Set Unlabeled Set (All Images)

% Simulate a Labeled and Unlabeled set
labeledSet   = find(mnistLabels >= 0 & mnistLabels <= 4);  % 找到0-4数字的索引
unlabeledSet = find(mnistLabels >= 5);  %找到5-9数字的索引，作为无标签数据，用来训练自编码器

numTrain = round(numel(labeledSet)/2);  % 计算训练集的样本数量，取0-4中一般的样本作为
                                                                     % softmax的训练样本，一半的样本作为测试样本
trainSet = labeledSet(1:numTrain);     % 将0-4的数字分为两组，一半用来训练Softmax分类器
testSet  = labeledSet(numTrain+1:end);  %一半用来做测试集。

unlabeledData = mnistData(:, unlabeledSet);  %得到无标签数据矩阵

trainData   = mnistData(:, trainSet);  % 得到有标签训练数据集矩阵
trainLabels = mnistLabels(trainSet)' + 1; 
                      % Shift Labels to the Range 1-5，把原本的0-4的标签，变成了1-5的标签

testData   = mnistData(:, testSet);  %得到有标签测试数据集矩阵
testLabels = mnistLabels(testSet)' + 1;   
                     % Shift Labels to the Range 1-5，把原本的0-4的标签，变成了1-5的标签

% Output Some Statistics
fprintf('# examples in unlabeled set: %d\n', size(unlabeledData, 2));
fprintf('# examples in supervised training set: %d\n\n', size(trainData, 2));
fprintf('# examples in supervised testing set: %d\n\n', size(testData, 2));

%% ======================================================================
%  STEP 2: Train the sparse autoencoder
%  This trains the sparse autoencoder on the unlabeled training
%  images. 

%  Randomly initialize the parameters
theta = initializeParameters(hiddenSize, inputSize);

%% ----------------- YOUR CODE HERE ----------------------
%  Find opttheta by running the sparse autoencoder on
%  unlabeledTrainingImages

opttheta = theta; 

%  Use minFunc to minimize the function
addpath minFunc/
options.Method = 'lbfgs'; % Here, we use L-BFGS to optimize our cost
                          % function. Generally, for minFunc to work, you
                          % need a function pointer with two outputs: the
                          % function value and the gradient. In our problem,
                          % sparseAutoencoderCost.m satisfies this.
options.maxIter = maxIter;      % Maximum number of iterations of L-BFGS to run 
options.display = 'on';
                           
                          % 调用minFunc，参数列表中传入sparseAutoencoderCost的函数句柄，
                          % 这样在函数minFunc中就可以调用该cost函数，从而不断迭代得到最优解
[opttheta, cost] = minFunc( @(p) sparseAutoencoderCost(p, ...
                                   inputSize, hiddenSize, ...
                                   lambda, sparsityParam, ...
                                   beta, unlabeledData), ...   % 这里传入unlabeledData，用作训练的数据集
                              theta, options);

%% -----------------------------------------------------
                          
% Visualize weights
W1 = reshape(opttheta(1:hiddenSize * inputSize), hiddenSize, inputSize);
display_network(W1'); %？？？？为什么显示w1

%%======================================================================
%% STEP 3: Extract Features from the Supervised Dataset
%  
%  You need to complete the code in feedForwardAutoencoder.m so that the 
%  following command will extract features from the data.

% 把原始的特征转换成由稀疏自编码器提取的特征
trainFeatures = feedForwardAutoencoder(opttheta, hiddenSize, inputSize, ...
                                       trainData);

testFeatures = feedForwardAutoencoder(opttheta, hiddenSize, inputSize, ...
                                       testData);

%%======================================================================
%% STEP 4: Train the softmax classifier

softmaxModel = struct;  
%% ----------------- YOUR CODE HERE ----------------------
%  Use softmaxTrain.m from the previous exercise to train a multi-class
%  classifier. 

%  Use lambda = 1e-4 for the weight regularization for softmax

% You need to compute softmaxModel using softmaxTrain on trainFeatures and
% trainLabels

numClasses=5;
lambda=1e-4;
options.maxIter = 100;
featureSize=hiddenSize;  
             % 注意这个size要改成提取出的feature的size，而不是原始的inputSize

softmaxModel = softmaxTrain(featureSize, numClasses, lambda, ...
                            trainFeatures, trainLabels, options);


%% -----------------------------------------------------


%%======================================================================
%% STEP 5: Testing 

%% ----------------- YOUR CODE HERE ----------------------
% Compute Predictions on the test set (testFeatures) using softmaxPredict
% and softmaxModel

% 
[pred] = softmaxPredict(softmaxModel, testFeatures);


%% -----------------------------------------------------

% Classification Score
% 我在使用minfunc<span style="color:#FF6600;">迭代80次</span>的情况下，
% 得到的识别率是 <span style="color:#FF6600;">97.404890%</span>
 fprintf('Test Accuracy: %f%%\n', 100*mean(pred(:) == testLabels(:)));

% (note that we shift the labels by 1, so that digit 0 now corresponds to
%  label 1)
%
% Accuracy is the proportion of correctly classified images
% The results for our implementation was:
%
% Accuracy: 98.3%
%
% 

matlab中的使用函数

1 find（），可以返回矩阵a中满足条件的元素的索引值。

2 numel（a），返回矩阵a中的元素个数。

3 max（），上面代码中有一处用到[nop, pred] = max(theta * data); 这里max第一个返回值是参数矩阵每列的最大值，第二个返回值是参数矩阵每列最大值的行号。这里我们很方便的通过一个max函数，直接就得到了预测的结果，用的比较巧妙。