机器学习-吴恩达-ex3

文件说明

本次作业将实现one-vs-all的逻辑回归和神经网络来进行手写数字识别
文件有：
ex3.m - Octave/MATLAB script that steps you through part 1
ex3 nn.m - Octave/MATLAB script that steps you through part 2
ex3data1.mat - 手写数字识别的训练数据
ex3weights.mat - 网络初始化权重
submit.m - 提交至服务器
displayData.m - 可视化数据
fmincg.m - 最小化函数
sigmoid.m - 激活函数
[*] lrCostFunction.m - 逻辑回归损失函数
[*] oneVsAll.m - 训练一对多分类器
[*] predictOneVsAll.m - 预测一对多分类器
[*] predict.m - 神经网络预测
*表示需要实现的文件

1.多分类

1.1 数据

加载数据

load('ex3data1.mat');

数据存储在ex3data1.mat中，包含5000例样本，每个样本是一张20*20大小的手写数字图片。load之后变成400*1的列向量，5000个样本用X表示，大小为5000*400。训练集的标签数据是5000*1的列向量，用y表示，数值在0-9中，表示其真实值。
m=5000表示样本量，n=400表示特征数
$$x^{(1)}=\left\{\begin{array}{c}1\\ 2\\ ......\\ 0\end{array}\right\}一个样本为400\ast 1的列向量$$
$$X=\left\{\begin{array}{c}{}^{-{(x^{(1)})}^T-}\\ {}^{-{(x^{(2)})}^T-}\\ ......\\ {}^{-{(x^{(5000)})}^T-}\end{array}\right\}所有样本为5000\ast 400的矩阵$$
$$y=\left\{\begin{array}{c}1\\ 2\\ ......\\ 0\end{array}\right\}一个样本为5000\ast 1的列向量$$

1.2 可视化数据

先随机选择100个样例，通过displayData.m中的displayData()展示数据

1.3 矩阵化逻辑回归

$h_{\theta }(x)=g({\theta }^{T}X)$，其中$g(z)=\frac{1}{1+e^{-z}}$

1.3.1逻辑回归中的损失函数为：

$$J(\theta )=\frac{1}{m}\sum _{i=1}^m\left[\begin{array}{c}-y^{(i)}log(h_{\theta }(x^{(i)}))-(1-y^{(i)}log(1-h_{\theta }(x^{(i)}))\end{array}\right]$$
为了计算每一个元素的差值和，我们需要对每个样例计算$h_{\theta }(x^{(i)})$，其中$h_{\theta }(x^{(i)})=g({\theta }^T{x}^{(i)})$，并且激活函数为$g(z)=\frac{1}{1+e^{-z}}$。
定义X和$\theta$如下：

$X=\left\{\begin{array}{c}{}^{-{(x^{(1)})}^T-}\\ {}^{-{(x^{(2)})}^T-}\\ ......\\ {}^{-{(x^{(5000)})}^T-}\end{array}\right\}且\theta =\left\{\begin{array}{c}{}^{{\theta }_0}\\ {}^{{\theta }_1}\\ ......\\ {}^{{\theta }_{400}}\end{array}\right\}$
所以有$X\theta =\left\{\begin{array}{c}{}^{-{(x^{(1)})}^T\theta -}\\ {}^{-{(x^{(2)})}^T\theta -}\\ ......\\ {}^{-{(x^{(5000)})}^T\theta -}\end{array}\right\}=\left\{\begin{array}{c}{}^{-{\theta }^T(x^{(1)})-}\\ {}^{-{\theta }^T(x^{(2)})-}\\ ......\\ {}^{-{\theta }^T(x^{(5000)})-}\end{array}\right\}={\theta }^T{X}^T,大小为5000\ast 1$
要做的工作是在lrCostFunction.m中计算损失函数，输入设定好的$\theta 、X、y$通过公式

1.3.2 梯度

逻辑回归梯度公式为 $\frac{\partial J}{\partial {\theta }_j}=\frac{1}{m}{\sum }_{i=1}^m((h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)})$
计算每个样本的梯度如下，

$=\frac{1}{m}{\sum }_{i=1}^m((h_{\theta }^{(i)})-y^{(i)})x^{(i)}=\frac{1}{m}{X}^T(h_{\theta }(x)-y)，$

$其中(h_{\theta }(x)-y)=\left[\begin{array}{c}{h}_{\theta }(x^{(1)})-y^{(1)}\\ h_{\theta }(x^{(2)})-y^{(2)}\\ ......\\ h_{\theta }(x^{(5000)})-y^{(5000)}\end{array}\right]$
令${\beta }_i=(h_{\theta }(x^{(i)}-y^{(i)})$，则有
${\sum }_i{\beta }_i{x}^{(i)}=\left[\begin{array}{cccc}|& |& & |\\ x^{(1)}& x^{(2)}& ......& x^{(5000)}\\ |& |& & |\end{array}\right]\left[\begin{array}{c}{\beta }_1\\ {\beta }_2\\ ⋮\\ {\beta }_{5000}\end{array}\right]=X^T\beta$

1.3.3 对逻辑回归进行正则化

正则化后的逻辑回归损失函数：
$$J(\theta )=\frac{1}{m}\sum _{i=1}^m\left[\begin{array}{c}-y^{(i)}log(h_{\theta }(x^{(i)}))-(1-y^{(i)}log(1-h_{\theta }(x^{(i)}))+\frac{\lambda }{2m}{\sum }_{j=1}^n{\theta }_j^2\end{array}\right]$$
正则化后的梯度：
$$\frac{\partial J}{\partial {\theta }_j}=\frac{1}{m}\sum _{i=1}^m((h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)})+\frac{\lambda }{m}{\theta }_j$$

以上逻辑回归的损失函数及梯度计算代码为：
输入$\theta .shape=[400,1]、X.shape=[5000,400]、y.shape=[5000,1]、\lambda 为实数$，返回$J为实数、grad$

function [J, grad] = lrCostFunction(theta, X, y, lambda)
m = length(y); % number of training examples
J = 0;
grad = zeros(size(theta));

temp = theta(2:length(theta));
J = sum((-y).*log(sigmoid(X*theta)) - (1-y).*log(1 - sigmoid(X*theta)))/m + (lambda/(2*m))*sum(temp.^2);

for i=1:size(X,2)
    grad(i) = ((sigmoid(X*theta) - y)'*X(:,i))/m;
end

grad = grad + [0;((lambda/m)*temp)];
grad = grad(:);

end

1.4 one-vs-all $K$分类

对所有的样例进行分类，$X.shape=[5000,400]、y.shape=[5000,1]、num\_labels=10表示10类结果，\lambda =0.1$，返回所有参数$\theta .shape=[10,401]$

function [all_theta] = oneVsAll(X, y, num_labels, lambda)

m = size(X, 1);%5000
n = size(X, 2);%400

all_theta = zeros(num_labels, n + 1);%[5000,401]

X = [ones(m, 1) X];%[5000,400]

initial_theta = zeros(n + 1, 1);%[400,1]
options = optimset('GradObj', 'on', 'MaxIter', 50);   %最小化函数

for c = 1:num_labels
[theta] =  fmincg (@(t)(lrCostFunction(t, X, (y == c), lambda)), initial_theta, options);
all_theta(c,:) = theta';
end;

end

预测,输入所有$\theta .shape=[10,401]$和$X.shape=[5000,400]$

$h_{\theta }(x)=g(X\theta )$，其中$g(z)=\frac{1}{1+e^{-z}}$

function p = predictOneVsAll(all_theta, X)

m = size(X, 1); %测试样例数
num_labels = size(all_theta, 1); %10类

p = zeros(size(X, 1), 1); %预测结果
X = [ones(m, 1) X]; %

temp = zeros(size(X, 1), num_labels);
for i=1:num_labels,
    temp(:,i) = sigmoid(X*(all_theta(i,:)'));
end

for i=1:m,
[x, ix] = max(temp(i,:));
p(i) = ix;
end

end

2神经网络

仅实现前馈过程，即predict，网络结构如图
输入两层网络的$\theta 和X$

function p = predict(Theta1, Theta2, X)

m = size(X, 1);
num_labels = size(Theta2, 1);

p = zeros(size(X, 1), 1);

X = [ones(m, 1) X];

z2 = X*Theta1';
a2 = sigmoid(z2);
a2 = [ones(size(a2, 1), 1) a2];

z3 = a2*Theta2';
a3 = sigmoid(z3);

for i=1:m,
[x, ix] = max(a3(i,:));
p(i) = ix;
end


end