机器学习笔记（2）-- 梯度下降法求单变量线性回归

数据生成

通过下式产生数据，作为训练集
$$y=4x^4+3x^3+2x^2+x+1+rand$$
加入0~1之间的随机数使数据更加接近实际数据

假设

我们假设一个函数用于拟合上述数据
$$h(x)={\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2+{\theta }_3{x}^3+{\theta }_4{x}^4$$
令
$$\Theta =\begin{array}{c}[{\theta }_0,{\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4]\end{array}$$

代价函数

$$J(\Theta )=\frac{1}{2m}\sum _{i=1}^m(h(x^{(i)})-y^{(i)}{)}^2$$
$m$为数据个数

梯度下降法

$${\theta }_i^k={\theta }_i^{k-1}-\alpha \frac{\partial J(\Theta )}{\partial {\theta }_i}$$
$\alpha$为学习率，用于控制梯度下降得快慢
这个式子表示新的${\theta }_i$等于上一个${\theta }_i$减去$\alpha \frac{\partial J(\Theta )}{\partial {\theta }_i}$使得${\theta }_i$始终朝着使代价函数$J(\Theta )$下降得方向变化
使用梯度下降法循环100次，或者使$J(\Theta )$的值下降到可以容许的误差范围内
$\frac{\partial J(\Theta )}{\partial {\theta }_i}$的偏导结果
$\frac{\partial J(\Theta )}{\partial {\theta }_0}=\frac{1}{m}{\sum }_{i=1}^m((h(x^{(i)})-y^{(i)})\cdot 1)$
$\frac{\partial J(\Theta )}{\partial {\theta }_1}=\frac{1}{m}{\sum }_{i=1}^m((h(x^{(i)})-y^{(i)})\cdot x)$
$\frac{\partial J(\Theta )}{\partial {\theta }_2}=\frac{1}{m}{\sum }_{i=1}^m((h(x^{(i)})-y^{(i)})\cdot x^2)$
$\frac{\partial J(\Theta )}{\partial {\theta }_3}=\frac{1}{m}{\sum }_{i=1}^m((h(x^{(i)})-y^{(i)})\cdot x^3)$
$\frac{\partial J(\Theta )}{\partial {\theta }_4}=\frac{1}{m}{\sum }_{i=1}^m((h(x^{(i)})-y^{(i)})\cdot x^4)$

代码

%数据的产生,并加入噪声
close all;
x = 0:0.01:1;
y = 4*x.^4+3*x.^3+2*x.^2+x+1+rand(1,length(x));
figure(1);
hold on;
plot(x,y,'r');

%假设函数为h(x) = theata0 + theata1*x+ theata2*x^2 + theata3*x^4
theata = 10*rand(1,5);%随机产生系数
alpha = 1.2;%学习率

for i=1:100%迭代100次
%假设函数
h_theata = theata(1)*ones(1,length(x))+theata(2)*x+theata(3)*x.^2+theata(4)*x.^3+theata(5)*x.^4;
plot(x,h_theata,'y');
var(i) =  1/length(x)*sum((h_theata - y).^2);%方差
%梯度下降法迭代求解新系数，代价函数1/2m *sum((h(x) - y)^2)
theata(1) = theata(1) - alpha*1/length(x)*sum(h_theata - y);
theata(2) = theata(2) - alpha*1/length(x)*sum((h_theata - y).*x);
theata(3) = theata(3) - alpha*1/length(x)*sum((h_theata - y).*x.^2);
theata(4) = theata(4) - alpha*1/length(x)*sum((h_theata - y).*x.^3);
theata(5) = theata(5) - alpha*1/length(x)*sum((h_theata - y).*x.^4);
end

%最终图形
h_theata = theata(1)*ones(1,length(x))+theata(2)*x+theata(3)*x.^2+theata(4)*x.^3+theata(5)*x.^4;
plot(x,h_theata,'b');
title('数据变化');
hold off;

%代价函数1/2m *sum((h(x) - y)^2)的变化曲线
figure(2);
plot(1:100,var);
title('代价函数大小变化');

图中红线为训练集数据的原始图线，黄色为拟合过程中图线的变化，蓝色为最终拟合曲线

在拟合过程中方差或者说代价函数的值的大小变化