机器学习学习笔记--线性模型之一元线性回归

线性模型

一元线性回归

• 基本形式

$$f(\mathbf{x})=w_1{x}_1+w_2{x}_2+\dots +w_d{x}_d+b$$

向量形式
$$f(\mathbf{x})={\mathbf{w}}^{\mathrm{T}}\mathbf{x}+b$$
目标：均方误差最小化
$$\begin{array}{rl}\left(w^{\ast },b^{\ast }\right)& =\underset{(w,b)}{\mathrm{argmin}}\sum _{i=1}^m{\left(f\left(x_i\right)-y_i\right)}^2\\ & =\underset{(w,b)}{\mathrm{argmin}}\sum _{i=1}^m{\left(y_i-w{x}_i-b\right)}^2\end{array}$$
方法：线性回归模型的最小二乘“参数估计”。将$E_{w,b}$分别对$w$和$b$求导得到：
$$\frac{\partial E_{(w,b)}}{\partial w}=2\left(w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\left(y_i-b\right)x_i\right)$$

$$=\sum _{i=1}^m\frac{\partial }{\partial w}{\left(y_i-w{x}_i-b\right)}^2$$

$$=\sum _{i=1}^{m}2\cdot \left(y_i-w{x}_i-b\right)\cdot \left(-x_i\right)$$

$$\frac{\partial E_{(w,b)}}{\partial b}=2\left(mb-\sum _{i=1}^m\left(y_i-w{x}_i\right)\right)$$

这里$E_{w,b}$是关于w和b的凸函数，当它关于w和b的导数均为零时，得到w和b的最优解.

判断凹凸性：

设f(x,y)在区域D上具有二阶连续偏导数，记$A = f_{xx}’’(x,y),B = f_{xy}’’(x,y),C = f_{yy}’’(x,y) $则：

（1)D上恒有A>0，且AC-$B^2>=0$时，f(x,y)在区域D上是凸函数；

（2）D上恒有$A<0$且$AC-B^2\ge 0$时，f(x,y)在区域D上是凹函数

$$\begin{array}{rl}\frac{{\partial }^2{E}_{(w,b)}}{\partial w^2}& =\frac{\partial }{\partial w}\left(\frac{\partial E_{(w,b)}}{\partial w}\right)\\ & =\frac{\partial }{\partial w}\left[2\left(w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\left(y_i-b\right)x_i\right)\right]\\ & =\frac{\partial }{\partial w}\left[2w\sum _{i=1}^m{x}_i^2\right]\end{array}$$

$$=2\sum _{i=1}^m{x}_i^2$$

∂ 2 E ( w , b ) ∂ w ∂ b = ∂ ∂ b ( ∂ E ( w , b ) ∂ w ) = ∂ ∂ b [ 2 ( w ∑ i = 1 m x i 2 − ∑ i = 1 m ( y i − b ) x i ) ] = ∂ ∂ b [ − 2 ∑ i = 1 m y i x i + 2 ∑ i = 1 m b x i ) = ∂ ∂ b ( 2