一元线性回归

假设$(x_{1},y_{1}), （x_{2},y_{2}), ..., （x_{n},y_{n})$是总体的n个观测值，一元线性回归的hypothesis函数:
$h_{\theta }(x)=\theta_{0}+\theta_{1}x$
观测值标示为估计值加误差的形式：
$y_{i}=\theta_{0}+\theta_{1}x_{i}+e_{i}$
误差的平方和：

$$Q=\sum_{i=1}^{n}(y_{i}-\theta_{0}-\theta_{1}x_{i})^{2}$$

最小二乘法是通过最小化Q来求$\theta_{0},\theta_{1}$

求解方法1 偏导为0，Q取最小值：
求Q对于$\theta_{0},\theta_{1}$的偏导：

$$\frac{\partial Q}{\partial \theta_{1}}=-2\sum_{i=1}^{n}(y_{i}-\theta_{0}-\theta_{1}x_{i})x_{i}$$
$$\frac{\partial Q}{\partial \theta_{0}}=-2\sum_{i=1}^{n}(y_{i}-\theta_{0}-\theta_{1}x_{i})$$
偏导为0，Q取最小值得到：
$$\theta_{1}=\frac{n \sum x_{i}y_{i}- \sum x_{i} \sum y_{i}}{n \sum x_{i}^2- (\sum x_{i})^2}$$
$$\theta_{0}=\frac{\sum x_{i}^2 \sum y_{i}- \sum x_{i} \sum x_{i}y_{i}}{n \sum x_{i}^2- (\sum x_{i})^2}$$

求解方法2 直观图形：
首先把所以的点都标准化

$${x_{i}}^{'}=\frac{x_{i}-\mu _{x}}{\sigma _{x}}$$
$${y_{i}}^{'}=\frac{y_{i}-\mu _{y}}{\sigma _{y}}$$
最佳直线为
$${y}^{'}=r{x}^{'}$$
$r=\frac{\sum(x_{i}-\bar{x})(y_{i}-\bar{y})}{n\sigma _{x}\sigma _{y}}$为 $x_{i}$与 $y_{i}$的相关系数，即
$$\frac{y-\mu _{y}}{\sigma _{y}}＝r\frac{x-\mu _{x}}{\sigma _{x}}$$
$$\theta_{1}=r\frac{\sigma _{y}}{\sigma _{x}}$$
$$\theta_{0}=\mu _{y}-r\mu _{x}\frac{\sigma _{y}}{\sigma _{x}}$$
把r代入得到
$$\theta_{1}=\frac{n \sum x_{i}y_{i}- \sum x_{i} \sum y_{i}}{n \sum x_{i}^2- (\sum x_{i})^2}$$
$$\theta_{0}=\frac{\sum x_{i}^2 \sum y_{i}- \sum x_{i} \sum x_{i}y_{i}}{n \sum x_{i}^2- (\sum x_{i})^2}$$
求解方法3 梯度下降：
start with some $\theta_{0}$, $\theta_{1}$
repeat until convergence{
$$\theta_{0}=\theta_{0}-a\frac{\partial Q}{\partial \theta_{0}}$$
$$\theta_{1}=\theta_{1}-a\frac{\partial Q}{\partial \theta_{1}}$$
}
a是步长，learning rate
hopefully we have the minimum of Q

第一种偏导为零是直接法，第三种梯度下降是迭代法，都是求误差平方和的方法。$\theta_{0}$,$\theta_{1}$的初始值，步长都影响梯度下降的结果。