线性回归之最小二乘法推导

一元线性回归函数:
$y=\omega x+b$
假设有n组[x1,x2,x3…xn],[y1,y2,y3…yn]数据，它们之间的关系为 $f(x)=\omega x+b$ 线性回归的目的之让 $f(x)$ 与 $y$ 之间的差值最小当 $\omega 和b$ 取何值的时候二者之间的差值最小.设 $E_{(\omega ,b)}$ 为 $f(x)$ 与 $y$ 的差异值$E(\omega ,b)$为损失函数:
$$E_{(\omega ,b)}=\sum _{i=1}^n(f(x)-y_i{)}^2=\sum _{i=1}^n(\omega x_i+b-y_i{)}^2$$
最小二乘法来确定 $\omega 和b$ 的值因为 $E_{(\omega ,b)}$ 为凸函数求其偏导：
$$\frac{\partial E_{(\omega ,b)}}{\partial b}=\frac{{\sum }_{i=1}^n(\omega x_i+b-y_i{)}^2}{\partial b}=2\sum _{i=1}^n(\omega x_i+b-y_i)$$ $$=2[\sum _{i=1}^n\omega x_i+\sum _{i=1}^{n}b+-\sum _{i=1}^n{y}_i]$$
$$=2[\omega \sum _{i=1}^n{x}_i+nb-\sum _{i=1}^n{y}_i]$$令$$\bar{x}=\frac{{\sum }_{i=1}^n{x}_i}{n},\bar{y}=\frac{{\sum }_{i=1}^n{y}_i}{n}$$
$$\therefore \frac{\partial E_{(\omega ,b)}}{\partial b}=2[\omega n\bar{x}+nb-n\bar{y}]$$
$$=2n[\omega \bar{x}+b-\bar{y}]$$
令$b$的导数为0
$$2n[\omega \bar{x}+b-\bar{y}]=0$$
$$\because n>=1$$
$$\therefore \omega \bar{x}+b-\bar{y}=0$$
$$\therefore b=\bar{y}-\omega \bar{x}$$对$\omega$求导
$$\frac{\partial E_{(\omega ,b)}}{\partial \omega }=\frac{{\sum }_{i=1}^n(\omega x_i+b-y_i{)}^2}{\partial \omega }=2\sum _{i=1}^n(\omega x_i+b-y_i)x_i$$
$$=2[\sum _{i=1}^n\omega x_i^2+\sum _{i=1}^{n}b{x}_i-\sum _{i=1}^n{y}_i{x}_i]$$
$$=2[\sum _{i=1}^n\omega x_i^2+\sum _{i=1}^n(\bar{y}-\omega \bar{x})x_i-\sum _{i=1}^n{y}_i{x}_i]$$
$$=2[\sum _{i=1}^n\omega x_i^2+n\bar{x}(\bar{y}-\omega \bar{x})-\sum _{i=1}^n{y}_i{x}_i]$$
$$=2[\sum _{i=1}^n\omega x_i^2+n\bar{x}\bar{y}-n\omega {\bar{x}}^2-\sum _{i=1}^n{y}_i{x}_i]$$
$$=2[\omega (\sum _{i=1}^n{x}_i^2-n{\bar{x}}^2)+n\bar{x}\bar{y}-\sum _{i=1}^n{y}_i{x}_i]$$
令$\omega$的导数为0
$$\omega (\sum _{i=1}^n{x}_i^2-n{\bar{x}}^2)+n\bar{x}\bar{y}-\sum _{i=1}^n{y}_i{x}_i=0$$
$$\omega =\frac{{\sum }_{i=1}^n{y}_i{x}_i-n\bar{x}\bar{y}}{{\sum }_{i=1}^n{x}_i^2-n{\bar{x}}^2}$$
可化简
$$\because \bar{x}=\frac{{\sum }_{i=1}^n{x}_i}{n}$$
$$\therefore \sum _{i=1}^n{x}_i=n\bar{x}$$
$$\therefore \sum _{i=1}^n{x}_i^2-n{\bar{x}}^2=\sum _{i=1}^n{x}^2-2n{\bar{x}}^2+n{\bar{x}}^2$$
$$=\sum _{i=1}^n{x}_i^2-2\sum _{i=1}^n{x}_i\bar{x}+\sum _{i=1}^n{x}^2$$
$$=\sum _{i=1}^n(x_i-\bar{x}{)}^2$$
同理可得
$$\sum _{i=1}^n{y}_i{x}_i-n\bar{x}\bar{y}=\sum _{i=1}^n{y}_i{x}_i-2n\bar{x}\bar{y}+n\bar{x}\bar{y}$$
$$=\sum _{i=1}^n{y}_i{x}_i-\sum _{i=1}^n{x}_i\bar{y}-\sum _{i=1}^n{y}_i\bar{x}+\sum _{i=1}^n\bar{x}\bar{y}$$
$$=\sum _{i=1}^n(x_i-\bar{x})(y_i-\bar{y})$$
$$\therefore \omega =\frac{{\sum }_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}$$
$$\therefore b=\bar{y}-\omega \bar{x},\omega =\frac{{\sum }_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}$$
$$\bar{x},\bar{y}还原可得$$
$$b=\frac{{\sum }_{i=1}^n(y_i-\omega x_i)}{n},\omega =\frac{{\sum }_{i=1}^n{y}_i{x}_i-\frac{{\sum }_{i=1}^n{x}_i{y}_i}{n}}{{\sum }_{i=1}^n{x}_i^2+\frac{({\sum }_{i=1}^n{x}_i{)}^2}{n}}$$