线性回归学习笔记
推导过程
f
(
x
i
)
=
w
x
i
+
b
f(x_i)=wx_i+b
f(xi)=wxi+b
目标函数为
m
i
n
∑
i
=
1
m
(
f
(
x
i
)
−
y
i
)
2
=
m
i
n
∑
i
=
1
m
(
y
i
−
w
x
i
−
b
)
2
min \sum_{i=1}^{m}(f(x_i)-y_i)^2=min \sum_{i=1}^{m}(y_i-wx_i-b)^2
mini=1∑m(f(xi)−yi)2=mini=1∑m(yi−wxi−b)2
分别对
w
w
w和
b
b
b求偏导,
∂
E
(
w
,
b
)
∂
w
=
2
∑
i
=
1
m
(
w
x
i
2
−
x
i
(
y
i
−
b
)
)
\frac {\partial E_{(w,b)}}{\partial w}=2\sum_{i=1}^{m}\bigl(wx_i^2-x_i(y_i-b)\bigr)
∂w∂E(w,b)=2i=1∑m(wxi2−xi(yi−b))
∂
E
(
w
,
b
)
∂
b
=
2
(
∑
i
=
1
m
(
w
x
i
−
y
i
)
+
m
b
)
\frac {\partial E_{(w,b)}}{\partial b}=2\bigl( \sum_{i=1}^{m}(wx_i-y_i) + mb \bigr)
∂b∂E(w,b)=2(i=1∑m(wxi−yi)+mb)
令两式为0,得
∑
i
=
1
m
(
x
i
y
i
−
w
x
i
2
)
=
b
∑
i
=
1
m
x
i
\sum_{i=1}^{m}(x_iy_i-wx_i^2)=b\sum_{i=1}^{m}x_i
i=1∑m(xiyi−wxi2)=bi=1∑mxi
∑
i
=
1
m
(
y
i
−
w
x
i
)
=
m
b
\sum_{i=1}^{m}(y_i-wx_i) = mb
i=1∑m(yi−wxi)=mb
消去
b
b
b,得
m
∑
i
=
1
m
(
x
i
y
i
−
w
x
i
2
)
∑
i
=
1
m
x
i
=
∑
i
=
1
m
(
y
i
−
w
x
i
)
m\frac {\sum_{i=1}^{m}(x_iy_i-wx_i^2)}{\sum_{i=1}^{m}x_i}=\sum_{i=1}^{m}(y_i-wx_i)
m∑i=1mxi∑i=1m(xiyi−wxi2)=i=1∑m(yi−wxi)
m
∑
i
=
1
m
(
x
i
y
i
−
w
x
i
2
)
=
∑
i
=
1
m
x
i
∑
i
=
1
m
y
i
−
w
∑
i
=
1
m
x
i
∑
i
=
1
m
x
i
m\sum_{i=1}^{m}(x_iy_i-wx_i^2)=\sum_{i=1}^{m}x_i\sum_{i=1}^{m}y_i-w\sum_{i=1}^{m}x_i\sum_{i=1}^{m}x_i
mi=1∑m(xiyi−wxi2)=i=1∑mxii=1∑myi−wi=1∑mxii=1∑mxi
w
(
∑
i
=
1
m
x
i
∑
i
=
1
m
x
i
−
m
∑
i
=
1
m
x
i
2
)
=
∑
i
=
1
m
x
i
∑
i
=
1
m
y
i
−
m
∑
i
=
1
m
x
i
y
i
w(\sum_{i=1}^{m}x_i\sum_{i=1}^{m}x_i- m\sum_{i=1}^{m}x_i^2)= \sum_{i=1}^{m}x_i\sum_{i=1}^{m}y_i-m\sum_{i=1}^{m}x_iy_i
w(i=1∑mxii=1∑mxi−mi=1∑mxi2)=i=1∑mxii=1∑myi−mi=1∑mxiyi
w
=
x
‾
∑
i
=
1
m
y
i
−
∑
i
=
1
m
x
i
y
i
1
/
m
∑
i
=
1
m
x
i
∑
i
=
1
m
x
i
−
∑
i
=
1
m
x
i
2
w= \frac{\overline x\sum_{i=1}^{m}y_i-\sum_{i=1}^{m}x_iy_i}{1/m\sum_{i=1}^{m}x_i\sum_{i=1}^{m}x_i- \sum_{i=1}^{m}x_i^2}
w=1/m∑i=1mxi∑i=1mxi−∑i=1mxi2x∑i=1myi−∑i=1mxiyi
最终得
w
=
∑
i
=
1
m
y
i
(
x
i
−
x
‾
)
∑
i
=
1
m
x
i
2
−
1
m
(
∑
i
=
1
m
x
i
)
2
w= \frac{\sum_{i=1}^{m}y_i(x_i-\overline x)}{\sum_{i=1}^{m}x_i^2-\frac{1}{m}(\sum_{i=1}^{m}x_i)^2}
w=∑i=1mxi2−m1(∑i=1mxi)2∑i=1myi(xi−x)
其中
x
‾
=
∑
i
=
1
m
x
i
m
\overline x=\frac{\sum_{i=1}^{m}x_i}{m}
x=m∑i=1mxi
那么
b
b
b的结果为
b
=
∑
i
=
1
m
(
y
i
−
w
x
i
)
m
b=\frac{\sum_{i=1}^{m}(y_i-wx_i)}{m}
b=m∑i=1m(yi−wxi)
Python实现
import numpy as np
def linear_regression(x, y, px):
sx = np.sum(x)
sy = np.sum(y)
sxy = np.dot(x, y.reshape(np.size(x), 1))
sx2 = np.dot(x, x.reshape(np.size(x), 1))
w = (sxy-sx*sy/np.size(x))/(sx2-sx*sx/np.size(x))
b = (sy-w*sx)/np.size(x)
py = w * px + b
return py