线性回归两种实现

线性回归

1.梯度下降法

$$h_{\theta }(x)={\theta }^{T}x=\sum _{i=0}^n({\theta }_i{x}_i)$$
代价函数
$$J(\theta )=\frac{1}{2m}\sum _{j=1}^m(h_{\theta }(x{)}^{(j)}-y^{(j)}{)}^2$$
梯度下降求解
$${\theta }_i={\theta }_i-\alpha \frac{1}{m}\sum _{j=1}^m(h_{\theta }(x{)}^{(j)}-y^{(j)})(x_i{)}^{(j)}$$

矢量化

设i = 0、1，m = 3
$$\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]=\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]-\alpha \frac{1}{m}\sum _{j=1}^m(b^{(j)}){\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(j)},b^{(j)}=h_{\theta }(x{)}^{(j)}-y^{(j)}$$
$$\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]=\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]-\alpha \frac{1}{m}\left[\begin{array}{c}{b}^{(1)}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(1)}+b^{(2)}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(2)}+b^{(3)}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(3)}\end{array}\right],b^{(j)}=h_{\theta }(x{)}^{(j)}-y^{(j)}$$
$$\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]=\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]-\alpha \frac{1}{m}\left[\begin{array}{c}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(1)}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(2)}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(3)}\end{array}\right]\left[\begin{array}{c}{b}^{(1)}\\ b^{(2)}\\ b^{(3)}\end{array}\right],b^{(j)}=h_{\theta }(x{)}^{(j)}-y^{(j)}$$
其中:
$$\left[\begin{array}{c}{b}^{(1)}\\ b^{(2)}\\ b^{(3)}\end{array}\right]=\left[\begin{array}{c}{h}_{\theta }(x{)}^{(1)}\\ h_{\theta }(x{)}^{(2)}\\ h_{\theta }(x{)}^{(3)}\end{array}\right]-\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ y^{(3)}\end{array}\right]=\left[\begin{array}{c}{\left[\begin{array}{cc}{x}_0& x_1\end{array}\right]}^{(1)}\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]\\ {\left[\begin{array}{cc}{x}_0& x_1\end{array}\right]}^{(2)}\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]\\ {\left[\begin{array}{cc}{x}_0& x_1\end{array}\right]}^{(3)}\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]\end{array}\right]-\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ y^{(3)}\end{array}\right]=\left[\begin{array}{c}{\left[\begin{array}{cc}{x}_0& x_1\end{array}\right]}^{(1)}\\ {\left[\begin{array}{cc}{x}_0& x_1\end{array}\right]}^{(2)}\\ {\left[\begin{array}{cc}{x}_0& x_1\end{array}\right]}^{(3)}\end{array}\right]\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]-\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ y^{(3)}\end{array}\right]$$
则：
$$\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]=\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]-\alpha \frac{1}{m}\left[\begin{array}{c}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(1)}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(2)}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(3)}\end{array}\right]\left[\begin{array}{c}\left[\begin{array}{c}{\left[\begin{array}{cc}{x}_0& x_1\end{array}\right]}^{(1)}\\ {\left[\begin{array}{cc}{x}_0& x_1\end{array}\right]}^{(2)}\\ {\left[\begin{array}{cc}{x}_0& x_1\end{array}\right]}^{(3)}\end{array}\right]\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right]-\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ y^{(3)}\end{array}\right]\end{array}\right]$$
$$XX=\left[\begin{array}{c}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(1)}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(2)}{\left[\begin{array}{c}{x}_0\\ x_1\end{array}\right]}^{(3)}\end{array}\right],\theta \theta =\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right],yy=\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ y^{(3)}\end{array}\right]$$
 
 
 

$$最终计算公式:\theta \theta =\theta \theta -\alpha \frac{1}{m}XX({XX}^T\theta \theta -yy)$$

import numpy as np
import matplotlib.pyplot as plt

mean = (1.5, 1.5)
cov = [[1, 0.95], [0.95, 1]]
XY = np.random.multivariate_normal(mean, cov, 30).T
m = XY.shape[1]  # 样本数
# 显示数据
plt.scatter(XY[0,:], XY[1,:], c='b', s=10, edgecolor='none')
plt.xlabel("X", fontsize=14)
plt.ylabel("Y", fontsize=14)
plt.tick_params(axis='both', labelsize=14)
plt.axis([-1, 4, -1, 4])
plt.show()

# 回归函数 h=theta^T*X=theta0X0+theta1X1
# 要生成 y=kx+b 的回归函数，令其中 X0=1
X = np.concatenate((np.ones((1, m)), XY[0, :][np.newaxis, :]), axis=0)
Y = XY[1, :][np.newaxis, :].T
theta = np.random.random((2, 1)) - 0.5
alpha = 0.1  # 学习速率
epochs = 200  # 迭代次数
for i in range(epochs):
    theta = theta - alpha * 1/m * X.dot(np.dot(X.T, theta) - Y)

# 回归直线
p_x = np.array([-1, 4])
p_y = theta[0, 0] + theta[1, 0] * p_x
    
# 显示数据
plt.plot(XY[0,:], XY[1,:], 'x', c='b')
plt.plot(p_x, p_y, c='r')

plt.xlabel("X", fontsize=14)
plt.ylabel("Y", fontsize=14)
plt.tick_params(axis='both', labelsize=14)
plt.axis([-1, 4, -1, 4])
plt.show()

2.线性回归解析解

$$线性模型:{XX}^T=\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1d}\\ x_{21}& x_{22}& \cdots & x_{2d}\\ ⋮& ⋮& ⋮& ⋮\\ x_{n1}& x_{n2}& \cdots & x_{nd}\end{array}\right]yy=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_n\end{array}\right]$$
 
 
$$(显然R(X^T)\ne R(X^T|y),{XX}^T\theta \theta =yy无解)解决问题：{XX}^T\theta \theta \to yy$$
 
 
$$引入代价函数：J=\Vert {XX}^T\theta \theta -yy{\Vert }_2^2,使\frac{\partial J}{\partial \theta }=0而不是J=0$$
 
$$\begin{array}{rl}J& =\Vert {XX}^T\theta \theta -yy{\Vert }_2^2=({XX}^T\theta \theta -yy{)}^T({XX}^T\theta \theta -yy)=({\theta \theta }^{T}XX-{yy}^T)({XX}^T\theta \theta -yy)\\ & ={\theta \theta }^{T}XX{XX}^T\theta \theta -{\theta \theta }^{T}XXyy-{yy}^T{XX}^T\theta \theta +{yy}^{T}yy\end{array}$$
 
$$其中，对于{\theta \theta }^{T}XXyy，设X维数为2，样本数为3$$
$$\begin{array}{rl}{\theta \theta }^{T}XXyy& =\left[\begin{array}{c}{\theta }_1,{\theta }_2\end{array}\right]\left[\begin{array}{ccc}{x}_{11}& x_{21}& x_{31}\\ x_{12}& x_{22}& x_{32}\end{array}\right]\left[\begin{array}{c}{y}_1\\ y_2\\ y_3\end{array}\right]\\ & =\left[\begin{array}{c}{\theta }_1{x}_{11}+{\theta }_2{x}_{12},{\theta }_1{x}_{21}+{\theta }_2{x}_{22},{\theta }_1{x}_{31}+{\theta }_2{x}_{32}\end{array}\right]\left[\begin{array}{c}{y}_1\\ y_2\\ y_3\end{array}\right]\\ & =({\theta }_1{x}_{11}+{\theta }_2{x}_{12})y_1+({\theta }_1{x}_{21}+{\theta }_2{x}_{22})y_2+({\theta }_1{x}_{31}+{\theta }_2{x}_{32})y_3\end{array}$$

$$\frac{\partial {\theta \theta }^{T}XXyy}{\partial \theta }=\left[\begin{array}{c}\frac{\partial {\theta }^{T}Xy}{\partial {\theta }_1}\\ \frac{\partial {\theta }^{T}Xy}{\partial {\theta }_2}\end{array}\right]=\left[\begin{array}{c}{x}_{11}{y}_1+x_{21}{y}_2+x_{31}{y}_3\\ x_{12}{y}_1+x_{22}{y}_2+x_{32}{y}_3\end{array}\right]=\left[\begin{array}{ccc}{x}_{11}& x_{21}& x_{31}\\ x_{12}& x_{22}& x_{32}\end{array}\right]\left[\begin{array}{c}{y}_1\\ y_2\\ y_3\end{array}\right]=XXyy$$

$$求导：\frac{\partial J}{\partial \theta }=\frac{\partial ({\theta \theta }^{T}XX{XX}^T\theta \theta -{\theta \theta }^{T}XXyy-{yy}^T{XX}^T\theta \theta +{yy}^{T}yy)}{\partial \theta }=2XX{XX}^T\theta \theta -2XXyy=0$$
$$XX{XX}^T\theta \theta -XXyy=0\Rightarrow \theta \theta =(XX{XX}^T{)}^{-1}XXyy(如果XX{XX}^T可逆)$$

$$为使XX{XX}^T可逆，由于SS=XX{XX}^T为d\times d矩阵，为使SS可逆，SS必须满秩，即R(XX{XX}^T)=d，因此样本矩阵X须满足n>d$$

theta_2 = np.linalg.inv(X.dot(X.T)).dot(X).dot(Y)

# 回归直线
p_x2 = np.array([-1, 4])
p_y2 = theta_2[0, 0] + theta_2[1, 0] * p_x2

# 显示数据
plt.plot(XY[0,:], XY[1,:], 'x', c='b')
plt.plot(p_x2, p_y2, c='r')

plt.xlabel("X", fontsize=14)
plt.ylabel("Y", fontsize=14)
plt.tick_params(axis='both', labelsize=14)
plt.axis([-1, 4, -1, 4])
plt.show()