机器学习-线性回归原理

本文主要介绍用梯度下降求线性回归最优解的数学细节

线性回归

令$Y=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right]$，$X=\left[\begin{array}{cc}{x}_1& 1\\ x_2& 1\\ ⋮& \\ x_m& 1\end{array}\right]$，$w=\left[\begin{array}{c}{w}_1\\ w_2\\ ⋮\\ w_n\\ b\end{array}\right]$，$x_i\in R^n$，
$$\hat{Y}=Xw$$

objective function:

$$J(w)=\frac{1}{2m}(Y-\hat{Y}{)}^T(Y-\hat{Y})$$

gradient of objective function:

根据标量对向量求导公式：

$$(\frac{\partial u^{T}v}{\partial x}=\frac{\partial u}{\partial x}v+\frac{\partial v}{\partial x}u)$$

$$\frac{\partial J(w)}{\partial w}=\frac{1}{2m}[\frac{\partial (Y-Xw)}{\partial w}(Y-Xw)+\frac{\partial (Y-Xw{)}^T}{\partial w}(Y-Xw{)}^T]=-\frac{1}{m}{X}^T(Y-Xw)$$

gradient descent:

$$w=w-\eta \frac{\partial J(w)}{\partial w}=w+\eta \frac{1}{m}{X}^T(Y-Xw)$$

numpy实现

def compute_square_loss(X, y, theta):

    n_instance, n_feature = X.shape
    loss = 1 / (2*n_instance) * (y - X.dot(theta.T)).dot((y - X.dot(theta.T)).T)

    return loss

def compute_square_loss_gradient(X, y, theta):

    n_instance, n_feature = X.shape
    return -1/n_instance * X.T.dot((y - X.dot(theta.T)).T)

如果考虑加上正则化项，当为Ridge Regression时，
（考虑了加上bias一起正则化）

objective function:

$$J(w)=\frac{1}{2m}(Y-\hat{Y}{)}^T(Y-\hat{Y})+\lambda w^{T}w$$

gradient of objective function:

$$\frac{\partial J(w)}{\partial w}=\frac{1}{2m}[\frac{\partial (Y-Xw)}{\partial w}(Y-Xw)+\frac{\partial (Y-Xw{)}^T}{\partial w}(Y-Xw{)}^T]+2\lambda w=-\frac{1}{m}{X}^T(Y-Xw)+2\lambda w$$

gradient descent:

$$w=w-\eta \frac{\partial J(w)}{\partial w}=w+\eta (\frac{1}{m}{X}^T(Y-Xw)-2\lambda w)$$