Ridge回归与Lasso回归——含python代码

  线性回归分析对数据噪声特别敏感，当样本矩阵$X$存在多重共线性或者样本数没有远大于特征维度时，$X^TX$是不可逆的或接近不可逆的；此时利用线性回归公式求得的参数$\boldsymbol{\theta}$很容易过拟合。如果能限制模型的复杂度，让$\boldsymbol{\theta}$的参数值不至于变得非常大，模型对噪声的敏感度就会降低。这就是$Ridge$回归和$Lasso$回归的基本思想。而降低模型复杂度的一个有效方法是$L_1正则化和L_2$正则化。在这里，加上$L_2$正则化的线性回归就是$Ridge$回归，而加上$L_1$正则化的线性回归就是$Lasso$回归。
  根据线性回归的损失函数加上$L_2$正则化，我们可以得到Ridge回归的损失函数：

$$J=\frac{1}{N}||X\boldsymbol{\theta}-Y||^2+\alpha||\boldsymbol{\theta}||^2$$
为了方便计算，我们可以令损失函数两边同乘以 $N$，并令$\frac{\beta}{2}=N\alpha$，画简如下：
$$J'=||X\boldsymbol{\theta}-Y||^2+\frac{\beta}{2}||\boldsymbol{\theta}||^2$$
对 $\boldsymbol{\theta}$求偏导数并令之为0可得：
$$\begin{eqnarray} \frac{\partial J}{\partial \boldsymbol{\theta}}&=&\frac{\partial}{\partial \boldsymbol{\theta}}[(\boldsymbol{\theta}^TX^TX\boldsymbol{\theta}-2\boldsymbol{\theta}^TX^TY+Y^TY)+\beta\boldsymbol{\theta}] \\ &=&2X^TX\boldsymbol{\theta}-2X^TY+\beta I\\ &=&0 \end{eqnarray}$$
解得： $\boldsymbol{\theta}=(X^TX+\beta I)^{-1}X^TY$，这便是Ridge回归的公式解。
  因为 $Lasso$回归的损失函数中 $L_1$正则化项没有固定导数，所以 $Lasso$回归只能通过梯度下降法来进行优化，不存在通解。

代码块

和线性回归的代码很类似，稍微修改就得到了Ridge回归和Lass回归的代码：

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import scale
from random import random

class LassoRegression(object):
    weight = np.array([])
    def __init__(self):
        return 
    def dif(self, W, c):#L1正则项求导
        w = np.array(W)
        w[W>0] = 1
        w[W<0] = -1
        return c*w

    def gradientDescent(self, X, Y, alpha, epoch, c):
        W = np.random.normal(0,1,size=(X.shape[1],))
        for i in range(epoch):
            W -= alpha*(X.T).dot(X.dot(W)-Y)/X.shape[0] + self.dif(W, c)
        return W

    def fit(self, train_data, train_target, alpha = 0.1, epoch = 300, c = 0.05):
        X = np.ones((train_data.shape[0], train_data.shape[1]+1))
        X[:,0:-1] = train_data
        Y = train_target
        self.weight = self.gradientDescent(X, Y, alpha, epoch, c)

    def predict(self, test_data):
        X = np.ones((test_data.shape[0], test_data.shape[1]+1))
        X[:,0:-1] = test_data
        return X.dot(self.weight)

    def evaluate(self, predict_target, test_target):
        predict_target[predict_target>=0.5] = 1
        predict_target[predict_target<0.5] = 0
        return sum(predict_target==test_target)/len(predict_target)

class RidgeRegression(object):
    weight = np.array([])
    way = 'gd'
    def __init__(self, training_way = 'gd'):
        self.way = training_way
    def gradientDescent(self, X, Y, alpha, epoch, c):
        W = np.random.normal(0,1,size=(X.shape[1],))
        for i in range(epoch):
            W -= alpha*(X.T).dot(X.dot(W)-Y)/X.shape[0] + c*W
        return W

    def fit(self, train_data, train_target, alpha = 0.1, epoch = 300, c = 0.05):
        X = np.ones((train_data.shape[0], train_data.shape[1]+1))
        X[:,0:-1] = train_data
        Y = train_target
        if self.way == 'gd':
            self.weight = self.gradientDescent(X, Y, alpha, epoch, c)
        else:
            I = np.eye(X.shape[1])
            self.weight = np.linalg.inv((X.T).dot(X)+c*I).dot(X.T).dot(Y)

    def predict(self, test_data):
        X = np.ones((test_data.shape[0], test_data.shape[1]+1))
        X[:,0:-1] = test_data
        return X.dot(self.weight)

    def evaluate(self, predict_target, test_target):
        predict_target[predict_target>=0.5] = 1
        predict_target[predict_target<0.5] = 0
        return sum(predict_target==test_target)/len(predict_target)

if __name__ == "__main__":
    lasso = LassoRegression()
    lasso.fit(train_data, train_target, 0.05, 1000, 0.01)
    lassoPredict = lasso.predict(test_data)
    print('lasso regression accruacy:',lasso.evaluate(lassoPredict,test_target))
    ridge = RidgeRegression(training_way = 'gd')
    ridge.fit(train_data, train_target, 0.05, 1000, 0.01)
    ridgePredict = ridge.predict(test_data)
    print('ridge regression accuracy:',ridge.evaluate(ridgePredict, test_target)