机器学习模型正则化与岭回归、LASSO回归

模型正则化

为了解决机器学习中方差过大问题，常用的手段是模型正则化，其原理是限制多项式模型中特征系数$\theta$，不让其过大，导致过拟合。
在线性回归模型中，目标是使得损失函数尽可能小
$$J(\theta )=\sum _{i=1}^m(y^{(i)}-{\theta }_0-{\theta }_1{X}_1^{(i)}-\dots \dots -{\theta }_n{X}_n^{(i)}{)}^2$$
当模型过拟合时，$\theta$就会非常大，当损失函数$J(\theta )$加上
$$\alpha \frac{1}{2}\sum _{i=1}^n{\theta }_i^2$$
此时要使得损失函数尽可能小，就要考虑到 $\theta$ 尽可能小，损失函数加上了 $\alpha \frac{1}{2}{\sum }_{i=1}^n{\theta }_i^2$ 即为加入正则化项。

岭回归

损失函数加正则化模型 $\alpha \frac{1}{2}{\sum }_{i=1}^n{\theta }_i^2$ 的方式，这种模型称为岭回归。

使用多项式回归模型

import numpy as np
import matplotlib.pyplot as plt
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler

x = np.random.uniform(-3,3,size=100)
y = 0.5*x +3 +np.random.normal(0,1,size=100)
X = x.reshape(-1,1)

plt.scatter(x,y)

x_train,x_test,y_train,y_test = train_test_split(X,y)

def plt_model(model):
    plt_x = np.linspace(-3,3,100).reshape(100,1)
    y_predict = model.predict(plt_x)
    plt.scatter(x,y)
    plt.plot(plt_x[:,0],y_predict)
    plt.show()
    
def PolynomialDegression(degree):
    return Pipeline([
    ("poly_reg",PolynomialFeatures(degree)),
    ("std",StandardScaler()),
    ("liner_reg",LinearRegression())
    ])

poly_reg = PolynomialDegression(degree=20)
poly_reg.fit(x_train,y_train)
plt_model(poly_reg)

使用岭回归模型

import numpy as np
import matplotlib.pyplot as plt
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge

x = np.random.uniform(-3,3,size=100)
y = 0.5*x +3 +np.random.normal(0,1,size=100)
X = x.reshape(-1,1)

plt.scatter(x,y)

x_train,x_test,y_train,y_test = train_test_split(X,y)

def plt_model(model):
    plt_x = np.linspace(-3,3,100).reshape(100,1)
    y_predict = model.predict(plt_x)
    plt.scatter(x,y)
    plt.plot(plt_x[:,0],y_predict)
    plt.show()
    
def RidgeRegression(degree,aplpha):
    return Pipeline([
    ("poly_reg",PolynomialFeatures(degree)),
    ("std",StandardScaler()),
    ("ridge",Ridge())
    ])

ridge = RidgeRegression(20,10000)
ridge.fit(x_train,y_train)
plt_model(ridge)

从结果可以看出，使用岭回归模型训练，超参数$alpha$取值非常重要，当$alpha$取适合的值时，能非常有效的降低方差。

LASSO回归

LASSO回归与岭回归不同点是使得下面的目标函数尽可能小
$$J(\theta )=\sum _{i=1}^m(y^{(i)}-{\theta }_0-{\theta }_1{X}_1^{(i)}-\dots \dots -{\theta }_n{X}_n^{(i)}{)}^2+\alpha \sum _{i=1}^n|\theta |$$

import numpy as np
import matplotlib.pyplot as plt
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Lasso

x = np.random.uniform(-3,3,size=100)
y = 0.5*x +3 +np.random.normal(0,1,size=100)
X = x.reshape(-1,1)

x_train,x_test,y_train,y_test = train_test_split(X,y)

def plt_model(model):
    plt_x = np.linspace(-3,3,100).reshape(100,1)
    y_predict = model.predict(plt_x)
    plt.scatter(x,y)
    plt.plot(plt_x[:,0],y_predict,color='r')
    plt.show()
    
def LassoRegression(degree,alpha):
    return Pipeline([
    ("poly_reg",PolynomialFeatures(degree)),
    ("std",StandardScaler()),
    ("lasso",Lasso(alpha=alpha))
    ])

lasso = LassoRegression(20,0.1)
lasso.fit(x_train,y_train)
plt_model(lasso)

岭回归与LASSO回归区别

• 岭回归训练得到的模型很难得到一条直线，始终是保持弯曲的形状，而LASSO回归，更趋向是一条直线
• 岭回归是使得每个$\theta$趋于0，但不等于0，所以岭回归训练得到的模型始终保持弯曲形状；LASSO趋向于使得一部分的$\theta$值为0，所以LASSO训练出来的模型趋向于一条直线