岭回归与LASSO回归

岭回归

根据线性回归模型的参数估计公式 可知，得到的前提是矩阵可逆，但在实际应用中，可能会出现自变量个数多于样本量或者自变量间存在多重共线性的情况，即的行列式为0，此时将无法根据公式计算回归系数的估计值。

解决该问题可用岭回归模型，岭回归模型就是在线性回归模型的目标函数之上添加L2正则项（也称为惩罚项）。L2范式惩罚项的加入，使得满秩，保证了可逆。但是由于惩罚项的加入，回归系数不再是无偏估计。所以岭回归实质上是以损失部分信息、降低精度为代价获得回归系数更为符合实际、更可靠的回归方法

import numpy as np
from sklearn import model_selection
from sklearn.linear_model import RidgeCV,Ridge
from sklearn import datasets

# 下载数据并将数据分成训练集和测试集
diabetes_X,diabetes_y = datasets.load_diabetes(return_X_y = True)
X_train,X_test,y_train,y_test = model_selection.train_test_split(diabetes_X,diabetes_y,test_size=0.2,random_state=1234)

# 构造不同的lambda值
Lambdas = np.logspace(-5,2,200)
#设置交叉验证的参数，对于每一个Lambda，都执行10重交叉验证，使用均方误差评估
ridge_cv = RidgeCV(alphas=Lambdas,normalize=True,scoring='neg_mean_squared_error',cv=10)
# 模型拟合
ridge_cv.fit(X_train,y_train)
# 返回最佳的lambda值
ridge_best_Lambda = ridge_cv.alpha_
# 基于最佳的Lambda值建模
ridge = Ridge(alpha = ridge_best_Lambda,normalize = True)
ridge.fit(X_train,y_train)
# 查看模型系数和常规项
print(ridge.intercept_,ridge.coef_.tolist())

#153.48394905886252 [-40.2209889451619, -272.34045468602784, 531.8338067664365, 336.14186195461497, -421.338662452308, 183.45943905307934, -31.34888375085302, 164.46712520385105, 564.142215674133, 123.34986255524717]

LASSO回归

岭回归模型解决线性回归模型中矩阵不可逆的办法是添加L2正则的惩罚项，但缺陷在于始终保留建模时的所有变量，无法降低模型的复杂度。对此，Lasso回归采用了L1正则的惩罚项

import numpy as np
from sklearn import model_selection
from sklearn.linear_model import LassoCV,Lasso
from sklearn import datasets

# 下载数据并将数据分成训练集和测试集
diabetes_X,diabetes_y = datasets.load_diabetes(return_X_y = True)
X_train,X_test,y_train,y_test = model_selection.train_test_split(diabetes_X,diabetes_y,test_size=0.2,random_state=1234)

# 构造不同的lambda值
Lambdas = np.logspace(-5,2,200)
#设置交叉验证的参数，对于每一个Lambda，都执行10重交叉验证，最大迭代次数10000
lasso_cv = LassoCV(alphas=Lambdas,normalize=True,cv=10,max_iter=10000)
# 模型拟合
lasso_cv.fit(X_train,y_train)
# 返回最佳的lambda值
lasso_best_alpha = lasso_cv.alpha_
# 基于最佳的Lambda值建模
lasso = Lasso(alpha = lasso_best_alpha,normalize = True,max_iter=100000)
lasso.fit(X_train,y_train)
# 查看模型系数和常规项
print(lasso.intercept_,lasso.coef_.tolist())
# 153.49343403565288 [-35.475769568542, -269.93284599620046, 535.5860339678649, 334.32842567082554, -486.43471877479504, 232.84006302504966, -0.0, 168.104072365242, 595.6288691782543, 117.8829315599235]

参考链接：

https://blog.csdn.net/weixin_43374551/article/details/83688913

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html#sklearn.linear_model.Ridge

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html#sklearn.linear_model.RidgeCV

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html#sklearn.linear_model.Lasso

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html#sklearn.linear_model.LassoCV