基本设置
导入包
import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model, discriminant_analysis,model_selection
使用数据集是scikit-learn自带的一个糖尿病人的数据集,该数据集的采样特点如下
- 数据集有442个样本
- 每个样本有10个特征
- 每个特征都是浮点数,数据都在 − 0.2 ∼ 0.2 -0.2\sim 0.2 −0.2∼0.2
- 样本的目标在整数 25 ∼ 346 25\sim 346 25∼346之间
数据加载函数
def load_data():
diabetes = datasets.load_diabetes()
print(diabetes.DESCR)
return model_selection.train_test_split(diabetes.data, diabetes.target, test_size=0.25,random_state=8)
**返回值:**一个元组,元组依次是:训练样本集、测试样本集、训练标签集、测试标签集。
load_data
函数加载数据集并随机切分数据集为两个部分
线性回归模型
LinearRegression
是scikit-learn
提供的线性回归模型,原型为
class LinearRegression(fit_intercept=True, normalize=False,copy_x=True, n_jobs=1)
参数
- fit_intercept:一个布尔值,表示是否需要计算 b b b值。
- normalize:一个布尔值,如果为True,那么样本会在回归之前被归一化
- copy_x:一个布尔值,如果为True,则会复制x。
- n_jobs:一个正数。任务并行是指定的CPU数量,如果为-1表示使用所有的CPU
属性
- coef_:权重向量
- intercept_:b值
方法:
-
fit(X,y[,sample_weight])
:训练模型 -
predict(X)
:用模型来预测,返回预测值 -
score(X,y[,sample_weight])
:返回预测性能得分,设预测集为 T t e s t T_{test} Ttest,真实值为 y i y_i yi,真实值的均值为 y ‾ \overline{y} y,预测值为 y ^ \hat y y^
s c o r e = 1 − ∑ T t e s t ( y i − y ^ i ) 2 ( y i − y ‾ ) 2 score = 1-\frac{\sum_{T_{test}}(y_i-\hat y_i)^2}{(y_i-\overline{y})^2} score=1−(yi−y)2∑Ttest(yi−y^i)2- score不超过1,可能为负值
- score越大,预测性能越好
def test_LinearRegression(*data):
X_train,X_test,y_train,y_test = data
regr = linear_model.LinearRegression()
regr.fit(X_train,y_train)
print("Coefficients:%s, intercept %.2f"%(regr.coef_, regr.intercept_))
print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
print("Score: %.2f" %regr.score(X_test,y_test))
调用函数
X_train,X_test,y_train,y_test = load_data()
test_LinearRegression(X_train,X_test,y_train,y_test)
输出的结果
Coefficients:[ 11.5106203 -282.51347161 534.20455671 401.73142674
-1043.89718398 634.92464089 186.43262636 204.93373199
762.47149733 91.9460394 ], intercept 152.56
Residual sum of squares: 3108.04
Score: 0.46
线性回归模型的正则化
岭回归和LASSO是目前最流行的两种线性回归正则化方法
- Ridge Regression:正则化为 α ∣ ∣ ω ∣ ∣ 2 2 , α ≥ 0 \alpha||\pmb\omega||_2^2,\alpha \ge 0 α∣∣ωωω∣∣22,α≥0
- Lasso Regression:正则化为 α ∣ ∣ ω ∣ ∣ 1 , α ≥ 0 \alpha ||\pmb \omega||_1,\alpha\ge 0 α∣∣ωωω∣∣1,α≥0
- Elastic Net:正则化为 α ρ ∣ ∣ ω ∣ ∣ 1 + α ( 1 − ρ ) 2 ∣ ∣ ω ∣ ∣ 2 2 , α ≥ 0 , 1 ≥ ρ ≥ 0 \alpha \rho||\pmb \omega||_1 + \frac{\alpha(1-\rho)}{2}||\pmb\omega||_2^2,\alpha \ge 0,1\ge \rho \ge 0 αρ∣∣ωωω∣∣1+2α(1−ρ)∣∣ωωω∣∣22,α≥0,1≥ρ≥0
其中正则项系数 α \alpha α的选择十分关键,为learning rate
岭回归(Ridge Regression)
原型
Ridge(alpha=1.0, fit_intercept=True, normalize=False,copy_X=True, max_iter=None, tol=1e-3, solver="auto",random_state=None):
参数
- alpha: α \alpha α值,越大则正则化项占的比例越大
- fit_intercept:是否计算b值
- max_iter:一个整数,指定最大迭代次数
- normlize:是否归一化
- copy_X:是否复制X
- solver:一个字符串,指定求解最优化问题的算法
- auto:自动选择算法
- svd:使用奇异值分解来计算回归系数
- cholesky:使用
scipy.linalg.solve
函数来求解 - sparse_cg:使用
scipy.sparse.linalg.cg
函数来求解 - lsqr:使用
scipy.sparse.linalg.lsqr
函数来求解,运算速度快,但是老版本不支持 - sag:使用Stochastic Average Gradient descent 算法那
- tol:指定判断迭代收敛与否的阈值
- random_state:一个整数或一个RandomState实例
- 如果为整数,则指定随机数生成器的种子
- 如果为RandomState,则使用指定的随机数生成器
- 如果为None,则使用默认的随机数生成器
属性
- coef_:权重向量
- intercept_:b值
- n_iter_:实际迭代次数
方法
同线性回归的方法
使用Ridge函数
def test_Ridge(*data):
X_train,X_test,y_train,y_test = data
regr = linear_model.Ridge()
regr.fit(X_train,y_train)
print("Coefficients:%s, intercept %.2f"%(regr.coef_, regr.intercept_))
print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
print("Score: %.2f" %regr.score(X_test,y_test))
X_train,X_test,y_train,y_test = load_data()
test_Ridge(X_train,X_test,y_train,y_test)
输出的结果如下
Coefficients:[ 36.8262072 -75.80823733 282.42652716 207.39314972 -1.46580263
-27.81750835 -134.3740951 98.97724793 222.67543268 117.97255343], intercept 152.55
Residual sum of squares: 3262.23
Score: 0.43
下面测试不同的 α \alpha α的值的预测性能的影响
def test_Ridge_alpha(*data):
X_train, X_test, y_train, y_test = data
#设置不同的alpha值
alphas = [0.01, 0.02, 0.05, 0.1, 0.2, 0.5,
1, 2, 5, 10, 20, 50, 100, 200, 500, 1000]
scores = []
for i, alpha in enumerate(alphas):
regr = linear_model.Ridge(alpha=alpha)
regr.fit(X_train, y_train)
scores.append(regr.score(X_test, y_test))
# 绘图
fig=plt.figure()
ax=fig.add_subplot(1, 1, 1)
ax.plot(alphas, scores)
ax.set_xlabel(r"$\alpha$")
ax.set_ylabel(r"score")
ax.set_xscale('log')
ax.set_title("Ridge")
plt.show()
当
α
\alpha
α超过1之后,随着
α
\alpha
α的增长,预测性能急剧下降,因为
α
\alpha
α较大时,正则化项影像比较大,模型简单。
Lasso回归
Lasso回归与岭回归的区别在于它的惩罚项是基于 L 1 L_1 L1范数。他可以将系数控制收缩到0,从而达到变量选择的效果。
函数为Lasso()
参数:
之前已经叙述过的变量不在描述
- precompute:是否提前计算Gram矩阵来加速计算
- warm_start:从头训练还是使用前一次的训练结果继续训练
- positive:如果为True,那么要求所有的权重向量的分量都是正数
- selection:一个字符串,可以为‘cyclic’或者‘random’,选择权重向量的哪个分量来更新
- random 更新的时候,随机选择权重向量的一个分量来更新
- cyclic:更新的时候,从前向后依次选择权重向量的一个分量来更新。
def test_Lasso(*data):
X_train, X_test, y_train, y_test = data
regr = linear_model.Lasso()
regr.fit(X_train, y_train)
print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
print("Score: %.2f" % regr.score(X_test, y_test))
结果
Coefficients:[ 0. -0. 384.73421807 72.69325545 0.
0. -0. 0. 247.88881314 0. ], intercept 152.69
Residual sum of squares: 3646.84
Score: 0.37
测试不同 α \alpha α的影响
def test_Lasso_alpha(*data):
X_train, X_test, y_train, y_test = data
#设置不同的alpha值
alphas = [0.01, 0.02, 0.05, 0.1, 0.2, 0.5,
1, 2, 5, 10, 20, 50, 100, 200, 500, 1000]
scores = []
for i, alpha in enumerate(alphas):
regr = linear_model.Lasso(alpha=alpha)
regr.fit(X_train, y_train)
scores.append(regr.score(X_test, y_test))
# 绘图
fig=plt.figure()
ax=fig.add_subplot(1, 1, 1)
ax.plot(alphas, scores)
ax.set_xlabel(r"$\alpha$")
ax.set_ylabel(r"score")
ax.set_xscale('log')
ax.set_title("Lasso")
plt.show()
ELasticNet回归
ElasticNet回归是对Lasso回归和岭回归的融合。其惩罚项为 L 1 L_1 L1和 L 2 L_2 L2范数的一个权衡。
def test_ElasticNet(*data):
X_train, X_test, y_train, y_test = data
regr = linear_model.ElasticNet()
regr.fit(X_train, y_train)
print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
print("Score: %.2f" % regr.score(X_test, y_test))
输出的结果为
Coefficients:[ 0.21886047 0. 3.32172605 2.40903271 0.39329589 0.0813468
-1.66664402 1.86756919 3.0113635 1.88112395], intercept 152.84
Residual sum of squares: 5704.63
Score: 0.01
对于不同的 α , ρ \alpha,\rho α,ρ
def test_ElasticNet_alpha_rho(*data):
X_train, X_test, y_train, y_test = data
alphas = np.logspace(-2,2)
rhos = np.linspace(0.01, 1)
scores = []
for alpha in alphas:
for rho in rhos:
regr = linear_model.ElasticNet(alpha=alpha, l1_ratio= rho)
regr.fit(X_train, y_train)
scores.append(regr.score(X_test,y_test))
# 绘图
alphas,rhos = np.meshgrid(alphas, rhos)
scores = np.array(scores).reshape(alphas.shape)
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
fig = plt.figure()
ax = Axes3D(fig)
surf = ax.plot_surface(alphas, rhos, scores, rstride=1, cstride=1, cmap=cm.jet, linewidth=0, antialiased = False)
fig.colorbar(surf, shrink = 0.5, aspect=5)
ax.set_xlabel(r"$\alpha$")
ax.set_ylabel(r"$\rho$")
ax.set_zlabel("score")
ax.set_title("ElasticNet")
plt.show()
所有代码
import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model, discriminant_analysis, model_selection
def load_data():
diabetes = datasets.load_diabetes()
# print(diabetes.DESCR)
return model_selection.train_test_split(diabetes.data, diabetes.target, test_size=0.25, random_state=8)
def test_LinearRegression(*data):
X_train, X_test, y_train, y_test = data
regr = linear_model.LinearRegression()
regr.fit(X_train, y_train)
print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
print("Residual sum of squares: %.2f" %
np.mean((regr.predict(X_test)-y_test)**2))
print("Score: %.2f" % regr.score(X_test, y_test))
def test_Ridge(*data):
X_train, X_test, y_train, y_test = data
regr = linear_model.Ridge()
regr.fit(X_train, y_train)
print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
print("Residual sum of squares: %.2f" %
np.mean((regr.predict(X_test)-y_test)**2))
print("Score: %.2f" % regr.score(X_test, y_test))
def test_Ridge_alpha(*data):
X_train, X_test, y_train, y_test = data
#设置不同的alpha值
alphas = [0.01, 0.02, 0.05, 0.1, 0.2, 0.5,
1, 2, 5, 10, 20, 50, 100, 200, 500, 1000]
scores = []
for i, alpha in enumerate(alphas):
regr = linear_model.Ridge(alpha=alpha)
regr.fit(X_train, y_train)
scores.append(regr.score(X_test, y_test))
# 绘图
fig=plt.figure()
ax=fig.add_subplot(1, 1, 1)
ax.plot(alphas, scores)
ax.set_xlabel(r"$\alpha$")
ax.set_ylabel(r"score")
ax.set_xscale('log')
ax.set_title("Ridge")
plt.show()
def test_Lasso(*data):
X_train, X_test, y_train, y_test = data
regr = linear_model.Lasso()
regr.fit(X_train, y_train)
print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
print("Score: %.2f" % regr.score(X_test, y_test))
def test_Lasso_alpha(*data):
X_train, X_test, y_train, y_test = data
#设置不同的alpha值
alphas = [0.01, 0.02, 0.05, 0.1, 0.2, 0.5,
1, 2, 5, 10, 20, 50, 100, 200, 500, 1000]
scores = []
for i, alpha in enumerate(alphas):
regr = linear_model.Lasso(alpha=alpha)
regr.fit(X_train, y_train)
scores.append(regr.score(X_test, y_test))
# 绘图
fig=plt.figure()
ax=fig.add_subplot(1, 1, 1)
ax.plot(alphas, scores)
ax.set_xlabel(r"$\alpha$")
ax.set_ylabel(r"score")
ax.set_xscale('log')
ax.set_title("Lasso")
plt.show()
def test_ElasticNet(*data):
X_train, X_test, y_train, y_test = data
regr = linear_model.ElasticNet()
regr.fit(X_train, y_train)
print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
print("Score: %.2f" % regr.score(X_test, y_test))
def test_ElasticNet_alpha_rho(*data):
X_train, X_test, y_train, y_test = data
alphas = np.logspace(-2,2)
rhos = np.linspace(0.01, 1)
scores = []
for alpha in alphas:
for rho in rhos:
regr = linear_model.ElasticNet(alpha=alpha, l1_ratio= rho)
regr.fit(X_train, y_train)
scores.append(regr.score(X_test,y_test))
# 绘图
alphas,rhos = np.meshgrid(alphas, rhos)
scores = np.array(scores).reshape(alphas.shape)
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
fig = plt.figure()
ax = Axes3D(fig)
surf = ax.plot_surface(alphas, rhos, scores, rstride=1, cstride=1, cmap=cm.jet, linewidth=0, antialiased = False)
fig.colorbar(surf, shrink = 0.5, aspect=5)
ax.set_xlabel(r"$\alpha$")
ax.set_ylabel(r"$\rho$")
ax.set_zlabel("score")
ax.set_title("ElasticNet")
plt.show()
if __name__ == '__main__':
X_train, X_test, y_train, y_test=load_data()
test_ElasticNet_alpha_rho(X_train,X_test,y_train,y_test)