机器学习实战——线性回归模型


基本设置

导入包

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model, discriminant_analysis,model_selection

使用数据集是scikit-learn自带的一个糖尿病人的数据集,该数据集的采样特点如下

  • 数据集有442个样本
  • 每个样本有10个特征
  • 每个特征都是浮点数,数据都在 − 0.2 ∼ 0.2 -0.2\sim 0.2 0.20.2
  • 样本的目标在整数 25 ∼ 346 25\sim 346 25346之间

数据加载函数

def load_data():
    diabetes = datasets.load_diabetes()
    print(diabetes.DESCR)
    return model_selection.train_test_split(diabetes.data, diabetes.target, test_size=0.25,random_state=8)

**返回值:**一个元组,元组依次是:训练样本集、测试样本集、训练标签集、测试标签集。

load_data函数加载数据集并随机切分数据集为两个部分

线性回归模型

LinearRegressionscikit-learn提供的线性回归模型,原型为

class LinearRegression(fit_intercept=True, normalize=False,copy_x=True, n_jobs=1)

参数

  • fit_intercept:一个布尔值,表示是否需要计算 b b b值。
  • normalize:一个布尔值,如果为True,那么样本会在回归之前被归一化
  • copy_x:一个布尔值,如果为True,则会复制x。
  • n_jobs:一个正数。任务并行是指定的CPU数量,如果为-1表示使用所有的CPU

属性

  • coef_:权重向量
  • intercept_:b值

方法

  • fit(X,y[,sample_weight]):训练模型

  • predict(X):用模型来预测,返回预测值

  • score(X,y[,sample_weight]):返回预测性能得分,设预测集为 T t e s t T_{test} Ttest,真实值为 y i y_i yi,真实值的均值为 y ‾ \overline{y} y,预测值为 y ^ \hat y y^
    s c o r e = 1 − ∑ T t e s t ( y i − y ^ i ) 2 ( y i − y ‾ ) 2 score = 1-\frac{\sum_{T_{test}}(y_i-\hat y_i)^2}{(y_i-\overline{y})^2} score=1(yiy)2Ttest(yiy^i)2

    • score不超过1,可能为负值
    • score越大,预测性能越好
def test_LinearRegression(*data):
    X_train,X_test,y_train,y_test = data
    regr = linear_model.LinearRegression()
    regr.fit(X_train,y_train)
    print("Coefficients:%s, intercept %.2f"%(regr.coef_, regr.intercept_))
    print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
    print("Score: %.2f" %regr.score(X_test,y_test))

调用函数

X_train,X_test,y_train,y_test = load_data()
test_LinearRegression(X_train,X_test,y_train,y_test)

输出的结果

Coefficients:[   11.5106203   -282.51347161   534.20455671   401.73142674
 -1043.89718398   634.92464089   186.43262636   204.93373199
   762.47149733    91.9460394 ], intercept 152.56
Residual sum of squares: 3108.04
Score: 0.46

线性回归模型的正则化

岭回归和LASSO是目前最流行的两种线性回归正则化方法

  • Ridge Regression:正则化为 α ∣ ∣ ω ∣ ∣ 2 2 , α ≥ 0 \alpha||\pmb\omega||_2^2,\alpha \ge 0 αωωω22,α0
  • Lasso Regression:正则化为 α ∣ ∣ ω ∣ ∣ 1 , α ≥ 0 \alpha ||\pmb \omega||_1,\alpha\ge 0 αωωω1,α0
  • Elastic Net:正则化为 α ρ ∣ ∣ ω ∣ ∣ 1 + α ( 1 − ρ ) 2 ∣ ∣ ω ∣ ∣ 2 2 , α ≥ 0 , 1 ≥ ρ ≥ 0 \alpha \rho||\pmb \omega||_1 + \frac{\alpha(1-\rho)}{2}||\pmb\omega||_2^2,\alpha \ge 0,1\ge \rho \ge 0 αρωωω1+2α(1ρ)ωωω22,α0,1ρ0

其中正则项系数 α \alpha α的选择十分关键,为learning rate

岭回归(Ridge Regression)

原型

Ridge(alpha=1.0, fit_intercept=True, normalize=False,copy_X=True, max_iter=None, tol=1e-3, solver="auto",random_state=None):
参数
  • alpha: α \alpha α值,越大则正则化项占的比例越大
  • fit_intercept:是否计算b值
  • max_iter:一个整数,指定最大迭代次数
  • normlize:是否归一化
  • copy_X:是否复制X
  • solver:一个字符串,指定求解最优化问题的算法
    • auto:自动选择算法
    • svd:使用奇异值分解来计算回归系数
    • cholesky:使用scipy.linalg.solve函数来求解
    • sparse_cg:使用scipy.sparse.linalg.cg函数来求解
    • lsqr:使用scipy.sparse.linalg.lsqr函数来求解,运算速度快,但是老版本不支持
    • sag:使用Stochastic Average Gradient descent 算法那
  • tol:指定判断迭代收敛与否的阈值
  • random_state:一个整数或一个RandomState实例
    • 如果为整数,则指定随机数生成器的种子
    • 如果为RandomState,则使用指定的随机数生成器
    • 如果为None,则使用默认的随机数生成器

属性

  • coef_:权重向量
  • intercept_:b值
  • n_iter_:实际迭代次数

方法

同线性回归的方法

使用Ridge函数

def test_Ridge(*data):
    X_train,X_test,y_train,y_test = data
    regr = linear_model.Ridge()
    regr.fit(X_train,y_train)
    print("Coefficients:%s, intercept %.2f"%(regr.coef_, regr.intercept_))
    print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
    print("Score: %.2f" %regr.score(X_test,y_test))
   
X_train,X_test,y_train,y_test = load_data()
test_Ridge(X_train,X_test,y_train,y_test)

输出的结果如下

Coefficients:[  36.8262072   -75.80823733  282.42652716  207.39314972   -1.46580263
  -27.81750835 -134.3740951    98.97724793  222.67543268  117.97255343], intercept 152.55
Residual sum of squares: 3262.23
Score: 0.43

下面测试不同的 α \alpha α的值的预测性能的影响

def test_Ridge_alpha(*data):
    X_train, X_test, y_train, y_test = data
    #设置不同的alpha值
    alphas = [0.01, 0.02, 0.05, 0.1, 0.2, 0.5,
        1, 2, 5, 10, 20, 50, 100, 200, 500, 1000]
    scores = []
    for i, alpha in enumerate(alphas):
        regr = linear_model.Ridge(alpha=alpha)
        regr.fit(X_train, y_train)
        scores.append(regr.score(X_test, y_test))
  
    # 绘图
    fig=plt.figure()
    ax=fig.add_subplot(1, 1, 1)
    ax.plot(alphas, scores)
    ax.set_xlabel(r"$\alpha$")
    ax.set_ylabel(r"score")
    ax.set_xscale('log')
    ax.set_title("Ridge")
    plt.show()

在这里插入图片描述
α \alpha α超过1之后,随着 α \alpha α的增长,预测性能急剧下降,因为 α \alpha α较大时,正则化项影像比较大,模型简单。

Lasso回归

Lasso回归与岭回归的区别在于它的惩罚项是基于 L 1 L_1 L1范数。他可以将系数控制收缩到0,从而达到变量选择的效果。

函数为Lasso()

参数:

之前已经叙述过的变量不在描述

  • precompute:是否提前计算Gram矩阵来加速计算
  • warm_start:从头训练还是使用前一次的训练结果继续训练
  • positive:如果为True,那么要求所有的权重向量的分量都是正数
  • selection:一个字符串,可以为‘cyclic’或者‘random’,选择权重向量的哪个分量来更新
    • random 更新的时候,随机选择权重向量的一个分量来更新
    • cyclic:更新的时候,从前向后依次选择权重向量的一个分量来更新。
def test_Lasso(*data):
    X_train, X_test, y_train, y_test = data
    regr = linear_model.Lasso()
    regr.fit(X_train, y_train)
    print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
    print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
    print("Score: %.2f" % regr.score(X_test, y_test))

结果

Coefficients:[  0.          -0.         384.73421807  72.69325545   0.
   0.          -0.           0.         247.88881314   0.        ], intercept 152.69
Residual sum of squares: 3646.84
Score: 0.37

测试不同 α \alpha α的影响

def test_Lasso_alpha(*data):
    X_train, X_test, y_train, y_test = data
    #设置不同的alpha值
    alphas = [0.01, 0.02, 0.05, 0.1, 0.2, 0.5,
        1, 2, 5, 10, 20, 50, 100, 200, 500, 1000]
    scores = []
    for i, alpha in enumerate(alphas):
        regr = linear_model.Lasso(alpha=alpha)
        regr.fit(X_train, y_train)
        scores.append(regr.score(X_test, y_test))
  
    # 绘图
    fig=plt.figure()
    ax=fig.add_subplot(1, 1, 1)
    ax.plot(alphas, scores)
    ax.set_xlabel(r"$\alpha$")
    ax.set_ylabel(r"score")
    ax.set_xscale('log')
    ax.set_title("Lasso")
    plt.show()

在这里插入图片描述

ELasticNet回归

ElasticNet回归是对Lasso回归和岭回归的融合。其惩罚项为 L 1 L_1 L1 L 2 L_2 L2范数的一个权衡。

def test_ElasticNet(*data):
    X_train, X_test, y_train, y_test = data
    regr = linear_model.ElasticNet()
    regr.fit(X_train, y_train)
    print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
    print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
    print("Score: %.2f" % regr.score(X_test, y_test))

输出的结果为

Coefficients:[ 0.21886047  0.          3.32172605  2.40903271  0.39329589  0.0813468
 -1.66664402  1.86756919  3.0113635   1.88112395], intercept 152.84
Residual sum of squares: 5704.63
Score: 0.01

对于不同的 α , ρ \alpha,\rho α,ρ

def test_ElasticNet_alpha_rho(*data):
    X_train, X_test, y_train, y_test = data
    alphas = np.logspace(-2,2)
    rhos = np.linspace(0.01, 1)
    scores = []
    for alpha in alphas:
        for rho in rhos:
            regr = linear_model.ElasticNet(alpha=alpha, l1_ratio= rho)
            regr.fit(X_train, y_train)
            scores.append(regr.score(X_test,y_test))

    # 绘图
    alphas,rhos = np.meshgrid(alphas, rhos)
    scores = np.array(scores).reshape(alphas.shape)
    from mpl_toolkits.mplot3d import Axes3D
    from matplotlib import cm
    fig = plt.figure()
    ax = Axes3D(fig)
    surf = ax.plot_surface(alphas, rhos, scores, rstride=1, cstride=1, cmap=cm.jet, linewidth=0, antialiased = False)
    fig.colorbar(surf, shrink = 0.5, aspect=5)
    ax.set_xlabel(r"$\alpha$")
    ax.set_ylabel(r"$\rho$")
    ax.set_zlabel("score")
    ax.set_title("ElasticNet")
    plt.show()

在这里插入图片描述

所有代码

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model, discriminant_analysis, model_selection


def load_data():
    diabetes = datasets.load_diabetes()
    # print(diabetes.DESCR)
    return model_selection.train_test_split(diabetes.data, diabetes.target, test_size=0.25, random_state=8)


def test_LinearRegression(*data):
    X_train, X_test, y_train, y_test = data
    regr = linear_model.LinearRegression()
    regr.fit(X_train, y_train)
    print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
    print("Residual sum of squares: %.2f" %
          np.mean((regr.predict(X_test)-y_test)**2))
    print("Score: %.2f" % regr.score(X_test, y_test))


def test_Ridge(*data):
    X_train, X_test, y_train, y_test = data
    regr = linear_model.Ridge()
    regr.fit(X_train, y_train)
    print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
    print("Residual sum of squares: %.2f" %
          np.mean((regr.predict(X_test)-y_test)**2))
    print("Score: %.2f" % regr.score(X_test, y_test))


def test_Ridge_alpha(*data):
    X_train, X_test, y_train, y_test = data
    #设置不同的alpha值
    alphas = [0.01, 0.02, 0.05, 0.1, 0.2, 0.5,
        1, 2, 5, 10, 20, 50, 100, 200, 500, 1000]
    scores = []
    for i, alpha in enumerate(alphas):
        regr = linear_model.Ridge(alpha=alpha)
        regr.fit(X_train, y_train)
        scores.append(regr.score(X_test, y_test))
  
    # 绘图
    fig=plt.figure()
    ax=fig.add_subplot(1, 1, 1)
    ax.plot(alphas, scores)
    ax.set_xlabel(r"$\alpha$")
    ax.set_ylabel(r"score")
    ax.set_xscale('log')
    ax.set_title("Ridge")
    plt.show()


def test_Lasso(*data):
    X_train, X_test, y_train, y_test = data
    regr = linear_model.Lasso()
    regr.fit(X_train, y_train)
    print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
    print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
    print("Score: %.2f" % regr.score(X_test, y_test))

def test_Lasso_alpha(*data):
    X_train, X_test, y_train, y_test = data
    #设置不同的alpha值
    alphas = [0.01, 0.02, 0.05, 0.1, 0.2, 0.5,
        1, 2, 5, 10, 20, 50, 100, 200, 500, 1000]
    scores = []
    for i, alpha in enumerate(alphas):
        regr = linear_model.Lasso(alpha=alpha)
        regr.fit(X_train, y_train)
        scores.append(regr.score(X_test, y_test))
  
    # 绘图
    fig=plt.figure()
    ax=fig.add_subplot(1, 1, 1)
    ax.plot(alphas, scores)
    ax.set_xlabel(r"$\alpha$")
    ax.set_ylabel(r"score")
    ax.set_xscale('log')
    ax.set_title("Lasso")
    plt.show()

def test_ElasticNet(*data):
    X_train, X_test, y_train, y_test = data
    regr = linear_model.ElasticNet()
    regr.fit(X_train, y_train)
    print("Coefficients:%s, intercept %.2f" % (regr.coef_, regr.intercept_))
    print("Residual sum of squares: %.2f" %np.mean((regr.predict(X_test)-y_test)**2))
    print("Score: %.2f" % regr.score(X_test, y_test))

def test_ElasticNet_alpha_rho(*data):
    X_train, X_test, y_train, y_test = data
    alphas = np.logspace(-2,2)
    rhos = np.linspace(0.01, 1)
    scores = []
    for alpha in alphas:
        for rho in rhos:
            regr = linear_model.ElasticNet(alpha=alpha, l1_ratio= rho)
            regr.fit(X_train, y_train)
            scores.append(regr.score(X_test,y_test))

    # 绘图
    alphas,rhos = np.meshgrid(alphas, rhos)
    scores = np.array(scores).reshape(alphas.shape)
    from mpl_toolkits.mplot3d import Axes3D
    from matplotlib import cm
    fig = plt.figure()
    ax = Axes3D(fig)
    surf = ax.plot_surface(alphas, rhos, scores, rstride=1, cstride=1, cmap=cm.jet, linewidth=0, antialiased = False)
    fig.colorbar(surf, shrink = 0.5, aspect=5)
    ax.set_xlabel(r"$\alpha$")
    ax.set_ylabel(r"$\rho$")
    ax.set_zlabel("score")
    ax.set_title("ElasticNet")
    plt.show()

if __name__ == '__main__':
    X_train, X_test, y_train, y_test=load_data()
    test_ElasticNet_alpha_rho(X_train,X_test,y_train,y_test)
   

  • 2
    点赞
  • 12
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值