7.5 sklearn中的逻辑回归中及正则化的学习笔记

在逻辑回归中添加多项式项，从而得到不规则的决策边界，进而对非线性的数据进行很好的分类。但是众所周知，添加多项式项之后，模型会变变得很复杂，非常容易出现过拟合。因此就需要使用正则化，且sklearn中的逻辑回归，都是使用的正则化。

1.逻辑回归中使用正则化

对损失函数增加L1正则或L2正则。可以引入一个新的参数$\alpha$来调节损失函数和正则项的权重，如：$ J(\theta)+\alpha L_{1} $。（对于L1、L2正则项的内容，不是本篇介绍的重点）

如果在损失函数前引入一个超参数$C$，即：$ C \cdot J(\theta)+ L_{1} $，如果C越大，优化损失函数时越应该集中火力，将损失函数减小到最小；C非常小时，此时L1和L2的正则项就显得更加重要。其实损失函数前的参数，作用相当于参数$\alpha$前的一个倒数。在逻辑回归中，对模型正则化更喜欢使用$ C \cdot J(\theta)+ L_{1} $ 这种方式。

2.sklearn中的逻辑回归

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(666)
# 构建服从标准差为0，方差为1的分布，200个样本，有两个特征
X = np.random.normal(0, 1, size=(200, 2))
# 构建输入空间X与标签y的关系：是一个抛物线，通过布尔向量转为int类型
y = np.array((X[:,0]**2+X[:,1])<1.5, dtype='int')
# 随机在样本中挑20个点，强制分类为1（相当于噪音）
for _ in range(20):
    y[np.random.randint(200)] = 1
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])

<matplotlib.collections.PathCollection at 0x1ea2076d710>

使用sklearn中的逻辑回归

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=666)

log_reg = LogisticRegression()
log_reg.fit(X_train, y_train)

C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)





LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=100,
                   multi_class='warn', n_jobs=None, penalty='l2',
                   random_state=None, solver='warn', tol=0.0001, verbose=0,
                   warm_start=False)

注意观察，有一个参数penalty的默认参数是l2，这说明sklearn中默认是使用L2正则项的，且超参数C默认1。

log_reg.score(X_train, y_train)

0.7933333333333333

log_reg.score(X_test, y_test)

0.86

我们发现准确不高，这很正常！因为设置的就是非线性的数据，而现在用的还是没加多项式的逻辑回归。

下面可以可视化一下决策边界：

def plot_decision_boundary(model, axis):   
    x0, x1 = np.meshgrid(
        np.linspace(axis[0], axis[1], int((axis[1]-axis[0])*100)).reshape(-1, 1),
        np.linspace(axis[2], axis[3], int((axis[3]-axis[2])*100)).reshape(-1, 1),
    )
    X_new = np.c_[x0.ravel(), x1.ravel()]
    y_predict = model.predict(X_new)
    zz = y_predict.reshape(x0.shape)
    from matplotlib.colors import ListedColormap
    custom_cmap = ListedColormap(['#EF9A9A','#FFF59D','#90CAF9'])
    plt.contourf(x0, x1, zz, linewidth=5, cmap=custom_cmap)

plot_decision_boundary(log_reg, axis=[-4, 4, -4, 4])
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])
plt.show()

C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:11: UserWarning: The following kwargs were not used by contour: 'linewidth'
  # This is added back by InteractiveShellApp.init_path()

3.多项式项逻辑回归

3.1 实现多项式项逻辑回归

尝试使用多项式项进行逻辑回归。使用pipeline方式组合三个步骤，得到一个使用多项式项的逻辑回归的方法

from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

def PolynomialLogisticRegression(degree):
    return Pipeline([
        ('poly', PolynomialFeatures(degree=degree)),
        ('std_scaler', StandardScaler()),
        ('log_reg', LogisticRegression())
    ])

poly_log_reg = PolynomialLogisticRegression(degree=2)
poly_log_reg.fit(X_train, y_train)

C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)





Pipeline(memory=None,
         steps=[('poly',
                 PolynomialFeatures(degree=2, include_bias=True,
                                    interaction_only=False, order='C')),
                ('std_scaler',
                 StandardScaler(copy=True, with_mean=True, with_std=True)),
                ('log_reg',
                 LogisticRegression(C=1.0, class_weight=None, dual=False,
                                    fit_intercept=True, intercept_scaling=1,
                                    l1_ratio=None, max_iter=100,
                                    multi_class='warn', n_jobs=None,
                                    penalty='l2', random_state=None,
                                    solver='warn', tol=0.0001, verbose=0,
                                    warm_start=False))],
         verbose=False)

看一下训练得到的结果：

poly_log_reg.score(X_train, y_train)

0.9133333333333333

poly_log_reg.score(X_test, y_test)

0.94

plot_decision_boundary(poly_log_reg, axis=[-4, 4, -4, 4])
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])
plt.show()

C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:11: UserWarning: The following kwargs were not used by contour: 'linewidth'
  # This is added back by InteractiveShellApp.init_path()

下面设置一个比较大的阶数（目的就是看看过拟合的样子）

poly_log_reg2 = PolynomialLogisticRegression(degree=20)
poly_log_reg2.fit(X_train, y_train)
poly_log_reg2.score(X_train, y_train)

C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)





0.94

poly_log_reg2.score(X_test, y_test)

0.92

plot_decision_boundary(poly_log_reg2, axis=[-4, 4, -4, 4])
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])
plt.show()

C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:11: UserWarning: The following kwargs were not used by contour: 'linewidth'
  # This is added back by InteractiveShellApp.init_path()

其实单看结果，还看不出来太多，因为我们的模型设置的很简单。但是看决策边界的话，就知道，这个奇奇怪怪的曲线，一瞅就是过拟合了。

下面就在这个过拟合的基础上，加入模型的正则化。

3.2 模型的正则化

3.2.1 L2正则

使用参数$C$进行模型正则化，在构建管道时，用参数C去覆盖。同时在生成多项式逻辑回归实例参数时，同样设置阶数为20，然后设置一个比较小的损失函数的权重参数$C=0.1$，相当于让模型正则化的项起到更大的作用，让分类准确度损失函数起到小一点的作用。

def PolynomialLogisticRegression(degree, C):
    return Pipeline([
        ('poly', PolynomialFeatures(degree=degree)),
        ('std_scaler', StandardScaler()),
        ('log_reg', LogisticRegression(C=C))
    ])

poly_log_reg3 = PolynomialLogisticRegression(degree=20, C=0.1)
poly_log_reg3.fit(X_train, y_train)

C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)





Pipeline(memory=None,
         steps=[('poly',
                 PolynomialFeatures(degree=20, include_bias=True,
                                    interaction_only=False, order='C')),
                ('std_scaler',
                 StandardScaler(copy=True, with_mean=True, with_std=True)),
                ('log_reg',
                 LogisticRegression(C=0.1, class_weight=None, dual=False,
                                    fit_intercept=True, intercept_scaling=1,
                                    l1_ratio=None, max_iter=100,
                                    multi_class='warn', n_jobs=None,
                                    penalty='l2', random_state=None,
                                    solver='warn', tol=0.0001, verbose=0,
                                    warm_start=False))],
         verbose=False)

在训练数据集及测试数据集上的表现如下：

poly_log_reg3.score(X_train, y_train)

0.8533333333333334

poly_log_reg3.score(X_test, y_test)

0.92

plot_decision_boundary(poly_log_reg3, axis=[-4, 4, -4, 4])
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])
plt.show()

C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:11: UserWarning: The following kwargs were not used by contour: 'linewidth'
  # This is added back by InteractiveShellApp.init_path()

与没正则的情况下相比较，貌似是好了一点。

3.2.2 L1正则

def PolynomialLogisticRegression(degree, C, penalty):
    return Pipeline([
        ('poly', PolynomialFeatures(degree=degree)),
        ('std_scaler', StandardScaler()),
        ('log_reg', LogisticRegression(C=C, penalty=penalty))
    ])

poly_log_reg4 = PolynomialLogisticRegression(degree=20, C=0.1, penalty='l1')
poly_log_reg4.fit(X_train, y_train)
poly_log_reg4.score(X_train, y_train)

C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)





0.8266666666666667

poly_log_reg4.score(X_test, y_test)

0.9

plot_decision_boundary(poly_log_reg4, axis=[-4, 4, -4, 4])
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])
plt.show()

C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:11: UserWarning: The following kwargs were not used by contour: 'linewidth'
  # This is added back by InteractiveShellApp.init_path()

虽然分类准确率比较低，但是没有过拟合，分类决策边界非常接近原本的真实数据了。

总结

在这一篇文章中，就介绍了sklearn中如何使用逻辑回归，并对不同的正则化项得到的效果进行了展示。在实际使用中，阶数degree，参数C以及正则化项，都是超参数，使用网格搜索的方式得到最佳的组合。