scikit-learn linearRegression 1.1.10 逻辑回归

最新推荐文章于 2024-09-14 13:58:53 发布

瑟瑟发抖的菜鸡望

最新推荐文章于 2024-09-14 13:58:53 发布

阅读量1.9k

点赞数 1

分类专栏：机器学习文章标签：博客

机器学习专栏收录该内容

37 篇文章 2 订阅

订阅专栏

逻辑回归形如其名，是一个线性分类模型而不是回归模型。逻辑回归在文献中也称为logit回归、最大熵分类(MaxEnt) 或者 log-linear classifier。在这个模型中，描述单次可能结果输出概率使用 logistic function 来建模。

scikit-learn中逻辑回归的实现为 LogisticRegression 类。它可以拟合含L2或者L1正则化项的多类逻辑回归问题。

作为一个优化问题，二分类L2 通过下方的代价函数来惩罚逻辑回归:

$\underset{w, c}{min\,} \frac{1}{2}w^T w + C \sum_{i=1}^n \log(\exp(- y_i (X_i^T w + c)) + 1) .$

类似的，L1 正则化逻辑回归解决下述的优化问题:

$\underset{w, c}{min\,} \|w\|_1 + C \sum_{i=1}^n \log(\exp(- y_i (X_i^T w + c)) + 1) .$

LogisticRegression 中的实现是solver “liblinear” (一个扩展的C++ library,LIBLINEAR), “newton-cg”, “lbfgs” and “sag”。

“lbfgs” 和 “newton-cg” 只支持L2罚项，并且对于一些高维数据收敛非常快。L1罚项产生稀疏预测的权重。

“liblinear” 使用了基于Liblinear的坐标下降法(CD)。对于F1罚项， sklearn.svm.l1_min_c 允许计算C的下界以获得一个非”null” 的模型（所有特征权重为0）。这依赖于非常棒的一个库 LIBLINEAR library ,用在scikit-learn中。然而，CD算法在liblinear中的实现无法学习一个真正的多维（多类）的模型。反而，最优问题被分解为 “one-vs-rest” 多个二分类问题来解决多分类。由于底层是这样实现的，所以使用了该库的 LogisticRegression 类就可以作为多类分类器了。

LogisticRegression 使用 “lbfgs” 或者 “newton-cg” 程序来设置 multi_class 为 “multinomial”，则该类学习了一个真正的多类逻辑回归模型，也就是说这种概率估计应该比默认 “one-vs-rest” 设置要更加准确。但是 “lbfgs”, “newton-cg” 和 “sag” 程序无法优化含L1罚项的模型，所以”multinomial” 的设置无法学习稀疏模型。

“sag” 程序使用了随机平均梯度下降（ Stochastic Average Gradient descent [3]）。它无法解决多分类问题，而且对于含L2罚项的模型有局限性。然而在超大数据集下计算要比其他程序快很多，当样本数量和特征数量都非常大的时候。

简单概括下，可以按照以下规则来选择solver:

Case	Solver
Small dataset or L1 penalty	“liblinear”
Multinomial loss	“lbfgs” or newton-cg”
Large dataset	“sag”

对于超大数据集，你同样可以考虑使用带log损失的 SGDClassifier

Examples:

Differences from liblinear:

There might be a difference in the scores obtained between LogisticRegression with solver=liblinear or LinearSVC and the external liblinear library directly, when fit_intercept=False and the fit coef_ (or) the data to be predicted are zeroes. This is because for the sample(s) with decision_functionzero, LogisticRegression and LinearSVC predict the negative class, while liblinear predicts the positive class. Note that a model with fit_intercept=False and having many samples with decision_function zero, is likely to be a underfit, bad model and you are advised to set fit_intercept=True and increase the intercept_scaling.

Note

Feature selection with sparse logistic regression

A logistic regression with L1 penalty yields sparse models, and can thus be used to perform feature selection, as detailed in 基于L1的特征选择(L1-based feature selection).

LogisticRegressionCV 实现了一个内建的交叉验证来寻找最优的参数C的逻辑回归模型。”newton-cg”,”sag” 和　”lbfgs” 程序在高维稠密数据上计算更快,原因在于warm-starting.对于多类问题,如果 multi_class 选项设置为 “ovr” ,那么最优的C从每个类别中获得，如果 multi_class 选项设置为　”multinomial” ,那么最优的Ｃ通过最小化交叉熵损失得到。

Examples:

logistic回归中的L1罚和稀疏性

对不同的C值使用L1和L2惩罚时的稀疏性（零系数百分比）的比较，我们可以看出C的大值给模型带来了更多的自由度。相反，较小的C值限制了模型的更多。在L1处罚的情况下，这会导致稀疏的解决方案。
我们把数字8x8图像可分为两类：0-4对5-9。可视化显示了不同C模型的系数。

../../_images/plot_logistic_l1_l2_sparsity_001.png

Script output:

 
    C=100.00
Sparsity with L1 penalty: 6.25%
score with L1 penalty: 0.9104
Sparsity with L2 penalty: 4.69%
score with L2 penalty: 0.9098
C=1.00
Sparsity with L1 penalty: 10.94%
score with L1 penalty: 0.9098
Sparsity with L2 penalty: 4.69%
score with L2 penalty: 0.9093
C=0.01
Sparsity with L1 penalty: 85.94%
score with L1 penalty: 0.8614
Sparsity with L2 penalty: 4.69%
score with L2 penalty: 0.8915
 
   

Python source code: plot_logistic_l1_l2_sparsity.py

 
    print(__doc__)

# Authors: Alexandre Gramfort <alexandre.gramfort@inria.fr>
#          Mathieu Blondel <mathieu@mblondel.org>
#          Andreas Mueller <amueller@ais.uni-bonn.de>
# License: BSD 3 clause

import numpy as np
import matplotlib.pyplot as plt

from sklearn.linear_model import LogisticRegression
from sklearn import datasets
from sklearn.preprocessing import StandardScaler

digits = datasets.load_digits()

X, y = digits.data, digits.target
X = StandardScaler().fit_transform(X)

# classify small against large digits
y = (y > 4).astype(np.int)


# Set regularization parameter
for i, C in enumerate((100, 1, 0.01)):
    # turn down tolerance for short training time
    clf_l1_LR = LogisticRegression(C=C, penalty='l1', tol=0.01)
    clf_l2_LR = LogisticRegression(C=C, penalty='l2', tol=0.01)
    clf_l1_LR.fit(X, y)
    clf_l2_LR.fit(X, y)

    coef_l1_LR = clf_l1_LR.coef_.ravel()
    coef_l2_LR = clf_l2_LR.coef_.ravel()

    # coef_l1_LR contains zeros due to the
    # L1 sparsity inducing norm

    sparsity_l1_LR = np.mean(coef_l1_LR == 0) * 100
    sparsity_l2_LR = np.mean(coef_l2_LR == 0) * 100

    print("C=%.2f" % C)
    print("Sparsity with L1 penalty: %.2f%%" % sparsity_l1_LR)
    print("score with L1 penalty: %.4f" % clf_l1_LR.score(X, y))
    print("Sparsity with L2 penalty: %.2f%%" % sparsity_l2_LR)
    print("score with L2 penalty: %.4f" % clf_l2_LR.score(X, y))

    l1_plot = plt.subplot(3, 2, 2 * i + 1)
    l2_plot = plt.subplot(3, 2, 2 * (i + 1))
    if i == 0:
        l1_plot.set_title("L1 penalty")
        l2_plot.set_title("L2 penalty")

    l1_plot.imshow(np.abs(coef_l1_LR.reshape(8, 8)), interpolation='nearest',
                   cmap='binary', vmax=1, vmin=0)
    l2_plot.imshow(np.abs(coef_l2_LR.reshape(8, 8)), interpolation='nearest',
                   cmap='binary', vmax=1, vmin=0)
    plt.text(-8, 3, "C = %.2f" % C)

    l1_plot.set_xticks(())
    l1_plot.set_yticks(())
    l2_plot.set_xticks(())
    l2_plot.set_yticks(())

plt.show() 
   

L1 logistic回归路径

Computes path on IRIS dataset.

../../_images/plot_logistic_path_001.png

Script output:

 
    Computing regularization path ...
This took  0:00:00.147946

Python source code: plot_logistic_path.py

 
    print(__doc__)

# Author: Alexandre Gramfort <alexandre.gramfort@inria.fr>
# License: BSD 3 clause

from datetime import datetime
import numpy as np
import matplotlib.pyplot as plt

from sklearn import linear_model
from sklearn import datasets
from sklearn.svm import l1_min_c

iris = datasets.load_iris()
X = iris.data
y = iris.target

X = X[y != 2]
y = y[y != 2]

X -= np.mean(X, 0)

###############################################################################
# Demo path functions

cs = l1_min_c(X, y, loss='log') * np.logspace(0, 3)


print("Computing regularization path ...")
start = datetime.now()
clf = linear_model.LogisticRegression(C=1.0, penalty='l1', tol=1e-6)
coefs_ = []
for c in cs:
    clf.set_params(C=c)
    clf.fit(X, y)
    coefs_.append(clf.coef_.ravel().copy())
print("This took ", datetime.now() - start)

coefs_ = np.array(coefs_)
plt.plot(np.log10(cs), coefs_)
ymin, ymax = plt.ylim()
plt.xlabel('log(C)')
plt.ylabel('Coefficients')
plt.title('Logistic Regression Path')
plt.axis('tight')
plt.show() 
   

瑟瑟发抖的菜鸡望

关注

1
点赞
踩
2

收藏

觉得还不错? 一键收藏
0
评论
复制链接

分享到 QQ

分享到新浪微博

扫一扫

专栏目录