sklearn实现决策树

sklearn.tree.DecisionTreeClassifier

class sklearn.tree.DecisionTreeClassifier(*, criterion=‘gini’, splitter=‘best’, max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=None, random_state=None, max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, class_weight=None, presort=‘deprecated’, ccp_alpha=0.0)
scikit-learn 使用 CART 算法的优化版本

参数

criterion: {‘gini’,‘entropy’}, default=‘gini’

注意：当使用entropy时，sklearn实际计算的是基于信息熵的信息增益（information gain），即父节点的信息熵和子节点的信息熵之差。比起基尼系数，信息熵对不纯度更加敏感，对不纯度的惩罚最强。但在实际使用中，信息熵和基尼系数的效果基本相同。信息熵的计算比基尼系数缓慢一些，因为基尼系数的计算不涉及对数。另外，由于信息熵对不纯度更加敏感，所以信息熵作为指标时，决策树的生长会更加精细，因此对于高维数据或者噪音数据很多的数据，信息熵容易过拟合，基尼系数在这种情况下往往效果比较好。当模型欠拟合的时候，可考虑使用信息熵。

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

from sklearn import tree
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split #训练集和测试集的切分函数

#导入数据集并探索数据
wine = load_wine() #实例化数据集

wine

{'data': array([[1.423e+01, 1.710e+00, 2.430e+00, ..., 1.040e+00, 3.920e+00,
         1.065e+03],
        [1.320e+01, 1.780e+00, 2.140e+00, ..., 1.050e+00, 3.400e+00,
         1.050e+03],
        [1.316e+01, 2.360e+00, 2.670e+00, ..., 1.030e+00, 3.170e+00,
         1.185e+03],
        ...,
        [1.327e+01, 4.280e+00, 2.260e+00, ..., 5.900e-01, 1.560e+00,
         8.350e+02],
        [1.317e+01, 2.590e+00, 2.370e+00, ..., 6.000e-01, 1.620e+00,
         8.400e+02],
        [1.413e+01, 4.100e+00, 2.740e+00, ..., 6.100e-01, 1.600e+00,
         5.600e+02]]),
 'target': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2,
        2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
        2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
        2, 2]),
 'target_names': array(['class_0', 'class_1', 'class_2'], dtype='<U7'),
 'DESCR': '.. _wine_dataset:\n\nWine recognition dataset\n------------------------\n\n**Data Set Characteristics:**\n\n    :Number of Instances: 178 (50 in each of three classes)\n    :Number of Attributes: 13 numeric, predictive attributes and the class\n    :Attribute Information:\n \t\t- Alcohol\n \t\t- Malic acid\n \t\t- Ash\n\t\t- Alcalinity of ash  \n \t\t- Magnesium\n\t\t- Total phenols\n \t\t- Flavanoids\n \t\t- Nonflavanoid phenols\n \t\t- Proanthocyanins\n\t\t- Color intensity\n \t\t- Hue\n \t\t- OD280/OD315 of diluted wines\n \t\t- Proline\n\n    - class:\n            - class_0\n            - class_1\n            - class_2\n\t\t\n    :Summary Statistics:\n    \n    ============================= ==== ===== ======= =====\n                                   Min   Max   Mean     SD\n    ============================= ==== ===== ======= =====\n    Alcohol:                      11.0  14.8    13.0   0.8\n    Malic Acid:                   0.74  5.80    2.34  1.12\n    Ash:                          1.36  3.23    2.36  0.27\n    Alcalinity of Ash:            10.6  30.0    19.5   3.3\n    Magnesium:                    70.0 162.0    99.7  14.3\n    Total Phenols:                0.98  3.88    2.29  0.63\n    Flavanoids:                   0.34  5.08    2.03  1.00\n    Nonflavanoid Phenols:         0.13  0.66    0.36  0.12\n    Proanthocyanins:              0.41  3.58    1.59  0.57\n    Colour Intensity:              1.3  13.0     5.1   2.3\n    Hue:                          0.48  1.71    0.96  0.23\n    OD280/OD315 of diluted wines: 1.27  4.00    2.61  0.71\n    Proline:                       278  1680     746   315\n    ============================= ==== ===== ======= =====\n\n    :Missing Attribute Values: None\n    :Class Distribution: class_0 (59), class_1 (71), class_2 (48)\n    :Creator: R.A. Fisher\n    :Donor: Michael Marshall (MARSHALL%[email protected])\n    :Date: July, 1988\n\nThis is a copy of UCI ML Wine recognition datasets.\nhttps://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data\n\nThe data is the results of a chemical analysis of wines grown in the same\nregion in Italy by three different cultivators. There are thirteen different\nmeasurements taken for different constituents found in the three types of\nwine.\n\nOriginal Owners: \n\nForina, M. et al, PARVUS - \nAn Extendible Package for Data Exploration, Classification and Correlation. \nInstitute of Pharmaceutical and Food Analysis and Technologies,\nVia Brigata Salerno, 16147 Genoa, Italy.\n\nCitation:\n\nLichman, M. (2013). UCI Machine Learning Repository\n[https://archive.ics.uci.edu/ml]. Irvine, CA: University of California,\nSchool of Information and Computer Science. \n\n.. topic:: References\n\n  (1) S. Aeberhard, D. Coomans and O. de Vel, \n  Comparison of Classifiers in High Dimensional Settings, \n  Tech. Rep. no. 92-02, (1992), Dept. of Computer Science and Dept. of  \n  Mathematics and Statistics, James Cook University of North Queensland. \n  (Also submitted to Technometrics). \n\n  The data was used with many others for comparing various \n  classifiers. The classes are separable, though only RDA \n  has achieved 100% correct classification. \n  (RDA : 100%, QDA 99.4%, LDA 98.9%, 1NN 96.1% (z-transformed data)) \n  (All results using the leave-one-out technique) \n\n  (2) S. Aeberhard, D. Coomans and O. de Vel, \n  "THE CLASSIFICATION PERFORMANCE OF RDA" \n  Tech. Rep. no. 92-01, (1992), Dept. of Computer Science and Dept. of \n  Mathematics and Statistics, James Cook University of North Queensland. \n  (Also submitted to Journal of Chemometrics).\n',
 'feature_names': ['alcohol',
  'malic_acid',
  'ash',
  'alcalinity_of_ash',
  'magnesium',
  'total_phenols',
  'flavanoids',
  'nonflavanoid_phenols',
  'proanthocyanins',
  'color_intensity',
  'hue',
  'od280/od315_of_diluted_wines',
  'proline']}

wine.data.shape

(178, 13)

wine.target 

array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2])

pd.concat([pd.DataFrame(wine.data), pd.DataFrame(wine.target)],axis=1).head()

	0	1	2	3	4	5	6	7	8	9	10	11	12	0
0	14.23	1.71	2.43	15.6	127.0	2.80	3.06	0.28	2.29	5.64	1.04	3.92	1065.0	0
1	13.20	1.78	2.14	11.2	100.0	2.65	2.76	0.26	1.28	4.38	1.05	3.40	1050.0	0
2	13.16	2.36	2.67	18.6	101.0	2.80	3.24	0.30	2.81	5.68	1.03	3.17	1185.0	0
3	14.37	1.95	2.50	16.8	113.0	3.85	3.49	0.24	2.18	7.80	0.86	3.45	1480.0	0
4	13.24	2.59	2.87	21.0	118.0	2.80	2.69	0.39	1.82	4.32	1.04	2.93	735.0	0

wine.feature_names

['alcohol',
 'malic_acid',
 'ash',
 'alcalinity_of_ash',
 'magnesium',
 'total_phenols',
 'flavanoids',
 'nonflavanoid_phenols',
 'proanthocyanins',
 'color_intensity',
 'hue',
 'od280/od315_of_diluted_wines',
 'proline']

wine.target_names

array(['class_0', 'class_1', 'class_2'], dtype='<U7')

#切分训练集和测试集
xtrain, xtest, ytrain, ytest = train_test_split(wine.data, wine.target, test_size=0.3)

xtrain.shape

(124, 13)

xtest.shape

(54, 13)

#模型训练
clf = tree.DecisionTreeClassifier(criterion='entropy')
clf = clf.fit(xtrain, ytrain)
score = clf.score(xtest, ytest) #返回预测的准确度
score

0.8703703703703703

# 绘制决策树
tree.plot_tree(clf.fit(xtrain, ytrain),feature_names=wine.feature_names, class_names=wine.target_names )

[Text(167.4, 195.696, 'proline <= 953.5\nentropy = 1.571\nsamples = 124\nvalue = [40, 49, 35]\nclass = class_1'),
 Text(111.60000000000001, 152.208, 'color_intensity <= 3.825\nentropy = 1.202\nsamples = 88\nvalue = [4, 49, 35]\nclass = class_1'),
 Text(55.800000000000004, 108.72, 'entropy = 0.0\nsamples = 44\nvalue = [0, 44, 0]\nclass = class_1'),
 Text(167.4, 108.72, 'flavanoids <= 1.4\nentropy = 0.934\nsamples = 44\nvalue = [4, 5, 35]\nclass = class_2'),
 Text(111.60000000000001, 65.232, 'entropy = 0.0\nsamples = 35\nvalue = [0, 0, 35]\nclass = class_2'),
 Text(223.20000000000002, 65.232, 'proline <= 679.0\nentropy = 0.991\nsamples = 9\nvalue = [4, 5, 0]\nclass = class_1'),
 Text(167.4, 21.744, 'entropy = 0.0\nsamples = 5\nvalue = [0, 5, 0]\nclass = class_1'),
 Text(279.0, 21.744, 'entropy = 0.0\nsamples = 4\nvalue = [4, 0, 0]\nclass = class_0'),
 Text(223.20000000000002, 152.208, 'entropy = 0.0\nsamples = 36\nvalue = [36, 0, 0]\nclass = class_0')]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-msI2sR3a-1589803106915)(output_16_1.png)]

#特征重要性
clf.feature_importances_

array([0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.1650984 , 0.        , 0.        , 0.33215949,
       0.        , 0.        , 0.50274211])

random_state：int, RandomState instance, default=None

1. 随机生成器种子设置，默认设置为None，如此，则每次模型结果都会有所不同。否生成一个最优决策树的问题在于，不管你是需要一个在多方面都是最优的决策树，还是说仅仅只需要一个理论上最优的决策树，他都是一个NP完全问题。所以在实际使用中的决策树学习算法都是基于像启发式算法般这样的贪婪算法，使得在每个节点上都取其局部最优值。当然，使用这样的算法并不能保证产生的树是一个全局最优树，所以可以让一组训练器训练出多个树，其中训练器的特征和数据点都使用随机获取。然后再去评估这些树以选出"全局最优树"。
2. random_state在高纬度数据集中随机性会表现得更为明显，低纬度的数据集（如鸢尾花）中， 随机性几乎不会显现。输入任意整数，会一直长出同一棵树，让模型稳定。
3. 一般会设置0或者42

clf = tree.DecisionTreeClassifier(criterion='gini', random_state=42)
clf = clf.fit(xtrain, ytrain)
score = clf.score(xtest, ytest)
score

0.8888888888888888

splitter：{“best”, “random”}, default=”best”

用于在每个节点上选择拆分的策略。 best在特征的所有划分点中找出最优的划分点。random是随机的在部分划分点中找局部最优的划分点。默认的"best"适合样本量不大的时候，而如果样本数据量非常大，此时决策树构建推荐"random"。

clf = tree.DecisionTreeClassifier(criterion='entropy', splitter='random', random_state=42)
clf = clf.fit(xtrain, ytrain)
score = clf