【机器学习】决策树和sklearn运用实战

决策树

决策树既可以用于分类问题，也可以用于回归问题。
熵是表示随机变量不确定性的度量，熵越大，随机变量的不确定性就越大。熵的计算如下：$H(X)=-\sum _{i=1}^n{p}_{i}log{p}_i$
决策树有三种选择特征的方式：
信息增益：$g(D,A)=H(D)-H(D|A)$
信息增益表示选择特征$A$时，对随机变量不确定性减小的程度。信息增益偏向于选择取值较多的特征，信息增益比可以对这一问题矫正
信息增益比：$g_R(D,A)=\frac{g(D,A)}{H_A(D)}$
基尼指数：$Gini(p)=\sum _{k=1}^K{p}_k(1-p_k)=1-\sum _{k=1}^K{p}_k^2$

决策树实例

用sklearn中的数据集iris
导入需要的包

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn import tree
from sklearn.metrics import (brier_score_loss, precision_score, recall_score,f1_score)
from sklearn.model_selection import learning_curve
import numpy as np
import matplotlib.pyplot as plt

取出需要的数据，其中iris_X有4个属性，共有150个样本点，iris_y的取值有3个，分别是0,1,2，将所有的数据分为测试集和训练集，训练集有420个点

iris = load_iris()
iris_X = iris.data   #x有4个属性，共有150个样本点
iris_y = iris.target #y的取值有3个，分别是0,1,2
X_train, X_test, y_train, y_test = train_test_split(iris_X, iris_y, test_size=0.3)

选择决策树作为分类器

clf = tree.DecisionTreeClassifier()

训练模型

clf.fit(X_train, y_train)

在测试集合上做预测，输出预测的类和正确的类

y_pred = clf.predict(X_test)
print(clf.predict(X_test))
print(y_test)

输出准确率、召回率、F1测度，由于是多分类，不是二分类，因此需要加average=“micro”

print("\tPrecision: %1.3f" % precision_score(y_test, y_pred, average="micro"))
print("\tRecall: %1.3f" % recall_score(y_test, y_pred, average="micro"))
print("\tF1: %1.3f\n" % f1_score(y_test, y_pred, average="micro"))

k折交叉验证，模型可视化

X_train, X_test, y_train, y_test = train_test_split(iris_X, iris_y, test_size=0.0)
train_sizes, train_loss, test_loss = learning_curve( clf, X_train, y_train, cv=10,
    train_sizes=[0.1, 0.25, 0.5, 0.75, 1])
train_loss_mean = 1-np.mean(train_loss, axis=1)
test_loss_mean = 1-np.mean(test_loss, axis=1)
plt.plot(train_sizes, train_loss_mean, 'o-', color="r",
         label="Training")
plt.plot(train_sizes, test_loss_mean, 'o-', color="g",
        label="Cross-validation")

plt.xlabel("Training examples")
plt.ylabel("Loss")
plt.legend(loc="best")
plt.show()

决策树参数

输入以下代码，可以看到决策树可调节的参数

from sklearn import tree
clf = tree.DecisionTreeClassifier()
clf

决策树参数如下

DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,
            max_features=None, max_leaf_nodes=None,
            min_impurity_decrease=0.0, min_impurity_split=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, presort=False, random_state=None,
            splitter='best')