决策树
决策树既可以用于分类问题,也可以用于回归问题。
熵是表示随机变量不确定性的度量,熵越大,随机变量的不确定性就越大。熵的计算如下:
H
(
X
)
=
−
∑
i
=
1
n
p
i
l
o
g
p
i
H(X)=-\sum\limits_{i=1}^np_ilogp_i
H(X)=−i=1∑npilogpi
决策树有三种选择特征的方式:
信息增益:
g
(
D
,
A
)
=
H
(
D
)
−
H
(
D
∣
A
)
g(D,A)=H(D)-H(D|A)
g(D,A)=H(D)−H(D∣A)
信息增益表示选择特征
A
A
A时,对随机变量不确定性减小的程度。信息增益偏向于选择取值较多的特征,信息增益比可以对这一问题矫正
信息增益比:
g
R
(
D
,
A
)
=
g
(
D
,
A
)
H
A
(
D
)
g_R(D,A)=\frac{g(D,A)}{H_A(D)}
gR(D,A)=HA(D)g(D,A)
基尼指数:
G
i
n
i
(
p
)
=
∑
k
=
1
K
p
k
(
1
−
p
k
)
=
1
−
∑
k
=
1
K
p
k
2
Gini(p)=\sum\limits_{k=1}^{K}p_k(1-p_k)=1-\sum\limits_{k=1}^{K}p_k^2
Gini(p)=k=1∑Kpk(1−pk)=1−k=1∑Kpk2
决策树实例
用sklearn中的数据集iris
导入需要的包
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn import tree
from sklearn.metrics import (brier_score_loss, precision_score, recall_score,f1_score)
from sklearn.model_selection import learning_curve
import numpy as np
import matplotlib.pyplot as plt
取出需要的数据,其中iris_X有4个属性,共有150个样本点,iris_y的取值有3个,分别是0,1,2,将所有的数据分为测试集和训练集,训练集有420个点
iris = load_iris()
iris_X = iris.data #x有4个属性,共有150个样本点
iris_y = iris.target #y的取值有3个,分别是0,1,2
X_train, X_test, y_train, y_test = train_test_split(iris_X, iris_y, test_size=0.3)
选择决策树作为分类器
clf = tree.DecisionTreeClassifier()
训练模型
clf.fit(X_train, y_train)
在测试集合上做预测,输出预测的类和正确的类
y_pred = clf.predict(X_test)
print(clf.predict(X_test))
print(y_test)
输出准确率、召回率、F1测度,由于是多分类,不是二分类,因此需要加average=“micro”
print("\tPrecision: %1.3f" % precision_score(y_test, y_pred, average="micro"))
print("\tRecall: %1.3f" % recall_score(y_test, y_pred, average="micro"))
print("\tF1: %1.3f\n" % f1_score(y_test, y_pred, average="micro"))
k折交叉验证,模型可视化
X_train, X_test, y_train, y_test = train_test_split(iris_X, iris_y, test_size=0.0)
train_sizes, train_loss, test_loss = learning_curve( clf, X_train, y_train, cv=10,
train_sizes=[0.1, 0.25, 0.5, 0.75, 1])
train_loss_mean = 1-np.mean(train_loss, axis=1)
test_loss_mean = 1-np.mean(test_loss, axis=1)
plt.plot(train_sizes, train_loss_mean, 'o-', color="r",
label="Training")
plt.plot(train_sizes, test_loss_mean, 'o-', color="g",
label="Cross-validation")
plt.xlabel("Training examples")
plt.ylabel("Loss")
plt.legend(loc="best")
plt.show()
决策树参数
输入以下代码,可以看到决策树可调节的参数
from sklearn import tree
clf = tree.DecisionTreeClassifier()
clf
决策树参数如下
DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,
max_features=None, max_leaf_nodes=None,
min_impurity_decrease=0.0, min_impurity_split=None,
min_samples_leaf=1, min_samples_split=2,
min_weight_fraction_leaf=0.0, presort=False, random_state=None,
splitter='best')