决策树是一个递归的过程,每层使用不同判断标准。
- 熵
- 信息增益
- 信息增益率
- GINI系数
- 剪枝策略
- 举例
1.熵
熵表示随机变量不确定性的度量(即内部混乱程度,分布越混乱熵值越大。)
E
(
x
)
=
−
∑
(
p
)
∗
l
o
g
(
p
)
i
=
1
,
2
,
3......
E(x) = -\sum (p)*log(p) i = 1,2,3......
E(x)=−∑(p)∗log(p)i=1,2,3......
例如A [1 1 1 1 2 2 2 2] B[1 2 3 4 5 6 7 8] C[1 1 1 1 1 1 1 1]
A的集熵值较低,B熵值更大 C最小因为C的熵值为0(元素概率为1)
2.信息增益 ID3
熵值增加或减少的值:
3.信息增益率 ID3
考虑自身的熵值。
G
a
i
n
(
D
,
a
)
′
=
G
a
i
n
(
D
,
a
)
/
I
V
Gain(D, a)' = Gain(D, a) / IV
Gain(D,a)′=Gain(D,a)/IV
IV即自身的熵值。
4.GINI系数
G
a
i
n
(
D
)
=
1
−
∑
k
=
1
K
p
2
Gain(D) = 1 - \sum_{k = 1}^K p^2
Gain(D)=1−k=1∑Kp2
和熵的衡量标准基本一致,只是计算方式不同
5.剪枝策略(控制节点个数等等)
预剪枝策略:边建立决策树边剪枝
后剪枝:建立完成之后再剪枝
6.例子
决策树之前,根节点包含所有样本。根节点的信息熵为:
E
n
t
(
D
)
=
−
(
(
8
/
17
)
∗
l
o
g
2
8
/
17
+
(
9
/
17
)
∗
l
o
g
2
9
/
17
)
=
0.998
Ent(D) = -((8/17) * log~2^{8/17} + (9/17) * log~2^{9/17})=0.998
Ent(D)=−((8/17)∗log 28/17+(9/17)∗log 29/17)=0.998
如果按照色泽划分:
D1:青绿
D2:乌黑
D3:浅白
同样算出三个熵值,
E
(
D
1
)
=
−
(
(
3
/
6
)
∗
l
o
g
2
3
/
6
+
(
3
/
6
)
∗
l
o
g
2
3
/
6
)
=
1
E(D1) = -((3/6) * log~2^{3/6} + (3/6) * log~2^{3/6})=1
E(D1)=−((3/6)∗log 23/6+(3/6)∗log 23/6)=1
…
信息增益
G
a
i
n
(
D
,
色
泽
)
=
E
n
t
(
D
)
−
(
(
6
/
17
)
∗
1
+
(
6
/
17
)
∗
0.918
+
(
5
/
17
)
∗
0.722
=
0.109
Gain(D, 色泽) = Ent(D) - ((6/17) * 1 + (6/17)*0.918 + (5/17)*0.722 = 0.109
Gain(D,色泽)=Ent(D)−((6/17)∗1+(6/17)∗0.918+(5/17)∗0.722=0.109
同理可以计算出按照其他属性的值:
例如*Gain(D, 根蒂) = 0.143 Gain(D, 纹理) = 0.143 *…
得到纹理增益最大。对纹理进一步子节点进一步划分。
信息增益率 例如通过上述步骤得到触感属性进行划分信息增益0.006
信息增益率 = 0.006/0.874
0.874即公式中的IV
I
V
=
−
(
(
12
/
17
)
∗
l
o
g
2
12
/
17
+
(
5
/
17
)
∗
l
o
g
2
5
/
17
)
IV = -((12/17) * log~2^{12/17} + (5/17) * log~2^{5/17})
IV=−((12/17)∗log 212/17+(5/17)∗log 25/17)
使用skLearn实现基本决策树
import matplotlib.pyplot as plt
import pandas as pd
%matplotlib inline
#在线数据 联网下载
from sklearn.datasets.california_housing import fetch_california_housing
housing = fetch_california_housing()
from sklearn.model_selection import train_test_split
data_train, data_test, target_train, target_test = train_test_split(housing.data, housing.target, test_size = 0.1, random_state = 42)
decisionTree = tree.DecisionTreeRegressor(random_state = 42)
#housing.data[:, [0, 1]] 输入样本x housing.target样本标签
decisionTree.fit(data_train, target_train)
decisionTree.score(data_test, target_test)
如果想更换不同的参数得到最高的效果,可以使用GridSearchCV
from sklearn.grid_search import GridSearchCV
# criterion='mse', max_depth=None, max_features=None,
# max_leaf_nodes=None, min_impurity_decrease=0.0,
# min_impurity_split=None, min_samples_leaf=1,
# min_samples_split=3, min_weight_fraction_leaf=0.0,
# presort=False, random_state=None, splitter='best'
tree_param_grid = { 'min_samples_split': list((3,6,9)),'max_depth':list((2,4,6,8,16, 32))}
grid = GridSearchCV(tree.DecisionTreeRegressor(),param_grid=tree_param_grid, cv=5)
grid.fit(data_train, target_train)
grid.grid_scores_, grid.best_params_, grid.best_score_
输出得到最好结果的组合:
([mean: 0.44506, std: 0.00625, params: {'max_depth': 2, 'min_samples_split': 3},
mean: 0.44506, std: 0.00625, params: {'max_depth': 2, 'min_samples_split': 6},
mean: 0.44506, std: 0.00625, params: {'max_depth': 2, 'min_samples_split': 9},
mean: 0.57607, std: 0.00765, params: {'max_depth': 4, 'min_samples_split': 3},
mean: 0.57607, std: 0.00765, params: {'max_depth': 4, 'min_samples_split': 6},
mean: 0.57607, std: 0.00765, params: {'max_depth': 4, 'min_samples_split': 9},
mean: 0.64758, std: 0.00481, params: {'max_depth': 6, 'min_samples_split': 3},
mean: 0.64659, std: 0.00582, params: {'max_depth': 6, 'min_samples_split': 6},
mean: 0.64700, std: 0.00511, params: {'max_depth': 6, 'min_samples_split': 9},
mean: 0.68846, std: 0.00426, params: {'max_depth': 8, 'min_samples_split': 3},
mean: 0.68738, std: 0.00471, params: {'max_depth': 8, 'min_samples_split': 6},
mean: 0.68722, std: 0.00515, params: {'max_depth': 8, 'min_samples_split': 9},
mean: 0.62133, std: 0.00743, params: {'max_depth': 16, 'min_samples_split': 3},
mean: 0.63677, std: 0.00863, params: {'max_depth': 16, 'min_samples_split': 6},
mean: 0.65077, std: 0.00892, params: {'max_depth': 16, 'min_samples_split': 9},
mean: 0.60382, std: 0.00944, params: {'max_depth': 32, 'min_samples_split': 3},
mean: 0.62400, std: 0.01184, params: {'max_depth': 32, 'min_samples_split': 6},
mean: 0.64117, std: 0.00866, params: {'max_depth': 32, 'min_samples_split': 9}],
{'max_depth': 8, 'min_samples_split': 3},
0.6884636618151855)
完整代码:完整代码