【机器学习实战学习笔记】决策树

特点

–优点：计算复杂度不高，输出结果易于理解，对中间值的缺失不敏感，可以处理不相关特征数据
–缺点：可能会产生过度匹配的问题
–使用数据类型：数值型和标准型

决策树首要解决的问题就是：如何选取具有决定性作用的数据特征

算法伪代码

def createBranch():
	If so return 类标签
	Else
		寻找划分数据集的最好特征
		划分数据集
		创建分支节点
			for 每个划分的子集
				调用函数createBranch()并增加返回结果到分支节点中
		return 分支节点

一般流程

（1）收集数据：可以采用各种方法
（2）准备数据：树构造算法只适用于标称型数据，因此数值型数据必须离散化
（3）分析数据：可以使用任何方法，构造树完成之后，我们应该检查图形是否符合预期
（4）训练算法：构造树的数据结构
（5）测试算法：使用经验树计算错误率
（6）使用算法：此步骤可以适用于任何监督学习算法，而使用决策树可以更好地理解数据的内在含义

示例1

如何确定最佳的划分特征—信息增益
（1）计算香浓熵

from math import *

'''
Function:
    calculate the shannon entropy according to the dataset
Parameter:
    dataset: the training dataset
Ouput/Return:
    shannonEntropy: the shannon entropy we calculate out
'''
def calculate_Shannon_Entropy(dataset):
    n_samples = len(dataset)
    labelCounts = {}
    for feat in dataset:
        currentLabel = feat[-1] # 获取分类
        if currentLabel not in labelCounts.keys():
            labelCounts[currentLabel] = 1
        else:
            labelCounts[currentLabel] += 1
    shannonEntropy = 0.0
    for key in labelCounts.keys():
        probability = float(labelCounts[key]) / n_samples
        shannonEntropy -= probability * log(probability,2)
    return shannonEntropy

（2）选取最优的特征分割数据集

'''
Funtion:
    split the dataset according to the axis-th feature's value
Parameters:
    dataset: the dataset need to be split
    axis: the axis-th feature
    value: the axis-th feature's value
'''
def splitDataset(dataset,axis,value):
    retDataset = []
    for feat in dataset:
        if feat[axis] == value:
            reducedDataset = feat[:axis]
            reducedDataset.extend(feat[axis+1:])
            retDataset.append(reducedDataset)
    return retDataset

'''
Function:  
    choose the best feature to split the dataset according to the information gain
Parameters:
    dataset: the dataset need to be split
Output/Return:
    bestFeature: the index of the best feature to split the dataset
'''
def chooseBestFeatureToSplit(dataset):
    num_feature = len(dataset[0]) - 1
    num_sample = len(dataset)
    baseEntropy = calculate_Shannon_Entropy(dataset)
    maxInfoGain = 0.0
    bestFeature = -1
    for i in range(num_feature):
        feat_value_list = []
        for j in range(num_sample):
            feat_value_list.append(dataset[j][i])
        feat_value_unique = set(feat_value_list)
        newEntropy = 0.0
        for value in feat_value_unique:
            subDataset = splitDataset(dataset,i,value) # 根据第i个特征划分数据集
            prob = len(subDataset) / float(len(dataset))
            newEntropy += prob * calculate_Shannon_Entropy(subDataset) # 计算新的信息熵
        infoGain = baseEntropy - newEntropy # 获取信息增益
        if infoGain > maxInfoGain:
            bestFeature = i
            maxInfoGain = infoGain
    return bestFeature

（3）构建决策树

'''
Function:
    get the feature that appears most frequently
Parameter:
    classList: the class list
Output/Return:
    sortedClassCount[0][0]: the feature that appears most frequently    
'''
def majorityCnt(classList):
    classCount = {}
    for vote in classList:
        if vote not in classCount.keys():
            classCount[vote] = 0
        classCount[vote] += 1
    sortedClassCount = sorted(classCount.iteritems(),key=operator.itemgetter(1),reverse=True)
    return sortedClassCount[0][0]

'''
Function:
    create the decision tree
Parameter:
    dataset: the dataset need to be split
    labels: the labels used to split the dataset
Output/Return:
    myTree: the decision tree
'''
def createTree(dataset,labels):
    classList = []
    for sample in dataset:
        classList.append(sample[-1])
    # 第一种情况：如果当前数据集类别完全相同，停止继续划分
    if classList.count(classList[0]) == len(classList):
        return classList[0]
    # 第二种情况：如果已经划分完所有特征后，返回出现次数最多的类别
    if len(dataset[0]) == 1:
        return majorityCnt(classList)
    # get the index of the best feature to split the dataset
    best_feature = chooseBestFeatureToSplit(dataset)
    best_feature_label = labels[best_feature] # 获取对应的标签
    myTree = {best_feature_label:{}}
    del(labels[best_feature])
    # get the value of the best feature
    feat_value = []
    for sample in dataset:
        feat_value.append(sample[best_feature])
    unique_value = set(feat_value)
    # create the tree
    for value in unique_value:
        subLabels = labels[:]
        myTree[best_feature_label][value] = createTree(splitDataset(dataset,best_feature,value),subLabels)
    return myTree

构建的决策树：

完整代码：

from math import *
import operator

'''
Function:
    calculate the shannon entropy according to the dataset
Parameter:
    dataset: the training dataset
Ouput/Return:
    shannonEntropy: the shannon entropy we calculate out
'''
def calculate_Shannon_Entropy(dataset):
    n_samples = len(dataset)
    labelCounts = {}
    for feat in dataset:
        currentLabel = feat[-1] # 获取分类
        if currentLabel not in labelCounts.keys():
            labelCounts[currentLabel] = 1
        else:
            labelCounts[currentLabel] += 1
    shannonEntropy = 0.0
    for key in labelCounts.keys():
        probability = float(labelCounts[key]) / n_samples
        shannonEntropy -= probability * log(probability,2)
    return shannonEntropy

'''
Funtion:
    split the dataset according to the axis-th feature's value
Parameters:
    dataset: the dataset need to be split
    axis: the axis-th feature
    value: the axis-th feature's value
'''
def splitDataset(dataset,axis,value):
    retDataset = []
    for feat in dataset:
        if feat[axis] == value:
            reducedDataset = feat[:axis]
            reducedDataset.extend(feat[axis+1:])
            retDataset.append(reducedDataset)
    return retDataset

'''
Function:  
    choose the best feature to split the dataset according to the information gain
Parameters:
    dataset: the dataset need to be split
Output/Return:
    bestFeature: the index of the best feature to split the dataset
'''
def chooseBestFeatureToSplit(dataset):
    num_feature = len(dataset[0]) - 1
    num_sample = len(dataset)
    baseEntropy = calculate_Shannon_Entropy(dataset)
    maxInfoGain = 0.0
    bestFeature = -1
    for i in range(num_feature):
        feat_value_list = []
        for j in range(num_sample):
            feat_value_list.append(dataset[j][i])
        feat_value_unique = set(feat_value_list)
        newEntropy = 0.0
        for value in feat_value_unique:
            subDataset = splitDataset(dataset,i,value) # 根据第i个特征划分数据集
            prob = len(subDataset) / float(len(dataset))
            newEntropy += prob * calculate_Shannon_Entropy(subDataset) # 计算新的信息熵
        infoGain = baseEntropy - newEntropy # 获取信息增益
        if infoGain > maxInfoGain:
            bestFeature = i
            maxInfoGain = infoGain
    return bestFeature

'''
Function:
    get the feature that appears most frequently
Parameter:
    classList: the class list
Output/Return:
    sortedClassCount[0][0]: the feature that appears most frequently    
'''
def majorityCnt(classList):
    classCount = {}
    for vote in classList:
        if vote not in classCount.keys():
            classCount[vote] = 0
        classCount[vote] += 1
    sortedClassCount = sorted(classCount.iteritems(),key=operator.itemgetter(1),reverse=True)
    return sortedClassCount[0][0]

'''
Function:
    create the decision tree
Parameter:
    dataset: the dataset need to be split
    labels: the labels used to split the dataset
Output/Return:
    myTree: the decision tree
'''
def createTree(dataset,labels):
    classList = []
    for sample in dataset:
        classList.append(sample[-1])
    # 第一种情况：如果当前数据集类别完全相同，停止继续划分
    if classList.count(classList[0]) == len(classList):
        return classList[0]
    # 第二种情况：如果已经划分完所有特征后，返回出现次数最多的类别
    if len(dataset[0]) == 1:
        return majorityCnt(classList)
    # get the index of the best feature to split the dataset
    best_feature = chooseBestFeatureToSplit(dataset)
    best_feature_label = labels[best_feature] # 获取对应的标签
    myTree = {best_feature_label:{}}
    del(labels[best_feature])
    # get the value of the best feature
    feat_value = []
    for sample in dataset:
        feat_value.append(sample[best_feature])
    unique_value = set(feat_value)
    # create the tree
    for value in unique_value:
        subLabels = labels[:]
        myTree[best_feature_label][value] = createTree(splitDataset(dataset,best_feature,value),subLabels)
    return myTree



def createData():
    dataset = [[1,1,'yes'],
               [1,1,'yes'],
               [1,0,'no'],
               [0,1,'no'],
               [0,1,'no']]
    labels = ['no surfacing','flippers']
    return dataset,labels

if __name__ == '__main__':
    dataset,labels = createData()
    shannonEntropy = calculate_Shannon_Entropy(dataset)
    best_split_feature = chooseBestFeatureToSplit(dataset)
    myTree = createTree(dataset,labels)
    print(myTree)