决策树实现

1. Python机器学习的库：scikit-learn

1.1： 特性：

• 简单高效的数据挖掘和机器学习分析

• 对所有用户开放，根据不同需求高度可重用性

• 基于Numpy, SciPy和matplotlib

• 开源，商用级别：获得 BSD许可

1.2 覆盖问题领域：

• 分类（classification)
• 回归（regression)
• 聚类（clustering)
• 降维(dimensionality reduction)
• 模型选择(model selection)
• 预处理(preprocessing)

1.3 使用用scikit-learn

安装scikit-learn: pip, easy_install, windows installer

安装必要的package：numpy， SciPy和matplotlib， 可使用Anaconda (包含numpy, scipy等科学计算常用 package）

2.例子

例子

2.1 使用sklearn之前要对数据进行预处理：

2.1.1表头：

headers = ['RID', 'age', 'income', 'student', 'credit_rating', 'class_buys_computer']

2.1.2标签类（表格最后一列class_buys_computer的内容）：

labelList = ['no', 'no', 'yes', 'yes', 'yes', 'no', 'yes', 'no', 'yes', 'yes', 'yes', 'yes', 'yes', 'no']

2.1.3特征值：

featureList = [{'age': 'youth', 'income': 'high', 'student': 'no', 'credit_rating': 'fair'}, {'age': 'youth', 'income': 'high', 'student': 'no', 'credit_rating': 'excellent'}, {'age': 'middle_aged', 'income': 'high', 'student': 'no', 'credit_rating': 'fair'}, {'age': 'senior', 'income': 'medium', 'student': 'no', 'credit_rating': 'fair'}, {'age': 'senior', 'income': 'low', 'student': 'yes', 'credit_rating': 'fair'}, {'age': 'senior', 'income': 'low', 'student': 'yes', 'credit_rating': 'excellent'}, {'age': 'middle_aged', 'income': 'low', 'student': 'yes', 'credit_rating': 'excellent'}, {'age': 'youth', 'income': 'medium', 'student': 'no', 'credit_rating': 'fair'}, {'age': 'youth', 'income': 'low', 'student': 'yes', 'credit_rating': 'fair'}, {'age': 'senior', 'income': 'medium', 'student': 'yes', 'credit_rating': 'fair'}, {'age': 'youth', 'income': 'medium', 'student': 'yes', 'credit_rating': 'excellent'}, {'age': 'middle_aged', 'income': 'medium', 'student': 'no', 'credit_rating': 'excellent'}, {'age': 'middle_aged', 'income': 'high', 'student': 'yes', 'credit_rating': 'fair'}, {'age': 'senior', 'income': 'medium', 'student': 'no', 'credit_rating': 'excellent'}]

2.2 利用DictVectorizer将 featureList 和labelList转换为向量dummyX和dummyY

2.2.1 dummyX

dummyX是一个10X14的矩阵，由10个列向量组成，每个列向量代表feature_names中的一项。

feature_names = ['age=middle_aged', 'age=senior', 'age=youth', 'credit_rating=excellent', 'credit_rating=fair', 'income=high', 'income=low', 'income=medium', 'student=no', 'student=yes']

dummyX =
[[ 0. 0. 1. 0. 1. 1. 0. 0. 1. 0.]
[ 0. 0. 1. 1. 0. 1. 0. 0. 1. 0.]
[ 1. 0. 0. 0. 1. 1. 0. 0. 1. 0.]
[ 0. 1. 0. 0. 1. 0. 0. 1. 1. 0.]
[ 0. 1. 0. 0. 1. 0. 1. 0. 0. 1.]
[ 0. 1. 0. 1. 0. 0. 1. 0. 0. 1.]
[ 1. 0. 0. 1. 0. 0. 1. 0. 0. 1.]
[ 0. 0. 1. 0. 1. 0. 0. 1. 1. 0.]
[ 0. 0. 1. 0. 1. 0. 1. 0. 0. 1.]
[ 0. 1. 0. 0. 1. 0. 0. 1. 0. 1.]
[ 0. 0. 1. 1. 0. 0. 0. 1. 0. 1.]
[ 1. 0. 0. 1. 0. 0. 0. 1. 1. 0.]
[ 1. 0. 0. 0. 1. 1. 0. 0. 0. 1.]
[ 0. 1. 0. 1. 0. 0. 0. 1. 1. 0.]]

2.2.2dummyY

dummyY是一个1X14的列向量，代表这标签类的内容，也就是class_buys_computer的内容
dummyY =
[[0]
[0]
[1]
[1]
[1]
[0]
[1]
[0]
[1]
[1]
[1]
[1]
[1]
[0]]

文档： http://scikit-learn.org/stable/modules/tree.html

3.可视化工具 Graphviz的安装

3.1下载安装

下载地址：http://www.graphviz.org/

3.2 环境变量配置

将安装路径C:\Program Files (x86)\Graphviz2.38\bin添加到环境变量中

转化dot文件至pdf可视化决策树：
进入allElectronicInformationGainOri.dot文件所在的目录，输入如下命令：

dot -Tpdf allElectronicInformationGainOri.dot -o outpu.pdf

可以将.dot文件转换为pdf文件
allElectronicInformationGainOri.dot文件如下：

digraph Tree {
node [shape=box] ;
0 [label="age=middle_aged <= 0.5\nentropy = 0.94\nsamples = 14\nvalue = [5, 9]"] ;
1 [label="student=yes <= 0.5\nentropy = 1.0\nsamples = 10\nvalue = [5, 5]"] ;
0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ;
2 [label="age=senior <= 0.5\nentropy = 0.722\nsamples = 5\nvalue = [4, 1]"] ;
1 -> 2 ;
3 [label="entropy = 0.0\nsamples = 3\nvalue = [3, 0]"] ;
2 -> 3 ;
4 [label="credit_rating=excellent <= 0.5\nentropy = 1.0\nsamples = 2\nvalue = [1, 1]"] ;
2 -> 4 ;
5 [label="entropy = 0.0\nsamples = 1\nvalue = [0, 1]"] ;
4 -> 5 ;
6 [label="entropy = 0.0\nsamples = 1\nvalue = [1, 0]"] ;
4 -> 6 ;
7 [label="credit_rating=excellent <= 0.5\nentropy = 0.722\nsamples = 5\nvalue = [1, 4]"] ;
1 -> 7 ;
8 [label="entropy = 0.0\nsamples = 3\nvalue = [0, 3]"] ;
7 -> 8 ;
9 [label="age=youth <= 0.5\nentropy = 1.0\nsamples = 2\nvalue = [1, 1]"] ;
7 -> 9 ;
10 [label="entropy = 0.0\nsamples = 1\nvalue = [1, 0]"] ;
9 -> 10 ;
11 [label="entropy = 0.0\nsamples = 1\nvalue = [0, 1]"] ;
9 -> 11 ;
12 [label="entropy = 0.0\nsamples = 4\nvalue = [0, 4]"] ;
0 -> 12 [labeldistance=2.5, labelangle=-45, headlabel="False"] ;
}

outpu.pdf文件的内容如下：

生成的决策树

4 最终的代码

代码如下：

from sklearn.feature_extraction import DictVectorizer
import csv
from sklearn import tree
from sklearn import preprocessing


# Read in the csv file and put features into list of dict and list of class label

allElectronicsData = open(r'AllElectronics.csv', 'rt')
reader = csv.reader(allElectronicsData)

headers = next(reader)    # 读取表格的第一行，即表头


featureList = []  # 特征值列表
labelList = []    # 标签列表，即class_buys_computer的内容


# 从表格的第二行开始，一行一行的循环整个cvs
for row in reader:
    labelList.append(row[len(row)-1])  # 表格最后一列class_buys_computer的内容
    rowDict = {}
    for i in range(1, len(row)-1):   # 不读取第一列RID
        rowDict[headers[i]] = row[i]
    featureList.append(rowDict)


# 将特征值列表中的内容转化成向量形式
vec = DictVectorizer()
dummyX = vec.fit_transform(featureList) .toarray()
print(dummyX)

print("dummyX: " + str(dummyX))
print(vec.get_feature_names())

print("labelList: " + str(labelList))

# 将标签类的内容转化成向量形式
lb = preprocessing.LabelBinarizer()
dummyY = lb.fit_transform(labelList)
print("dummyY: " + str(dummyY))

# Using decision tree for classification
# clf = tree.DecisionTreeClassifier()
clf = tree.DecisionTreeClassifier(criterion='entropy')
clf = clf.fit(dummyX, dummyY)
print("clf: " + str(clf))


# Visualize model
# 生成allElectronicInformationGainOri.dot文件
with open("allElectronicInformationGainOri.dot", 'w') as f:
    f = tree.export_graphviz(clf, feature_names=vec.get_feature_names(), out_file=f)


# 利用原来的数据生成新的数据进行预测
# 取第一行
oneRowX = dummyX[0, :]
print("oneRowX: " + str(oneRowX))

# 修改后生成新的数据
newRowX = oneRowX
newRowX[0] = 1
newRowX[2] = 0
print("newRowX: " + str(newRowX))


# 进行预测
predictedY = clf.predict([newRowX])
print("predictedY: " + str(predictedY))

注意的问题：
最后进行预测是 clf.predict([newRowX])传入的是一个向量，不是数组，如果是：

# 进行预测
predictedY = clf.predict(newRowX)
print("predictedY: " + str(predictedY))

则会报错：

ValueError: Expected 2D array, got 1D array instead:
array=[ 1.  0.  0.  0.  1.  1.  0.  0.  1.  0.].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

这一点要注意，当然也可以用 array.reshape(1, -1)

            【注】：本文为麦子学院机器学习课程的学习笔记