数据挖掘领域十大经典算法之—C4.5算法（超详细附代码）

最新推荐文章于 2023-10-03 14:30:39 发布

谦卑t

最新推荐文章于 2023-10-03 14:30:39 发布

阅读量583

点赞数

分类专栏：算法小结文章标签： C4.5算法（超详细附代码）

算法小结专栏收录该内容

2 篇文章 0 订阅

订阅专栏

				版权声明：本文为博主原创文章，未经博主允许不得转载。					https://blog.csdn.net/fuqiuai/article/details/79456971				</div>
							            <div id="content_views" class="markdown_views prism-atom-one-dark">
						<!-- flowchart 箭头图标 勿删 -->
						<svg xmlns="http://www.w3.org/2000/svg" style="display: none;"><path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path></svg>
						<blockquote>

相关文章：

数据挖掘十大经典算法如下：
这里写链接内容

简介

C4.5是决策树算法的一种。决策树算法作为一种分类算法，目标就是将具有p维特征的n个样本分到c个类别中去。常见的决策树算法有ID3,C4.5,CART。

基本思想

下面以一个例子来详细说明C4.5的基本思想

上述数据集有四个属性，属性集合A={ 天气，温度，湿度，风速}，类别标签有两个，类别集合L={进行，取消}。

1. 计算类别信息熵
类别信息熵表示的是所有样本中各种类别出现的不确定性之和。根据熵的概念，熵越大，不确定性就越大，把事情搞清楚所需要的信息量就越多。
这里写链接内容

2. 计算每个属性的信息熵
每个属性的信息熵相当于一种条件熵。他表示的是在某种属性的条件下，各种类别出现的不确定性之和。属性的信息熵越大，表示这个属性中拥有的样本类别越不“纯”。
这里写链接内容

3. 计算信息增益
信息增益的 = 熵 - 条件熵，在这里就是类别信息熵 - 属性信息熵，它表示的是信息不确定性减少的程度。如果一个属性的信息增益越大，就表示用这个属性进行样本划分可以更好的减少划分后样本的不确定性，当然，选择该属性就可以更快更好地完成我们的分类目标。

信息增益就是ID3算法的特征选择指标。
这里写链接内容
但是我们假设这样的情况，每个属性中每种类别都只有一个样本，那这样属性信息熵就等于零，根据信息增益就无法选择出有效分类特征。所以，C4.5选择使用信息增益率对ID3进行改进。

4.计算属性分裂信息度量
用分裂信息度量来考虑某种属性进行分裂时分支的数量信息和尺寸信息，我们把这些信息称为属性的内在信息（instrisic information）。信息增益率用信息增益 / 内在信息，会导致属性的重要性随着内在信息的增大而减小（也就是说，如果这个属性本身不确定性就很大，那我就越不倾向于选取它），这样算是对单纯用信息增益有所补偿。
这里写链接内容

5. 计算信息增益率
（下面写错了。。应该是IGR = Gain / H ）
这里写链接内容

天气的信息增益率最高，选择天气为分裂属性。发现分裂了之后，天气是“阴”的条件下，类别是”纯“的，所以把它定义为叶子节点，选择不“纯”的结点继续分裂。
这里写链接内容

在子结点当中重复过程1~5。
至此，这个数据集上C4.5的计算过程就算完成了，一棵树也构建出来了。

总结算法流程为：

while (当前节点”不纯“)  
	(1)计算当前节点的类别信息熵Info(D) （以类别取值计算）  
	(2)计算当前节点各个属性的信息熵Info(Ai) （以属性取值下的类别取值计算）  
	(3)计算各个属性的信息增益Gain(Ai)=Info(D)-Info(Ai)  
	(4)计算各个属性的分类信息度量H(Ai) （以属性取值计算）  
	(5)计算各个属性的信息增益率IGR(Ai)=Gain(Ai)/H(Ai)  
end while  
当前节点设置为叶子节点  

 
 
 
 1
2
3
4
5
6
7
8

优缺点

优点

产生的分类规则易于理解，准确率较高。

缺点

在构造树的过程中，需要对数据集进行多次的顺序扫描和排序，因而导致算法的低效。

代码

代码已在github上实现，这里也贴出来

# encoding=utf-8

import cv2
import time
import numpy as np
import pandas as pd

from sklearn.cross_validation import train_test_split
from sklearn.metrics import accuracy_score

# 二值化
def binaryzation(img):
cv_img = img.astype(np.uint8)
cv2.threshold(cv_img,50,1,cv2.THRESH_BINARY_INV,cv_img)
return cv_img

def binaryzation_features(trainset):
features = []

<span class="token keyword">for</span> img <span class="token keyword">in</span> trainset<span class="token punctuation">:</span>
    img <span class="token operator">=</span> np<span class="token punctuation">.</span>reshape<span class="token punctuation">(</span>img<span class="token punctuation">,</span><span class="token punctuation">(</span><span class="token number">28</span><span class="token punctuation">,</span><span class="token number">28</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
    cv_img <span class="token operator">=</span> img<span class="token punctuation">.</span>astype<span class="token punctuation">(</span>np<span class="token punctuation">.</span>uint8<span class="token punctuation">)</span>

    img_b <span class="token operator">=</span> binaryzation<span class="token punctuation">(</span>cv_img<span class="token punctuation">)</span>
    <span class="token comment"># hog_feature = np.transpose(hog_feature)</span>
    features<span class="token punctuation">.</span>append<span class="token punctuation">(</span>img_b<span class="token punctuation">)</span>

features <span class="token operator">=</span> np<span class="token punctuation">.</span>array<span class="token punctuation">(</span>features<span class="token punctuation">)</span>
features <span class="token operator">=</span> np<span class="token punctuation">.</span>reshape<span class="token punctuation">(</span>features<span class="token punctuation">,</span><span class="token punctuation">(</span><span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">,</span>feature_len<span class="token punctuation">)</span><span class="token punctuation">)</span>

<span class="token keyword">return</span> features

class Tree(object):
def init(self,node_type,Class = None, feature = None):
self.node_type = node_type # 节点类型（internal或leaf）
self.dict = {} # dict的键表示特征Ag的可能值ai，值表示根据ai得到的子树
self.Class = Class # 叶节点表示的类，若是内部节点则为none
self.feature = feature # 表示当前的树即将由第feature个特征划分（即第feature特征是使得当前树中信息增益最大的特征）

<span class="token keyword">def</span> <span class="token function">add_tree</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span>key<span class="token punctuation">,</span>tree<span class="token punctuation">)</span><span class="token punctuation">:</span>
    self<span class="token punctuation">.</span><span class="token builtin">dict</span><span class="token punctuation">[</span>key<span class="token punctuation">]</span> <span class="token operator">=</span> tree

<span class="token keyword">def</span> <span class="token function">predict</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span>features<span class="token punctuation">)</span><span class="token punctuation">:</span> 
    <span class="token keyword">if</span> self<span class="token punctuation">.</span>node_type <span class="token operator">==</span> <span class="token string">'leaf'</span> <span class="token operator">or</span> <span class="token punctuation">(</span>features<span class="token punctuation">[</span>self<span class="token punctuation">.</span>feature<span class="token punctuation">]</span> <span class="token operator">not</span> <span class="token keyword">in</span> self<span class="token punctuation">.</span><span class="token builtin">dict</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        <span class="token keyword">return</span> self<span class="token punctuation">.</span>Class

    <span class="token class-name">tree</span> <span class="token operator">=</span> self<span class="token punctuation">.</span><span class="token builtin">dict</span><span class="token punctuation">.</span>get<span class="token punctuation">(</span>features<span class="token punctuation">[</span>self<span class="token punctuation">.</span>feature<span class="token punctuation">]</span><span class="token punctuation">)</span>
    <span class="token keyword">return</span> tree<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>features<span class="token punctuation">)</span>

# 计算数据集x的经验熵H(x)
def calc_ent(x):
x_value_list = set([x[i] for i in range(x.shape[0])])
ent = 0.0
for x_value in x_value_list:
p = float(x[x == x_value].shape[0]) / x.shape[0]
logp = np.log2(p)
ent -= p * logp

<span class="token keyword">return</span> ent

# 计算条件熵H(y/x)
def calc_condition_ent(x, y):
x_value_list = set([x[i] for i in range(x.shape[0])])
ent = 0.0
for x_value in x_value_list:
sub_y = y[x == x_value]
temp_ent = calc_ent(sub_y)
ent += (float(sub_y.shape[0]) / y.shape[0]) * temp_ent

<span class="token keyword">return</span> ent

# 计算信息增益
def calc_ent_grap(x,y):
base_ent = calc_ent(y)
condition_ent = calc_condition_ent(x, y)
ent_grap = base_ent - condition_ent

<span class="token keyword">return</span> ent_grap

# C4.5算法
def recurse_train(train_set,train_label,features):

LEAF <span class="token operator">=</span> <span class="token string">'leaf'</span>
INTERNAL <span class="token operator">=</span> <span class="token string">'internal'</span>

<span class="token comment"># 步骤1——如果训练集train_set中的所有实例都属于同一类Ck</span>
label_set <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span>train_label<span class="token punctuation">)</span>
<span class="token keyword">if</span> <span class="token builtin">len</span><span class="token punctuation">(</span>label_set<span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">:</span>
    <span class="token keyword">return</span> Tree<span class="token punctuation">(</span>LEAF<span class="token punctuation">,</span>Class <span class="token operator">=</span> label_set<span class="token punctuation">.</span>pop<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>

<span class="token comment"># 步骤2——如果特征集features为空</span>
class_len <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">(</span>i<span class="token punctuation">,</span><span class="token builtin">len</span><span class="token punctuation">(</span><span class="token builtin">list</span><span class="token punctuation">(</span><span class="token builtin">filter</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> x<span class="token punctuation">:</span>x<span class="token operator">==</span>i<span class="token punctuation">,</span>train_label<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>class_num<span class="token punctuation">)</span><span class="token punctuation">]</span> <span class="token comment"># 计算每一个类出现的个数</span>
<span class="token punctuation">(</span>max_class<span class="token punctuation">,</span>max_len<span class="token punctuation">)</span> <span class="token operator">=</span> <span class="token builtin">max</span><span class="token punctuation">(</span>class_len<span class="token punctuation">,</span>key <span class="token operator">=</span> <span class="token keyword">lambda</span> x<span class="token punctuation">:</span>x<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span>

<span class="token keyword">if</span> <span class="token builtin">len</span><span class="token punctuation">(</span>features<span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token number">0</span><span class="token punctuation">:</span>
    <span class="token keyword">return</span> Tree<span class="token punctuation">(</span>LEAF<span class="token punctuation">,</span>Class <span class="token operator">=</span> max_class<span class="token punctuation">)</span>

<span class="token comment"># 步骤3——计算信息增益,并选择信息增益最大的特征</span>
max_feature <span class="token operator">=</span> <span class="token number">0</span>
max_gda <span class="token operator">=</span> <span class="token number">0</span>
D <span class="token operator">=</span> train_label
<span class="token keyword">for</span> feature <span class="token keyword">in</span> features<span class="token punctuation">:</span>
    <span class="token comment"># print(type(train_set))</span>
    A <span class="token operator">=</span> np<span class="token punctuation">.</span>array<span class="token punctuation">(</span>train_set<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span>feature<span class="token punctuation">]</span><span class="token punctuation">.</span>flat<span class="token punctuation">)</span> <span class="token comment"># 选择训练集中的第feature列（即第feature个特征）</span>
    gda <span class="token operator">=</span> calc_ent_grap<span class="token punctuation">(</span>A<span class="token punctuation">,</span>D<span class="token punctuation">)</span>
    <span class="token keyword">if</span> calc_ent<span class="token punctuation">(</span>A<span class="token punctuation">)</span> <span class="token operator">!=</span> <span class="token number">0</span><span class="token punctuation">:</span>  <span class="token comment">####### 计算信息增益比，这是与ID3算法唯一的不同</span>
        gda <span class="token operator">/=</span> calc_ent<span class="token punctuation">(</span>A<span class="token punctuation">)</span>
    <span class="token keyword">if</span> gda <span class="token operator">&gt;</span> max_gda<span class="token punctuation">:</span>
        max_gda<span class="token punctuation">,</span>max_feature <span class="token operator">=</span> gda<span class="token punctuation">,</span>feature

<span class="token comment"># 步骤4——信息增益小于阈值</span>
<span class="token keyword">if</span> max_gda <span class="token operator">&lt;</span> epsilon<span class="token punctuation">:</span>
    <span class="token keyword">return</span> Tree<span class="token punctuation">(</span>LEAF<span class="token punctuation">,</span>Class <span class="token operator">=</span> max_class<span class="token punctuation">)</span>

<span class="token comment"># 步骤5——构建非空子集</span>
sub_features <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span><span class="token builtin">filter</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> x<span class="token punctuation">:</span>x<span class="token operator">!=</span>max_feature<span class="token punctuation">,</span>features<span class="token punctuation">)</span><span class="token punctuation">)</span>
tree <span class="token operator">=</span> Tree<span class="token punctuation">(</span>INTERNAL<span class="token punctuation">,</span>feature<span class="token operator">=</span>max_feature<span class="token punctuation">)</span>

max_feature_col <span class="token operator">=</span> np<span class="token punctuation">.</span>array<span class="token punctuation">(</span>train_set<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span>max_feature<span class="token punctuation">]</span><span class="token punctuation">.</span>flat<span class="token punctuation">)</span>
feature_value_list <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span>max_feature_col<span class="token punctuation">[</span>i<span class="token punctuation">]</span> <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>max_feature_col<span class="token punctuation">.</span>shape<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token comment"># 保存信息增益最大的特征可能的取值 (shape[0]表示计算行数)</span>
<span class="token keyword">for</span> feature_value <span class="token keyword">in</span> feature_value_list<span class="token punctuation">:</span>

    index <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
    <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>train_label<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        <span class="token keyword">if</span> train_set<span class="token punctuation">[</span>i<span class="token punctuation">]</span><span class="token punctuation">[</span>max_feature<span class="token punctuation">]</span> <span class="token operator">==</span> feature_value<span class="token punctuation">:</span>
            index<span class="token punctuation">.</span>append<span class="token punctuation">(</span>i<span class="token punctuation">)</span>

    sub_train_set <span class="token operator">=</span> train_set<span class="token punctuation">[</span>index<span class="token punctuation">]</span>
    sub_train_label <span class="token operator">=</span> train_label<span class="token punctuation">[</span>index<span class="token punctuation">]</span>

    sub_tree <span class="token operator">=</span> recurse_train<span class="token punctuation">(</span>sub_train_set<span class="token punctuation">,</span>sub_train_label<span class="token punctuation">,</span>sub_features<span class="token punctuation">)</span>
    tree<span class="token punctuation">.</span>add_tree<span class="token punctuation">(</span>feature_value<span class="token punctuation">,</span>sub_tree<span class="token punctuation">)</span>

<span class="token keyword">return</span> tree

def train(train_set,train_label,features):
return recurse_train(train_set,train_label,features)

def predict(test_set,tree):
result = []
for features in test_set:
tmp_predict = tree.predict(features)
result.append(tmp_predict)
return np.array(result)

class_num = 10 # MINST数据集有10种labels，分别是“0,1,2,3,4,5,6,7,8,9”
feature_len = 784 # MINST数据集每个image有28*28=784个特征（pixels）
epsilon = 0.001 # 设定阈值

if name == ‘main’:

<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"Start read data..."</span><span class="token punctuation">)</span>

time_1 <span class="token operator">=</span> time<span class="token punctuation">.</span>time<span class="token punctuation">(</span><span class="token punctuation">)</span>

raw_data <span class="token operator">=</span> pd<span class="token punctuation">.</span>read_csv<span class="token punctuation">(</span><span class="token string">'../data/train.csv'</span><span class="token punctuation">,</span> header<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">)</span>  <span class="token comment"># 读取csv数据</span>
data <span class="token operator">=</span> raw_data<span class="token punctuation">.</span>values

imgs <span class="token operator">=</span> data<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token punctuation">]</span>
features <span class="token operator">=</span> binaryzation_features<span class="token punctuation">(</span>imgs<span class="token punctuation">)</span> <span class="token comment"># 图片二值化(很重要，不然预测准确率很低)</span>
labels <span class="token operator">=</span> data<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span>

<span class="token comment"># 避免过拟合，采用交叉验证，随机选取33%数据作为测试集，剩余为训练集</span>
train_features<span class="token punctuation">,</span> test_features<span class="token punctuation">,</span> train_labels<span class="token punctuation">,</span> test_labels <span class="token operator">=</span> train_test_split<span class="token punctuation">(</span>features<span class="token punctuation">,</span> labels<span class="token punctuation">,</span> test_size<span class="token operator">=</span><span class="token number">0.33</span><span class="token punctuation">,</span> random_state<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">)</span>
time_2 <span class="token operator">=</span> time<span class="token punctuation">.</span>time<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'read data cost %f seconds'</span> <span class="token operator">%</span> <span class="token punctuation">(</span>time_2 <span class="token operator">-</span> time_1<span class="token punctuation">)</span><span class="token punctuation">)</span>

<span class="token comment"># 通过C4.5算法生成决策树</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Start training...'</span><span class="token punctuation">)</span>
tree <span class="token operator">=</span> train<span class="token punctuation">(</span>train_features<span class="token punctuation">,</span>train_labels<span class="token punctuation">,</span><span class="token builtin">list</span><span class="token punctuation">(</span><span class="token builtin">range</span><span class="token punctuation">(</span>feature_len<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
time_3 <span class="token operator">=</span> time<span class="token punctuation">.</span>time<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'training cost %f seconds'</span> <span class="token operator">%</span> <span class="token punctuation">(</span>time_3 <span class="token operator">-</span> time_2<span class="token punctuation">)</span><span class="token punctuation">)</span>

<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Start predicting...'</span><span class="token punctuation">)</span>
test_predict <span class="token operator">=</span> predict<span class="token punctuation">(</span>test_features<span class="token punctuation">,</span>tree<span class="token punctuation">)</span>
time_4 <span class="token operator">=</span> time<span class="token punctuation">.</span>time<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'predicting cost %f seconds'</span> <span class="token operator">%</span> <span class="token punctuation">(</span>time_4 <span class="token operator">-</span> time_3<span class="token punctuation">)</span><span class="token punctuation">)</span>

<span class="token comment"># print("预测的结果为：")</span>
<span class="token comment"># print(test_predict)</span>
<span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>test_predict<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">if</span> test_predict<span class="token punctuation">[</span>i<span class="token punctuation">]</span> <span class="token operator">==</span> <span class="token boolean">None</span><span class="token punctuation">:</span>
        test_predict<span class="token punctuation">[</span>i<span class="token punctuation">]</span> <span class="token operator">=</span> epsilon
score <span class="token operator">=</span> accuracy_score<span class="token punctuation">(</span>test_labels<span class="token punctuation">,</span> test_predict<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"The accruacy score is %f"</span> <span class="token operator">%</span> score<span class="token punctuation">)</span>

测试数据集为MNIST数据集，获取地址为train.csv

运行结果

这里写链接内容

        </div>
					<link href="https://csdnimg.cn/release/phoenix/mdeditor/markdown_views-7b4cdcb592.css" rel="stylesheet">
            </div>

谦卑t

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
复制链接

分享到 QQ

分享到新浪微博

扫一扫

专栏目录