ID3决策树与C4.5决策树分类算法简述

最新推荐文章于 2024-04-30 13:14:31 发布

止于至玄

最新推荐文章于 2024-04-30 13:14:31 发布

阅读量431

点赞数

分类专栏： Machine Learning 文章标签：机器学习

本文链接：https://blog.csdn.net/philthinker/article/details/69230019

版权

Machine Learning 专栏收录该内容

23 篇文章 3 订阅

订阅专栏

本文详细介绍了两种经典的决策树算法：ID3和C4.5。首先探讨了ID3算法如何通过信息增益选择最优特征进行树的构建，并指出了其存在的问题。接着引入了C4.5算法，该算法使用增益率作为特征选择标准，解决了ID3的一些不足之处。

摘要由CSDN通过智能技术生成

Let’s begin with ID3 decision tree:
The ID3 algorithm tries to get the most information gain when grow the decision trees. The information gain is defined as

Gain (A) = I (s 1, s 2, \dots, s m) - E (A)

$\text{Gain}(A)=I(s_{1},s_{2},\dots,s_{m})-E(A)$
where

I $I$ is the information entropy of a given sample setting,

I (s 1, s 2, \dots, s m) = - \sum i = 1 m p i log 2 (p i)

$I(s_{1},s_{2},\dots,s_{m})=-\sum_{i=1}^{m}p_{i}\log_{2}(p_{i})$

E(A) $E(A)$ is the information entropy of the subset classified by attribute

A=(a1,a2,…,av) $A=(a_{1},a_{2},\dots,a_{v})$ ,

E (A) = \sum j = 1 v s i j + s 2 j + \dots + s m j s I (s 1, s 2, \dots, s m)

$E(A)=\sum_{j=1}^{v}\frac{s_{ij}+s_{2j}+\cdots+s_{mj}}{s}I(s_{1},s_{2},\dots,s_{m})$
Moreover,

pi $p_{i}$ is the probability of an sample belonging to class

Ci $C_{i}$ , which can be estimated as

pi=si|S| $p_{i}=\frac{s_{i}}{|S|}$ and

pij $p_{ij}$ is the probability an sample belonging to class

Ci $C_{i}$ with attribute

A=aj $A=a_{j}$ , i.e.

pij+sij|Sj| $p_{ij}+\frac{s_{ij}}{|S_{j}|}$ .
ID3 algorithm can be simplified as follows:

For every attribute $A$ , we calculate its information gain $E(A)$ .
Pick up the attribute who is of the largest $E(A)$ as the root node or internal node.
Get rid of the grown attribute $A$ , and for every value $a_{j}$ of attribute $A$ , calculate the next node to be grown.
Keep steps 1~3 until each subset has only one label/class $C_{i}$ .

ID3 algorithm is an old machine learning algorithm created in 1979 based on information entropy, however, there are several problems of it:

ID3 prefers the attribute with more values, though it turns out not to be the optimal one.
ID3 has to calculate the information entropy of every value of every attribute. Hence it always leads to many levels and branches with very little probability, as a result of which it tends to overfit classification in the test set.

C4.5 decision tree
C4,.5 algorithm makes use of Grain Ratio instead of Gain to select attributes.