Study notes for Decision Trees

Decision Trees are biased to prefer classifiers that can be expressed as trees. Good tutorials are given by Andrew Moore:  Decision Tree and  Information Gain. The most well-known algorithms are ID3,  C4.5 (a successor of ID3),  C5.0 (an extension to C4.5) and  CART ( classification and regression trees). For a comparison, refer to: A Comparative Study on Decision Tree Classification Algorithms in Data Mining.

1. Definitions

• Decision stump is a decision tree with a single node. 
• Arity is the number of arguments that a function can take. 

2. Advantages

• The construction of decision tree classifiers does not require any domain knowledge or parameter setting, and hence is appropriate for exploratory knowledge discovery. 
• It can handle multiple dimensional data. The acquired knowledge is represented in the tree form that is intuitive. 
• Decision trees can be easily converted to classification rules. 
• The learning and classification steps of decision tree inductions are simple and fast. 

3. Disadvantages

• The division of input space is based on hard splits in which only one model is responsible for making predictions for any given value of the input variables. 
• Decision trees suffer from repetition and replication. Repetition occurs when an attribute is repeatedly tested along a given branch of the tree. In replication, duplicate subtrees exist within the tree. 
• The aforementioned three algorithms cannot adapt to large-scale real-world data sets since the tree construction is memory-consuming. Some scalable decision tree algorithms have been proposed, such as RainForest and BOAT. BOAT was found to be two to three times faster than RainForest, while constructing exactly the same tree. 

4. Remarks

• Attribute selection measure is a heuristic for selecting the splitting criterion that "best" separates a give data partition of class-labeled raining examples into individual classes, also known as splitting rules. ID3 uses information gain, C4.5 adopts an extension to information gain known as gain ratio, and the bookclassification and regression tree (CART) uses the Gini index. 
  • Information gain is biased toward multi-valued attributes (i.e., more than 2) and has difficulty when the number of classes is large. It tends to favor tests that result in equal-size partitions and purity in both partitions.
  • Gini index is in favor of binary classes. 
  • Other measures include minimum description length, multivariate splits. 
  • No golden rules to select the best attribute selection measure, a practice tip is to prefer the one producing shallower trees. 
• When decision trees are built, many of the branches may reflect noise or outliers in the training data.Tree pruning attempts to identify and remove such branches, aiming to avoid the problem of overfitting the training data and improve the performance on the testing set. 
  • Prepruning. a tree is "pruned" by halting its construction early, e.g., by deciding not to further split or partition the subset of training examples at a given node. 
  • Postpruning. it is more common used which removes the subtrees from a "fully grown" tree. A subtree at a given node is pruned by removing its branches and replacing it with a leaf. The leaf is labeled with the most frequent class among the subtree being replaced. The concerns could be error rate, cost complexity,the number of bits required for encoding, etc. 
  • Combining prepruning and postpruning. Postpruning requires more computation than prepruning, yet generally leads to a more reliable tree. No single pruning method is found to be superior ovall all others.
• Bayesian Decision Trees can take a different approach to preventing overfitting
• Blog: Information Theory 信息论简介

5. Randomized Decision Forest

• RandomForest is an ensemble of multiple randomized decision trees, a fast tool used for classification, clustering or regression. Specifically, for a new input vector, it is put down to each decision tree in the forest, and each tree gives a classification. The final class is determined by the "votes" of the trees in the forest. 
• Random Decision Tree (RDT)
• Randomized Decision Forest = Randomized Forest = Random Forest (Trade Mark)
  • Good generalization for through randomized training
  • Inherently for multi-class
  • Simple training / testing algorithms
• Random forest wikipedia
• Blog: 决策树模型组合之随机森林与GBDT
• An example of the use of Random Forest to resolve a real problem