CART分类与回归树的原理与实现

原创 2015年04月16日 22:31:09

算法概述

CART(Classification And Regression Tree)算法是一种决策树分类方法。

它采用一种二分递归分割的技术,分割方法采用基于最小距离的基尼指数估计函数,将当前的样本集分为两个子样本集,使得生成的的每个非叶子节点都有两个分支。因此,CART算法生成的决策树是结构简洁的二叉树。

分类树

如果目标变量是离散变量,则是classfication Tree。

分类树是使用树结构算法将数据分成离散类的方法。

回归树

如果目标是连续变量,则是Regression Tree。

CART树是二叉树,不像多叉树那样形成过多的数据碎片。


分类树两个关键点

(1)将训练样本进行递归地划分自变量空间进行建树

(2)用验证数据进行剪枝。

a.对于离散变量X(x1…xn)

分别取X变量各值的不同组合,将其分到树的左枝或右枝,并对不同组合而产生的树,进行评判,找出最佳组合。如果只有两个取值,好办,直接根据这两个值就可以划分树。取值多于两个的情况就复杂一些了,如变量年纪,其值有“少年”、“中年”、“老年”,则分别生产{少年,中年}和{老年},{上年、老年}和{中年},{中年,老年}和{少年},这三种组合,最后评判对目标区分最佳的组合。因为CART二分的特性,当训练数据具有两个以上的类别,CART需考虑将目标类别合并成两个超类别,这个过程称为双化。这里可以说一个公式,n个属性,可以分出(2^n-2)/2种情况。

b.对于连续变量X(x1…xn)

首先将值排序,分别取其两相邻值的平均值点作为分隔点,将树一分成左枝和右枝,不断扫描,进而判断最佳分割点。特征值大于分裂值就走左子树,或者就走右子树。


这里有一个问题,这次选中的分裂属性在下次还可以被选择吗?对于离散变量XD,如果XD只有两种取值,那么在这一次分裂中,根据XD分裂后,左子树中的subDataset中每个数据的XD属性一样,右子树中的subDataset中每个数据的XD属性也一样,所以在这个节点以后,XD都不起作用了,就不用考虑XD了。XD取3种,4种。。。的情况大家自己想想,不难想明白。至于连续变量XC,离散化后相当于一个可以取n个值的离散变量,按刚刚离散变量的情况分析。除非XC的取值都一样,否则这次用了XC作为分裂属性,下次还要考虑XC。


变量和最佳切分点选择原则

树的生长,总的原则是,让枝比树更纯,而度量原则是根据不纯对指标来衡量,对于分类树,则用GINI指标、Twoing指标、Order Twoing等;如果是回归树则用,最小平方残差、最小绝对残差等指标衡量

(1)GINI指标(Gini越小,数据越纯)——针对离散目标





(2)最小平方残差——针对连续目标


其思想是,让组内方差最小,对应组间方差最大,这样两组,也即树分裂的左枝和右枝差异化最大。

通过以上不纯度指标,分别计算每个变量的各种切分/组合情况,找出该变量的最佳值组合/切分点;再比较各个变量的最佳值组合/切分点,最终找出最佳变量和该变量的最佳值组合/切分点



整个树的生长是一个递归过程,直到终止条件

终止条件

(1)节点是纯结点,即所有的记录的目标变量值相同

(2)树的深度达到了预先指定的最大值

(3)混杂度的最大下降值小于一个预先指定的值

(4)节点的记录量小于预先指定的最小节点记录量

(5)一个节点中的所有记录其预测变量值相同

直观的情况,当节点包含的数据记录都属于同一个类别时就可以终止分裂了。这只是一个特例,更一般的情况我们计算χ2值来判断分类条件和类别的相关程度,当χ2很小时说明分类条件和类别是独立的,即按照该分类条件进行分类是没有道理的,此时节点停止分裂。注意这里的“分类条件”是指按照GINI_Gain最小原则得到的“分类条件”。


终止条件(3)混杂度的最大下降值小于一个预先指定的值,该枝的分化即停止。所有枝节的分化都停止后,树形模型即成。其实你也可以不使用这个终止条件,让树生长到最大,因为CART有剪枝算法。


建树过程




这里面误分类成本和先验概率是需要提前设定好的参数。这里为node标定label如果考虑一些unbalanced data,比如训练样本里有100个正样本,只有1个负样本,这样的数据就是unbalanced,就不能简单的majority归类了。上面的这个mark label的方法对不均衡数据就有一定的鲁棒性。

要注意对于每一个树结点,不管是否叶子结点,该node都要标上label,因为后面剪枝时非叶节点可能变为叶节点。


树生长完之后就是剪枝,剪枝非常重要。剪枝目的是避免决策树过拟合(Overfitting)样本。在一般的数据集中,过拟合的决策树的错误率比经过简化的决策树的错误率要高。


剪枝算法CCP(Cost-Complexity Pruning)

这一部分参考http://blog.csdn.net/u010159842/article/details/46458973

Cost-Complexity Pruning(CCP、代价复杂度)
CCP方法包含两个步骤:
1:从原始决策树T0开始生成一个子树序列{T0、T1、T2、...、Tn},其中Ti+1是从Ti总产生,Tn为根节点
2:从子树序列中,根据树的真实误差估计选择最佳决策树。

对于分类回归树中的每一个非叶子节点计算它的表面误差率增益值α。

是子树中包含的叶子节点个数;

是节点t的误差代价,如果该节点被剪枝;

r(t)是节点t的误差率;

p(t)是节点t上的数据占所有数据的比例。

是子树Tt的误差代价,如果该节点不被剪枝。它等于子树Tt上所有叶子节点的误差代价之和。

比如有个非叶子节点t4如图所示:

比如有个非叶子节点t4如图所示:

已知所有的数据总共有60条,则节点t4的节点误差代价为:

子树误差代价为:

以t4为根节点的子树上叶子节点有3个,最终:

找到α值最小的非叶子节点,令其左右孩子为NULL。当多个非叶子节点的α值同时达到最小时,取最大的进行剪枝。

剪枝过程特别重要,所以在最优决策树生成过程中占有重要地位。有研究表明,剪枝过程的重要性要比树生成过程更为重要,对于不同的划分标准生成的最大树(Maximum Tree),在剪枝之后都能够保留最重要的属性划分,差别不大。反而是剪枝方法对于最优树的生成更为关键。


好了,再来看一个例子


很明白了吧


用一幅图解释一下


29-30之间的水平线以下的几个点代表的树都满足:


但箭头所指的树的叶节点最少,所以选择这棵树作为best tree。



缺失值的处理 

对于某些采样数据,可能会缺少属性值。在这种情况下,处理缺少属性值的通常做法是赋予该属性的常见值,或者属性均值。另外一种比较好的方法是为该属性的每个可能值赋予一个概率,即将该属性以概率形式赋值。例如给定Boolean属性B,已知采样数据有12个B=0和88个B=1实例,那么在赋值过程中,B属性的缺失值被赋值为B(0)=0.12、B(1)=0.88;所以属性B的缺失值以12%概率被分到False的分支,以88%概率被分到True的分支。这种处理的目的是计算信息增益,使得这种属性值缺失的样本也能处理。





最终结果

(1)分类树:最终叶子中概率最大的类
(2)回归树:最终叶子的均值或者中位数


优点

1)非常灵活,可以允许有部分错分成本,还可指定先验概率分布,可使用自动的成本复杂性剪枝来得到归纳性更强的树。
2)在面对诸如存在缺失值、变量数多等问题时CART显得非常稳健。



其实一直想看Cart:Classification and Regression Trees-Leo Breiman原版的书,可惜找不到,大家有谁找到能不能分享一下。

下面是我的实现,回归部分其实还没写,以后写了会更新一下。
数据集用的是UCI  adult数据集,大家可以搜搜
// cart.cpp : 定义控制台应用程序的入口点。
//

#include "stdafx.h"
#include<vector>
#include<set>
#include<algorithm>
#include<iostream>
#include<iterator>
#include<fstream>
#include<string>
#include<map>
/*******************************************/
/************author Marshall****************/
/**********date 2015.10.3*******************/
/**************version 1.0******************/
/************copyright reserved*************/
/*******************************************/
using namespace std;



class cart
{
private:
	vector<int>nums_of_value_each_discreteAttri;
	int num_of_continuousAttri;
	int ContinuousAttriNums;
	int labelNums;//how many kinds of label
	unsigned int CL_max_height;
	//double miniumginigain;//not need,we have prune method

	//define the record
	class Record
	{
	public:
		vector<int>discrete_attri;//for each discrete attribute,it's value can be 0,1...increased by 1
		vector<double>continuous_attti;
		int label;//0,1,2...
	};

	//define the node
	struct CartNode
	{
		vector<int>remianDiscreteAttriID;
		int selectedAttriID;
		vector<int>selectedDiscreteAttriValues;
		bool isSelectedAttriIDDiscrete;
		double continuousAttriPartitionValue;//
		int label;//if the record drop in this node,its' label should be
		int height;//current node's height
		vector<int>labelcount;//a counter for the records' label that current node holds
		double alpha;//for nonleaf,for prune
		int record_number;//该节点上涵盖的记录个数
		CartNode*lnode, *rnode;
		CartNode()
		{
			label = -1;
			selectedAttriID = -1;
			isSelectedAttriIDDiscrete = true;
			lnode = rnode = NULL;
			record_number = 0;
		}
	};
	CartNode*root;


	//double threshold;

private:
	//calculate gini index,for classify
	double calGiniIndex(vector<int>&subdatasetbyID, const vector<Record>*dataset, CartNode*node = NULL);
	double calSquaredresiduals();//calculate squaredresiduals,for regression
	void CL_split_dataset();
	void RE_split_dataset();
	void CL_trim(const vector<Record>*validationdataset);
	void RE_trim();
	//void make_discrete();
	//pair.first is majority label in subdataset,pair.second is it's number
	int allthesame(vector<int>&subdatasetbyID, const vector<Record>*dataset);
	/*如果某特征取值有3个,那么二分序列组合就有3种,4个取值就有7种组合,5个取值就有15种组合*/
	vector<pair<vector<int>, vector<int>>>make_two_heap(const int kk);
	pair<vector<int>, vector<int>>split_dataset(const int&selectedDiscreteAttriID,
		vector<int>&selected, const vector<int>&subdatasetbyID, const vector<Record>*dataset);
	pair<vector<int>, vector<int>>split_dataset(const int&selectedContiuousAttriID,
		const double partition, const vector<int>&subdatasetbyID, const vector<Record>*dataset);
	CartNode* copytree(CartNode*src, CartNode*dst);//deepcopy of a tree,dst should be NUll
	void copynode(CartNode*src, CartNode*dst);
	void cal_alpha(CartNode*node);
	vector<CartNode*>getLeaf(CartNode*node);
	void destroyTree(CartNode*node);
	int labelNode(CartNode*node);
	void create_root();
	void build_tree_classify(vector<int>&subdatasetbyID,
		CartNode*node, const vector<Record>*dataset);
	void build_tree_regression();
public:
	void load_adult_dataset();
	int CART_classify(const Record dataset, CartNode*root = NULL);
	void CART_regression();
	void CART_trian(const vector<Record>*dataset, const vector<Record>*validationdataset);
	void CART_trian()
	{
		CART_trian(traindataset, validatedataset);
	}
	void set_paras();
	~cart()
	{
		destroyTree(root);
		if (traindataset != NULL)
			delete traindataset;
		if (validatedataset != NULL)
			delete validatedataset;
	}
	vector<Record>*traindataset;//as it's name
	vector<Record>*validatedataset;
	vector<Record>*testdataset;
	void test(CartNode*node);
	void test();
};
void cart::test(CartNode*node)
{
	int errorNum = 0;
	for (int j = 0; j < testdataset->size(); j++)
	{
		errorNum += CART_classify((*testdataset)[j], node) == (*testdataset)[j].label ? 0 : 1;
	}
	cout << "测试集上的错误率为" << double(errorNum) / testdataset->size();

}

void cart::test()
{
	test(this->root);

}



void cart::set_paras()
{
	CL_max_height = 6;


}
void cart::CART_trian(const vector<Record>*dataset, const vector<Record>*validationdataset)
{
	create_root();
	set_paras();
	vector<int>subset;
	for (int i = 0; i < dataset->size(); i++)
		subset.push_back(i);
	build_tree_classify(subset, root, dataset);
	CL_trim(validationdataset);
}


void cart::destroyTree(CartNode*treeroot)
{
	_ASSERTE(treeroot != NULL);
	vector<CartNode*>pool, que;
	que.push_back(treeroot);
	while (!que.empty())
	{
		CartNode*node = que.back();
		que.pop_back();
		pool.push_back(node);
		if (node->lnode != NULL)
		{
			_ASSERTE(node->rnode != NULL);
			pool.push_back(node->lnode);
			pool.push_back(node->rnode);
		}
	}
	for (int i = 0; i < pool.size(); i++)
		delete pool[i];
}

void cart::copynode(CartNode*src, CartNode*dst)
{
	_ASSERTE(dst != NULL);
	_ASSERTE(src != NULL);
	dst->alpha = src->alpha;
	dst->continuousAttriPartitionValue = src->continuousAttriPartitionValue;
	dst->height = src->height;
	dst->isSelectedAttriIDDiscrete = src->isSelectedAttriIDDiscrete;
	dst->label = src->label;
	dst->labelcount = src->labelcount;
	dst->record_number = src->record_number;
	dst->remianDiscreteAttriID = src->remianDiscreteAttriID;
	dst->selectedAttriID = src->selectedAttriID;
	dst->selectedDiscreteAttriValues = src->selectedDiscreteAttriValues;

}

//implementation of tree copy
cart::CartNode* cart::copytree(CartNode*Srctreeroot, CartNode*Dsttreeroot)
{
	_ASSERTE(Dsttreeroot == NULL);
	_ASSERTE(Srctreeroot != NULL);

	vector<CartNode*>pool, parentpool;
	Dsttreeroot = new CartNode;
	copynode(Srctreeroot, Dsttreeroot);
	if (Srctreeroot->lnode == NULL)
	{
		_ASSERTE(Srctreeroot->rnode == NULL);
		return Dsttreeroot;
	}
	pool.push_back(Srctreeroot->lnode);
	pool.push_back(Srctreeroot->rnode);
	parentpool.push_back(Dsttreeroot);

	bool lnodeflag = false;
	while (!pool.empty())
	{
		CartNode*node = pool.back();
		pool.pop_back();
		CartNode*newnode = new CartNode;
		copynode(node, newnode);
		if (!lnodeflag)
			parentpool.back()->rnode = newnode;
		else
			parentpool.back()->lnode = newnode;
		if (node->lnode != NULL)
		{
			_ASSERTE(node->rnode != NULL);
			if (lnodeflag)
				parentpool.pop_back();
			lnodeflag = false;
			pool.push_back(node->lnode);
			pool.push_back(node->rnode);

			parentpool.push_back(newnode);
		}
		else
		{
			if (lnodeflag)
				parentpool.pop_back();
			else
				lnodeflag = !lnodeflag;
		}
	}
	_ASSERTE(parentpool.empty());
	_ASSERTE(Dsttreeroot);
	return Dsttreeroot;
}

int cart::CART_classify(const Record rd, CartNode*treeroot)
{
	if (treeroot == NULL)
		treeroot = this->root;
	CartNode*node = treeroot;
	while (true)
	{
		if (node->lnode == NULL)
		{
			_ASSERTE(node->rnode == NULL);
			return node->label;
		}
		if (node->isSelectedAttriIDDiscrete)
		{
			if (find(node->selectedDiscreteAttriValues.begin(),
				node->selectedDiscreteAttriValues.end(),
				rd.discrete_attri[node->selectedAttriID])
				== node->selectedDiscreteAttriValues.end())
			{
				node = node->rnode;
			}
			else
			{
				node = node->lnode;
			}
		}
		else
		{
			if (rd.continuous_attti[node->selectedAttriID] >= node->continuousAttriPartitionValue)
			{
				node = node->rnode;
			}
			else
			{
				node = node->lnode;
			}
		}
	}
	//should not run here
	_ASSERTE(false);
}


void cart::CL_trim(const vector<Record>*validationdataset)
{
	vector<CartNode*>candidateBestTree;
	CartNode*curretroot = root;
	while (curretroot->lnode != NULL)//&&root->rnode!=NULL
	{
		vector<CartNode*>pool;
		pool.push_back(curretroot);
		double min_alpha = 10000000;
		CartNode*tobecut = NULL;
		while (!pool.empty())
		{
			CartNode*node = pool.back();
			pool.pop_back();
			if (node->lnode != NULL)
			{
				_ASSERTE(node->rnode != NULL);
				cal_alpha(node);
				if (node->alpha < min_alpha)
				{
					min_alpha = node->alpha;
					tobecut = node;
				}
				pool.push_back(node->rnode);
				pool.push_back(node->lnode);
			}
		}
		_ASSERTE(tobecut != NULL);
		//then delete tobecut's child and son node
		vector<CartNode*>alltodel, temppool;
		temppool.push_back(tobecut);
		while (!temppool.empty())
		{
			CartNode*nn = temppool.back();
			temppool.pop_back();
			alltodel.push_back(nn);
			if (nn->lnode != NULL)
			{
				_ASSERTE(nn->rnode != NULL);
				temppool.push_back(nn->lnode);
				temppool.push_back(nn->rnode);
			}
		}
		alltodel.erase(find(alltodel.begin(), alltodel.end(), tobecut));
		for (int i = 0; i < alltodel.size(); i++)
			delete alltodel[i];
		tobecut->lnode = tobecut->rnode = NULL;



		candidateBestTree.push_back(curretroot);
		CartNode*nextroot = NULL;
		nextroot = copytree(curretroot, nextroot);
		_ASSERTE(nextroot);
		curretroot = nextroot;
	}

	//get the best tree
	int minError = validationdataset->size();
	CartNode*besttree = NULL;
	int th = -1;
	vector<int>candidateBestTreeErrorNums;
	for (int i = 0; i < candidateBestTree.size(); i++)
	{
		int errorNum = 0;
		for (int j = 0; j < validationdataset->size(); j++)
		{
			errorNum += CART_classify((*validationdataset)[j],
				candidateBestTree[i]) == (*validationdataset)[j].label ? 0 : 1;
		}
		//error /= (*validationdataset).size();
		candidateBestTreeErrorNums.push_back(errorNum);
		if (errorNum < minError)
		{
			minError = errorNum;
			th = i;
		}
	}

	test(candidateBestTree[th]);

	double SE = sqrt(double(minError*(validationdataset->size() - minError)) / validationdataset->size());
	for (int i = candidateBestTree.size() - 1; i >= 0; i--)
	{
		if (candidateBestTreeErrorNums[i] <= minError + SE)
		{
			besttree = candidateBestTree[i];
			th = i;
			break;
		}
	}
	candidateBestTree.erase(candidateBestTree.begin() + th);
	for (int i = 0; i < candidateBestTree.size(); i++)
		destroyTree(candidateBestTree[i]);
	_ASSERTE(besttree != NULL);
	root = besttree;
	cout << "剪枝后在验证集上的错误为" << (double)candidateBestTreeErrorNums[th] / validationdataset->size() << endl;
}


void cart::cal_alpha(CartNode*node)
{
	_ASSERTE(node->lnode != NULL&&node->rnode != NULL);
	int max_nodelabel = -1;
	for (int i = 0; i < labelNums; i++)
	{
		if (node->labelcount[i] > max_nodelabel)
		{
			max_nodelabel = node->labelcount[i];
		}
	}
	double Rt = double(max_nodelabel) / node->record_number*node->record_number / traindataset->size();
	double RTt = 0;
	vector<CartNode*>leafpool = getLeaf(node);
	for (int i = 0; i < leafpool.size(); i++)
	{
		RTt += double(leafpool[i]->record_number - leafpool[i]->labelcount[leafpool[i]->label]) /
			traindataset->size();
	}
	node->alpha = (Rt - RTt) / (leafpool.size() - 1);
}


vector<cart::CartNode*>cart::getLeaf(CartNode*node)
{
	vector<CartNode*>leafpool, que;
	que.push_back(node);
	while (!que.empty())
	{
		CartNode*nn = que.back();
		que.pop_back();
		if (nn->lnode != NULL)
			que.push_back(nn->lnode);
		else
		{
			_ASSERTE(nn->rnode == NULL);
			if (find(leafpool.begin(), leafpool.end(), nn) == leafpool.end())
				leafpool.push_back(nn);
		}

		if (nn->rnode != NULL)
			que.push_back(nn->rnode);
		else
		{
			_ASSERTE(nn->lnode == NULL);
			if (find(leafpool.begin(), leafpool.end(), nn) == leafpool.end())
				leafpool.push_back(nn);
		}
	}
	return leafpool;
}


pair<vector<int>, vector<int>>cart::split_dataset(const int&selectedDiscreteAttriID,
	vector<int>&selected, const vector<int>&subdatasetbyID, const vector<Record>*dataset)
{
	vector<int>aa, bb;
	for (int i = 0; i < subdatasetbyID.size(); i++)
	{
		if (find(selected.begin(), selected.end(), (*dataset)[subdatasetbyID[i]].
			discrete_attri[selectedDiscreteAttriID]) == selected.end())
		{
			bb.push_back(subdatasetbyID[i]);
		}
		else
			aa.push_back(subdatasetbyID[i]);
	}
	return pair<vector<int>, vector<int>>(aa, bb);
}

pair<vector<int>, vector<int>>cart::split_dataset(const int&selectedContiuousAttriID,
	const double partition, const vector<int>&subdatasetbyID, const vector<Record>*dataset)
{
	vector<int>aa, bb;
	for (int i = 0; i < subdatasetbyID.size(); i++)
	{
		if ((*dataset)[subdatasetbyID[i]].continuous_attti[selectedContiuousAttriID] >= partition)
		{
			bb.push_back(subdatasetbyID[i]);
		}
		else
			aa.push_back(subdatasetbyID[i]);
	}
	return pair<vector<int>, vector<int>>(aa, bb);

}
set<set<int>>solu;
void select(set<int>&selected, vector<int>&remain, int toselect)
{
	if (selected.size() == toselect)
	{
		if (solu.find(selected) == solu.end())
		{
			solu.insert(selected);
			//for (set<int>::iterator it = selected.begin(); it != selected.end(); it++)
			//	cout << *it << ",";
			//cout << endl;
		}
		return;
	}
	for (int i = 0; i < remain.size(); i++)
	{
		vector<int> re = remain;
		set<int>se = selected;
		se.insert(re[i]);
		re.erase(re.begin() + i);
		select(se, re, toselect);
	}
}
void Combination(vector<int>remain, int toselect)//组合  
{
	solu.clear();
	set<int>selected;
	select(selected, remain, toselect);
	//cout << "共有" << solu.size() << "种组合" << endl;
}

vector<pair<vector<int>, vector<int>>>cart::make_two_heap(const int kk)
{
	vector<pair<vector<int>, vector<int>>>toret;
	int len = nums_of_value_each_discreteAttri[kk];
	set<set<int>>re;
	vector<int>remain;
	for (int i = 0; i < len; i++)
		remain.push_back(i);
	for (int i = 1; i < len / 2 + 1; i++)
	{
		Combination(vector<int>(remain), i);
		re.insert(solu.begin(), solu.end());
	}
	for (set<set<int>>::iterator it = re.begin(); it != re.end(); it++)
	{
		vector<int>aa, bb;//bb(*it);
		set_difference(it->begin(), it->end(),
			remain.begin(), remain.end(), inserter(aa, aa.begin()));
		bb.insert(bb.begin(), it->begin(), it->end());

		toret.push_back(pair<vector<int>, vector<int>>(aa, bb));
	}
	return toret;
}

void cart::create_root()
{
	if (root == NULL)
	{
		root = new CartNode;
		for (int i = 0; i < nums_of_value_each_discreteAttri.size(); i++)
			root->remianDiscreteAttriID.push_back(i);
		root->height = 1;

	}
}

int cart::allthesame(vector<int>&subdatasetbyID, const vector<Record>*dataset)
{
	vector<int>count(labelNums);
	int label = ((*dataset)[subdatasetbyID[0]]).label;
	for (int i = 1; i < subdatasetbyID.size(); i++)
		if (((*dataset)[subdatasetbyID[i]]).label != label)
			return -1;
	return label;
}

//build classify tree recursively
void cart::build_tree_classify(vector<int>&subdatasetbyID,
	CartNode*node, const vector<Record>*dataset)
{
	node->record_number = subdatasetbyID.size();
	double basegini = calGiniIndex(subdatasetbyID, dataset, node);
	int currentlabel = allthesame(subdatasetbyID, dataset);
	if (currentlabel >= 0)
	{
		node->label = currentlabel;
		return;
	}
	if (node->height >= CL_max_height)
	{
		node->label = labelNode(node);
		return;
	}
	node->label = labelNode(node);
	double mingini = 10000000000;
	int selected = -1;
	bool isSelectedDiscrete = true;
	vector<int>selectedDiscreteAttriValues;
	pair<vector<int>, vector<int>>splited_subdataset;
	bool lnodeDecreaseDiscreteAttri = false;//is node's lnode's discrete attribute nums decrease
	bool rnodeDecreaseDiscreteAttri = false;



	//for discrete features,calculate giniindex
	for (int i = 0; i < node->remianDiscreteAttriID.size(); i++)
	{
		if (nums_of_value_each_discreteAttri[node->remianDiscreteAttriID[i]] > 2)
		{
			vector<pair<vector<int>, vector<int>>>bipart = make_two_heap(node->remianDiscreteAttriID[i]);
			for (int j = 0; j < bipart.size(); j++)
			{
				pair<vector<int>, vector<int>>two_subdataset = split_dataset(
					node->remianDiscreteAttriID[i], bipart[i].first, subdatasetbyID, dataset);
				if (two_subdataset.first.size() > 0 && two_subdataset.second.size() > 0)
				{
					double gini1 = calGiniIndex(two_subdataset.first, dataset);
					double gini2 = calGiniIndex(two_subdataset.second, dataset);
					double gini = double(two_subdataset.first.size()) / subdatasetbyID.size()*gini1
						+ double(two_subdataset.second.size()) / subdatasetbyID.size()*gini2;
					if (gini < mingini)
					{
						if (bipart[i].first.size() == 1)
							lnodeDecreaseDiscreteAttri = true;
						else
							lnodeDecreaseDiscreteAttri = false;
						if (bipart[i].second.size() == 1)
							rnodeDecreaseDiscreteAttri = true;
						else
							rnodeDecreaseDiscreteAttri = false;
						mingini = gini;
						selected = node->remianDiscreteAttriID[i];
						splited_subdataset = two_subdataset;
						selectedDiscreteAttriValues = bipart[i].first;
					}
				}
			}
		}
		else
		{
			vector<int>aa;
			aa.push_back(0);
			pair<vector<int>, vector<int>>two_subdataset = split_dataset(node->remianDiscreteAttriID[i],
				aa, subdatasetbyID, dataset);
			if (two_subdataset.first.size() > 0 && two_subdataset.second.size() > 0)
			{
				double gini1 = calGiniIndex(two_subdataset.first, dataset);
				double gini2 = calGiniIndex(two_subdataset.second, dataset);
				double gini = double(two_subdataset.first.size()) / subdatasetbyID.size()*gini1
					+ double(two_subdataset.second.size()) / subdatasetbyID.size()*gini2;
				if (gini < mingini)
				{
					mingini = gini;
					selected = node->remianDiscreteAttriID[i];
					splited_subdataset = two_subdataset;
					lnodeDecreaseDiscreteAttri = true;
					rnodeDecreaseDiscreteAttri = true;
					selectedDiscreteAttriValues.clear();
					selectedDiscreteAttriValues.push_back(0);
				}
			}
		}
	}
	// 利用函数对象实现升降排序    
	struct CompNameEx{
		CompNameEx(bool asce, int k, const vector<Record>*dataset) : asce_(asce), kk(k), dataset(dataset)
		{}
		bool operator()(int const& pl, int const& pr)
		{
			return asce_ ? (*dataset)[pl].continuous_attti[kk] < (*dataset)[pr].continuous_attti[kk]
				: (*dataset)[pr].continuous_attti[kk] < (*dataset)[pl].continuous_attti[kk];
			// 《Eff STL》条款21: 永远让比较函数对相等的值返回false    
		}
	private:
		bool asce_;
		int kk;
		const vector<Record>*dataset;
	};

	//for continuous features,calculate giniindex
	double partitionpoint;
	for (int i = 0; i < ContinuousAttriNums; i++)
	{
		sort(subdatasetbyID.begin(), subdatasetbyID.end(),
			CompNameEx(true, i, dataset));
		for (int j = 0; j < subdatasetbyID.size() - 1; j++)
		{
			double partition = 0.5*(*dataset)[subdatasetbyID[j]].continuous_attti[i] +
				0.5*(*dataset)[subdatasetbyID[j + 1]].continuous_attti[i];
			pair<vector<int>, vector<int>>two_subdataset =
				split_dataset(i, partition, subdatasetbyID, dataset);
			if (two_subdataset.first.size() > 0 && two_subdataset.second.size() > 0)
			{
				double gini1 = calGiniIndex(two_subdataset.first, dataset);
				double gini2 = calGiniIndex(two_subdataset.second, dataset);
				double gini = double(two_subdataset.first.size()) / subdatasetbyID.size()*gini1
					+ double(two_subdataset.second.size()) / subdatasetbyID.size()*gini2 + log(double(subdatasetbyID.size() - 2) / dataset->size()) / log(2.0);
				if (gini < mingini)
				{
					partitionpoint = partition;
					mingini = gini;
					selected = i;
					isSelectedDiscrete = false;
					splited_subdataset = two_subdataset;
				}
			}
		}
	}

	//we have prune,so regardless of ginigain
	//double ginigain = basegini - mingini;//if not greater than miniumginigain;current node should not grow 

	if (splited_subdataset.first.size() > 0 && splited_subdataset.second.size() > 0)//&&ginigain>miniumginigain)
	{
		CartNode*lchild = new CartNode;
		CartNode*rchild = new CartNode;
		node->lnode = lchild;
		node->rnode = rchild;
		lchild->height = node->height + 1;
		rchild->height = node->height + 1;
		lchild->remianDiscreteAttriID = node->remianDiscreteAttriID;
		rchild->remianDiscreteAttriID = node->remianDiscreteAttriID;
		node->selectedAttriID = selected;
		if (isSelectedDiscrete)
		{
			if (lnodeDecreaseDiscreteAttri)
			{
				lchild->remianDiscreteAttriID.erase(find(lchild->
					remianDiscreteAttriID.begin(), lchild->remianDiscreteAttriID.end(), selected));
			}
			if (rnodeDecreaseDiscreteAttri)
			{
				rchild->remianDiscreteAttriID.erase(find(rchild->
					remianDiscreteAttriID.begin(), rchild->remianDiscreteAttriID.end(), selected));
			}
			node->selectedDiscreteAttriValues = selectedDiscreteAttriValues;
		}
		else
		{
			node->isSelectedAttriIDDiscrete = false;
			node->continuousAttriPartitionValue = partitionpoint;
		}
		//recursively call 	build_tree_classify()
		build_tree_classify(splited_subdataset.first, lchild, dataset);

		build_tree_classify(splited_subdataset.second, rchild, dataset);
	}
}


double cart::calGiniIndex(vector<int>&subdatasetbyID, const vector<Record>*dataset, CartNode*node)
{
	_ASSERTE(subdatasetbyID.size() > 0);
	_ASSERTE(dataset != NULL);
	vector<int>count;
	count.resize(labelNums);
	for (int i = 0; i < subdatasetbyID.size(); i++)
	{
		count[((*dataset)[subdatasetbyID[i]]).label]++;
	}
	if (node != NULL)
	{
		node->labelcount = count;
		node->record_number = subdatasetbyID.size();
	}
	vector<double> probalblity;
	probalblity.resize(labelNums);
	double re = 1;
	for (int i = 0; i < labelNums; i++)
	{
		probalblity[i] = double(count[i]) / subdatasetbyID.size();
		re -= pow(probalblity[i], 2);
	}
	_ASSERTE(re >= 0);
	return re;
}

int cart::labelNode(CartNode*node)
{
	int label = -1;
	double maxpro = 0;
	for (int i = 0; i < labelNums; i++)
	{
		double temppro = double(node->labelcount[i]) / node->record_number;
		temppro /= double(root->labelcount[i]) / root->record_number;
		if (temppro > maxpro)
		{
			maxpro = temppro;
			label = i;
		}
	}
	_ASSERTE(label >= 0);
	return label;
}






int split(const std::string& str, std::vector<std::string>& ret_, std::string sep = ",")
{
	if (str.empty())
	{
		return 0;
	}

	std::string tmp;
	std::string::size_type pos_begin = str.find_first_not_of(sep);
	std::string::size_type comma_pos = 0;

	while (pos_begin != std::string::npos)
	{
		comma_pos = str.find(sep, pos_begin);
		if (comma_pos != std::string::npos)
		{
			tmp = str.substr(pos_begin, comma_pos - pos_begin);
			pos_begin = comma_pos + sep.length();
		}
		else
		{
			tmp = str.substr(pos_begin);
			pos_begin = comma_pos;
		}

		if (!tmp.empty())
		{
			ret_.push_back(tmp);
			tmp.clear();
		}
	}
	return 0;
}





//说明,因为education,workclass,marital-status,occupation,native country属性太多,不作考虑
void cart::load_adult_dataset()
{
	vector<Record>*traindataset;//as it's name
	vector<Record>*validatedataset;
	string filename = "adult.data";
	ifstream infile(filename.c_str());
	string temp;
	cout << endl;
	int count = 0;
	//vector<vector<std::string>>ss;
	traindataset = new vector < Record > ;
	validatedataset = new vector < Record > ;
	this->traindataset = traindataset;
	this->validatedataset = validatedataset;
	testdataset = new vector < Record > ;
	//Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked

	/*map<string, int>workclass;
	workclass["Private"] = 0;
	workclass["Self-emp-not-inc"] = 1;
	workclass["Self-emp-inc"] = 2;
	workclass["Federal-gov"] = 3;
	workclass["Local-gov"] = 4;
	workclass["State-gov"] = 5;
	workclass["Without-pay"] = 6;
	workclass["Never-worked"] = 7;*/

	//education: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th,
	// 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.

	/*map<string, int>education;
	education["Bachelors"] = 0;
	education["Some-college"] = 1;
	education["11th"] = 2;
	education["HS-grad"] = 3;
	education["Prof-school"] = 4;
	education["Assoc-acdm"] = 5;
	education["Assoc-voc"] = 6;
	education["9th"] = 7;
	education["7th-8th"] = 8;
	education["12th"] = 9;
	education["Masters"] = 10;
	education["1st-4th"] = 11;
	education["10th"] = 12;
	education["Doctorate"] = 13;
	education["5th-6th"] = 14;
	education["Preschool"] = 15;
	*/
	//marital-status: Married-civ-spouse, Divorced, Never-married, Separated, Widowed,
	// Married-spouse-absent, Married-AF-spouse.
	/*map<string, int>marital_status;
	marital_status["Married-civ-spouse"] = 0;
	marital_status["Divorced"] = 1;
	marital_status["Never-married"] = 2;
	marital_status["Separated"] = 3;
	marital_status["Widowed"] = 4;
	marital_status["Married-spouse-absent"] = 5;
	marital_status["Married-AF-spouse"] = 6;*/

	//occupation: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, 
	//Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing,
	// Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.
	/*map<string, int>occupation;
	occupation["Tech-support"] = 0;
	occupation["Craft-repair"] = 1;
	occupation["Other-service"] = 2;
	occupation["Sales"] = 3;
	occupation["Exec-managerial"] = 4;
	occupation["Prof-specialty"] = 5;
	occupation["Handlers-cleaners"] = 6;
	occupation["Machine-op-inspct"] = 7;
	occupation["Adm-clerical"] = 8;
	occupation["Farming-fishing"] = 9;
	occupation["Transport-moving"] = 10;
	occupation["Priv-house-serv"] = 11;
	occupation["Protective-serv"] = 12;
	occupation["Armed-Forces"] = 13;
	*/

	//relationship: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.

	map<string, int>relationship;
	relationship["Wife"] = 0;
	relationship["Own-child"] = 1;
	relationship["Husband"] = 2;
	relationship["Not-in-family"] = 3;
	relationship["Other-relative"] = 4;
	relationship["Unmarried"] = 5;

	//race: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.

	map<string, int>race;
	race["White"] = 0;
	race["Asian-Pac-Islander"] = 1;
	race["Amer-Indian-Eskimo"] = 2;
	race["Other"] = 3;
	race["Black"] = 4;

	//sex: Female, Male.
	map<string, int>sex;
	sex["Female"] = 0;
	sex["Male"] = 1;

	//native-country: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, 
	//Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran,
	// Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, 
	//Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia,
	// Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, 
	//Trinadad&Tobago, Peru, Hong, Holand-Netherlands.
	map<string, int>label;
	label["<=50K"] = 0;
	label[">50K"] = 1;


	while (getline(infile, temp) && count < 7000)
	{

		Record rd;
		rd.continuous_attti.resize(6);
		rd.discrete_attri.resize(3);
		//cout << temp << endl;

		std::vector<std::string>re;
		split(temp, re, std::string(", "));
		bool desert = false;
		if (re.size() == 15)
		{

			/*age: continuous.
			workclass: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.
			fnlwgt: continuous.
			education: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.
			education-num: continuous.
			marital-status: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.
			occupation: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.
			relationship: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.
			race: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.
			sex: Female, Male.
			capital-gain: continuous.
			capital-loss: continuous.
			hours-per-week: continuous.
			native-country: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.*/


			//age continuous
			rd.continuous_attti[0] = atoi(re[0].c_str());

			//workclass discrete
			/*if (workclass.find(re[1]) != workclass.end())
				rd.discrete_attri[0] = workclass[re[1]];
				else
				desert=true;*/

			//fnlwgt: continuous
			rd.continuous_attti[1] = atoi(re[2].c_str());

			//education: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.
			/*if (education.find(re[3]) != education.end())
				rd.discrete_attri[1] = education[re[3]];
				else
				desert=true;*/

			//education-num: continuous.
			rd.continuous_attti[2] = atoi(re[4].c_str());

			//marital-status
			/*if (marital_status.find(re[5]) != marital_status.end())
				rd.discrete_attri[1] = marital_status[re[5]];
				else
				desert=true;*/

			//relationship: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.
			if (relationship.find(re[7]) != relationship.end())
				rd.discrete_attri[0] = relationship[re[7]];
			else
				desert = true;

			//race
			if (race.find(re[8]) != race.end())
				rd.discrete_attri[1] = race[re[8]];
			else
				desert = true;

			//sex
			if (sex.find(re[9]) != sex.end())
				rd.discrete_attri[2] = sex[re[9]];
			else
				desert = true;

			//capital-gain: continuous.
			rd.continuous_attti[3] = atoi(re[10].c_str());

			//capital-loss: continuous.

			rd.continuous_attti[4] = atoi(re[11].c_str());
			//hours-per-week: continuous
			rd.continuous_attti[5] = atoi(re[12].c_str());

			if (label.find(re[14]) != label.end())
				rd.label = label[re[14]];
			else
				desert = true;
			if (!desert)
				if (count < 3500)
				{
					traindataset->push_back(rd);
				}
				else if (count < 4500)
				{
					validatedataset->push_back(rd);
				}
				else
					testdataset->push_back(rd);
		}
		count++;
	}
	ContinuousAttriNums = 6;
	labelNums = 2;
	int aa[3] = { 6, 5, 2 };
	nums_of_value_each_discreteAttri.push_back(6);
	nums_of_value_each_discreteAttri.push_back(5);
	nums_of_value_each_discreteAttri.push_back(2);


}

int _tmain(int argc, _TCHAR* argv[])
{

	cart cart;
	cart.load_adult_dataset();
	cart.CART_trian();
	cart.test();
	system("pause");
	return 0;
}

可能不太完善,大体框架是这样了,具体细节可能处理不好。欢迎大家指点。



遗留问题:
先验概率和分类平衡
误分类成本的引入
支持权重,对于不同的样本赋予不同的权重值
动态特征构架
值敏感学习
概率树
回归树细节
模型树


版权声明:

简单易学的机器学习算法——分类回归树CART

一、树回归的概念 二、

数据挖掘十大经典算法--CART: 分类与回归树

一、决策树的类型  在数据挖掘中,决策树主要有两种类型: 分类树 的输出是样本的类标。 回归树 的输出是一个实数 (例如房子的价格,病人呆在医院的时间等)。 术语分类和回归树 (CART) 包含了...

CART回归树对于特征的处理

CART算法的重要基础包含以下三个方面: (1)二分(Binary Split):在每次判断过程中,都是对观察变量进行二分。 CART算法采用一种二分递归分割的技术,算法总是将当前样本集分割为两...

CART分类回归树

这一篇主要是CART,有几个重点的词语先写下来,重点哦:基尼指数(Gini index,跟经济学的基尼系数长这么像,八杆子打得着吗)、最小二乘回归树(least squares regression ...
  • learnee
  • learnee
  • 2015年11月20日 23:03
  • 4576

机器学习算法之CART(分类和回归树)

CART算法介绍: 分类和回归树(CART)是应用广泛的决策树学习方法。CART同样由特征选择,树的生成和减枝组成,既可以用于分类也可以用于回归。CART的生成就是递归的构建二叉决策树的过程。对回归...

CART算法原理及实现

1.算法介绍     分类回归树算法:CART(Classification And Regression Tree)算法采用一种二分递归分割的技术,将当前的样本集分为两个子样本集,使得生成的的每个...

Cart分类树算法之原理篇

 CART算法由一下两步组成: (1)决策树的生成:基于训练数据集生成决策树,生成的决策树要尽量大; (2)决策树的剪枝:用验证数据集对已生成的树进行剪枝并选择最优子树,这时用损失函数最小作...

决策树之CART算法

在之前介绍过决策树的ID3算法实现,今天主要来介绍决策树的另一种实现,即CART算法。   Contents      1. CART算法的认识    2. CART算法的原理    3. CART算...

简单介绍树回归的算法原理

线性回归方法可以有效的拟合所有样本点(局部加权线性回归除外)。当数据拥有众多特征并且特征之间关系十分复杂时,构建全局模型的想法一个是困难一个是笨拙。此外,实际中很多问题为非线性的,例如常见到的分段函数...

Scikit-Learn与回归树

回归算法原理 CART(Classification and Regression Tree)算法是目前决策树算法中最为成熟的一类算法,应用范围也比较广泛。它既可以用于分类。 西方预测理论一...
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:CART分类与回归树的原理与实现
举报原因:
原因补充:

(最多只允许输入30个字)