使用Adaboost方法,以一级决策树树桩(Stump)为基础建立弱分类器,形式为:(feature, threshold, positive/negtive)。
设定的最大轮数为20,最终使用了16轮就使总误差就降到了0,得到了准确率100%的分类器。
分类的数据包括:business,sports,auto三类。
由于写的Adaboost是二分类模型,所以将语料分为business和非business两类划分,分别为1和-1。feature的选取就是分词的结果,选择两个字以上的词。threshold也只做是否在文档中出现的0/1划分。
最终结果如下,可以看出,选择的特征是有重复的,如“体育”特征,被选择了好几次,每次的树桩也相同。在中间训练过程中,总体的分类正确率也是有波动和反复的。
从选择的特征词可以看出,前半部分的特征词还比较靠谱,后半部分就不一定了,比如“编辑”,“第一”,“来源”什么的,可见模型是有一定的过拟合的。
result stumplist is: [(2404, 0, -1), (32590, 0, -1), (19569, 0, 1), (12171, 0, 1), (29965, 0, -1), (15667, 0, 1), (12171, 0, 1), (32687, 0, -1), (25944, 0, 1), (12171, 0, 1), (32890, 0, -1), (2404, 0, -1), (4840, 0, -1), (15667, 0, 1), (9642, 0, 1), (8630, 0, -1)]
result features are:
财经
股票
汽车
体育
银行
编辑
体育
作者
责任
体育
教育
财经
指出
编辑
第一
来源
单个树桩的训练过程中的输出如下:
-------------------- Train stump round 9 --------------------
>>train featureindex is 0
get new min stump (0, 0, -1) feature is 石块 , error is 0.408032946166
get new min stump (1, 0, -1) feature is 基建 , error is 0.402227800978
get new min stump (12, 0, -1) feature is 律师 , error is 0.397832761677
get new min stump (25, 0, -1) feature is 合理 , error is 0.385203075673
get new min stump (108, 0, -1) feature is 首席 , error is 0.382699443012
get new min stump (316, 0, -1) feature is 证券 , error is 0.344348559772
>>train featureindex is 1000
>>train featureindex is 2000
get new min stump (2258, 0, -1) feature is 政府 , error is 0.341484530448
get new min stump (2404, 0, -1) feature is 财经 , error is 0.27536370978
>>train featureindex is 3000
>>train featureindex is 4000
>>train featureindex is 5000
>>train featureindex is 6000
>>train featureindex is 7000
>>train featureindex is 8000
>>train featureindex is 9000
>>train featureindex is 10000
>>train featureindex is 11000
>>train featureindex is 12000
get new min stump (12171, 0, 1) feature is 体育 , error is 0.261865947444
>>train featureindex is 13000
>>train featureindex is 14000
>>train featureindex is 15000
>>train featureindex is 16000
>>train featureindex is 17000
>>train featureindex is 18000
>>train featureindex is 19000
>>train featureindex is 20000
>>train featureindex is 21000
>>train featureindex is 22000
>>train featureindex is 23000
>>train featureindex is 24000
>>train featureindex is 25000
>>train featureindex is 26000
>>train featureindex is 27000
>>train featureindex is 28000
>>train featureindex is 29000
>>train featureindex is 30000
>>train featureindex is 31000
>>train featureindex is 32000
>>train featureindex is 33000
>>train featureindex is 34000
>>train featureindex is 35000
>>train featureindex is 36000
>>train featureindex is 37000
>>train featureindex is 38000
this round stump is (12171, 0, 1)
totallabelerror is 0.00293542074364
在写Adaboost程序的时候,要注意一下,在每一轮中,训练树桩的时候,根据新的weightlist来训练单个树桩,此时计算单个树桩的weightlist加权整体错误率,据此来选择最好的树桩;在本轮训练树桩结束后(即已选择好本轮最好树桩,跳出来进入下一轮训练之前),需要计算整个Adaboost模型的分类错误率,此时需要将之前的所有树桩的加权值(即cm值,或有的材料中叫alpha值)的预测结果和本轮树桩的预测结果叠加,之后使用sign函数进行分类(之前的总的预测结果可以保留,这样每次单独加上本轮树桩的更新值即可,无需再从头计算),就得到了整体的分类正确率。这两个分类正确率是不同的,分别是为了选择最优树桩和训练整个Adaboost模型,前者只需要对新的weightlist进行加权即可,后者需要考虑之前得到的所有树桩的预测值。我之前就弄混了,所以写出来的总是预测的不对。
Python版的Adaboost算法代码如下:
# /usr/bin/env python
# -*- coding: utf-8 -*-
from numpy import *
import os
def getwordset(doclist):
wordset = set(range(0))
docwordset = []
for doc in doclist:
f = open(os.path.join(os.getcwd(), DIR, doc))
content = f.read()
words = content.split(' ')
words = [word.strip() for word in words if len(word) > 4]
wordset |= set(words)
docwordset.append(list(set(words)))
f.close()
return list(wordset), docwordset
def savefeaturelist(featurelist):
f = open('featurelist','w')
for feature in featurelist:
f.write(feature + ' ')
f.close()
def classifydoclist(stump, doclist, weightlist):
labellist = []
for i in range(len(doclist)):
#print 'classify doc',doclist[i]
exist = 0
words = docfeaturelist[i]
if featurelist[stump[0]] in words: #Notice:'feature in words', NOT 'featureindex in words'
exist = 1
#append classify doc label
if exist == stump[1]:
labellist.append(stump[2])
else:
labellist.append(-1 * stump[2])
return labellist
def trainstump(doclist, featurelist, weightlist):
#stump is (featureindex, threshold, positive/negtive)
minstump = (featurelist[0], 0, 0)
minerror = 1.0
minlabellist = []
for featureindex in range(len(featurelist)):
if featureindex % 1000 == 0: print '>>train featureindex is', featureindex
for threshold in [0, 1]:
for symbol in [-1, 1]:
stump = (featureindex, threshold, symbol)
#print 'train stump',stump
labellist = classifydoclist(stump, doclist, weightlist)
error = float(abs(array(doclabellist) - array(labellist))/2 * mat(weightlist).T)
#print 'featureindex',featureindex,'error is',error
if error < minerror:
minstump = stump
minerror = error
minlabellist = labellist
print 'get new min stump',stump,'feature is',featurelist[minstump[0]],', error is',error
if minerror == 0.0: break
return minstump, minerror, minlabellist
def getcm(error):
error = max(error, 1e-16)
return log((1.0-error)/error) #sometime this will muptiply 0.5.
def updateweightlist(weightlist, cm, labellist):
#print 'original weightlist is:\n',weightlist
#minus = list(abs(array(doclabellist) - array(labellist))/2)
minus = getclassifydiff(labellist)
#print 'minus is:\n',minus
weightlist = [weightlist[i] * exp(cm * minus[i]) for i in range(len(weightlist))]
#print 'new weightlist0 is:\n',weightlist
weightlist = [weight/sum(weightlist) for weight in weightlist]
#print 'new weightlist is:\n',weightlist
return weightlist
def sign(plist):
result = [-1 for i in range(len(plist))]
for i in range(len(plist)):
if plist[i] > 0:
result[i] = 1
return result
def getclassifydiff(plabellist):
return list(abs(array(doclabellist) - array(plabellist))/2)
def getclassifyerror(plabellist):
#print 'predict labellist is',plabellist
minus = getclassifydiff(plabellist)
#print 'minus is',minus
return 1.0 * minus.count(1) / len(plabellist)
def traindata(doclist, featurelist):
stumplist = []
cmlist = []
max_k = 20
totallabelpredict = array([0.0 for i in range(len(doclabellist))])
weightlist = [1.0/len(doclist) for i in range(len(doclist))]
for i in range(max_k):
print '\n','-' * 20,'Train stump round',i, '-' * 20
#print 'new weightlist is',weightlist
stump, error, labellist = trainstump(doclist, featurelist, weightlist)
print 'this round stump is',stump
cm = getcm(error)
stumplist.append(stump)
cmlist.append(cm)
#check total predict result error.
#print 'cm is',cm
totallabelpredict += cm * array(labellist)
#print 'doclabellist is',doclabellist
#print 'totallabelpredict is',totallabelpredict
totallabelerror = getclassifyerror(sign(totallabelpredict))
print 'totallabelerror is',totallabelerror
#print 'cm is',cm
if totallabelerror == 0.0:
break
#update weight list.
weightlist = updateweightlist(weightlist, cm, labellist)
print '\n\nTrain data done!'
#save model to file
model = open('Adaboostmodel','w')
model.write('cmlist:\n')
model.write(str(cmlist)+'\n')
model.write('stumplist:\n')
model.write(str(stumplist)+'\n')
model.write('stump features are:\n')
print 'result stumplist is:', stumplist
print 'result features are:'
for s in stumplist:
print featurelist[s[0]]
model.write(str(featurelist[s[0]])+'\n')
model.close()
print 'save model to file done!'
def getdoclabellist(doclist):
'''sports is -1, business is 1.
two-class classify(business and not-business).'''
labellist = [-1 for i in range(len(doclist))]
for i in range(len(doclist)):
if 'business' in doclist[i]:
labellist[i] = 1
return labellist
def adaboost():
global DIR
global doclist, featurelist, docfeaturelist, doclabellist
DIR = 'news'
print 'Arthur adaboost test begin...'
print 'doc path DIR is:',DIR
doclist = os.listdir(os.path.join(os.getcwd(), DIR))
doclist.sort()
print 'total doc size:',len(doclist)
#Get doc real label. train stump with this list!
doclabellist = getdoclabellist(doclist)
featurelist, docfeaturelist = getwordset(doclist)
print 'total feature size:',len(featurelist)
#train data to get stumps.
traindata(doclist, featurelist)
if __name__ == '__main__':
adaboost()
Adaboost树桩训练完毕,之后在预测的时候直接使用cm作为每个树桩的权重,之后对整体的预测结果使用sign函数即可进行分类预测。