简介及相关链接:
在b站上看的一个视频B站xgboost课程。讲了xgboost相关的理论知识后,用kaggle上的Otto Group Product Classification数据集进行分类任务。
1、首先导入模块
from xgboost import XGBClassifier
import xgboost as xgb
import numpy as np
import pandas as pd
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import log_loss #-log似然损失
from matplotlib import pyplot
import seaborn as sns
%matplotlib inline
2、读取数据,用到pandas里面pd.read_csv()。之前读取碰到过读取json数据,用pd.read_json()
path='D:\\data\\kaggle\\otto-group-product-classification-challenge'
train=pd.read_csv(path+'\\train.csv')
test=pd.read_csv(path+'\\test.csv')
3、一般在处理问题前都要看下各种特征的分布。该数据集的变量确定,数据特征单一,且经过脱敏(肉眼看不出来他的实际意义),在特征工程方面少做些工作,主要精力在参数调优。
先看一下target的分布,看看各类样本是否均衡。
sns.countplot(train.target)
pyplot.xlabel('target')
pyplot.ylabel('number of occurrences')
4、分类不是很均衡,各类数据的个数差异很大,所以在交叉检验的时候要按各类比例抽取。所以这里用StratifiedKFold,在每折采样时各类样本按比例采样,确保训练集,测试集中各类别样本的比例与原始数据集中相同。
y_train=train['target']
y_train=y_train.map(lambda s:s[6:])
y_train=y_train.map(lambda s:int(s)-1)
train=train.drop(['id','target'],axis=1)#维度1,丢掉id和target列。target作为label单独存储,id不作为特征
x_train=np.array(train)
kfold=StratifiedKFold(n_splits=5,shuffle=True,random_state=3)
5、默认参数,此时学习率为0.1,比较大,观察弱分类器数目的大致范围。(采用默认参数,看模型是过拟合还是欠拟合)
def modelfit(alg,x_train,y_train,useTrainCV=True,cv_folds=None,early_stopping_rounds=50):
if useTrainCV:
xgb_param=alg.get_xgb_params()
xgb_param['num_class']=9
xgtrain=xgb.DMatrix(x_train,label=y_train)
cvresult=xgb.cv(xgb_param,xgtrain,num_boost_round=alg.get_params()['n_estimators'],folds=cv_folds,
metrics='mlogloss',early_stopping_rounds=early_stopping_rounds)
n_estimators=cvresult.shape[0]
alg.set_params(n_estimators=n_estimators)
print(cvresult)
cvresult.to_csv(path+'\\my_preds_4_1.csv',index_label='n_estimators')
#plot
test_means=cvresult['test-mlogloss-mean']#平均
test_stds=cvresult['test-mlogloss-std']#标准差
train_means=cvresult['train-mlogloss-mean']
train_stds=cvresult['train-mlogloss-std']
x_axis=range(0,n_estimators)
pyplot.errorbar(x_axis,test_means,yerr=test_stds,label='test')
pyplot.errorbar(x_axis,train_means,yerr=train_stds,label='train')
pyplot.title('XGBoost n_estimators vs Log Loss')
pyplot.xlabel('n_estimators')
pyplot.ylabel('Log Loss')
pyplot.savefig(path+'\\n_estimators.png')
#在数据上训练模型
alg.fit(x_train,y_train,eval_metric='mlogloss')
#在训练集上预测
train_predprob=alg.predict_proba(x_train)
logloss=log_loss(y_train,train_predprob)
#print
print("logloss of train:")
print(logloss)
xgbl=XGBClassifier(
learning_rate=0.1,
n_estimators=1000,
max_depth=5,
min_child_weight=1,
gamma=0,
subsample=0.3,
colsample_bytree=0.8,
colsample_bylevel=0.7,
objective='multi:softprob',#多分类问题
seed=3)
modelfit(xgbl,x_train,y_train,cv_folds=kfold)
这一步好花费很长时间:在弱分类器在742的时候停止,所以接下来调参,我们将弱分类器个数固定,调整每棵树的深度和叶子节点权重。
6、调整树的参数:max_depth&min_child_weight。粗调参数的步长为2,下一步是在粗调的最佳参数周围,将步长降为1,进行精细调整¶
max_depth=range(3,10,2)
min_child_weight=range(1,6,2)
param_test2_1=dict(max_depth=max_depth,min_child_weight=min_child_weight)
param_test2_1
xgb2_1=XGBClassifier(
learning_rate=0.1,
n_estimators=742, #这里用上面找到的最优弱分类器个数
max_depth=5,
min_child_weight=1,
gamma=0,
subsample=0.3,
colsample_bytree=0.8,
colsample_bylevel=0.7,
objective='multi:softprob',#多分类问题
seed=3)
qsearch2_1=GridSearchCV(xgb2_1,param_grid=param_test2_1,scoring='neg_log_loss',n_jobs=-1,cv=kfold)
qsearch2_1.fit(x_train,y_train)
qsearch2_1.grid_scores_,qsearch2_1.best_params_,qsearch2_1.best_score_
7、可视化,寻找各个参数和损失函数的关系
#可视化
print("best:{} using {}".format(qsearch2_1.best_score_,qsearch2_1.best_params_))
test_means=qsearch2_1.cv_results_['mean_test_score']
test_stds=qsearch2_1.cv_results_['std_test_score']
train_means=qsearch2_1.cv_results_['mean_train_score']
train_stds=qsearch2_1.cv_results_['std_train_score']
pd.DataFrame(qsearch2_1.cv_results_).to_csv(path+'\\my_preds_maxdepth_min_child_weight')
test_scores=np.array(test_means).reshape(len(max_depth),len(min_child_weight))
train_scores=np.array(train_means).reshape(len(max_depth),len(min_child_weight))
for i,value in enumerate(max_depth):
pyplot.plot(min_child_weight,-test_scores[i],label='test_max_depth')
pyplot.legend()
pyplot.xlabel('max_depth')
pyplot.ylabel('log_loss')
pyplot.savefig('max_depth_vs_min_child_weight_1.png')
8、微调,步长设为1
max_depth=[6,7,8]
min_child_weight=[4,5,6]
通过以上步骤得到较优参数值max_depth和min_child_weight
调整max_depth=6和min_child_weight=4后,再次调整n_estimators
gamma参数调整
正则参数调整
降低学习率,调整树的数目
大家看开头的视频,讲的很详细!