用xgboost获取特征重要性原理及实践

1.xgboost对特征重要性排序的原理

xgboost根据结构分数的增益情况计算出来选择哪个特征作为分割点,而某个特征的重要性就是它在所有树中出现的次数之和。也就是说一个属性越多的被用来在模型中构建决策树，它的重要性就相对越高
2 xgboost特征重要性排序的方法

1. xgboost可以通过get_score获取特征重要性
   for importance_type in (‘weight’, ‘gain’, ‘cover’, ‘total_gain’, ‘total_cover’):
   print(’%s: ’ % importance_type, bst.get_score(importance_type=importance_type))
   weight - 该特征在所有树中被用作分割样本的特征的次数。
   gain - 在所有树中的平均增益。
   cover - 在树中使用该特征时的平均覆盖范围。(还不是特别明白)

2. 利用plot_importance画出各个特征的重要性排序

3. 可以通过测试多个阈值，借助衡量分类器优劣的指标，来从特征重要性中选择特征。
   下面利用kaggle的heartdisease数据实证分析

import numpy as np
import pandas as pd

from sklearn.model_selection import train_test_split,StratifiedKFold,train_test_split,GridSearchCV
from sklearn.metrics import accuracy_score, confusion_matrix, mean_squared_error,roc_auc_score
from xgboost import plot_importance
from matplotlib import pyplot as plt
import xgboost as xgb
#利用xgb.train中的get_score得到weight，gain，以及cover
params={        'max_depth':7,
                'n_estimators':80,
                'learning_rate':0.1, 
                'nthread':4,
                'subsample':1.0,
                'colsample_bytree':0.5,
                'min_child_weight' : 3,
                'seed':1301}
bst = xgb.train(params, xgtrain, num_boost_round=1)
for importance_type in ('weight', 'gain', 'cover', 'total_gain', 'total_cover'):
    print('%s: ' % importance_type, bst.get_score(importance_type=importance_type))
import graphviz
xgb.plot_tree(bst)
plt.show()

得到的结果如下：
weight: {‘slope’: 2, ‘sex’: 2, ‘age’: 7, ‘chol’: 13, ‘trestbps’: 9, ‘restecg’: 2}
gain: {‘slope’: 4.296458304, ‘sex’: 2.208011625, ‘age’: 0.8395543860142858, ‘chol’: 0.6131722695384615, ‘trestbps’: 0.49512829022222227, ‘restecg’: 0.679761901}
cover: {‘slope’: 116.5, ‘sex’: 106.0, ‘age’: 24.714285714285715, ‘chol’: 22.846153846153847, ‘trestbps’: 18.555555555555557, ‘restecg’: 18.0}
total_gain: {‘slope’: 8.592916608, ‘sex’: 4.41602325, ‘age’: 5.8768807021, ‘chol’: 7.971239503999999, ‘trestbps’: 4.456154612000001, ‘restecg’: 1.359523802}
total_cover: {‘slope’: 233.0, ‘sex’: 212.0, ‘age’: 173.0, ‘chol’: 297.0, ‘trestbps’: 167.0, ‘restecg’: 36.0}

from sklearn.feature_selection import SelectFromModel
model = xgb.XGBClassifier()
model.fit(X_train, y_train)
#plot_importance；利用plot_importance画出各个特征的重要性排序
from xgboost import plot_importance
plot_importance(model)
plt.show()

得到结果如下：

#我们可以通过测试多个阈值，来从特征重要性中选择特征。具体而言，每个输入变量的特征重要性，本质上允许我们#通过重要性来测试每个特征子集。
# make predictions for test data and evaluate
y_pred = model.predict(X_test)
predictions = [round(value) for value in y_pred]
accuracy = accuracy_score(y_test, predictions)
print("Accuracy: %.2f%%" % (accuracy * 100.0))
# Fit model using each importance as a threshold
thresholds = np.sort(model.feature_importances_)
for thresh in thresholds:
	# select features using threshold
	selection = SelectFromModel(model, threshold=thresh, prefit=True)
	select_X_train = selection.transform(X_train)
	# train model
	selection_model = xgb.XGBClassifier()
	selection_model.fit(select_X_train, y_train)
	# eval model
	select_X_test = selection.transform(X_test)
	y_pred = selection_model.predict(select_X_test)
	predictions = [round(value) for value in y_pred]
	accuracy = accuracy_score(y_test, predictions)
	print("Thresh=%.3f, n=%d, Accuracy: %.2f%%" % (thresh, select_X_train.shape[1], accuracy*100.0))

Accuracy: 84.62%
Thresh=0.025, n=13, Accuracy: 84.62%
Thresh=0.026, n=12, Accuracy: 80.22%
Thresh=0.026, n=11, Accuracy: 79.12%
Thresh=0.028, n=10, Accuracy: 76.92%
Thresh=0.032, n=9, Accuracy: 78.02%
Thresh=0.036, n=8, Accuracy: 80.22%
Thresh=0.041, n=7, Accuracy: 76.92%
Thresh=0.066, n=6, Accuracy: 76.92%
Thresh=0.085, n=5, Accuracy: 84.62%
Thresh=0.146, n=4, Accuracy: 80.22%
Thresh=0.151, n=3, Accuracy: 76.92%
Thresh=0.163, n=2, Accuracy: 74.73%
Thresh=0.174, n=1, Accuracy: 78.02%
由上述结果可以看出，随着阈值的增大，特征数目的减少，精确度有减小的趋势，一般情况下用交叉验证作为模型评估方案可能是更有用的策略。这在下一篇的xgb.cv中将实现。xgb.cv