python 梯度提升树_梯度提升树（Gradient Boosted Trees）：模型理解

Note:我们的 TensorFlow 社区翻译了这些文档。因为社区翻译是尽力而为， 所以无法保证它们是最准确的，并且反映了最新的

官方英文文档。如果您有改进此翻译的建议， 请提交 pull request 到

tensorflow/docs GitHub 仓库。要志愿地撰写或者审核译文，请加入

docs-zh-cn@tensorflow.org Google Group。

对于梯度提升模型(Gradient Boosting model)的端到端演示(end-to-end walkthrough)，请查阅在 Tensorflow 中训练提升树(Boosted Trees)模型。在本教程中，您将：

学习到如何对提升树模型(Boosted Trees model)进行局部解释和全局解释

了解到提升树模型在数据集上的表现。

如何对提升树模型(Boosted Trees model)进行局部解释和全局解释

局部可解释性指模型的预测在单一样例层面上的理解程度，而全局可解释性指模型作为一个整体的理解能力。这种技术可以帮助使用机器学习的人在模型开发阶段检测偏差(bias)和bug。

对于局部可解释性，您将了解到如何创造并可视化每个实例(per-instance)的贡献度。区别于特征重要性，这种贡献被称为 DFCs(定向特征贡献，directional feature contributions)。

对于全局可解释性，您将学习并可视化基于增益的特征重要性(gain-based feature importances)，排列特征重要性(permutation feature importances)和总DFCs。

加载泰坦尼克数据集(titanic)

本教程使用泰坦尼克数据集，旨在已知乘客的性别，年龄和客舱等级等特征的情况下预测的存活率。

import numpy as np

import pandas as pd

from IPython.display import clear_output

# 加载数据集。

dftrain = pd.read_csv('https://storage.googleapis.com/tf-datasets/titanic/train.csv')

dfeval = pd.read_csv('https://storage.googleapis.com/tf-datasets/titanic/eval.csv')

y_train = dftrain.pop('survived')

y_eval = dfeval.pop('survived')import tensorflow as tf

tf.random.set_seed(123)

TensorFlow 2.x selected.

有关特征的描述，请参阅之前的教程。

创建特征列, 输入函数并训练 estimator

数据预处理

特征处理，使用原始的数值特征和独热编码(one-hot-encoding)处理过的非数值特征(如性别，舱位)别建立数据集。

fc = tf.feature_column

CATEGORICAL_COLUMNS = ['sex', 'n_siblings_spouses', 'parch', 'class', 'deck',

'embark_town', 'alone']

NUMERIC_COLUMNS = ['age', 'fare']

def one_hot_cat_column(feature_name, vocab):

return fc.indicator_column(

fc.categorical_column_with_vocabulary_list(feature_name,

vocab))

feature_columns = []

for feature_name in CATEGORICAL_COLUMNS:

# 需要使用独热编码(one-hot-encoding)处理非数值特征。

vocabulary = dftrain[feature_name].unique()

feature_columns.append(one_hot_cat_column(feature_name, vocabulary))

for feature_name in NUMERIC_COLUMNS:

feature_columns.append(fc.numeric_column(feature_name,

dtype=tf.float32))

构建输入 pipeline

使用 API tf.data 中的 from_tensor_slices 方法建立输入方程来从 Pandas 中直接读取数据。

# 当数据集小的时候，将整个数据集作为一个 batch。

NUM_EXAMPLES = len(y_train)

def make_input_fn(X, y, n_epochs=None, shuffle=True):

def input_fn():

dataset = tf.data.Dataset.from_tensor_slices((X.to_dict(orient='list'), y))

if shuffle:

dataset = dataset.shuffle(NUM_EXAMPLES)

# 训练时让数据迭代尽可能多次 (n_epochs=None)。

dataset = (dataset

.repeat(n_epochs)

.batch(NUM_EXAMPLES))

return dataset

return input_fn

# 训练并评估输入函数。

train_input_fn = make_input_fn(dftrain, y_train)

eval_input_fn = make_input_fn(dfeval, y_eval, shuffle=False, n_epochs=1)

训练模型

params = {

'n_trees': 50,

'max_depth': 3,

'n_batches_per_layer': 1,

# 为了得到 DFCs，请设置 center_bias = True。这将强制

# 模型在使用特征(例如：回归中训练集标签的均值，分类中使

# 用交叉熵损失函数时的对数几率)前做一个初始预测。

'center_bias': True

}

est = tf.estimator.BoostedTreesClassifier(feature_columns, **params)

# 训练模型。

est.train(train_input_fn, max_steps=100)

# 评估。

results = est.evaluate(eval_input_fn)

clear_output()

pd.Series(results).to_frame()

出于性能方面的原因，当您的数据是内存数据集时，我们推荐您使用 boosted_trees_classifier_train_in_memory 函数。此外，如果您对训练时间没有要求抑或是您的数据集很大且不愿做分布式训练，请使用上面显示的 tf.estimator.BoostedTrees API。

当您使用此方法时，请不要对数据分批(batch)，而是对整个数据集进行操作。

in_memory_params = dict(params)

in_memory_params['n_batches_per_layer'] = 1

# 在内存中的输入方程请不要对数据分批。

def make_inmemory_train_input_fn(X, y):

y = np.expand_dims(y, axis=1)

def input_fn():

return dict(X), y

return input_fn

train_input_fn = make_inmemory_train_input_fn(dftrain, y_train)

# 训练模型。

est = tf.estimator.BoostedTreesClassifier(

feature_columns,

train_in_memory=True,

**in_memory_params)

est.train(train_input_fn)

print(est.evaluate(eval_input_fn))

INFO:tensorflow:Using default config.

WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpec8e696f

INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpec8e696f', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true

graph_options {

rewrite_options {

meta_optimizer_iterations: ONE

}

}

, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

WARNING:tensorflow:Issue encountered when serializing resources.

Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.

'_Resource' object has no attribute 'name'

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

WARNING:tensorflow:Issue encountered when serializing resources.

Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.

'_Resource' object has no attribute 'name'

INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpec8e696f/model.ckpt.

WARNING:tensorflow:Issue encountered when serializing resources.

Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.

'_Resource' object has no attribute 'name'

INFO:tensorflow:loss = 0.6931472, step = 0

WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.

INFO:tensorflow:global_step/sec: 80.2732

INFO:tensorflow:loss = 0.34654337, step = 99 (1.249 sec)

INFO:tensorflow:Saving checkpoints for 153 into /tmp/tmpec8e696f/model.ckpt.

WARNING:tensorflow:Issue encountered when serializing resources.

Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.

'_Resource' object has no attribute 'name'

INFO:tensorflow:Loss for final step: 0.31796658.

INFO:tensorflow:Calling model_fn.

WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Starting evaluation at 2020-03-09T21:21:14Z

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmp/tmpec8e696f/model.ckpt-153

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Inference Time : 0.55945s

INFO:tensorflow:Finished evaluation at 2020-03-09-21:21:15

INFO:tensorflow:Saving dict for global step 153: accuracy = 0.8030303, accuracy_baseline = 0.625, auc = 0.8679216, auc_precision_recall = 0.8527449, average_loss = 0.4203342, global_step = 153, label/mean = 0.375, loss = 0.4203342, precision = 0.7473684, prediction/mean = 0.38673538, recall = 0.7171717

WARNING:tensorflow:Issue encountered when serializing resources.

Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.

'_Resource' object has no attribute 'name'

INFO:tensorflow:Saving 'checkpoint_path' summary for global step 153: /tmp/tmpec8e696f/model.ckpt-153

{'accuracy': 0.8030303, 'accuracy_baseline': 0.625, 'auc': 0.8679216, 'auc_precision_recall': 0.8527449, 'average_loss': 0.4203342, 'label/mean': 0.375, 'loss': 0.4203342, 'precision': 0.7473684, 'prediction/mean': 0.38673538, 'recall': 0.7171717, 'global_step': 153}

模型说明与绘制

import matplotlib.pyplot as plt

import seaborn as sns

sns_colors = sns.color_palette('colorblind')

局部可解释性(Local interpretability)

接下来，您将输出定向特征贡献(DFCs)来解释单个预测。输出依据 Palczewska et al 和 Saabas 在 解释随机森林(Interpreting Random Forests) 中提出的方法产生(scikit-learn 中随机森林相关的包 treeinterpreter 使用原理相同的远离). 使用以下语句输出 DFCs:

pred_dicts = list(est.experimental_predict_with_explanations(pred_input_fn))

(注意：带 “experimental” 前缀为实验版本(开发中)，在正式版发布前可能对其修改。)

pred_dicts = list(est.experimental_predict_with_explanations(eval_input_fn))

INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpec8e696f', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true

graph_options {

rewrite_options {

meta_optimizer_iterations: ONE

}

}

, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Restoring parameters from /tmp/tmpec8e696f/model.ckpt-153

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

# 创建 DFCs 的 DataFrame。

labels = y_eval.values

probs = pd.Series([pred['probabilities'][1] for pred in pred_dicts])

df_dfc = pd.DataFrame([pred['dfc'] for pred in pred_dicts])

df_dfc.describe().T

DFCs 有个不错的特性：贡献和 + 偏差(bias) = 给出样例的预测值。

# DFCs的和 + 偏差(bias) == 可能性

bias = pred_dicts[0]['bias']

dfc_prob = df_dfc.sum(axis=1) + bias

np.testing.assert_almost_equal(dfc_prob.values,

probs.values)

为单个乘客绘制 DFCs，绘图时按贡献的方向性对其进行涂色并添加特征的值。

# 绘制模版 :)

def _get_color(value):

"""正的 DFCs 标为绿色，负的为红色。"""

green, red = sns.color_palette()[2:4]

if value >= 0: return green

return red

def _add_feature_values(feature_values, ax):

"""在图的左侧显示特征的值"""

x_coord = ax.get_xlim()[0]

OFFSET = 0.15

for y_coord, (feat_name, feat_val) in enumerate(feature_values.items()):

t = plt.text(x_coord, y_coord - OFFSET, '{}'.format(feat_val), size=12)

t.set_bbox(dict(facecolor='white', alpha=0.5))

from matplotlib.font_manager import FontProperties

font = FontProperties()

font.set_weight('bold')

t = plt.text(x_coord, y_coord + 1 - OFFSET, 'feature\nvalue',

fontproperties=font, size=12)

def plot_example(example):

TOP_N = 8 # 显示前8个特征。

sorted_ix = example.abs().sort_values()[-TOP_N:].index # 按值排序。

example = example[sorted_ix]

colors = example.map(_get_color).tolist()

ax = example.to_frame().plot(kind='barh',

color=[colors],

legend=None,

alpha=0.75,

figsize=(10,6))

ax.grid(False, axis='y')

ax.set_yticklabels(ax.get_yticklabels(), size=14)

# 添加特征的值。

_add_feature_values(dfeval.iloc[ID][sorted_ix], ax)

return ax# 绘制结果。

ID = 182

example = df_dfc.iloc[ID] # 从评估集中选择第 i 个样例。

TOP_N = 8 # 显示前8个特征。

sorted_ix = example.abs().sort_values()[-TOP_N:].index

ax = plot_example(example)

ax.set_title('Feature contributions for example {}\n pred: {:1.2f}; label: {}'.format(ID, probs[ID], labels[ID]))

ax.set_xlabel('Contribution to predicted probability', size=14)

plt.show()

更大的贡献值意味着对模型的预测有更大的影响。负的贡献表示此样例该特征的值减小了减小了模型的预测，正贡献值表示增加了模型的预测。

您也可以使用小提琴图(violin plot)来绘制该样例的 DFCs 并与整体分布比较。

# 绘制代码模版。

def dist_violin_plot(df_dfc, ID):

# 初始化画布。

fig, ax = plt.subplots(1, 1, figsize=(10, 6))

# 创建样例 DataFrame。

TOP_N = 8 # 显示前8个特征。

example = df_dfc.iloc[ID]

ix = example.abs().sort_values()[-TOP_N:].index

example = example[ix]

example_df = example.to_frame(name='dfc')

# 添加整个分布的贡献。

parts=ax.violinplot([df_dfc[w] for w in ix],

vert=False,

showextrema=False,

widths=0.7,

positions=np.arange(len(ix)))

face_color = sns_colors[0]

alpha = 0.15

for pc in parts['bodies']:

pc.set_facecolor(face_color)

pc.set_alpha(alpha)

# 添加特征的值。

_add_feature_values(dfeval.iloc[ID][sorted_ix], ax)

# 添加局部贡献。

ax.scatter(example,

np.arange(example.shape[0]),

color=sns.color_palette()[2],

s=100,

marker="s",

label='contributions for example')

# 图例。

# 生成小提琴图的详细图例。

ax.plot([0,0], [1,1], label='eval set contributions\ndistributions',

color=face_color, alpha=alpha, linewidth=10)

legend = ax.legend(loc='lower right', shadow=True, fontsize='x-large',

frameon=True)

legend.get_frame().set_facecolor('white')

# 调整格式。

ax.set_yticks(np.arange(example.shape[0]))

ax.set_yticklabels(example.index)

ax.grid(False, axis='y')

ax.set_xlabel('Contribution to predicted probability', size=14)

绘制此样例。

dist_violin_plot(df_dfc, ID)

plt.title('Feature contributions for example {}\n pred: {:1.2f}; label: {}'.format(ID, probs[ID], labels[ID]))

plt.show()

最后，第三方的工具，如：LIME 和 shap 也可以帮助理解模型的各个预测。

全局特征重要性(Global feature importances)

此外，您或许想了解模型这个整体而不是单个预测。接下来，您将计算并使用：

通过 est.experimental_feature_importances 得到基于增益的特征重要性(Gain-based feature importances)

排列特征重要性(Permutation feature importances)

使用 est.experimental_predict_with_explanations 得到总 DFCs。

基于增益的特征重要性在分离特定特征时测量损失的变化。而排列特征重要性是在评估集上通过每次打乱一个特征后观察模型性能的变化计算而出。

一般来说，排列特征重要性要优于基于增益的特征重要性，尽管这两种方法在潜在预测变量的测量范围或类别数量不确定时和特征相关联时不可信(来源)。 对不同种类特征重要性的更透彻概括和更翔实讨论请参考 这篇文章 。

基于增益的特征重要性(Gain-based feature importances)

TensorFlow 的提升树估算器(estimator)内置了函数 est.experimental_feature_importances 用于计算基于增益的特征重要性。

importances = est.experimental_feature_importances(normalize=True)

df_imp = pd.Series(importances)

# 可视化重要性。

N = 8

ax = (df_imp.iloc[0:N][::-1]

.plot(kind='barh',

color=sns_colors[0],

title='Gain feature importances',

figsize=(10, 6)))

ax.grid(False, axis='y')

平均绝对 DFCs

您还可以得到绝对DFCs的平均值来从全局的角度分析影响。

# 绘图。

dfc_mean = df_dfc.abs().mean()

N = 8

sorted_ix = dfc_mean.abs().sort_values()[-N:].index # 求平均并按绝对值排序。

ax = dfc_mean[sorted_ix].plot(kind='barh',

color=sns_colors[1],

title='Mean |directional feature contributions|',

figsize=(10, 6))

ax.grid(False, axis='y')

您可以看到 DFCs 如何随特征的值变化而变化。

FEATURE = 'fare'

feature = pd.Series(df_dfc[FEATURE].values, index=dfeval[FEATURE].values).sort_index()

ax = sns.regplot(feature.index.values, feature.values, lowess=True)

ax.set_ylabel('contribution')

ax.set_xlabel(FEATURE)

ax.set_xlim(0, 100)

plt.show()

排列特征重要性(Permutation feature importances)

def permutation_importances(est, X_eval, y_eval, metric, features):

"""

分别对每列，打散列中的值并观察其对评估集的影响。