此部分为零基础入门金融风控的 Task4 建模调参部分，带你来了解各种模型以及模型的评价和调参策略
4.1学习目标
- 学习在金融风控领域常见的机器学习模型
- 学习机器学习模型的建模任务与调参流程
- 完成相应的学习打卡任务
4.2内容介绍
- 逻辑回归模型
- 理解逻辑回归模型；
- 逻辑回归模型的应用；
- 逻辑回归的优缺点；
- 树模型
- 理解树模型；
- 树模型的应用；
- 树模型的优缺点；
- 集成模型
- 基于bagging思想的集成模型
- 随机森林模型
- 基于boosting思想的集成模型
- XGBoost模型
- LightGBM模型
- CatBoost模型
- 基于bagging思想的集成模型
- 模型对比与性能评估
- 回归模型/树模型/集成模型；
- 模型评估方法；
- 模型评价结果；
- 建模调参
- 贪心调参方法；
- 网格调参方法；
- 贝叶斯调参方法；
4.3模型相关原理介绍
4.3.1逻辑回归模型
https://blog.csdn.net/han_xiaoyang/article/details/49123419
4.3.2决策树模型
https://blog.csdn.net/c406495762/article/details/76262487
4.3.3GBDT模型
https://zhuanlan.zhihu.com/p/45145899
4.3.4XGBoost模型
https://blog.csdn.net/wuzhongqiang/article/details/104854890
4.3.5LightGBM模型
https://blog.csdn.net/wuzhongqiang/article/details/105350579
4.3.6Catboost模型
https://mp.weixin.qq.com/s/xloTLr5NJBgBspMQtxPoFA
4.3.7时间序列模型
RNN:https://zhuanlan.zhihu.com/p/45289691
LSTM:https://zhuanlan.zhihu.com/p/83496936
4.3.8推荐教材
《机器学习》 https://book.douban.com/subject/26708119/
《统计学习方法》 https://book.douban.com/subject/10590856/
《面向机器学习的特征工程》 https://book.douban.com/subject/26826639/
《信用评分模型技术与应用》https://book.douban.com/subject/1488075/
《数据化风控》https://book.douban.com/subject/30282558/
4.4模型对比与性能评估
4.4.1 逻辑回归
- 优点
- 训练速度较快，分类的时候，计算量仅仅只和特征的数目相关；
- 简单易理解，模型的可解释性非常好，从特征的权重可以看到不同的特征对最后结果的影响；
- 适合二分类问题，不需要缩放输入特征；
- 内存资源占用小，只需要存储各个维度的特征值
- 缺点
- 逻辑回归需要预先处理缺失值和异常值【可参考task3特征工程】；
- 不能用Logistic回归去解决非线性问题，因为Logistic的决策面是线性的；
- 对多重共线性数据较为敏感，且很难处理数据不平衡的问题；
- 准确率并不是很高，因为形式非常简单，很难去拟合数据的真实分布；
4.4.2 决策树模型
- 优点
- 简单直观，生成的决策树可以可视化展示
- 数据不需要预处理，不需要归一化，不需要处理缺失数据
- 既可以处理离散值，也可以处理连续值
- 缺点
- 决策树算法非常容易过拟合，导致泛化能力不强（可进行适当的剪枝）
- 采用的是贪心算法，容易得到局部最优解
4.4.3集成模型集成方法
-
通过组合多个学习器来完成学习任务，通过集成方法，可以将多个弱学习器组合成一个强分类器，因此集成学习的泛化能力一般比单一分类器要好。
-
集成方法主要包括Bagging和Boosting，Bagging和Boosting都是将已有的分类或回归算法通过一定方式组合起来，形成一个更加强大的分类。两种方法都是把若干个分类器整合为一个分类器的方法，只是整合的方式不一样，最终得到不一样的效果。常见的基于Bagging思想的集成模型有：随机森林、基于Boosting思想的集成模型有：Adaboost、GBDT、XgBoost、LightGBM等。
Baggin和Boosting的区别总结如下：
- 样本选择上： Bagging方法的训练集是从原始集中有放回的选取，所以从原始集中选出的各轮训练集之间是独立的；而Boosting方法需要每一轮的训练集不变，只是训练集中每个样本在分类器中的权重发生变化。而权值是根据上一轮的分类结果进行调整
- 样例权重上： Bagging方法使用均匀取样，所以每个样本的权重相等；而Boosting方法根据错误率不断调整样本的权值，错误率越大则权重越大
- 预测函数上： Bagging方法中所有预测函数的权重相等；而Boosting方法中每个弱分类器都有相应的权重，对于分类误差小的分类器会有更大的权重
- 并行计算上： Bagging方法中各个预测函数可以并行生成；而Boosting方法各个预测函数只能顺序生成，因为后一个模型参数需要前一轮模型的结果。
4.4.4模型评估方法
-
对于模型来说，其在训练集上面的误差我们称之为训练误差或者经验误差，而在测试集上的误差称之为测试误差。
-
对于我们来说，我们更关心的是模型对于新样本的学习能力，即我们希望通过对已有样本的学习，尽可能的将所有潜在样本的普遍规律学到手，而如果模型对训练样本学的太好，则有可能把训练样本自身所具有的一些特点当做所有潜在样本的普遍特点，这时候我们就会出现过拟合的问题。
-
因此我们通常将已有的数据集划分为训练集和测试集两部分，其中训练集用来训练模型，而测试集则是用来评估模型对于新样本的判别能力。
对于数据集的划分，我们通常要保证满足以下两个条件：
- 训练集和测试集的分布要与样本真实分布一致，即训练集和测试集都要保证是从样本真实分布中独立同分布采样而得；
- 训练集和测试集要互斥
对于数据集的划分有三种方法：留出法，交叉验证法和自助法，下面挨个介绍： - ①留出法:是直接将数据集D划分为两个互斥的集合，其中一个集合作为训练集S，另一个作为测试集T。需要注意的是在划分的时候要尽可能保证数据分布的一致性，即避免因数据划分过程引入额外的偏差而对最终结果产生影响。为了保证数据分布的一致性，通常我们采用分层采样的方式来对数据进行采样。
Tips： 通常会将数据集D中大约2/3~4/5的样本作为训练集，其余的作为测试集。
- ②交叉验证法:k折交叉验证通常将数据集D分为k份，其中k-1份作为训练集，剩余的一份作为测试集，这样就可以获得k组训练/测试集，可以进行k次训练与测试，最终返回的是k个测试结果的均值。交叉验证中数据集的划分依然是依据分层采样的方式来进行。
对于交叉验证法，其k值的选取往往决定了评估结果的稳定性和保真性，通常k值选取10。
当k=1的时候，我们称之为留一法
- ③自助法:我们每次从数据集D中取一个样本作为训练集中的元素，然后把该样本放回，重复该行为m次，这样我们就可以得到大小为m的训练集，在这里面有的样本重复出现，有的样本则没有出现过，我们把那些没有出现过的样本作为测试集。
进行这样采样的原因是因为在D中约有36.8%的数据没有在训练集中出现过。留出法与交叉验证法都是使用分层采样的方式进行数据采样与划分，而自助法则是使用有放回重复采样的方式进行数据采样
数据划分总结
- 对于数据量充足的时候，通常采用留出法或者k折交叉验证法来进行训练/测试集的划分；
- 对于数据集小且难以有效划分训练/测试集时使用自助法；
- 对于数据集小且可有效划分的时候最好使用留一法来进行划分，因为这种方法最为准确
4.4.5模型评价标准
对于本次比赛，我们选用auc作为模型评价标准，类似的评价标准还有ks、f1-score等，具体介绍与实现大家可以回顾下task1中的内容
AUC
- 在逻辑回归里面，对于正负例的界定，通常会设一个阈值，大于阈值的为正类，小于阈值为负类。如果我们减小这个阀值，更多的样本会被识别为正类，提高正类的识别率，但同时也会使得更多的负类被错误识别为正类。为了直观表示这一现象，引入ROC。
- 根据分类结果计算得到ROC空间中相应的点，连接这些点就形成ROC curve，横坐标为False Positive Rate(FPR：假正率)，纵坐标为True Positive Rate(TPR：真正率)。 一般情况下，这个曲线都应该处于(0,0)和(1,1)连线的上方
- https://blog.csdn.net/weixin_46180512/article/details/108607240
ROC曲线中的四个点：
- 点(0,1)：即FPR=0, TPR=1，意味着FN＝0且FP＝0，将所有的样本都正确分类；
- 点(1,0)：即FPR=1，TPR=0，最差分类器，避开了所有正确答案；
- 点(0,0)：即FPR=TPR=0，FP＝TP＝0，分类器把每个实例都预测为负类；
- 点(1,1)：分类器把每个实例都预测为正类
总之：ROC曲线越接近左上角，该分类器的性能越好，其泛化性能就越好。而且一般来说，如果ROC是光滑的，那么基本可以判断没有太大的overfitting。
但是对于两个模型，我们如何判断哪个模型的泛化性能更优呢？这里我们有主要以下两种方法：
- 如果模型A的ROC曲线完全包住了模型B的ROC曲线，那么我们就认为模型A要优于模型B；
- 如果两条曲线有交叉的话，我们就通过比较ROC与X，Y轴所围得曲线的面积来判断，面积越大，模型的性能就越优，这个面积我们称之为AUC(area under ROC curve)
4.5代码示例
4.5.1导入相关设置
import pandas as pd
import numpy as np
import warnings
import os
import seaborn as sns
import matplotlib.pyplot as plt
"""
sns 相关设置
@return:
"""
#声明使用 Seaborn 样式
sns.set()
# 有五种seaborn的绘图风格，它们分别是：darkgrid, whitegrid, dark, white, ticks。默认的主题是darkgrid。
sns.set_style("whitegrid")
# 有四个预置的环境，按大小从小到大排列分别为：paper, notebook, talk, poster。其中，notebook是默认的。
sns.set_context('talk')
# 中文字体设置-黑体
plt.rcParams['font.sans-serif']=['SimHei']
# 解决保存图像是负号'-'显示为方块的问题
plt.rcParams['axes.unicode_minus']=False
# 解决Seaborn中文显示问题并调整字体大小
sns.set(font='SimHei')
#读取文件
data_train=pd.read_csv('train.csv')
data_test_a=pd.read_csv('testA.csv')
data_train_sample=pd.read_csv('train.csv',nrows=5)
4.5.2 读取数据
reduce_mem_usage 函数通过调整数据类型，帮助我们减少数据在内存中占用的空间
def reduce_mem_usage(df):
start_mem=df.memory_usage().sum()
print('Memeory usage of dataframe is{:.2f} MB'.format(start_mem))#{:.2f} 表示打印输出小数点保留两位小数
for col in df.columns:
col_type=df[col].dtype
if col_type != object:
c_min=df[col].min()
c_max=df[col].max()
if str(col_type)[:3]=='int':
if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
df[col] = df[col].astype(np.int8)
elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
df[col] = df[col].astype(np.int16)
elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
df[col] = df[col].astype(np.int32)
elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
df[col] = df[col].astype(np.int64)
else:
if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
df[col] = df[col].astype(np.float16)
elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
df[col] = df[col].astype(np.float32)
else:
df[col] = df[col].astype(np.float64)
else:
df[col] = df[col].astype('category')
end_mem=df.memory_usage().sum()
print('Memory usage after optimization is {:.2f}MB'.format(end_mem))
print('Decreased by {:.1f}%'.format(100*(start_mem-end_mem)/start_mem))
return df
Python中内存的优化
- https://blog.csdn.net/wushaowu2014/article/details/86561141
x_train=pd.read_csv('data_for_model.csv')
x_train = reduce_mem_usage(x_train)
y_train=pd.read_csv('label_for_model.csv')['isDefault'].astype(np.int8)
Memeory usage of dataframe is377449200.00 MB
Memory usage after optimization is 92524170.00MB
Decreased by 75.5%
5.3简单建模
使用lightgbm建模
"""
对训练集数据进行划分，分成训练集和验证集，并进行相应的操作
"""
from sklearn.model_selection import train_test_split
import lightgbm as lgb
#数据集划分
X_train_split,X_val,y_train_split,y_val=train_test_split(x_train, y_train, test_size=0.2)
train_matrix=lgb.Dataset(X_train_split,label=y_train_split)
valid_matrix=lgb.Dataset(X_val,label=y_val)
params={
'boosting_type': 'gbdt',
'objective': 'binary',
'learning_rate': 0.1,
'metric': 'auc',
'min_child_weight': 1e-3,
'num_leaves': 31,
'max_depth': -1,
'reg_lambda': 0,
'reg_alpha': 0,
'feature_fraction': 1,
'bagging_fraction': 1,
'bagging_freq': 0,
'seed': 2020,
'nthread': 8,
'silent': True,
'verbose': -1
}
#使用训练集数据进行模型训练
model=lgb.train(params,train_set=train_matrix,valid_sets=valid_matrix,
num_boost_round=20000,verbose_eval=1000,early_stopping_rounds=200
)
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
[LightGBM] [Warning] Unknown parameter: silent
Training until validation scores don't improve for 200 rounds
Early stopping, best iteration is:
[391] valid_0's auc: 0.730961
对验证集进行预测
from sklearn import metrics
from sklearn.metrics import roc_auc_score
"""
预测并计算roc的相关指标
"""
val_pre_lgb=model.predict(X_val,num_iteration=model.best_iteration)
fpr,tpr,threshold=metrics.roc_curve(y_val,val_pre_lgb)
roc_auc=metrics.auc(fpr,tpr )
print('未调参前lightgbm单模型在验证集上的AUC:{}'.format(roc_auc))
"画出roc曲线"
plt.figure(figsize=(8,8))
plt.title('Validation ROC')
plt.plot(fpr,tpr,'b',label='Val AUC=%.4f%roc_auc')
plt.ylim(0,1)
plt.xlim(0,1)
plt.legend(loc='best')
# plt.title('ROC')
plt.ylabel('True Positive Rate')
plt.xlabel('Fasle Positive Rate')
"画出对角线"
plt.plot([0, 1], [0, 1], 'r--')
plt.show()
未调参前lightgbm单模型在验证集上的AUC:0.7309614198008657
进一步地，使用5折交叉验证进行模型性能评估
from sklearn.model_selection import KFold
#5折交叉验证
folds=5
seed=2020
kf=KFold(n_splits=folds,shuffle=True,random_state=seed)
params = {
'boosting_type': 'gbdt',
'objective': 'binary',
'learning_rate': 0.1,
'metric': 'auc',
'min_child_weight': 1e-3,
'num_leaves': 31,
'max_depth': -1,
'reg_lambda': 0,
'reg_alpha': 0,
'feature_fraction': 1,
'bagging_fraction': 1,
'bagging_freq': 0,
'seed': seed,
'nthread': 8,
'silent': True,
'verbose': -1
}
cv_scores=[]
for i,(train_index, valid_index) in enumerate(kf.split(x_train, y_train)):
print('*'*30, str(i+1), '*'*30)
X_train_split, y_train_split, X_val, y_val = (x_train.iloc[train_index],
y_train.iloc[train_index],
x_train.iloc[valid_index],
y_train.iloc[valid_index])
train_matrix = lgb.Dataset(X_train_split, label=y_train_split)
valid_matrix = lgb.Dataset(X_val, label=y_val)
model = lgb.train(params, train_set=train_matrix, valid_sets=valid_matrix,
num_boost_round=20000, verbose_eval=1000, early_stopping_rounds=200)
val_pred = model.predict(X_val, num_iteration=model.best_iteration)
cv_scores.append(roc_auc_score(y_val, val_pred))
print(cv_scores)
print('lgb_scotrainre_list: ', cv_scores)
print('lgb_score_mean: ', np.mean(cv_scores))
print('lgb_score_std: ', np.std(cv_scores))
****************************** 1 ******************************
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
[LightGBM] [Warning] Unknown parameter: silent
Training until validation scores don't improve for 200 rounds
Early stopping, best iteration is:
[361] valid_0's auc: 0.729021
[0.729021161055615]
****************************** 2 ******************************
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
[LightGBM] [Warning] Unknown parameter: silent
Training until validation scores don't improve for 200 rounds
Early stopping, best iteration is:
[457] valid_0's auc: 0.731255
[0.729021161055615, 0.7312548206123745]
****************************** 3 ******************************
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
[LightGBM] [Warning] Unknown parameter: silent
Training until validation scores don't improve for 200 rounds
Early stopping, best iteration is:
[481] valid_0's auc: 0.731781
[0.729021161055615, 0.7312548206123745, 0.7317810172654411]
****************************** 4 ******************************
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
[LightGBM] [Warning] Unknown parameter: silent
Training until validation scores don't improve for 200 rounds
Early stopping, best iteration is:
[377] valid_0's auc: 0.728142
[0.729021161055615, 0.7312548206123745, 0.7317810172654411, 0.7281417426420325]
****************************** 5 ******************************
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
[LightGBM] [Warning] Unknown parameter: silent
Training until validation scores don't improve for 200 rounds
Early stopping, best iteration is:
[249] valid_0's auc: 0.732503
[0.729021161055615, 0.7312548206123745, 0.7317810172654411, 0.7281417426420325, 0.7325031206130581]
lgb_scotrainre_list: [0.729021161055615, 0.7312548206123745, 0.7317810172654411, 0.7281417426420325, 0.7325031206130581]
lgb_score_mean: 0.7305403724377041
lgb_score_std: 0.0016711340043747977
5.4 模型调参
5.4.1 贪心调参
先使用当前对模型影响最大的参数进行调优，达到当前参数下的模型最优化，再使用对模型影响次之的参数进行调优，如此下去，直到所有的参数调整完毕。
这个方法的缺点就是可能会调到局部最优而不是全局最优，但是只需要一步一步的进行参数最优化调试即可，容易理解。
需要注意的是在树模型中参数调整的顺序，也就是各个参数对模型的影响程度，这里列举一下日常调参过程中常用的参数和调参顺序：
①：max_depth（树模型深度）、num_leaves（叶子节点数，树模型复杂度）
②：min_data_in_leaf（一个叶子上最小数据量，可以用来处理过拟合）、min_child_weight（决定最小叶子节点样本权重和。当它的值较大时,可以避免模型学习到局部的特殊样本。但如果这个值过高,会导致欠拟合。）
③：bagging_fraction（不进行重采样的情况下随机选择部分数据）、 feature_fraction（每次迭代中随机选择特征的比例）、bagging_freq（bagging的次数）
④：reg_lambda(权重的L2正则化项)、reg_alpha(权重的L1正则化项)
⑤：min_split_gain（执行切分的最小增益）
objective 可选参数值详见：https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters.rst#objective
from sklearn.model_selection import cross_val_score
best_obj=dict()
objective=['regression','rgression_l1','binary','cross_entropy','cross_entropy_lambda']
for obj in objective:
model=lgb.LGBMRegressor(objective=obj)
score=cross_val_score(model,x_train,y_train,cv=5,scoring='roc_auc').mean()
best_obj[obj]=score
best_leaves=dict()
num_leaves=range(10,80)
for leaves in num_leaves:
model=lgb.LGBMRegressor(objective=min(best_obj.items(),key=lambda x: x[1])[0],num_leaves=leaves)
score=cross_val_score(model,x_train,y_train,cv=5,scoring='roc_auc').mean()
best_leaves[leaves]=score
best_depth = dict()
max_depth = range(3, 10)
for depth in max_depth:
model = lgb.LGBMRegressor(objective=min(best_obj.items(), key=lambad x: x[1])[0],
num_leaves=min(best_leaves.items(), key=lambad x: x[1])[0],
max_depth=depth)
score = cross_val_score(model, x_train, y_train, cv=5, scoring='roc_auc').mean()
best_depth[depth] = score
5.4.2网格搜索
相比贪心调参效果更优，但时间开销大，一旦数据量过大，就很难得出结论。
from sklearn.model_selection import GridSearchCV, StratifiedKFold
def get_best_cv_params(learning_rate=0.1, n_estimators=581,
num_leaves=31, max_depth=-1, bagging_fraction=1.0,
feature_fraction=1.0, bagging_req=0, min_data_in_leaf=20,
min_child_weight=0.001, min_split_gain=0, reg_lambda=0,
reg_alpha=0, param_grid=None):
cv_fold = StratifiedKFold(n_splits=5, random_state=0, shuffle=True)
model_lgb = lgb.LGBMClassifier(learning_rate=learning_rate,
n_estimators=n_estimators,
num_leaves=num_leaves,
max_depth=max_depth,
bagging_fraction=bagging_fraction,
feature_fraction=feature_fraction,
bagging_req=bagging_req,
min_data_in_leaf=min_data_in_leaf,
min_child_weight=min_child_weight,
min_split_gain=min_split_gain,
reg_lambda=reg_lambda,
reg_alpha=reg_alpha,
n_jobs=8
)
grid_searh = GridSearchCV(estimator=model_lgb,
cv=cv_fold,
param_grid=param_grid,
scoring='roc_auc'
)
grid_searh.fit(x_train, y_train)
print('模型当前最优参数为：', grid_searh.best_parmas_)
print('模型当前最优得分为：', grid_searh.best_score_)
"""以下代码未运行，耗时较长，请谨慎运行，且每一步的最优参数需要在下一步进行手动更新，请注意"""
"""
需要注意一下的是，除了获取上面的获取num_boost_round时候用的是原生的lightgbm（因为要用自带的cv）
下面配合GridSearchCV时必须使用sklearn接口的lightgbm。
"""
#设置n_estimators 为581，调整num_leaves和max_depth，这里选择先粗调再细调
lgb_params = {'num_leaves': range(10, 80, 5), 'max_depth': range(3, 10, 2)}
get_best_cv_params(learning_rate=0.1, n_estimators=581, num_leaves=None, max_depth=None,
min_data_in_leaf=20, min_child_weight=0.001, bagging_fraction=1.0,
feature_fraction=1.0, bagging_req=0, min_split_gain=0, reg_lambda=0,
reg_alpha=0, param_grid=lgb_params)
# num_leaves为30，max_depth为7，进一步细调num_leaves和max_depth
lgb_params = {'num_leaves': range(25, 35, 1), 'max_depth': range(5, 9, 1)}
get_best_cv_params(learning_rate=0.1, n_estimators=85, num_leaves=None, max_depth=None,
min_data_in_leaf=20, min_child_weight=0.001, bagging_fraction=1.0,
feature_fraction=1.0, bagging_req=1.0, min_split_gain=0,
reg_lambda=0, reg_lambda=0, reg_alpha=0, param_grid=lgb_params)
# 确定min_data_in_leaf为45，min_child_weight为0.001
# 再进行bagging_fraction、feature_fraction和bagging_freq的调参
lgb_params = {'reg_lambda': [i/10 for i in range(5, 10, 1)],
'reg_alpha': [i/10 for i in range(5, 10, 1)],
'bagging_freq': range(0, 81, 10)}
get_best_cv_params(learning_rate=0.1, n_estimators=85, num_leaves=29, max_depth=7,
min_data_in_leaf=45, min_child_weight=0.001, bagging_fraction=None,
feature_fraction=None, bagging_req=None, min_split_gain=0,
reg_lambda=0, reg_lambda=0, param_grid=lgb_params)
# 确定bagging_fraction为0.4、feature_fraction为0.6、bagging_freq为
# 再进行reg_lambda、reg_alpha的调参
lgb_params = {'reg_lambda': [0, 0.001, 0.01, 0.03, 0.08, 0.3, 0.5],
'reg_alpha': [0, 0.001, 0.01, 0.03, 0.08, 0.3, 0.5]}
get_best_cv_params(learning_rate=0.1, n_estimators=85, num_leaves=29, max_depth=7,
min_data_in_leaf=45, min_child_weight=0.001, bagging_fraction=0.9,
feature_fraction=0.9, bagging_req=40, min_split_gain=0,
reg_lambda=None, reg_lambda=None, param_grid=lgb_params)
# 确定reg_lambda、reg_alpha都为0
# 再进行min_split_gain的调参
lgb_params = {'min_split_gain': [i/10 for i in range(0, 11, 1)]}
get_best_cv_params(learning_rate=0.1, n_estimators=85, num_leaves=29, max_depth=7,
min_data_in_leaf=45, min_child_weight=0.001, bagging_fraction=0.9,
feature_fraction=0.9, bagging_req=40, min_split_gain=0,
reg_lambda=None, reg_lambda=None, param_grid=lgb_params)
"通过网格搜索确定最优参数"
final_parmas = {'boosting_type': 'gbdt',
'learning_rate': 0.01,
'num_leaves': 29,
'max_depth': 7,
'min_data_in_leaf': 45,
'min_child_weight': 0.001,
'bagging_fraction': 0.9,
'feature_fraction': 0.9,
'bagging_freq': 40,
'min_split_gain': 0,
'reg_lambda': 0,
'reg_alpha': 0,
'nthread': 6
}
cv_result = lgb.cv(train_set=lgb_train,
early_stopping_rounds=20,
num_boost_round=5000,
nfold=5,
stratified=True,
params=final_parmas,
metrics='auc',
seed=2020)
print('迭代次数: ', len(cv_result['auc-mean']))
print('交叉验证的AUC为：', max(cv_result['auc-mean']))
5.4.3贝叶斯调参
贝叶斯调参的主要思想是：给定优化的目标函数(广义的函数，只需指定输入和输出即可，无需知道内部结构以及数学性质)，通过不断的添加样本点来更新目标函数的后验分布(高斯过程，直到后验分布基本贴合于真实分布)。简单地说，就是考虑了上一次参数的信息，从而更好的调整当前参数。
贝叶斯调参的步骤如下：
- 定义优化函数
- 建立模型
- 定义待优化的参数
- 得到优化结果，并返回要优化的分数指标
# pip install bayesian-optimization
from sklearn.model_selection import cross_val_score
def rf_cv_lgb(num_leaves, max_depth, bagging_fraction, feature_fraction, bagging_freq,
min_data_in_leaf, min_child_weight, min_split_gain, reg_lambda, reg_alpha):
model_lgb = lgb.LGBMClassifier(boosting_type='gbdt', objective='binary', metric='auc',
learning_rate=0.1, n_estimators=5000, num_leaves=int(num_leaves),
max_depth=int(max_depth), bagging_fraction=round(bagging_fraction, 2),
feature_fraction=round(feature_fraction, 2), bagging_freq=int(bagging_freq),
min_data_in_leaf=int(min_data_in_leaf), min_child_weight=min_child_weight,
min_split_gain=min_split_gain, reg_lambda=reg_lambda, reg_alpha=reg_alpha,
n_jobs=8)
val = cross_val_score(model_lgb, X_train_split, y_train_split, cv=5, scoring='roc_auc').mean()
return val
from bayes_opt import BayesianOptimization
"定义需要优化的参数"
bayes_lgb = BayesianOptimization(rf_cv_lgb,
{
'num_leaves': (10, 200),
'max_depth': (3, 20),
'bagging_fraction': (0.5, 1.0),
'feature_fraction': (0.5, 1.0),
'bagging_freq': (0, 100),
'min_data_in_leaf': (10, 100),
'min_child_weight': (0, 10),
'min_split_gain': (0.0, 1.0),
'reg_alpha': (0.0, 10),
'reg_lambda': (0.0, 10)
}
)
"开始优化"
bayes_lgb.maximize(n_iter=10)
| iter | target | baggin... | baggin... | featur... | max_depth | min_ch... | min_da... | min_sp... | num_le... | reg_alpha | reg_la... |
-------------------------------------------------------------------------------------------------------------------------------------------------
[LightGBM] [Warning] feature_fraction is set=0.6, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6
[LightGBM] [Warning] min_data_in_leaf is set=77, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=77
[LightGBM] [Warning] bagging_fraction is set=0.76, subsample=1.0 will be ignored. Current value: bagging_fraction=0.76
[LightGBM] [Warning] bagging_freq is set=32, subsample_freq=0 will be ignored. Current value: bagging_freq=32
[LightGBM] [Warning] feature_fraction is set=0.6, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6
[LightGBM] [Warning] min_data_in_leaf is set=77, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=77
[LightGBM] [Warning] bagging_fraction is set=0.76, subsample=1.0 will be ignored. Current value: bagging_fraction=0.76
[LightGBM] [Warning] bagging_freq is set=32, subsample_freq=0 will be ignored. Current value: bagging_freq=32
[LightGBM] [Warning] feature_fraction is set=0.6, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6
[LightGBM] [Warning] min_data_in_leaf is set=77, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=77
[LightGBM] [Warning] bagging_fraction is set=0.76, subsample=1.0 will be ignored. Current value: bagging_fraction=0.76
[LightGBM] [Warning] bagging_freq is set=32, subsample_freq=0 will be ignored. Current value: bagging_freq=32
[LightGBM] [Warning] feature_fraction is set=0.6, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6
[LightGBM] [Warning] min_data_in_leaf is set=77, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=77
[LightGBM] [Warning] bagging_fraction is set=0.76, subsample=1.0 will be ignored. Current value: bagging_fraction=0.76
[LightGBM] [Warning] bagging_freq is set=32, subsample_freq=0 will be ignored. Current value: bagging_freq=32
| [0m 1 [0m | [0m 0.7259 [0m | [0m 0.7568 [0m | [0m 32.34 [0m | [0m 0.5954 [0m | [0m 11.03 [0m | [0m 3.17 [0m | [0m 77.86 [0m | [0m 0.6699 [0m | [0m 27.38 [0m | [0m 2.782 [0m | [0m 6.878 [0m |
[LightGBM] [Warning] feature_fraction is set=0.87, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.87
[LightGBM] [Warning] min_data_in_leaf is set=36, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=36
[LightGBM] [Warning] bagging_fraction is set=0.62, subsample=1.0 will be ignored. Current value: bagging_fraction=0.62
[LightGBM] [Warning] bagging_freq is set=14, subsample_freq=0 will be ignored. Current value: bagging_freq=14
[LightGBM] [Warning] feature_fraction is set=0.87, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.87
[LightGBM] [Warning] min_data_in_leaf is set=36, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=36
[LightGBM] [Warning] bagging_fraction is set=0.62, subsample=1.0 will be ignored. Current value: bagging_fraction=0.62
[LightGBM] [Warning] bagging_freq is set=14, subsample_freq=0 will be ignored. Current value: bagging_freq=14
[LightGBM] [Warning] feature_fraction is set=0.87, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.87
[LightGBM] [Warning] min_data_in_leaf is set=36, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=36
[LightGBM] [Warning] bagging_fraction is set=0.62, subsample=1.0 will be ignored. Current value: bagging_fraction=0.62
[LightGBM] [Warning] bagging_freq is set=14, subsample_freq=0 will be ignored. Current value: bagging_freq=14
[LightGBM] [Warning] feature_fraction is set=0.87, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.87
[LightGBM] [Warning] min_data_in_leaf is set=36, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=36
[LightGBM] [Warning] bagging_fraction is set=0.62, subsample=1.0 will be ignored. Current value: bagging_fraction=0.62
[LightGBM] [Warning] bagging_freq is set=14, subsample_freq=0 will be ignored. Current value: bagging_freq=14
[LightGBM] [Warning] feature_fraction is set=0.87, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.87
[LightGBM] [Warning] min_data_in_leaf is set=36, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=36
[LightGBM] [Warning] bagging_fraction is set=0.62, subsample=1.0 will be ignored. Current value: bagging_fraction=0.62
[LightGBM] [Warning] bagging_freq is set=14, subsample_freq=0 will be ignored. Current value: bagging_freq=14
| [0m 2 [0m | [0m 0.7019 [0m | [0m 0.6182 [0m | [0m 14.01 [0m | [0m 0.872 [0m | [0m 11.06 [0m | [0m 6.923 [0m | [0m 36.27 [0m | [0m 0.2546 [0m | [0m 62.03 [0m | [0m 6.643 [0m | [0m 6.671 [0m |
[LightGBM] [Warning] feature_fraction is set=0.89, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.89
[LightGBM] [Warning] min_data_in_leaf is set=73, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=73
[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59
[LightGBM] [Warning] bagging_freq is set=50, subsample_freq=0 will be ignored. Current value: bagging_freq=50
[LightGBM] [Warning] feature_fraction is set=0.89, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.89
[LightGBM] [Warning] min_data_in_leaf is set=73, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=73
[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59
[LightGBM] [Warning] bagging_freq is set=50, subsample_freq=0 will be ignored. Current value: bagging_freq=50
[LightGBM] [Warning] feature_fraction is set=0.89, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.89
[LightGBM] [Warning] min_data_in_leaf is set=73, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=73
[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59
[LightGBM] [Warning] bagging_freq is set=50, subsample_freq=0 will be ignored. Current value: bagging_freq=50
[LightGBM] [Warning] feature_fraction is set=0.89, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.89
[LightGBM] [Warning] min_data_in_leaf is set=73, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=73
[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59
[LightGBM] [Warning] bagging_freq is set=50, subsample_freq=0 will be ignored. Current value: bagging_freq=50
[LightGBM] [Warning] feature_fraction is set=0.89, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.89
[LightGBM] [Warning] min_data_in_leaf is set=73, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=73
[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59
[LightGBM] [Warning] bagging_freq is set=50, subsample_freq=0 will be ignored. Current value: bagging_freq=50
| [0m 3 [0m | [0m 0.7206 [0m | [0m 0.5888 [0m | [0m 50.46 [0m | [0m 0.8861 [0m | [0m 16.39 [0m | [0m 0.7554 [0m | [0m 73.52 [0m | [0m 0.862 [0m | [0m 134.6 [0m | [0m 0.04269 [0m | [0m 4.55 [0m |
[LightGBM] [Warning] feature_fraction is set=0.9, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9
[LightGBM] [Warning] min_data_in_leaf is set=55, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=55
[LightGBM] [Warning] bagging_fraction is set=0.54, subsample=1.0 will be ignored. Current value: bagging_fraction=0.54
[LightGBM] [Warning] bagging_freq is set=69, subsample_freq=0 will be ignored. Current value: bagging_freq=69
[LightGBM] [Warning] feature_fraction is set=0.9, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.9
[LightGBM] [Warning] min_data_in_leaf is set=55, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=55
[LightGBM] [Warning] bagging_fraction is set=0.54, subsample=1.0 will be ignored. Current value: bagging_fraction=0.54
[LightGBM] [Warning] bagging_freq is set=69, subsample_freq=0 will be ignored. Current value: bagging_freq=69
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\bayes_opt\target_space.py in probe(self, params)
190 try:
--> 191 target = self._cache[_hashable(x)]
192 except KeyError:
KeyError: (0.5438826238302319, 69.59212457859161, 0.8984290543444566, 5.851070486497143, 6.157080131679358, 55.391326349403265, 0.797801407577217, 179.56696344446573, 5.940206783590173, 5.648257414566444)
During handling of the above exception, another exception occurred:
KeyboardInterrupt Traceback (most recent call last)
<ipython-input-17-02aebdcc545d> in <module>
18
19 "开始优化"
---> 20 bayes_lgb.maximize(n_iter=10)
C:\ProgramData\Anaconda3\lib\site-packages\bayes_opt\bayesian_optimization.py in maximize(self, init_points, n_iter, acq, kappa, kappa_decay, kappa_decay_delay, xi, **gp_params)
183 iteration += 1
184
--> 185 self.probe(x_probe, lazy=False)
186
187 if self._bounds_transformer:
C:\ProgramData\Anaconda3\lib\site-packages\bayes_opt\bayesian_optimization.py in probe(self, params, lazy)
114 self._queue.add(params)
115 else:
--> 116 self._space.probe(params)
117 self.dispatch(Events.OPTIMIZATION_STEP)
118
C:\ProgramData\Anaconda3\lib\site-packages\bayes_opt\target_space.py in probe(self, params)
192 except KeyError:
193 params = dict(zip(self._keys, x))
--> 194 target = self.target_func(**params)
195 self.register(x, target)
196 return target
<ipython-input-15-2b3948d8ee8f> in rf_cv_lgb(num_leaves, max_depth, bagging_fraction, feature_fraction, bagging_freq, min_data_in_leaf, min_child_weight, min_split_gain, reg_lambda, reg_alpha)
13 n_jobs=8)
14
---> 15 val = cross_val_score(model_lgb, X_train_split, y_train_split, cv=5, scoring='roc_auc').mean()
16 return val
C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py in inner_f(*args, **kwargs)
71 FutureWarning)
72 kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
---> 73 return f(**kwargs)
74 return inner_f
75
C:\ProgramData\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py in cross_val_score(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, error_score)
399 scorer = check_scoring(estimator, scoring=scoring)
400
--> 401 cv_results = cross_validate(estimator=estimator, X=X, y=y, groups=groups,
402 scoring={'score': scorer}, cv=cv,
403 n_jobs=n_jobs, verbose=verbose,
C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py in inner_f(*args, **kwargs)
71 FutureWarning)
72 kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
---> 73 return f(**kwargs)
74 return inner_f
75
C:\ProgramData\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py in cross_validate(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, return_train_score, return_estimator, error_score)
240 parallel = Parallel(n_jobs=n_jobs, verbose=verbose,
241 pre_dispatch=pre_dispatch)
--> 242 scores = parallel(
243 delayed(_fit_and_score)(
244 clone(estimator), X, y, scorers, train, test, verbose, None,
C:\ProgramData\Anaconda3\lib\site-packages\joblib\parallel.py in __call__(self, iterable)
1030 self._iterating = self._original_iterator is not None
1031
-> 1032 while self.dispatch_one_batch(iterator):
1033 pass
1034
C:\ProgramData\Anaconda3\lib\site-packages\joblib\parallel.py in dispatch_one_batch(self, iterator)
845 return False
846 else:
--> 847 self._dispatch(tasks)
848 return True
849
C:\ProgramData\Anaconda3\lib\site-packages\joblib\parallel.py in _dispatch(self, batch)
763 with self._lock:
764 job_idx = len(self._jobs)
--> 765 job = self._backend.apply_async(batch, callback=cb)
766 # A job can complete so quickly than its callback is
767 # called before we get here, causing self._jobs to
C:\ProgramData\Anaconda3\lib\site-packages\joblib\_parallel_backends.py in apply_async(self, func, callback)
206 def apply_async(self, func, callback=None):
207 """Schedule a func to be run"""
--> 208 result = ImmediateResult(func)
209 if callback:
210 callback(result)
C:\ProgramData\Anaconda3\lib\site-packages\joblib\_parallel_backends.py in __init__(self, batch)
570 # Don't delay the application, to avoid keeping the input
571 # arguments in memory
--> 572 self.results = batch()
573
574 def get(self):
C:\ProgramData\Anaconda3\lib\site-packages\joblib\parallel.py in __call__(self)
250 # change the default number of processes to -1
251 with parallel_backend(self._backend, n_jobs=self._n_jobs):
--> 252 return [func(*args, **kwargs)
253 for func, args, kwargs in self.items]
254
C:\ProgramData\Anaconda3\lib\site-packages\joblib\parallel.py in <listcomp>(.0)
250 # change the default number of processes to -1
251 with parallel_backend(self._backend, n_jobs=self._n_jobs):
--> 252 return [func(*args, **kwargs)
253 for func, args, kwargs in self.items]
254
C:\ProgramData\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py in _fit_and_score(estimator, X, y, scorer, train, test, verbose, parameters, fit_params, return_train_score, return_parameters, return_n_test_samples, return_times, return_estimator, error_score)
529 estimator.fit(X_train, **fit_params)
530 else:
--> 531 estimator.fit(X_train, y_train, **fit_params)
532
533 except Exception as e:
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\sklearn.py in fit(self, X, y, sample_weight, init_score, eval_set, eval_names, eval_sample_weight, eval_class_weight, eval_init_score, eval_metric, early_stopping_rounds, verbose, feature_name, categorical_feature, callbacks, init_model)
826 valid_sets[i] = (valid_x, self._le.transform(valid_y))
827
--> 828 super(LGBMClassifier, self).fit(X, _y, sample_weight=sample_weight,
829 init_score=init_score, eval_set=valid_sets,
830 eval_names=eval_names,
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\sklearn.py in fit(self, X, y, sample_weight, init_score, group, eval_set, eval_names, eval_sample_weight, eval_class_weight, eval_init_score, eval_group, eval_metric, early_stopping_rounds, verbose, feature_name, categorical_feature, callbacks, init_model)
593 init_model = init_model.booster_
594
--> 595 self._Booster = train(params, train_set,
596 self.n_estimators, valid_sets=valid_sets, valid_names=eval_names,
597 early_stopping_rounds=early_stopping_rounds,
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\engine.py in train(params, train_set, num_boost_round, valid_sets, valid_names, fobj, feval, init_model, feature_name, categorical_feature, early_stopping_rounds, evals_result, verbose_eval, learning_rates, keep_training_booster, callbacks)
250 evaluation_result_list=None))
251
--> 252 booster.update(fobj=fobj)
253
254 evaluation_result_list = []
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py in update(self, train_set, fobj)
2368 if self.__set_objective_to_none:
2369 raise LightGBMError('Cannot update due to null objective function.')
-> 2370 _safe_call(_LIB.LGBM_BoosterUpdateOneIter(
2371 self.handle,
2372 ctypes.byref(is_finished)))
KeyboardInterrupt:
因为调参时间实在是太久了。我就给先停止了。先试试3次调参后的效果怎么样。
'显示优化结果'
bayes_lgb.max
{'target': 0.7259101142518538,
'params': {'bagging_fraction': 0.756812760054594,
'bagging_freq': 32.33771733707418,
'feature_fraction': 0.5954069016709567,
'max_depth': 11.033948576753312,
'min_child_weight': 3.1697449726726656,
'min_data_in_leaf': 77.85639704189543,
'min_split_gain': 0.6699180209645704,
'num_leaves': 27.37930159390323,
'reg_alpha': 2.7815537356750255,
'reg_lambda': 6.878393002204629}}
参数优化完成后，可以根据优化后的参数建立新的模型，降低学习率并寻找最优模型迭代次数
"调整一个较小的学习率，并通过cv函数确定当前最优的迭代次数"
base_params_lgb ={'boosting_type': 'gbdt',
'objective': 'binary',
'metric': 'auc',
'learning_rate': 0.01,
'nthread': 8,
'seed': 2020,
'silent': True,
'verbose': -1
}
for fea in bayes_lgb.max['params']:
if fea in ['num_leaves', 'max_depth', 'min_data_in_leaf', 'bagging_freq']:
base_params_lgb[fea] = int(bayes_lgb.max['params'][fea])
else:
base_params_lgb[fea] = round(bayes_lgb.max['params'][fea], 2)
cv_result_lgb = lgb.cv(train_set=train_matrix,
early_stopping_rounds=1000,
num_boost_round=20000,
nfold=5,
stratified=True,
shuffle=True,
params=base_params_lgb,
metrics='auc',
seed=2020
)
print('迭代次数：', len(cv_result_lgb['auc-mean']))
print('最终模型的AUC为：', max(cv_result_lgb['auc-mean']))
迭代次数： 6586
最终模型的AUC为： 0.7324276609153875
5.5建立最终模型
模型参数已经确定，确立最终模型并对验证集进行验证
cv_scores = []
for i, (train_index, valid_index) in enumerate(kf.split(x_train, y_train)):
print('*'*30, str(i+1), '*'*30)
X_train_split, y_train_split, X_val, y_val = (x_train.iloc[train_index],
y_train.iloc[train_index],
x_train.iloc[valid_index],
y_train.iloc[valid_index])
train_matrix = lgb.Dataset(X_train_split, label=y_train_split)
valid_matrix = lgb.Dataset(X_val, label=y_val)
params = base_params_lgb
# params.pop('verbose')
# verbose_eval 每迭代多少次输出一次评估结果
model = lgb.train(params, train_set=train_matrix, valid_sets=valid_matrix,
num_boost_round=6586, verbose_eval=1000, early_stopping_rounds=200)
val_pred = model.predict(X_val, num_iteration=model.best_iteration)
cv_scores.append(roc_auc_score(y_val, val_pred))
print(cv_scores)
print('lgb_scotrainre_list: ', cv_scores)
print('lgb_score_mean: ', np.mean(cv_scores))
print('lgb_score_std: ', np.std(cv_scores))
****************************** 1 ******************************
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
Training until validation scores don't improve for 200 rounds
[1000] valid_0's auc: 0.728042
[2000] valid_0's auc: 0.730554
[3000] valid_0's auc: 0.731483
[4000] valid_0's auc: 0.731849
[5000] valid_0's auc: 0.732146
Early stopping, best iteration is:
[5044] valid_0's auc: 0.732178
[0.7321783280976049]
****************************** 2 ******************************
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
[LightGBM] [Warning] Unknown parameter: silent
Training until validation scores don't improve for 200 rounds
[1000] valid_0's auc: 0.729889
[2000] valid_0's auc: 0.732288
[3000] valid_0's auc: 0.733158
[4000] valid_0's auc: 0.733613
Early stopping, best iteration is:
[4016] valid_0's auc: 0.733616
[0.7321783280976049, 0.7336157776312788]
****************************** 3 ******************************
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
[LightGBM] [Warning] Unknown parameter: silent
Training until validation scores don't improve for 200 rounds
[1000] valid_0's auc: 0.730131
[2000] valid_0's auc: 0.733153
[3000] valid_0's auc: 0.734341
[4000] valid_0's auc: 0.734879
[5000] valid_0's auc: 0.735199
[6000] valid_0's auc: 0.735473
Early stopping, best iteration is:
[5984] valid_0's auc: 0.735481
[0.7321783280976049, 0.7336157776312788, 0.7354806643446536]
****************************** 4 ******************************
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
[LightGBM] [Warning] Unknown parameter: silent
Training until validation scores don't improve for 200 rounds
[1000] valid_0's auc: 0.72634
[2000] valid_0's auc: 0.729182
[3000] valid_0's auc: 0.729954
[4000] valid_0's auc: 0.730356
[5000] valid_0's auc: 0.730544
Early stopping, best iteration is:
[5270] valid_0's auc: 0.730585
[0.7321783280976049, 0.7336157776312788, 0.7354806643446536, 0.7305853364085984]
****************************** 5 ******************************
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
[LightGBM] [Warning] Unknown parameter: silent
Training until validation scores don't improve for 200 rounds
[1000] valid_0's auc: 0.730891
[2000] valid_0's auc: 0.733485
[3000] valid_0's auc: 0.734429
[4000] valid_0's auc: 0.734935
Early stopping, best iteration is:
[4544] valid_0's auc: 0.735132
[0.7321783280976049, 0.7336157776312788, 0.7354806643446536, 0.7305853364085984, 0.7351324863078958]
lgb_scotrainre_list: [0.7321783280976049, 0.7336157776312788, 0.7354806643446536, 0.7305853364085984, 0.7351324863078958]
lgb_score_mean: 0.7333985185580063
lgb_score_std: 0.0018325957167270564
通过5折交叉验证，可以发现模型迭代到13000次后会停止，那么在建立新模型时，可以直接设置最大迭代次数为13000，并使用验证集进行模型预测。
X_train_split, X_val, y_train_split, y_val = train_test_split(x_train, y_train, test_size=0.2)
train_matrix = lgb.Dataset(X_train_split, label=y_train_split)
valid_matrix = lgb.Dataset(X_val, label=y_val)
final_model_lgb = lgb.train(params, train_set=train_matrix, valid_sets=valid_matrix,
num_boost_round=13000, verbose_eval=1000, early_stopping_rounds=200)
C:\ProgramData\Anaconda3\lib\site-packages\lightgbm\basic.py:1075: UserWarning: silent keyword has been found in `params` and will be ignored.
Please use silent argument of the Dataset constructor to pass this parameter.
warnings.warn('{0} keyword has been found in `params` and will be ignored.\n'
[LightGBM] [Warning] Unknown parameter: silent
Training until validation scores don't improve for 200 rounds
[1000] valid_0's auc: 0.728216
[2000] valid_0's auc: 0.730926
[3000] valid_0's auc: 0.731853
[4000] valid_0's auc: 0.732385
Early stopping, best iteration is:
[4311] valid_0's auc: 0.73253
val_pre_lgb = final_model_lgb.predict(X_val)
fpr, tpr, threshold = metrics.roc_curve(y_val, val_pre_lgb)
roc_auc = metrics.auc(fpr, tpr)
print('调参后lightgbm单mox模型在验证集上的AUC：', roc_auc)
plt.figure(figsize=(8, 8))
plt.title('Validation ROC')
plt.plot(fpr, tpr, 'b', label='Val AUC = %.4f' % roc_auc)
plt.ylim(0, 1)
plt.xlim(0, 1)
plt.legend(loc='best')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.plot([0, 1], [0, 1], 'r-')
plt.show()
调参后lightgbm单mox模型在验证集上的AUC： 0.7325304803332136
"保存模型到本地"
import pickle
pickle.dump(final_model_lgb, open('model_lgb_best.pkl', 'wb'))
717
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
