基于LightGBM的数据实践

巷子里的猫X

于 2024-10-10 20:09:47 发布

阅读量201

点赞数 6

分类专栏：机器学习原理与实践文章标签：算法机器学习

本文链接：https://blog.csdn.net/qq_52421831/article/details/142830942

版权

机器学习原理与实践专栏收录该内容

10 篇文章 0 订阅

订阅专栏

一、LightGBM基础介绍

LightGBM可以说是xgboost上的延续，在xgboost上做了一些优化，从应用层面上来说最直接的两点就是：(1)：速度变快了很多，(2)：可以支持类别型特征(在xgb中是不可以的，需要one-hot)。

从算法层面上做了如下的几点优化：

1：直方图算法(Histogram algorithm)

该算法的思想是把连续的浮点值离散成若干个整数(其实也是分箱),然后构造一个宽度为K的直方图。在遍历的时候，只需要遍历离散化的索引。听起来好像和xgb中的近似算法有点类似，但是绝不是这样的。

上面是我们近似算法的例子，它做的只是减少了一些寻找分裂点的次数，其余的还是没有任何变化。下面是直方图算法的例子：

可以发现直方图算法在根据每个特征分箱的时候是将每个bin中的样本数量，样本一阶梯度和，二阶梯度和都计算出来的。

既然叫做直方图，我们就转化为直方图的方式：

直方图做差

我们可以想象地到，每分裂一次之后，之前所有特征的直方图就不在适用了，这时候需要重新计算。假设我们以特征A进行分裂，那么特征A的新的直方图很容易得到，根据某个bin一分为二就可以。每个bin里面的样本数量，梯度和都保持不变。但是其余特征呢？、

我们需要重新根据分裂后的情况，计算左侧样本和右侧样本关于所有特征的直方图。使用直方图另外一个好处就是直方图做差，一个叶子的直方图可以由它的父亲节点的直方图与它兄弟的直方图做差得到。这样的话我们只需要计算一个叶节点，然后就可迅速计算出另外一个叶节点，速度提升了一倍。上图是一个直方图做差的例子。

假设我们根据特征A进行分裂之后，我们需要重新计算在特征B中直方图中每个bin中样本的数量，以及一二阶梯度和。但是，我们只需要计算一侧，因为某个特征两个叶节点的直方图就等于父节点的直方图，使用这种方法可以让我们少计算一般的直方图，也就是节省了一半的时间。

2、单边梯度采样

单边梯度采样算法(GOSS)的目的是减少一些梯度很小的样本，保留梯度较大的样本，为了保持训练集的分布，然后又增加没有去除的梯度较小样本的权重。具体的算法步骤为：

比起直方分布算法，这个算法还是很好理解的。首先我们选取𝑎比例的梯度最大的样本 $TopN=a \times len(I)$ ,然后从剩下的样本中随机挑选𝑏比例的样本： $randN=b \times len(I)$ 。由于担心采样导致梯度较少的样本太少，因此对采样的𝑟𝑎𝑛𝑑𝑁样本进行提权，提权的比例为 $\frac{1-a}{b}$ (其实就是出去TopN样本的数量除以挑选入randN样本的数量)。其余无任何变化。

3、互斥捆绑算法

互斥捆绑算法实现起来很复杂。我这里简单地给大家介绍一下它的基本思路吧。该算法是主要目的是将多个互斥的特征合并成一个特征，在正常的数据中，很多特征会出现0或者NA(也就是稀疏性)。比如一个特征数据为：A=[0,0,1,0,1,0,2,0,0,0],另外一个特征数据为：B=[1,0,0,0,0,0,0,2,3,4]。这两个特征其实就是互斥的。当A特征不为0的时候，B特征为0，反之亦然。我们可以合并特征为C=[3,0,1,0,1,0,2,4,5,6]。(其实就是将B特征中所有的数都加上2，然后再加上A特征)。这样我们可以根据一个特征做到之前两个特征才能做到的事情。

比如：
A<=1 并且 B>0 我们可以根据：C>2来实现。得到的都是1，8，9，10四个数据。
A>1 并且B=0 我们可以根据 C=2 来实现

非常有意思一点是：我们常见的one-hot产生的特征就是明显的互斥特征，因此在Light中我们是鼓励使用自带的类别特征处理方法的。

4、其余细节

(1)：基于many vs many的类别特征处理方法。

下图中X是类别特征，左图是常见的one-hot的方法，右图就是many vs many的方法。ont_hot在树模型中的缺陷：当类型较多的时候在这个类别特征进行切分，也会把数据切分到很多零碎的小空间上，造成不均衡的切分。两种方法其实就我尝试的结果，效果差不多，不果还是推荐设置类别特征，使用many vs many的方法。

(2):带深度限制的 Leaf-wise 算法

上面是xgb的分裂方法，采用的是按层生长level(depth)-wise生长策略。可以按照多进程的方法同时分裂每一层的所有叶子，速度快不容易过拟合。但是对每一层不加区分的强行分裂(有些分裂收益很低，没必要分裂的)，同时又带来了多余的开销。

下面是lightGBM采用的leaf-wise的分裂策略。每次从当前所有叶子中找到分裂增益最大的那一个进行分裂，因此每次分裂前需要计算从各个叶节点分裂的收益，这种方法可以避免无效的分裂。在分裂次数相同的情况下，可以降低误差，得到更好的精度。为了防止过拟合可以控制参数，限制分裂的深度。

二、lightGBM模型的实现

与xgb模型一样，我们也介绍常见的两种调用方式。一种是基于原始的，一种是基于sklearn的。个人习惯不同会使用不同的方法(比起tensor的n种方法来说，已经很友好了)。我们这里也介绍两种实现。

1：lgb二分类简单实现

import numpy as np
from sklearn import datasets
import pandas as pd
from sklearn.model_selection import train_test_split
X=datasets.load_breast_cancer()['data']
Y=datasets.load_breast_cancer()['target']
fea=datasets.load_breast_cancer()['feature_names']
data=pd.DataFrame(X,columns=fea)
data['target']=Y
x_train=data[fea]
y_train=data['target'].copy()
data['target'].value_counts()

'''
1    357
0    212
Name: target, dtype: int64
'''

首先我们使用lgb的原生包进行训练：

from sklearn.model_selection import KFold,StratifiedKFold
import lightgbm as lgb
import datetime
import sklearn
import warnings 
warnings.filterwarnings('ignore')
folds = KFold(n_splits=5, shuffle=True, random_state=1996)


params = {
    'learning_rate': 0.05,
    'boosting_type': 'gbdt',
    'objective': 'binary',
    'metric': 'auc',
    'num_leaves': 32,
    'feature_fraction': 0.8,
    'bagging_fraction': 0.8,
    'seed': 2020,
    'min_data_in_leaf': 100,
    'n_jobs': -1, 
    'verbose': -1, }
oof=np.zeros(x_train.shape[0]) 
for fold_, (train_index, test_index) in enumerate(folds.split(x_train, y_train)):
    print("第{}折".format(fold_))
    train_x, test_x, train_y, test_y = x_train.iloc[train_index], x_train.iloc[test_index], y_train.iloc[train_index], y_train.iloc[test_index]   
    num_round=5
    dtrain = lgb.Dataset(train_x, train_y)
    dval = lgb.Dataset(test_x, test_y)
    lgb_model = lgb.train(
        params,
        dtrain,
        num_boost_round=10000,
        valid_sets=[dtrain, dval], 
        early_stopping_rounds=30,
        verbose_eval=50,
    )
    val_train=lgb_model.predict(test_x)
    oof[test_index]=val_train
print("最终auc为：",sklearn.metrics.roc_auc_score(y_train,oof))
    
'''
第0折
Training until validation scores don't improve for 30 rounds.
[50]	training's auc: 0.993987	valid_1's auc: 0.997327
Early stopping, best iteration is:
[63]	training's auc: 0.995779	valid_1's auc: 0.997661
第1折
Training until validation scores don't improve for 30 rounds.
[50]	training's auc: 0.995666	valid_1's auc: 0.987765
Early stopping, best iteration is:
[49]	training's auc: 0.995521	valid_1's auc: 0.988095
第2折
Training until validation scores don't improve for 30 rounds.
[50]	training's auc: 0.993742	valid_1's auc: 0.998712
Early stopping, best iteration is:
[50]	training's auc: 0.993742	valid_1's auc: 0.998712
第3折
Training until validation scores don't improve for 30 rounds.
[50]	training's auc: 0.995645	valid_1's auc: 0.988514
[100]	training's auc: 0.998829	valid_1's auc: 0.992568
Early stopping, best iteration is:
[85]	training's auc: 0.99811	valid_1's auc: 0.993581
第4折
Training until validation scores don't improve for 30 rounds.
[50]	training's auc: 0.998161	valid_1's auc: 0.965415
[100]	training's auc: 0.999752	valid_1's auc: 0.969697
[150]	training's auc: 0.999979	valid_1's auc: 0.975296
Early stopping, best iteration is:
[152]	training's auc: 1	valid_1's auc: 0.975955
最终auc为： 0.9865757623804239
'''

其次,我们采用sklearn进行训练：

from lightgbm.sklearn import LGBMClassifier
oof=np.zeros(x_train.shape[0]) 
for fold_, (train_index, test_index) in enumerate(folds.split(x_train, y_train)):
    print('Fold_{}'.format(fold_))
    train_x, test_x, train_y, test_y = x_train.iloc[train_index], x_train.iloc[test_index], y_train.iloc[train_index], y_train.iloc[test_index]


    model = LGBMClassifier(objective='binary',
                           boosting_type='gbdt',
                           num_leaves=28,
                           max_depth=8,
                           learning_rate=0.01,
                           n_estimators=10000,
                           random_state=1996,
                           is_unbalance=True,
                           metric='auc',
                           )
    
    lgb_model = model.fit(train_x,
                        train_y,
                        eval_names=['train', 'valid'],
                        eval_set=[(train_x, train_y), (test_x, test_y)],
                        verbose=100,
                        early_stopping_rounds=200)
    val_train=lgb_model.predict_proba(test_x, num_iteration=lgb_model.best_iteration_)[:, 1]
    oof[test_index]=val_train
print("最终auc为：",sklearn.metrics.roc_auc_score(y_train,oof))

'''
Fold_0
Training until validation scores don't improve for 200 rounds.
[100]	train's auc: 0.997303	valid's auc: 0.998664
[200]	train's auc: 0.999362	valid's auc: 0.998329
[300]	train's auc: 0.999876	valid's auc: 0.998329
Early stopping, best iteration is:
[103]	train's auc: 0.997303	valid's auc: 0.998998
Fold_1
Training until validation scores don't improve for 200 rounds.
[100]	train's auc: 0.997936	valid's auc: 0.970569
[200]	train's auc: 0.999216	valid's auc: 0.978505
[300]	train's auc: 0.999897	valid's auc: 0.981812
[400]	train's auc: 1	valid's auc: 0.98578
[500]	train's auc: 1	valid's auc: 0.985119
Early stopping, best iteration is:
[353]	train's auc: 1	valid's auc: 0.984127
Fold_2
Training until validation scores don't improve for 200 rounds.
[100]	train's auc: 0.995821	valid's auc: 0.996135
[200]	train's auc: 0.998545	valid's auc: 0.997746
[300]	train's auc: 0.999854	valid's auc: 0.997424
[400]	train's auc: 1	valid's auc: 0.998068
Early stopping, best iteration is:
[201]	train's auc: 0.998545	valid's auc: 0.998068
Fold_3
Training until validation scores don't improve for 200 rounds.
[100]	train's auc: 0.997637	valid's auc: 0.989189
[200]	train's auc: 0.999404	valid's auc: 0.988851
[300]	train's auc: 0.999897	valid's auc: 0.989865
[400]	train's auc: 1	valid's auc: 0.991892
[500]	train's auc: 1	valid's auc: 0.993919
Early stopping, best iteration is:
[354]	train's auc: 1	valid's auc: 0.990878
Fold_4
Training until validation scores don't improve for 200 rounds.
[100]	train's auc: 0.998347	valid's auc: 0.955204
[200]	train's auc: 0.99969	valid's auc: 0.965415
[300]	train's auc: 0.999979	valid's auc: 0.972003
[400]	train's auc: 1	valid's auc: 0.972991
[500]	train's auc: 1	valid's auc: 0.971673
Early stopping, best iteration is:
[309]	train's auc: 1	valid's auc: 0.972003
最终auc为： 0.9831932773109244
'''

在我们直接使用lgb的时候，我们在预测的时候直接model.predict就好，在二分类中输出的就是1的概率。而采用sklearn包装的，需要使用predict_proba，并且需要传入参数：num_iteration=lgb_model.best_iteration_。预测的结果有2列，第一列是结果为0的概率，第二列是结果为1的概率。

2：lgb多分类代码实现

为了方便，我这里把上面的数据其中一部分1直接设置为2

y_train[500:]=2
y_train.value_counts()

'''
1    305
0    195
2     69
Name: target, dtype: int64
'''


params = {
    'bagging_fraction': 0.85,
    'boost': 'gbdt',
    'feature_fraction': 0.85,
    'learning_rate': 0.02,
    'max_depth': 4,  
    'metric':'multi_logloss',
    'min_data_in_leaf': 1,
    'objective': 'multiclass',
    'num_class':3,
    'lambda_l1': 0.5,
    'lambda_l2': 1.2,
    'verbosity': -1,
}


folds = KFold(n_splits=5, shuffle=True, random_state=1996)

oof=np.zeros([x_train.shape[0],3]) 

val_label=[]

for fold_, (train_index, test_index) in enumerate(folds.split(x_train, y_train)):
    print("第{}折".format(fold_))
    train_x, test_x, train_y, test_y = x_train.iloc[train_index], x_train.iloc[test_index], y_train.iloc[train_index], y_train.iloc[test_index]
    
    trn_data = lgb.Dataset(train_x, train_y)
    val_data = lgb.Dataset(test_x, test_y)
    num_round=50000
    clf = lgb.train(params, 
                    trn_data, 
                    num_round, 
                    valid_sets = [trn_data, val_data], 
                    verbose_eval = 100,early_stopping_rounds = 500,
                  )

    val_train=clf.predict(test_x, num_iteration=clf.best_iteration)
    oof[test_index]=val_train    

'''
第0折
Training until validation scores don't improve for 500 rounds.
[100]	training's multi_logloss: 0.316381	valid_1's multi_logloss: 0.503319
[200]	training's multi_logloss: 0.17255	valid_1's multi_logloss: 0.446449
[300]	training's multi_logloss: 0.104334	valid_1's multi_logloss: 0.436903
[400]	training's multi_logloss: 0.0692727	valid_1's multi_logloss: 0.443223
[500]	training's multi_logloss: 0.0503941	valid_1's multi_logloss: 0.450412
[600]	training's multi_logloss: 0.0399517	valid_1's multi_logloss: 0.45708
[700]	training's multi_logloss: 0.0338657	valid_1's multi_logloss: 0.465057
Early stopping, best iteration is:
[295]	training's multi_logloss: 0.106774	valid_1's multi_logloss: 0.4361
第1折
Training until validation scores don't improve for 500 rounds.
[100]	training's multi_logloss: 0.319733	valid_1's multi_logloss: 0.500965
[200]	training's multi_logloss: 0.179031	valid_1's multi_logloss: 0.454095
[300]	training's multi_logloss: 0.109023	valid_1's multi_logloss: 0.444972
[400]	training's multi_logloss: 0.0729738	valid_1's multi_logloss: 0.447401
[500]	training's multi_logloss: 0.0523144	valid_1's multi_logloss: 0.453079
[600]	training's multi_logloss: 0.0415938	valid_1's multi_logloss: 0.461504
[700]	training's multi_logloss: 0.0351875	valid_1's multi_logloss: 0.463114
[800]	training's multi_logloss: 0.0323636	valid_1's multi_logloss: 0.465132
Early stopping, best iteration is:
[355]	training's multi_logloss: 0.0869976	valid_1's multi_logloss: 0.443858
第2折
Training until validation scores don't improve for 500 rounds.
[100]	training's multi_logloss: 0.322191	valid_1's multi_logloss: 0.514117
[200]	training's multi_logloss: 0.171936	valid_1's multi_logloss: 0.45388
[300]	training's multi_logloss: 0.102635	valid_1's multi_logloss: 0.445143
[400]	training's multi_logloss: 0.0673399	valid_1's multi_logloss: 0.451089
[500]	training's multi_logloss: 0.0491556	valid_1's multi_logloss: 0.460862
[600]	training's multi_logloss: 0.0392411	valid_1's multi_logloss: 0.46876
[700]	training's multi_logloss: 0.0340304	valid_1's multi_logloss: 0.473031
[800]	training's multi_logloss: 0.0316171	valid_1's multi_logloss: 0.4741
Early stopping, best iteration is:
[304]	training's multi_logloss: 0.100669	valid_1's multi_logloss: 0.444896
第3折
Training until validation scores don't improve for 500 rounds.
[100]	training's multi_logloss: 0.286728	valid_1's multi_logloss: 0.667402
[200]	training's multi_logloss: 0.141262	valid_1's multi_logloss: 0.639446
[300]	training's multi_logloss: 0.0838781	valid_1's multi_logloss: 0.65439
[400]	training's multi_logloss: 0.0560208	valid_1's multi_logloss: 0.688699
[500]	training's multi_logloss: 0.0416471	valid_1's multi_logloss: 0.715084
[600]	training's multi_logloss: 0.0346003	valid_1's multi_logloss: 0.727676
[700]	training's multi_logloss: 0.031354	valid_1's multi_logloss: 0.734887
Early stopping, best iteration is:
[210]	training's multi_logloss: 0.133505	valid_1's multi_logloss: 0.63655
第4折
Training until validation scores don't improve for 500 rounds.
[100]	training's multi_logloss: 0.307255	valid_1's multi_logloss: 0.540643
[200]	training's multi_logloss: 0.168882	valid_1's multi_logloss: 0.508212
[300]	training's multi_logloss: 0.103895	valid_1's multi_logloss: 0.513859
[400]	training's multi_logloss: 0.0677857	valid_1's multi_logloss: 0.526925
[500]	training's multi_logloss: 0.0492562	valid_1's multi_logloss: 0.536568
[600]	training's multi_logloss: 0.0389841	valid_1's multi_logloss: 0.547573
[700]	training's multi_logloss: 0.0331248	valid_1's multi_logloss: 0.561236
Early stopping, best iteration is:
[242]	training's multi_logloss: 0.137563	valid_1's multi_logloss: 0.507335
'''

3、设置类别的样本权重

一般我们在做分类任务的时候，类别之间总是不均衡的，比如很多风控场景，0-1的比例是几百比1，这时候我们需要更重视label为1的样本，因此可以在训练的时候可以使用类别权重提升某个类别分类错误时候的loss，用于提高模型的准确性。

对于二分类，我们可以直接设置is_unbalance，它会自动计算，某个类别的权重与该类别的样本数成反比。如果是多分类的话 "class_weight"参数需要输入一个字典，设置每个类别的权重。

y_train=data['target'].copy()
params = {
    'learning_rate': 0.05,
    'boosting_type': 'gbdt',
    'objective': 'binary',
    'metric': 'auc',
    'num_leaves': 32,
    'feature_fraction': 0.8,
    'bagging_fraction': 0.8,
    'seed': 2020,
    'min_data_in_leaf': 100,
    "class_weight ":"is_unbalance", 
    'n_jobs': -1, 
    'verbose': -1, }
folds = KFold(n_splits=5, shuffle=True, random_state=1996)

oof=np.zeros(x_train.shape[0]) 

val_label=[]

for fold_, (train_index, test_index) in enumerate(folds.split(x_train, y_train)):
    print("第{}折".format(fold_))
    train_x, test_x, train_y, test_y = x_train.iloc[train_index], x_train.iloc[test_index], y_train.iloc[train_index], y_train.iloc[test_index]
    
    trn_data = lgb.Dataset(train_x, train_y)
    val_data = lgb.Dataset(test_x, test_y)
    num_round=50000
    clf = lgb.train(params, 
                    trn_data, 
                    num_round, 
                    valid_sets = [trn_data, val_data], 
                    verbose_eval = 100,early_stopping_rounds = 500,
                  )

    val_train=clf.predict(test_x, num_iteration=clf.best_iteration)
    oof[test_index]=val_train   

'''
第0折
Training until validation scores don't improve for 500 rounds.
[100]	training's auc: 0.998229	valid_1's auc: 0.997661
[200]	training's auc: 1	valid_1's auc: 0.998329
[300]	training's auc: 1	valid_1's auc: 0.997661
[400]	training's auc: 1	valid_1's auc: 0.998329
[500]	training's auc: 1	valid_1's auc: 0.998664
[600]	training's auc: 1	valid_1's auc: 0.998664
Early stopping, best iteration is:
[135]	training's auc: 0.999629	valid_1's auc: 0.998664
第1折
Training until validation scores don't improve for 500 rounds.
[100]	training's auc: 0.998885	valid_1's auc: 0.987103
[200]	training's auc: 1	valid_1's auc: 0.99041
[300]	training's auc: 1	valid_1's auc: 0.991733
[400]	training's auc: 1	valid_1's auc: 0.991402
[500]	training's auc: 1	valid_1's auc: 0.991733
[600]	training's auc: 1	valid_1's auc: 0.991402
Early stopping, best iteration is:
[165]	training's auc: 1	valid_1's auc: 0.989087
第2折
Training until validation scores don't improve for 500 rounds.
[100]	training's auc: 0.998253	valid_1's auc: 0.99839
[200]	training's auc: 1	valid_1's auc: 0.998068
[300]	training's auc: 1	valid_1's auc: 0.997424
[400]	training's auc: 1	valid_1's auc: 0.997746
[500]	training's auc: 1	valid_1's auc: 0.997424
Early stopping, best iteration is:
[50]	training's auc: 0.993742	valid_1's auc: 0.998712
第3折
Training until validation scores don't improve for 500 rounds.
[100]	training's auc: 0.998829	valid_1's auc: 0.992568
[200]	training's auc: 1	valid_1's auc: 0.996622
[300]	training's auc: 1	valid_1's auc: 0.998649
[400]	training's auc: 1	valid_1's auc: 0.999324
[500]	training's auc: 1	valid_1's auc: 0.999324
[600]	training's auc: 1	valid_1's auc: 0.999324
Early stopping, best iteration is:
[173]	training's auc: 1	valid_1's auc: 0.995946
第4折
Training until validation scores don't improve for 500 rounds.
[100]	training's auc: 0.999752	valid_1's auc: 0.969697
[200]	training's auc: 1	valid_1's auc: 0.975955
[300]	training's auc: 1	valid_1's auc: 0.977931
[400]	training's auc: 1	valid_1's auc: 0.977931
[500]	training's auc: 1	valid_1's auc: 0.979578
[600]	training's auc: 1	valid_1's auc: 0.978261
Early stopping, best iteration is:
[152]	training's auc: 1	valid_1's auc: 0.975955
'''

4、设置样本权重

样本权重是给每个样本设置一个权重，比如我们在时间序列场景中，可能会对距离当前时间越近的样本越看重。这时候我们如果想要对部分样本加权的话，就要可以使用到样本权重了。

import random
#随机给所有样本一个权重
data_weight=np.array([random.random() for i in range(569)])
y_train=data['target'].copy()


oof=np.zeros(x_train.shape[0]) 

for fold_, (train_index, test_index) in enumerate(folds.split(x_train, y_train)):
    print('Fold_{}'.format(fold_))
    train_x, test_x, train_y, test_y = x_train.iloc[train_index], x_train.iloc[test_index], y_train.iloc[train_index], y_train.iloc[test_index]
    weights=data_weight[train_index]

    model = LGBMClassifier(objective='binary',
                           boosting_type='gbdt',
                           num_leaves=28,
                           max_depth=8,
                           learning_rate=0.01,
                           n_estimators=10000,
                           random_state=1996,
                           is_unbalance=True,
                           metric='auc',
                           )
    
    lgb_model = model.fit(train_x,
                        train_y,
                        sample_weight=weights,
                        eval_names=['train', 'valid'],
                        eval_set=[(train_x, train_y), (test_x, test_y)],
                        verbose=100,
                        early_stopping_rounds=200)
    val_train=lgb_model.predict_proba(test_x, num_iteration=lgb_model.best_iteration_)[:, 1]
    oof[test_index]=val_train
print("最终auc为：",sklearn.metrics.roc_auc_score(y_train,oof))

'''
Fold_0
Training until validation scores don't improve for 200 rounds.
[100]	train's auc: 0.997717	valid's auc: 0.995322
[200]	train's auc: 0.999408	valid's auc: 0.995657
[300]	train's auc: 0.999951	valid's auc: 0.996325
[400]	train's auc: 1	valid's auc: 0.996659
[500]	train's auc: 1	valid's auc: 0.997327
Early stopping, best iteration is:
[378]	train's auc: 1	valid's auc: 0.996659
Fold_1
Training until validation scores don't improve for 200 rounds.
[100]	train's auc: 0.998411	valid's auc: 0.95668
[200]	train's auc: 0.999287	valid's auc: 0.973876
[300]	train's auc: 0.999919	valid's auc: 0.974206
[400]	train's auc: 1	valid's auc: 0.977183
[500]	train's auc: 1	valid's auc: 0.978836
Early stopping, best iteration is:
[368]	train's auc: 1	valid's auc: 0.974206
Fold_2
Training until validation scores don't improve for 200 rounds.
[100]	train's auc: 0.995944	valid's auc: 0.987118
[200]	train's auc: 0.998981	valid's auc: 0.996135
[300]	train's auc: 0.999833	valid's auc: 0.995491
Early stopping, best iteration is:
[183]	train's auc: 0.998753	valid's auc: 0.996135
Fold_3
Training until validation scores don't improve for 200 rounds.
[100]	train's auc: 0.997182	valid's auc: 0.990541
[200]	train's auc: 0.999477	valid's auc: 0.991892
[300]	train's auc: 0.999874	valid's auc: 0.993243
[400]	train's auc: 0.999989	valid's auc: 0.993919
[500]	train's auc: 0.999999	valid's auc: 0.995946
[600]	train's auc: 1	valid's auc: 0.995946
Early stopping, best iteration is:
[467]	train's auc: 0.999999	valid's auc: 0.995946
Fold_4
Training until validation scores don't improve for 200 rounds.
[100]	train's auc: 0.997774	valid's auc: 0.95191
[200]	train's auc: 0.999448	valid's auc: 0.958827
[300]	train's auc: 0.999966	valid's auc: 0.963109
[400]	train's auc: 0.999995	valid's auc: 0.968709
[500]	train's auc: 0.999999	valid's auc: 0.970685
[600]	train's auc: 1	valid's auc: 0.971673
[700]	train's auc: 1	valid's auc: 0.969368
Early stopping, best iteration is:
[568]	train's auc: 1	valid's auc: 0.97332
最终auc为： 0.9828101051741451
'''

5、类别特征

lgb与xgb相比，类别特征的处理是很大的差异之处，lgb支持直接传入类别特征。只需要将类别特征设置为"category"后就可以直接放入lgb模型中跑了。

cols=['mean radius']
x_train['mean radius2']=x_train['mean radius'].apply(lambda x:str(int(x))) ##随便新建一个String类型特征
for i in cols:
    x_train[i]=x_train[i].astype('category') # 将某个需要为类别特征的特征设置为 ”category“

6：其余可调参数

params = {
    'bagging_fraction': 0.85,
    'boost': 'gbdt',
    'feature_fraction': 0.85,
    'learning_rate': 0.02,
    'max_depth': 4,  
    'metric':'multi_logloss',
    'min_data_in_leaf': 1,
    'objective': 'multiclass',
    'num_class':3,
    'lambda_l1': 0.5,
    'lambda_l2': 1.2,
    'verbosity': -1,
}

1：learning_rate 学习率

2：max_depth 树的深度，这个一个般看特征数量和样本数量而定，特征越多，可以稍微深点，样本越多反倒可以浅一点。

3：min_data_in_leaf 每个叶节点的样本数量。越大就越减少分裂，可以防止过拟合

4：lambda_l1，lambda_l2，正则化，这个一般取默认就可以了。

5：bagging_fraction，feature_fraction 样本特征的采样比例，

在我看来，树模型一般来说还是样本，特征的艺术。对于参数其实不需要和深度学习一样特别关注。