本系列基本不讲数学原理,只从代码角度去让读者们利用最简洁的Python代码实现机器学习方法。
集成模型发展到现在的XGboost,LightGBM,都是目前竞赛项目会采用的主流算法。是真正的具有做项目的价值。这两个方法都是具有很多GBM没有的特点,比如收敛快,精度好,速度快等等。但由于他们底层不是Python,没有进sklearn库,要自己单独安装,用法和sklearn库也不完全相同。
两种模型都有自己的原生用法和sklearn库接口的用法,下面把回归和分类问题案例都一一介绍。只是案例演示,因此使用都采用sklearn库的自带数据集,会方便一点。
模块安装
安装都很简单,在这里输入
或者Win+R,打开CMD,输入下面两句
pip install xgboost
pip install lightgbm
然后就等自动装了
参数详情
两个模型都要大量的超参数,后面可能用到。先了解一下,后面有不明白的再回头看
还有eta 表示学习率,默认值0.3;
gamma 表示叶子结点进一步分裂的阈值。即分裂这个节点让损失函数下降超过这个值才会进行分裂,默认值0;
max_leaves 表示最大叶子节点数,默认0;
max_bin 最大桶数量,默认值256;
min_child_weigh 表示子节点包含实例权重的最小和,防止过拟合用的,越大越不容易过拟合,默认值0;
subsmaple 表示训练样本的采样率,即划分多少去训练,若是训练前就划分了训练测试集就不用管,默认值1;
colsample_bytree 表示列采样率,默认值1;
colsample_bylevel 每一级每次分裂的采样率,默认值1;
scale_pos_weight 控制正负样本的权重平衡,取值应该设为负类样本量/正类样本量,默认值1;
predictor 预测器类型,默认值cpu_predictor, 还可以换显卡加速:gpu_predictor。
seed 随机数种子,默认0;
slient 打印运行的信息,默认打印,默认值0;
objective [默认值=reg:linear]
reg:linear– 线性回归
reg:logistic – 逻辑回归
binary:logistic – 二分类逻辑回归,输出为概率
binary:logitraw – 二分类逻辑回归,输出的结果为wTx
count:poisson – 计数问题的poisson回归,输出结果为poisson分布。在poisson回归中,max_delta_step的默认值为0.7 (used to safeguard optimization)
multi:softmax – 设置 XGBoost 使用softmax目标函数做多分类,需要设置参数num_class(类别个数)
multi:softprob – 如同softmax,但是输出结果为ndata*nclass的向量,其中的值是每个数据分为每个类的概率。
eval_metric [默认值=取决于目标函数选择]
rmse: 均方根误差
mae: 平均绝对值误差
logloss: negative log-likelihood
error: 二分类错误率。其值通过错误分类数目与全部分类数目比值得到。对于预测,预测值大于0.5被认为是正类,其它归为负类。 error@t: 不同的划分阈值可以通过 ‘t’进行设置
merror: 多分类错误率
mlogloss: 多分类log损失
auc: 曲线下的面积
ndcg: Normalized Discounted Cumulative Gain
map: 平均正确率
一般来说,我们都会使用xgboost.train(params, dtrain)函数来训练我们的模型。这里的params指的是booster参数。
XGboost原生用法
分类
import numpy as np
import pandas as pd
#import pickle
import xgboost as xgb
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
#鸢尾花
iris=load_iris()
X=iris.data
y=iris.target
X.shape,y.shape
最经典的3分类的鸢尾花数据集
划分训练测试集,将数据变为xgb需要的格式
# 做数据切分
X_train, X_test,y_train, y_test = train_test_split(X,y,test_size=0.2,random_state=0)
xgb_train = xgb.DMatrix(X_train, y_train)
xgb_test = xgb.DMatrix( X_test,y_test )
设置参数
params = {'objective':'multi:softmax','num_class':3,'booster':'gbtree','max_depth':5, 'eta':0.1, 'subsample':0.7, 'colsample_bytree':0.7}
训练
num_round=50
watchlist = [(xgb_train,'train'), (xgb_test,'test')]
model = xgb.train(params, xgb_train, num_round, watchlist)
预测
pred = model.predict(xgb_test)
pred
error_rate=np.sum(pred!=y_test)/y_test.shape[0]
error_rate #错误率
回归问题
from sklearn.datasets import load_boston
X,y=load_boston(return_X_y=True)
X.shape,y.shape
波士顿房价数据集,经典回归数据集
X_train,X_test,y_train,y_test= train_test_split(X,y,test_size=0.2,random_state=0)
print(X_train.shape,X_test.shape,y_train.shape,y_test.shape)
xgb_train = xgb.DMatrix(X_train, y_train)
xgb_test = xgb.DMatrix( X_test,y_test )
设置参数
params = {'objective':'reg:squarederror','booster':'gbtree','max_depth':5, 'eta':0.1, 'min_child_weight':1}
num_round=50
watchlist = [(xgb_train,'train'), (xgb_test,'test')]
model = xgb.train(params, xgb_train, num_round, watchlist)
pred = model.predict(xgb_test)
pred,y_test
计算均方误差,拟合优度
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error,r2_score
reg=LinearRegression()
reg.fit(y_test.reshape(-1, 1),pred.reshape(-1, 1))
reg.score(y_test.reshape(-1, 1),pred.reshape(-1, 1))
mean_squared_error(y_test,pred),r2_score(y_test,pred)
交叉验证
#交叉验证
result=xgb.cv(params=params,dtrain=xgb_train,nfold=10,metrics='rmse',#'auc'
num_boost_round=300,as_pandas=True,seed=123)
result.shape
result.head()
画出交叉验证误差图
# Plot CV Errors
import matplotlib.pyplot as plt
plt.plot(range(1, 301), result['train-rmse-mean'], 'k', label='Training Error')
plt.plot(range(1, 301), result['test-rmse-mean'], 'b', label='Test Error')
plt.xlabel('Number of Trees')
plt.ylabel('RMSE')
plt.axhline(0, linestyle='--', color='k', linewidth=1)
plt.legend()
plt.title('CV Errors for XGBoost')
plt.show()
自定义目标函数和损失函数
XGboost还可以自定义损失函数和评价函数
from sklearn.datasets import load_breast_cancer
X,y=load_breast_cancer(return_X_y=True)
X.shape,y.shape
乳腺癌数据集
X_train,X_test,y_train,y_test= train_test_split(X,y,test_size=0.2,random_state=8)
print(X_train.shape,X_test.shape,y_train.shape,y_test.shape)
xgb_train = xgb.DMatrix(X_train, y_train)
xgb_test = xgb.DMatrix( X_test,y_test )
params = {'booster':'gbtree','max_depth':5, 'eta':0.1}
num_round=50
watchlist = [(xgb_train,'train'), (xgb_test,'test')]
定义损失函数和评价函数
def logregobj(preds,dtrain):
labels=dtrain.get_label()
preds=1.0/(1.0+np.exp(-preds))
grad=preds-labels
hess=preds*(1.0-preds)
return grad,hess
def evalerror(preds,dtrain):
labels=dtrain.get_label()
return 'error',float(sum(labels!=(preds>0.0)))/len(labels)
训练
model = xgb.train(params, xgb_train, num_round, watchlist,obj=logregobj,feval=evalerror)
交叉验证也可以使用自定义函数
result=xgb.cv(params=params,dtrain=xgb_train,nfold=10,metrics='auc',#'auc'
num_boost_round=300,as_pandas=True,seed=123,obj=logregobj,feval=evalerror)
result.head()
# Plot CV Errors
import matplotlib.pyplot as plt
plt.plot(range(1, 301), result['train-error-mean'], 'k', label='Training Error')
plt.plot(range(1, 301), result['test-error-mean'], 'b', label='Test Error')
plt.xlabel('Number of Trees')
plt.ylabel('AUC')
plt.axhline(0, linestyle='--', color='k', linewidth=1)
plt.legend()
plt.title('CV Errors for XGBoost')
plt.show()
XGboost的sklearn库接口
sklearn库接口就好用很多,符合sklearn库的一些常用的函数,例如交叉验证,网格化搜参,变量筛选,都可以用。
回归
import xgboost as xgb
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold, train_test_split, GridSearchCV
from sklearn.metrics import confusion_matrix, mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris, load_boston
from sklearn.datasets import load_breast_cancer
X, y= load_boston(return_X_y=True)
拟合评价(经典sklearn用法)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)
model = xgb.XGBRegressor(objective='reg:squarederror', n_estimators=300, max_depth=6,
subsample=0.6, colsample_bytree=0.8, learning_rate=0.1, random_state=0)
model.fit(X_train, y_train)
model.score(X_test, y_test)
pred = model.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, pred))
rmse
回归交叉验证
rng = np.random.RandomState(123)
kf = KFold(n_splits=3, shuffle=True, random_state=rng)
print("在3折数据上的交叉验证")
for train_index, test_index in kf.split(X):
xgb_model = xgb.XGBRegressor(objective='reg:squarederror', n_estimators=300, max_depth=6,subsample=0.6,
colsample_bytree=0.8, learning_rate=0.1, random_state=0).fit(X[train_index],y[train_index])
predictions = xgb_model.predict(X[test_index])
actuals = y[test_index]
print("均方根误差:")
print(np.sqrt(mean_squared_error(actuals, predictions)))
print('拟合优度')
print(xgb_model.score(X[test_index],y[test_index]))
回归网格化搜索最优超参数
# 回归网格化搜索最优超参数
model = xgb.XGBRegressor(objective='reg:squarederror',subsample=0.6, colsample_bytree=0.8, random_state=0,nthread=8)
param_dict = {'max_depth': [5,6,7,8],'n_estimators': [100,200,300],'learning_rate':[0.05,0.1,0.2]}
clf = GridSearchCV(model, param_dict, cv=10,verbose=1 , scoring='r2')
clf.fit(X_train, y_train)
print(clf.best_score_)
print(clf.best_params_)
分类交叉验证
#二分类
rng = np.random.RandomState(123)
X,y=load_breast_cancer(return_X_y=True)
print(X.shape,y.shape)
kf = KFold(n_splits=3, shuffle=True, random_state=rng)
print("在3折数据上的交叉验证")
for train_index, test_index in kf.split(X):
xgb_model = xgb.XGBClassifier(objective='binary:logistic', n_estimators=300,random_state=0,eta=0.1,max_depth=6,
use_label_encoder=False,eval_metric=['logloss','auc','error']).fit(X[train_index],y[train_index])
predictions = xgb_model.predict(X[test_index])
actuals = y[test_index]
print("混淆矩阵:")
print(confusion_matrix(actuals, predictions))
# 多分类:混淆矩阵
print("\nIris: 多分类")
iris = load_iris()
y = iris['target']
X = iris['data']
kf = KFold(n_splits=5, shuffle=True, random_state=rng)
print("在5折数据上的交叉验证")
for train_index, test_index in kf.split(X):
xgb_model = xgb.XGBClassifier(objective='multi:softmax', n_estimators=300,random_state=0,eta=0.1,max_depth=6,
use_label_encoder=False,eval_metric=['logloss','auc','error']).fit(X[train_index],y[train_index])
predictions = xgb_model.predict(X[test_index])
actuals = y[test_index]
print("混淆矩阵:")
print(confusion_matrix(actuals, predictions))
分类网格化搜索最优超参数
# 网格化搜索最优超参数
print("参数最优化:")
X,y=load_breast_cancer(return_X_y=True)
xgb_model = xgb.XGBClassifier(objective='binary:logistic',random_state=0,use_label_encoder=False,eval_metric=['logloss','auc','error'])
param_dict = {'max_depth': [2,4,6],'n_estimators': [50,100,200],'eta':[0.05,0.1,0.2]}
clf = GridSearchCV(xgb_model, param_dict, verbose=1)
clf.fit(X,y)
print(clf.best_score_)
print(clf.best_params_)
早停
和神经网络一样,可以使用早停防止过拟合
#早停
X_train, X_val, y_train, y_val = train_test_split(X, y,test_size=0.2, random_state=0)
clf = xgb.XGBClassifier(objective='binary:logistic',use_label_encoder=False,random_state=0)
clf.fit(X_train, y_train, early_stopping_rounds=10,
eval_metric="auc",eval_set=[(X_val, y_val)])
变量重要性
xgb包自带的画图用法
#变量重要性
xgb.plot_importance(clf,height=0.5,importance_type='gain',max_num_features=10)
sklearn库用法
clf.feature_importances_
cancer=load_breast_cancer()
cancer.feature_names
sorted_index = clf.feature_importances_.argsort()
plt.figure(figsize=(10,5))
plt.barh(range(len(cancer.feature_names)), clf.feature_importances_[sorted_index])
plt.yticks(np.arange(len(cancer.feature_names)),cancer.feature_names[sorted_index])
plt.xlabel('Feature Importance')
plt.ylabel('Feature')
plt.title('GradientBoosting')
#plt.savefig('梯度提升特征排序.png')
plt.tight_layout()
变量筛选
根据变量重要性,小于阈值的变量就扔掉
from sklearn.feature_selection import SelectFromModel
selection =SelectFromModel(clf,threshold=0.05,prefit=True)
select_X_train=selection.transform(X_train)
select_X_train.shape
threshold=0.05,表示变量重要性小于0.05就扔掉,最后只留下了四个变量(和上图也一致)
将测试集也筛选一下
select_X_val=selection.transform(X_val)
select_X_val.shape
查看一些筛出来了那些变量
print(selection.get_support())
print(selection.get_support(True))
[cancer.feature_names[i] for i in selection.get_support(True)]
xgb到这了
lightgbm用起来其实和xgboost差不多,就是参数有细微的差别,用sklearn库会更加一致,当然也展示一下原生用法。
LightGBM原生用法
from sklearn.datasets import load_iris
import lightgbm as lgb
from lightgbm import plot_importance
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# 加载鸢尾花数据集
iris = load_iris()
X,y = iris.data,iris.target
# 数据集分割
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,random_state=123457)
# 参数
params = {
'booster': 'gbtree',
'objective': 'multiclass', #回归:'objective': 'regression'
'num_class': 3,
'num_leaves': 31,
'subsample': 0.8,
'bagging_freq': 1,
'feature_fraction ': 0.8,
'slient': 1,
'learning_rate ': 0.01,
'seed': 0
}
# 构造训练集
dtrain = lgb.Dataset(X_train,y_train)
dtest = lgb.Dataset(X_test,y_test)
num_rounds = 500
model = lgb.train(params,dtrain, num_rounds, valid_sets=[dtrain, dtest],
verbose_eval=100, early_stopping_rounds=10)
# 对测试集进行预测
y_pred = model.predict(X_test)
# 计算准确率
accuracy = accuracy_score(y_test, np.argmax(y_pred, axis=1))
print('accuarcy:%.2f%%'%(accuracy*100))
# 显示重要特征
plot_importance(model)
plt.show()
# 模型保存
#gbm.save_model('model.txt')
# 模型加载
#gbm = lgb.Booster(model_file='model.txt')
LightGBM的sklearn接口
回归
from lightgbm import LGBMRegressor
from lightgbm import plot_importance
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_boston
from sklearn.metrics import mean_squared_error
# 导入数据集
boston = load_boston()
X ,y = boston.data,boston.target
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,random_state=0)
model = LGBMRegressor(
boosting_type='gbdt',
num_leaves=31,
max_depth=-1,
learning_rate=0.1,
n_estimators=100,
objective='regression', # 默认是二分类
min_split_gain=0.0,
min_child_samples=20,
subsample=1.0,
subsample_freq=0,
colsample_bytree=1.0,
reg_alpha=0.0,
reg_lambda=0.0,
random_state=None,
silent=True
)
model.fit(X_train,y_train, eval_set=[(X_train, y_train), (X_test, y_test)],
verbose=100, early_stopping_rounds=50)
# 对测试集进行预测
y_pred = model.predict(X_test)
mse = mean_squared_error(y_test,y_pred)
print('mse', mse)
# 显示重要特征
plot_importance(model)
plt.show()
分类
from lightgbm import LGBMClassifier
from sklearn.datasets import load_iris
from lightgbm import plot_importance
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# 加载样本数据集
iris = load_iris()
X,y = iris.data,iris.target
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,random_state=12343)
model = LGBMClassifier(
max_depth=3,
learning_rate=0.1,
n_estimators=200, # 使用多少个弱分类器
objective='multiclass',
num_class=3,
booster='gbtree',
min_child_weight=2,
subsample=0.8,
colsample_bytree=0.8,
reg_alpha=0,
reg_lambda=1,
seed=0 # 随机数种子
)
model.fit(X_train,y_train, eval_set=[(X_train, y_train), (X_test, y_test)],
verbose=100, early_stopping_rounds=50)
# 对测试集进行预测
y_pred = model.predict(X_test)
model.predict_proba
#计算准确率
accuracy = accuracy_score(y_test,y_pred)
print('accuracy:%3.f%%'%(accuracy*100))
# 显示重要特征
plot_importance(model)
plt.show()
当然,上面使用验证集去早停和评估什么的比较麻烦,模型定义好后,训练可以简单一点
model.fit(X_train,y_train)
评价预测什么的和sklearn库完全一样,可以参考我以前的文章。原生用法也可以参考xgb。
print(model.score(X_test,y_test))
model.predict(X_test)
更新
LGBM的参数其实很多,再放一张图
调参思路其实是:
1.先选取较大的学习率,加速收敛
2.对决策树等参数进行调整:max_depth,num_leaves,subsample,colsample_bytree
3.然后再对正则化参数调整,min_child_weight,lambda...
4.降低学习率,配合估计器个数,进行最后的调整。
必要的时候还可以使用早停机制。