Task5 模型融合

5.1模型融合的方法

• 简单线性加权：
  • 回归（分类概率）：算术平均融合（Arithmetic mean），几何平均融合（Geometric mean）；
  • 分类：投票（Voting)
  • 综合：排序融合(Rank averaging)，log融合
• Stacking/Blending
  构建多层模型，并利用预测结果再拟合预测。
• boosting/bagging
  多树的提升方法

5.2 Stacking的一些笔记

理论推导（非数学的同学可以绕路）

5.3 代码

5.3.1 简单线性加权

In [1]:
## 生成一些简单的样本数据，test_prei 代表第i个模型的预测值
test_pre1 = [1.2, 3.2, 2.1, 6.2]
test_pre2 = [0.9, 3.1, 2.0, 5.9]
test_pre3 = [1.1, 2.9, 2.2, 6.0]
# y_test_true 代表第模型的真实值
y_test_true = [1, 3, 2, 6]

In [2]:
import numpy as np
import pandas as pd
## 定义结果的加权平均函数 def Weighted_method(test_pre1,test_pre2,test_pre3,w=[1/3,1/3,1/3]):
 Weighted_result = w[0]*pd.Series(test_pre1)+w[1]*pd.Series(test_pre2)+w[2]*pd.Series(test_pre3)
 return Weighted_result

In [3]:
from sklearn import metrics
# 各模型的预测结果计算MAE
print('Pred1 MAE:',metrics.mean_absolute_error(y_test_true, test_pre1))
print('Pred2 MAE:',metrics.mean_absolute_error(y_test_true, test_pre2))
print('Pred3 MAE:',metrics.mean_absolute_error(y_test_true, test_pre3))

输出结果：

由此可见，多个模型尽管是简单线性加权，对于结果都是有提升的，所以模型的融合是很有必要的.

5.3.2 Stacking融合(回归)：

库的导入

from sklearn import linear_model
def Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,test_pre1,test_pre2,test_pre3,mode
 model_L2.fit(pd.concat([pd.Series(train_reg1),pd.Series(train_reg2),pd.Series(train_reg3)],axis=
 Stacking_result = model_L2.predict(pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Serie
 return Stacking_result

模型准备

```python
## 生成一些简单的样本数据，test_prei 代表第i个模型的预测值
train_reg1 = [3.2, 8.2, 9.1, 5.2]
train_reg2 = [2.9, 8.1, 9.0, 4.9]
train_reg3 = [3.1, 7.9, 9.2, 5.0]
# y_test_true 代表第模型的真实值
y_train_true = [3, 8, 9, 5] 
test_pre1 = [1.2, 3.2, 2.1, 6.2]
test_pre2 = [0.9, 3.1, 2.0, 5.9]
test_pre3 = [1.1, 2.9, 2.2, 6.0]
# y_test_true 代表第模型的真实值
y_test_true = [1, 3, 2, 6]

model_L2= linear_model.LinearRegression()
Stacking_pre = Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,
 test_pre1,test_pre2,test_pre3,model_L2)
print('Stacking_pre MAE:',metrics.mean_absolute_error(y_test_true, Stacking_pre))

5.3.2 分类

引入库

from sklearn.datasets import make_blobs
from sklearn import datasets
from sklearn.tree import DecisionTreeClassifier
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from xgboost import XGBClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_moons
from sklearn.metrics import accuracy_score,roc_auc_score
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import StratifiedKFold

'''
5-Fold Stacking
'''
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import ExtraTreesClassifier,GradientBoostingClassifier
import pandas as pd
#创建训练的数据集
data_0 = iris.data
data = data_0[:100,:]
target_0 = iris.target
target = target_0[:100] #模型融合中使用到的各个单模型
clfs = [LogisticRegression(solver='lbfgs'),
 RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),
 ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),
 ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),
 GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)]
#切分一部分数据作为测试集 X, X_predict, y, y_predict = train_test_split(data, target, test_size=0.3, random_state=2020)
dataset_blend_train = np.zeros((X.shape[0], len(clfs)))
dataset_blend_test = np.zeros((X_predict.shape[0], len(clfs)))
#5折stacking
n_splits = 5
skf = StratifiedKFold(n_splits)
skf = skf.split(X, y) for j, clf in enumerate(clfs):
 #依次训练各个单模型
 dataset_blend_test_j = np.zeros((X_predict.shape[0], 5))
 for i, (train, test) in enumerate(skf):
 #5-Fold交叉训练，使用第i个部分作为预测，剩余的部分来训练模型，获得其预测的输出作为第i部分的新
 X_train, y_train, X_test, y_test = X[train], y[train], X[test], y[test]
 clf.fit(X_train, y_train)
 y_submission = clf.predict_proba(X_test)[:, 1]
 dataset_blend_train[test, j] = y_submission
 dataset_blend_test_j[:, i] = clf.predict_proba(X_predict)[:, 1]
 #对于测试集，直接用这k个模型的预测值均值作为新的特征。
 dataset_blend_test[:, j] = dataset_blend_test_j.mean(1)
 print("val auc Score: %f" % roc_auc_score(y_predict, dataset_blend_test[:, j]))
clf = LogisticRegression(solver='lbfgs')
clf.fit(dataset_blend_train, y)
y_submission = clf.predict_proba(dataset_blend_test)[:, 1]
print("Val auc Score of Stacking: %f" % (roc_auc_score(y_predict, y_submission)))