集成学习之Blending
Theory part
1.Blending是一种集成学习的方法,而非算法。
2.从train set取一部分数据再作为训练集(train_set)和验证集(val_set),并用于第二层模型的训练。
1.具体步骤
- (1) 将数据划分为训练集和测试集(test_set),而且训练集需要再拿一部分数据出来做验证集(val_set);
- (2) 创建第一层的多个Models,these models可以是同质的也可以是异质的(一般是异质的);
- (3) 使用train_set训练步骤(2)中的多个模型,然后拟合 val_set 和 test_set 得到 val_predict 和 test_predict,注意将新得到的 val_predict 和 test_predict加入 val_set 和 test_set;
- (4) 用新的 val_set 来 refresh models ;
- (5) 使用训练好的模型对第二层测试集test_predict进行预测,新的Model拟合出新的test set得到最终结果。
Homework
#导入所需包
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warning
warnings.filterwarnings('ignore')
#导入sklearn包
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegressio
#数据加载
iris = load_iris()
iris_data = pd.DataFrame(iris.data, columns=iris.feature_names)
iris_data['target'] = iris.target
iris_data = iris_data.loc[iris_data.target.isin([0, 1])]
y = iris_data.target
X = iris_data.drop('target', axis=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=200)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=200)
#因为集成学习上和中没来,这是参考了群里大佬的建议完成的
clpfs = [SVC(probability=True),
DecisionTreeClassifier(max_depth=2),
KNeighborsClassifier()]
val_features = np.zeros((X_val.shape[0], len(clfs)))
test_features = np.zeros((X_test.shape[0], len(clfs)))
for i,model in enumerate(clpfs):
model.fit(X_train, y_train)
val_feature = model.predict_proba(X_val)[:,1]
test_feature = model.predict_proba(X_test)[:,1]
val_features[:,i] = val_feature
test_features[:,i] = test_feature
#生成原始数据+预测特征
val_features = pd.DataFrame(val_features, index=X_val.index, columns=['svc', 'tree', 'knn'])
test_features = pd.DataFrame(test_features, index=X_test.index, columns=['svc', 'tree', 'knn'])
#比较原始数据,预测特征,原始数据+预测特征的得分
#因为集成学习上和中没来,这是参考了群里大佬的建议完成的
X_val1 = pd.concat([X_val, val_features], axis=1)
X_test1 = pd.concat([X_test, test_features], axis=1)
lr = LogisticRegression()
lr.fit(X_val1, y_val)
scores = cross_val_score(lr, X_test1, y_test, cv=5)
print('原始数据+特征分数:%.2f(%.2f)'%(np.mean(scores),np.std(scores)))
lr.fit(X_val, y_val)
scores = cross_val_score(lr, X_test, y_test, cv=5)
print('原始数据分数:%.2f(%.2f)'%(np.mean(scores),np.std(scores)))
lr.fit(val_features, y_val)
scores = cross_val_score(lr, test_features, y_test, cv=5)
print('特征分数:%.2f(%.2f)'%(np.mean(scores),np.std(scores)))
iris数据集数据少,测试效果相近。