【机器学习】python机器学习使用scikit-learn对模型进行微调：使用GridSearchCV及RandomizedSearchCV

端到端机器学习导航：
【机器学习】python借助pandas加载并显示csv数据文件，并绘制直方图
【机器学习】python使用matplotlib进行二维数据绘图并保存为png图片
【机器学习】python借助pandas及scikit-learn使用三种方法分割训练集及测试集
【机器学习】python借助pandas及matplotlib将输入数据可视化，并计算相关性
【机器学习】python机器学习借助scikit-learn进行数据预处理工作：缺失值填补，文本处理（一）
【机器学习】python机器学习scikit-learn和pandas进行Pipeline处理工作：归一化和标准化及自定义转换器(二)
【机器学习】python机器学习使用scikit-learn评估模型：基于普通抽样及分层抽样的k折交叉验证做模型选择
【机器学习】python机器学习使用scikit-learn对模型进行微调：使用GridSearchCV及RandomizedSearchCV
【机器学习】python机器学习使用scikit-learn对模型进行评估：使用t分布及z分布评估模型误差的95%置信空间
【机器学习】python机器学习使用scikit-learn对模型进行微调：RandomizedSearchCV的分布参数设置
【机器学习】python机器学习使用scikit-learn对模型进行微调：按特征贡献大小保留最重要k个特征的transform
【机器学习】python机器学习使用scikit-learn对模型进行微调：使用RandomizedSearchCV对pipline进行参数选择

数据准备：

import os

HOUSING_PATH = os.path.join("datasets", "housing")
import pandas as pd

def load_housing_data(housing_path=HOUSING_PATH):
    csv_path = os.path.join(housing_path, "housing.csv")
    return pd.read_csv(csv_path)
housing=load_housing_data()
housing2=housing.copy()
import numpy as np

#预处理前去掉带文字的指定列
from sklearn.preprocessing import MinMaxScaler,StandardScaler,OneHotEncoder

from sklearn.impute import SimpleImputer
housing_num = housing2.drop("ocean_proximity", axis=1)
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
num_pipeline = Pipeline([
        ('imputer', SimpleImputer(strategy="median")),
        ('std_scaler', StandardScaler())
    ])



from sklearn.compose import ColumnTransformer
#返回所有列名
num_attribs = list(housing_num)

cat_attribs = ["ocean_proximity"]
#找出待独热编码列的最大分类数，不然在进行测试集划分处理时，
#容易造成独热向量因测试集构成不同而列数不一致的情况
categories=housing2['ocean_proximity'].unique()
full_pipeline = ColumnTransformer([
        ("num", num_pipeline, num_attribs),
        ("cat", OneHotEncoder(categories=[categories]), cat_attribs),
    ])
#抽样后的数据，去除预测目标列，并拿出对应目标列准备数据训练
housing_labels = housing2["median_house_value"]

housing_prepared = full_pipeline.fit_transform(housing2)

1.先使用网格搜索：这里以随机森林算法做示例

from sklearn.metrics import mean_squared_error
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestRegressor
param_grid = [
    # 先尝试第一个大括号内的参数组合
    {'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]},
    # 再尝试第二个大括号内的参数组合
    {'bootstrap': [False], 'n_estimators': [3, 10], 'max_features': [2, 3, 4]},
  ]

forest_reg = RandomForestRegressor(random_state=42)
# 网格搜索进行了5折随机验证：
grid_search = GridSearchCV(forest_reg, param_grid, cv=5,
                           scoring='neg_mean_squared_error',
                           return_train_score=True)

grid_search.fit(housing_prepared, housing_labels)
#可查看最佳参数
print(grid_search.best_params_)
#获得训练后的最佳估计器
grid_search.best_estimator_
#再查看具体的得分和对应参数
cvres = grid_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
    print(np.sqrt(-mean_score), params)
#更直观的是输出结果的pd矩阵
pd.DataFrame(grid_search.cv_results_).head(n=5*(3*4+1*2*3))

输出结果：

{'max_features': 8, 'n_estimators': 30}
35443.48580361833 {'max_features': 2, 'n_estimators': 3}
33951.7206599765 {'max_features': 2, 'n_estimators': 10}
30769.58857145479 {'max_features': 2, 'n_estimators': 30}
16382.432452343699 {'max_features': 4, 'n_estimators': 3}
13028.203730939029 {'max_features': 4, 'n_estimators': 10}
11045.325540604028 {'max_features': 4, 'n_estimators': 30}
6985.258670858961 {'max_features': 6, 'n_estimators': 3}
4881.725705637295 {'max_features': 6, 'n_estimators': 10}
4542.256635926347 {'max_features': 6, 'n_estimators': 30}
4383.951766276588 {'max_features': 8, 'n_estimators': 3}
3226.539015978242 {'max_features': 8, 'n_estimators': 10}
2184.642855306284 {'max_features': 8, 'n_estimators': 30}
37260.542315007835 {'bootstrap': False, 'max_features': 2, 'n_estimators': 3}
31590.025336610437 {'bootstrap': False, 'max_features': 2, 'n_estimators': 10}
28205.910706549468 {'bootstrap': False, 'max_features': 3, 'n_estimators': 3}
19303.343587996253 {'bootstrap': False, 'max_features': 3, 'n_estimators': 10}
16574.537687528173 {'bootstrap': False, 'max_features': 4, 'n_estimators': 3}
12117.700717051643 {'bootstrap': False, 'max_features': 4, 'n_estimators': 10}

2.使用随机参数搜索：

from sklearn.metrics import mean_squared_error
from sklearn.ensemble import RandomForestRegressor

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_distribs = {
        'n_estimators': randint(low=1, high=200),
        'max_features': randint(low=1, high=8),
    }

forest_reg = RandomForestRegressor(random_state=42)
#每个迭代探索一种参数组合
rnd_search = RandomizedSearchCV(forest_reg, param_distributions=param_distribs,
                                n_iter=10, cv=5, scoring='neg_mean_squared_error', random_state=42)
rnd_search.fit(housing_prepared, housing_labels)
#可查看最佳参数
print(rnd_search.best_params_)
#获得训练后的最佳估计器
rnd_search.best_estimator_
#再查看具体的得分和对应参数
cvres = rnd_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
    print(np.sqrt(-mean_score), params)

输出结果：

{'max_features': 7, 'n_estimators': 180}
2240.6588119511284 {'max_features': 7, 'n_estimators': 180}
8498.979191166485 {'max_features': 5, 'n_estimators': 15}
16136.994063324326 {'max_features': 3, 'n_estimators': 72}
7610.943288328705 {'max_features': 5, 'n_estimators': 21}
2348.3590775352754 {'max_features': 7, 'n_estimators': 122}
16121.080130491993 {'max_features': 3, 'n_estimators': 75}
16254.342107487135 {'max_features': 3, 'n_estimators': 88}
6054.400552299054 {'max_features': 5, 'n_estimators': 100}
16498.617512610923 {'max_features': 3, 'n_estimators': 150}
11108.994981465223 {'max_features': 5, 'n_estimators': 2}
    

作为随机森林回归模型，可用如下代码显示特征重要性：

#返回所有训练特征的重要性
feature_importances = grid_search.best_estimator_.feature_importances_
cat_encoder = full_pipeline.named_transformers_["cat"]
#获取one-hot-encoder的具体编码类别，one-hot每个类别可看成一个新特征
cat_one_hot_attribs = list(cat_encoder.categories_[0])
attributes = num_attribs + cat_one_hot_attribs
#从大到小排序
sorted(zip(feature_importances, attributes), reverse=True)