笔记：tensorflow2.0中实现keras超参数搜索

 本文是tensorflow2.0版本下对tf.keras实现超参数搜索的代码笔记，使用sklearn对keras.model进行封装。代码实现是在jupyter notebook上，首先导入一些需要的库：

import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf

from tensorflow import keras

print(tf.__version__)
print(sys.version_info)
for module in mpl, np, pd, sklearn, tf, keras:
    print(module.__name__, module.__version__)

1.10.0
sys.version_info(major=3, minor=6, micro=5, releaselevel='final', serial=0)
matplotlib 3.0.2
numpy 1.14.5
pandas 0.23.4
sklearn 0.21.2
tensorflow 1.10.0
tensorflow.keras 2.1.6-tf

从sklearn中导入加州房价数据：

from sklearn.datasets import fetch_california_housing

housing = fetch_california_housing()
# print(housing.DESCR)
# print(housing.data.shape)
# print(housing.target.shape)

对数据进行分割：

from sklearn.model_selection import train_test_split

x_train_all, x_test, y_train_all, y_test = train_test_split(
    housing.data, housing.target, random_state = 7)
x_train, x_valid, y_train, y_valid = train_test_split(
    x_train_all, y_train_all, random_state = 11)
print(x_train.shape, y_train.shape)
print(x_valid.shape, y_valid.shape)
print(x_test.shape, y_test.shape)

 数据进行标准化处理，训练数据需要fit.trainform(),而验证与测试数据只需trainform():

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
x_train_scaled = scaler.fit_transform(x_train)
x_valid_scaled = scaler.transform(x_valid)
x_test_scaled = scaler.transform(x_test)

将tf.keras中的model转化为sklearn_model，采用EarlyStoping回调函数在设定的最优值下终止训练，这里是简单的回归问题，使用tensorflow中keras.wrappers.scikit_learn.KerasRegressor()进行sklearn_model()创建：

def build_model(hidden_layers = 1,
               layer_size = 30,
               learning_rate = 3e-3):
    model = keras.models.Sequential()
    model.add(keras.layers.Dense(layer_size,activation='relu',
                                input_shape = x_train.shape[1:]))
    for _ in range(hidden_layers-1):
        model.add(keras.layers.Dense(layer_size,
                                    activation = 'relu'))
    model.add(keras.layers.Dense(1))
    optimizer = keras.optimizers.SGD(learning_rate)
    model.compile(loss = 'mse',optimizer = optimizer)
    return model

sklearn_model = keras.wrappers.scikit_learn.KerasRegressor(build_model)
callbacks = [keras.callbacks.EarlyStopping(
        patience=5, min_delta=1e-2)]
history = sklearn_model.fit(x_train_scaled,y_train,epochs = 100,
                 validation_data = (x_valid_scaled, y_valid),
                  callbacks = callbacks)

对训练损失进行可视化：

def plot_learning_curves(history):
    pd.DataFrame(history.history).plot(figsize=(8, 5))
    plt.grid(True)
    plt.gca().set_ylim(0, 1)
    plt.show()

plot_learning_curves(history)

 ****接下来进行sklearn超参数搜索，定义参数集合，实现sklearn对超参数搜索：

from scipy.stats import reciprocal
#reciprocal分布函数表达式f(x) = 1/(x*log(b/a)) a<=x<=b

param_distribution = {
    "hidden_layers":[1,2,3,4],
    "layer_size":np.arange(1,100),
    "learning_rate":reciprocal(1e-4,1e-2),
}

from sklearn.model_selection import RandomizedSearchCV
random_search_cv = RandomizedSearchCV(sklearn_model,
                                     param_distribution,
                                     n_iter = 10,
                                     n_jobs = 1)
random_search_cv.fit(x_train_scaled,y_train,epochs = 100,
                    validation_data = (x_valid_scaled,y_valid),
                    callbacks = callbacks)

可以对reciprocal分布参数进行查看，并输出训练后的最优参数：

from scipy.stats import reciprocal
reciprocal.rvs(1e-4,1e-2,size=10)


print(random_search_cv.best_params_)
print(random_search_cv.best_score_)
print(random_search_cv.best_estimator_)

最后进行对最优model的测试：

model = random_search_cv.best_estimator_.model
model.evaluate(x_test_scaled,y_test)