使用多个特征来进一步提高模型的有效性,使用测试数据集检查模型是否过拟合验证数据

Github: https://github.com/DPnice/TensorFlowTest/blob/master/validation.ipynb

验证

学习目标：

• 使用多个特征而非单个特征来进一步提高模型的有效性
• 调试模型输入数据中的问题
• 使用测试数据集检查模型是否过拟合验证数据

与在之前的练习中一样，我们将使用加利福尼亚州住房数据集，尝试根据 1990 年的人口普查数据在城市街区级别预测 median_house_value。

设置

我们首先加载并准备数据。这一次，我们将使用多个特征，因此我们会将逻辑模块化，以对特征进行预处理：

In [12]:

from __future__ import print_function

import math

from IPython import display
from matplotlib import cm
from matplotlib import gridspec
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd
from sklearn import metrics
import tensorflow as tf
from tensorflow.python.data import Dataset

tf.logging.set_verbosity(tf.logging.ERROR)
pd.options.display.max_rows = 10
pd.options.display.float_format = '{:.1f}'.format

california_housing_dataframe = pd.read_csv("https://storage.googleapis.com/mledu-datasets/california_housing_train.csv", sep=",")

california_housing_dataframe = california_housing_dataframe.reindex(
    np.random.permutation(california_housing_dataframe.index))

In [13]:

# 特征预处理
def preprocess_features(california_housing_dataframe):
  """Prepares input features from California housing data set.

  Args:
    california_housing_dataframe: A Pandas DataFrame expected to contain data
      from the California housing data set.
  Returns:
    A DataFrame that contains the features to be used for the model, including
    synthetic features.
  """
  selected_features = california_housing_dataframe[
    ["latitude",
     "longitude",
     "housing_median_age",
     "total_rooms",
     "total_bedrooms",
     "population",
     "households",
     "median_income"]]
  processed_features = selected_features.copy()
  # Create a synthetic feature.
  processed_features["rooms_per_person"] = (
    california_housing_dataframe["total_rooms"] /
    california_housing_dataframe["population"])
  return processed_features
# 标签预处理
def preprocess_targets(california_housing_dataframe):
  """Prepares target features (i.e., labels) from California housing data set.

  Args:
    california_housing_dataframe: A Pandas DataFrame expected to contain data
      from the California housing data set.
  Returns:
    A DataFrame that contains the target feature.
  """
  output_targets = pd.DataFrame()
  # Scale the target to be in units of thousands of dollars.
  output_targets["median_house_value"] = (
    california_housing_dataframe["median_house_value"] / 1000.0)
  return output_targets

对于训练集，我们从共 17000 个样本中选择前 12000 个样本。

In [14]:

training_examples = preprocess_features(california_housing_dataframe.head(12000))
training_examples.describe()

Out[14]:

	latitude	longitude	housing_median_age	total_rooms	total_bedrooms	population	households	median_income	rooms_per_person
count	12000.0	12000.0	12000.0	12000.0	12000.0	12000.0	12000.0	12000.0	12000.0
mean	35.6	-119.6	28.6	2649.1	540.7	1434.8	502.4	3.9	2.0
std	2.1	2.0	12.5	2203.9	424.4	1177.9	388.0	1.9	1.2
min	32.5	-124.3