2-数据分析

陈大愚

已于 2023-04-21 15:27:14 修改

阅读量187

点赞数 1

分类专栏： TensorFlow笔记文章标签：数据分析 python tensorflow

于 2023-04-21 15:27:04 首次发布

本文链接：https://blog.csdn.net/qq_45190143/article/details/130289846

版权

TensorFlow笔记专栏收录该内容

4 篇文章 0 订阅

订阅专栏

文章目录

用 tf.data 加载 CSV 数据

通过一个示例展示了怎样将 CSV 格式的数据加载进 tf.data.Dataset。

使用的是泰坦尼克号乘客的数据。模型会根据乘客的年龄、性别、票务舱和是否独自旅行等特征来预测乘客生还的可能性。

# 设置
import functools
import pandas as pd
import numpy as np
import tensorflow as tf
import tensorflow_datasets as tfds

d:\anaconda3\envs\tensorflow\lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

# 训练&测试数据的url
TRAIN_DATA_URL = "https://storage.googleapis.com/tf-datasets/titanic/train.csv"
TEST_DATA_URL = "https://storage.googleapis.com/tf-datasets/titanic/eval.csv"

train_file_path = tf.keras.utils.get_file("train.csv", TRAIN_DATA_URL)
test_file_path = tf.keras.utils.get_file("eval.csv", TEST_DATA_URL)

# 让 numpy 数据更易读。
# precision：控制输出结果的精度(即小数点后的位数)，默认值为8
# suppress：小数是否需要以科学计数法的形式输出
np.set_printoptions(precision=3, suppress=True)

加载数据

先打印CSV文件的前几行了解文件格式。

pd.read_csv(train_file_path)

	survived	sex	age	n_siblings_spouses	parch	fare	class	deck	embark_town	alone
0	0	male	22.0	1	0	7.2500	Third	unknown	Southampton	n
1	1	female	38.0	1	0	71.2833	First	C	Cherbourg	n
2	1	female	26.0	0	0	7.9250	Third	unknown	Southampton	y
3	1	female	35.0	1	0	53.1000	First	C	Southampton	n
4	0	male	28.0	0	0	8.4583	Third	unknown	Queenstown	y
...	...	...	...	...	...	...	...	...	...	...
622	0	male	28.0	0	0	10.5000	Second	unknown	Southampton	y
623	0	male	25.0	0	0	7.0500	Third	unknown	Southampton	y
624	1	female	19.0	0	0	30.0000	First	B	Southampton	y
625	0	female	28.0	1	2	23.4500	Third	unknown	Southampton	n
626	0	male	32.0	0	0	7.7500	Third	unknown	Queenstown	y

627 rows × 10 columns


# 列名
CSV_COLUMNS = ['survived', 'sex', 'age', 'n_siblings_spouses', 'parch', 'fare', 'class', 'deck', 'embark_town', 'alone']

# 预测的值的列是需要显式指定的
LABEL_COLUMN = 'survived'
LABELS = [0, 1]

# 从文件中读取 CSV 数据并且创建 dataset
def get_dataset(file_path):
  dataset = tf.data.experimental.make_csv_dataset(
      file_path,
      batch_size=12, # 为了示例更容易展示，手动设置较小的值
      label_name=LABEL_COLUMN,
      na_value="?",
      num_epochs=1,
      ignore_errors=True)
  return dataset

raw_train_data = get_dataset(train_file_path)
raw_test_data = get_dataset(test_file_path)

# dataset 中的每个条目都是一个批次，用一个元组（多个样本，多个标签）表示
examples, labels = next(iter(raw_train_data)) # 第一个批次
print("examples: \n", examples, "\n")
print("labels: \n", labels)

examples: 
 OrderedDict([('sex', <tf.Tensor: shape=(12,), dtype=string, numpy=
array([b'male', b'male', b'male', b'male', b'female', b'male', b'male',
       b'female', b'male', b'female', b'male', b'female'], dtype=object)>), ('age', <tf.Tensor: shape=(12,), dtype=float32, numpy=
array([ 4., 28., 47., 22., 18., 51., 27., 28., 28., 26., 31., 40.],
      dtype=float32)>), ('n_siblings_spouses', <tf.Tensor: shape=(12,), dtype=int32, numpy=array([4, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0])>), ('parch', <tf.Tensor: shape=(12,), dtype=int32, numpy=array([2, 0, 0, 0, 2, 0, 0, 0, 0, 1, 0, 0])>), ('fare', <tf.Tensor: shape=(12,), dtype=float32, numpy=
array([ 31.275,   7.896,  38.5  ,   7.25 ,  13.   ,   8.05 ,   8.663,
        82.171,   0.   ,  26.   ,  52.   , 153.462], dtype=float32)>), ('class', <tf.Tensor: shape=(12,), dtype=string, numpy=
array([b'Third', b'Third', b'First', b'Third', b'Second', b'Third',
       b'Third', b'First', b'Second', b'Second', b'First', b'First'],
      dtype=object)>), ('deck', <tf.Tensor: shape=(12,), dtype=string, numpy=
array([b'unknown', b'unknown', b'E', b'unknown', b'unknown', b'unknown',
       b'unknown', b'unknown', b'unknown', b'unknown', b'B', b'C'],
      dtype=object)>), ('embark_town', <tf.Tensor: shape=(12,), dtype=string, numpy=
array([b'Southampton', b'Cherbourg', b'Southampton', b'Southampton',
       b'Southampton', b'Southampton', b'Southampton', b'Cherbourg',
       b'Southampton', b'Southampton', b'Southampton', b'Southampton'],
      dtype=object)>), ('alone', <tf.Tensor: shape=(12,), dtype=string, numpy=
array([b'n', b'y', b'y', b'n', b'n', b'y', b'y', b'n', b'y', b'n', b'n',
       b'y'], dtype=object)>)]) 

labels: 
 tf.Tensor([0 0 0 0 1 0 1 1 0 0 0 1], shape=(12,), dtype=int32)

数据预处理

【分类数据】

CSV 数据中的有些列是分类的列。也就是说，这些列只能在有限的集合中取值。

使用 tf.feature_column API 创建一个 tf.feature_column.indicator_column 集合，每个 tf.feature_column.indicator_column 对应一个分类的列。

CATEGORIES = {
    'sex': ['male', 'female'],
    'class' : ['First', 'Second', 'Third'],
    'deck' : ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J'],
    'embark_town' : ['Cherbourg', 'Southhampton', 'Queenstown'],
    'alone' : ['y', 'n']
}

categorical_columns = []
for feature, vocab in CATEGORIES.items():
  cat_col = tf.feature_column.categorical_column_with_vocabulary_list(
        key=feature, vocabulary_list=vocab)
  categorical_columns.append(tf.feature_column.indicator_column(cat_col))

# 刚才创建的内容
categorical_columns

[IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='sex', vocabulary_list=('male', 'female'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),
 IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='class', vocabulary_list=('First', 'Second', 'Third'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),
 IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='deck', vocabulary_list=('A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),
 IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='embark_town', vocabulary_list=('Cherbourg', 'Southhampton', 'Queenstown'), dtype=tf.string, default_value=-1, num_oov_buckets=0)),
 IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='alone', vocabulary_list=('y', 'n'), dtype=tf.string, default_value=-1, num_oov_buckets=0))]

【连续数据】

连续数据需要标准化。

写一个函数标准化这些值，然后将这些值改造成 2 维的张量。

def process_continuous_data(mean, data):
  # 标准化数据
  data = tf.cast(data, tf.float32) * 1/(2*mean)
  return tf.reshape(data, [-1, 1])

创建一个数值列的集合。tf.feature_columns.numeric_column API 会使用 normalizer_fn 参数。在传参的时候使用 functools.partial，functools.partial 由使用每个列的均值进行标准化的函数构成。

MEANS = {
    'age' : 29.631308,
    'n_siblings_spouses' : 0.545455,
    'parch' : 0.379585,
    'fare' : 34.385399
}

# 数值列集合
numerical_columns = []

for feature in MEANS.keys():
  num_col = tf.feature_column.numeric_column(feature, normalizer_fn=functools.partial(process_continuous_data, MEANS[feature]))
  numerical_columns.append(num_col)

# 刚才创建的数值列
numerical_columns

[NumericColumn(key='age', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=functools.partial(<function process_continuous_data at 0x00000235DA383E50>, 29.631308)),
 NumericColumn(key='n_siblings_spouses', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=functools.partial(<function process_continuous_data at 0x00000235DA383E50>, 0.545455)),
 NumericColumn(key='parch', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=functools.partial(<function process_continuous_data at 0x00000235DA383E50>, 0.379585)),
 NumericColumn(key='fare', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=functools.partial(<function process_continuous_data at 0x00000235DA383E50>, 34.385399))]

【创建预处理层】

将这两个特征列的集合相加，并且传给 tf.keras.layers.DenseFeatures 从而创建一个进行预处理的输入层。

preprocessing_layer = tf.keras.layers.DenseFeatures(categorical_columns+numerical_columns)

构建模型

从 preprocessing_layer （预处理层）开始构建 tf.keras.Sequential（层的线性叠加）。

model = tf.keras.Sequential([
  preprocessing_layer, # 预处理层
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(1, activation='sigmoid'),
])

model.compile(
    loss='binary_crossentropy',
    optimizer='adam',
    metrics=['accuracy'])

训练、评估和预测

train_data = raw_train_data.shuffle(500) # shuffle把数组中的元素按随机顺序重新排列
test_data = raw_test_data

model.fit(train_data, epochs=20)

Epoch 1/20
WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'collections.OrderedDict'> input: OrderedDict([('sex', <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>), ('age', <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float32>), ('n_siblings_spouses', <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int32>), ('parch', <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int32>), ('fare', <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float32>), ('class', <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>), ('deck', <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>), ('embark_town', <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>), ('alone', <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>)])
Consider rewriting this model with the Functional API.
WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'collections.OrderedDict'> input: OrderedDict([('sex', <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>), ('age', <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float32>), ('n_siblings_spouses', <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int32>), ('parch', <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int32>), ('fare', <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float32>), ('class', <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>), ('deck', <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>), ('embark_town', <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>), ('alone', <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>)])
Consider rewriting this model with the Functional API.
53/53 [==============================] - 1s 1ms/step - loss: 0.5403 - accuracy: 0.7496
Epoch 2/20
53/53 [==============================] - 0s 924us/step - loss: 0.4344 - accuracy: 0.8022
Epoch 3/20
53/53 [==============================] - 0s 885us/step - loss: 0.4174 - accuracy: 0.8198
Epoch 4/20
53/53 [==============================] - 0s 905us/step - loss: 0.4069 - accuracy: 0.8214
Epoch 5/20
53/53 [==============================] - 0s 905us/step - loss: 0.3980 - accuracy: 0.8341
Epoch 6/20
53/53 [==============================] - 0s 943us/step - loss: 0.3951 - accuracy: 0.8278
Epoch 7/20
53/53 [==============================] - 0s 905us/step - loss: 0.3849 - accuracy: 0.8325
Epoch 8/20
53/53 [==============================] - 0s 924us/step - loss: 0.3808 - accuracy: 0.8453
Epoch 9/20
53/53 [==============================] - 0s 924us/step - loss: 0.3744 - accuracy: 0.8325
Epoch 10/20
53/53 [==============================] - 0s 885us/step - loss: 0.3701 - accuracy: 0.8421
Epoch 11/20
53/53 [==============================] - 0s 905us/step - loss: 0.3677 - accuracy: 0.8437
Epoch 12/20
53/53 [==============================] - 0s 905us/step - loss: 0.3558 - accuracy: 0.8485
Epoch 13/20
53/53 [==============================] - 0s 885us/step - loss: 0.3572 - accuracy: 0.8453
Epoch 14/20
53/53 [==============================] - 0s 905us/step - loss: 0.3533 - accuracy: 0.8565
Epoch 15/20
53/53 [==============================] - 0s 924us/step - loss: 0.3525 - accuracy: 0.8549
Epoch 16/20
53/53 [==============================] - 0s 924us/step - loss: 0.3491 - accuracy: 0.8341
Epoch 17/20
53/53 [==============================] - 0s 943us/step - loss: 0.3420 - accuracy: 0.8517
Epoch 18/20
53/53 [==============================] - 0s 943us/step - loss: 0.3450 - accuracy: 0.8469
Epoch 19/20
53/53 [==============================] - 0s 905us/step - loss: 0.3355 - accuracy: 0.8565
Epoch 20/20
53/53 [==============================] - 0s 905us/step - loss: 0.3360 - accuracy: 0.8596





<tensorflow.python.keras.callbacks.History at 0x235e1f88550>

# 测试评估
test_loss, test_accuracy = model.evaluate(test_data)

print('\n\nTest Loss {}, Test Accuracy {}'.format(test_loss, test_accuracy))

WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'collections.OrderedDict'> input: OrderedDict([('sex', <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>), ('age', <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float32>), ('n_siblings_spouses', <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int32>), ('parch', <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int32>), ('fare', <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float32>), ('class', <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>), ('deck', <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>), ('embark_town', <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>), ('alone', <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>)])
Consider rewriting this model with the Functional API.
22/22 [==============================] - 0s 1ms/step - loss: 0.4383 - accuracy: 0.8333


Test Loss 0.43826165795326233, Test Accuracy 0.8333333134651184

# 预测
predictions = model.predict(test_data)

# 显示部分结果
for prediction, survived in zip(predictions[:10], list(test_data)[0][1][:10]):
  print("Predicted survival: {:.2%}".format(prediction[0]),
        " | Actual outcome: ",
        ("SURVIVED" if bool(survived) else "DIED"))

WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'collections.OrderedDict'> input: OrderedDict([('sex', <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>), ('age', <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float32>), ('n_siblings_spouses', <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int32>), ('parch', <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int32>), ('fare', <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float32>), ('class', <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>), ('deck', <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>), ('embark_town', <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>), ('alone', <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>)])
Consider rewriting this model with the Functional API.
Predicted survival: 9.68%  | Actual outcome:  SURVIVED
Predicted survival: 7.91%  | Actual outcome:  DIED
Predicted survival: 0.36%  | Actual outcome:  DIED
Predicted survival: 9.77%  | Actual outcome:  DIED
Predicted survival: 75.22%  | Actual outcome:  DIED
Predicted survival: 99.39%  | Actual outcome:  DIED
Predicted survival: 62.61%  | Actual outcome:  DIED
Predicted survival: 89.30%  | Actual outcome:  SURVIVED
Predicted survival: 87.70%  | Actual outcome:  SURVIVED
Predicted survival: 8.08%  | Actual outcome:  SURVIVED

参考

https://www.tensorflow.org/tutorials/load_data/csv?hl=zh-cn

陈大愚

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
打赏
0
评论
2-数据分析

创建一个数值列的集合。在传参的时候使用 functools.partial，functools.partial 由使用每个列的均值进行标准化的函数构成。使用 tf.feature_column API 创建一个 tf.feature_column.indicator_column 集合，每个 tf.feature_column.indicator_column 对应一个分类的列。将这两个特征列的集合相加，并且传给 tf.keras.layers.DenseFeatures 从而创建一个进行预处理的输入层。
复制链接

扫一扫