- 🍨 本文为🔗365天深度学习训练营 中的学习记录博客
- 🍖 原作者:K同学啊
电脑环境:
语言环境:Python 3.8.0
编译器:Jupyter Notebook
深度学习环境:tensorflow 2.15.0
一、前期工作
1.设置GPU(如果使用的是CPU可以忽略这步)
import tensorflow as tf
gpus = tf.config.list_physical_devices("GPU")
if gpus:
tf.config.experimental.set_memory_growth(gpus[0], True) #设置GPU显存用量按需使用
tf.config.set_visible_devices([gpus[0]],"GPU")
# 打印显卡信息,确认GPU可用
print(gpus)
2. 导入数据
import matplotlib.pyplot as plt
# 支持中文
plt.rcParams['font.sans-serif'] = ['SimHei'] # 用来正常显示中文标签
plt.rcParams['axes.unicode_minus'] = False # 用来正常显示负号
import os,PIL,pathlib
#隐藏警告
import warnings
warnings.filterwarnings('ignore')
data_dir = "./365-7-data"
data_dir = pathlib.Path(data_dir)
image_count = len(list(data_dir.glob('*/*')))
print("图片总数为:",image_count)
二、数据预处理
1、加载数据
使用image_dataset_from_directory方法将磁盘中的数据加载到tf.data.Dataset中。
batch_size = 64
img_height = 224
img_width = 224
"""
关于image_dataset_from_directory()的详细介绍可以参考文章:https://mtyjkh.blog.csdn.net/article/details/117018789
"""
train_ds = tf.keras.preprocessing.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="training",
seed=12,
image_size=(img_height, img_width),
batch_size=batch_size)
val_ds = tf.keras.preprocessing.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="validation",
seed=12,
image_size=(img_height, img_width),
batch_size=batch_size)
我们可以通过class_names输出数据集的标签。标签将按字母顺序对应于目录名称。
class_names = train_ds.class_names
print(class_names)
输出:
[‘cat’, ‘dog’]
2、再次检查数据
for image_batch, labels_batch in train_ds:
print(image_batch.shape)
print(labels_batch.shape)
break
输出:
(64, 224, 224, 3)
(64,)
3、可视化数据
plt.figure(figsize=(15, 10)) # 图形的宽为15高为10
for images, labels in train_ds.take(1):
for i in range(8):
ax = plt.subplot(5, 8, i + 1)
plt.imshow(images[i])
plt.title(class_names[labels[i]])
plt.axis("off")
3、数据增强、配置数据集
在上期的文章中,我们没有对数据进行数据增强,本次尝试数据增强改善模型性能。
AUTOTUNE = tf.data.AUTOTUNE
# 定义数据增强层
data_augmentation = tf.keras.Sequential([
tf.keras.layers.RandomFlip("horizontal"),
tf.keras.layers.RandomRotation(0.2),
tf.keras.layers.RandomZoom(0.2),
tf.keras.layers.RandomContrast(0.1)
])
def preprocess_image(image, label):
image = image / 255.0
image = data_augmentation(image)
return image, label
def preprocess_val_image(image, label):
image = image / 255.0
return image, label
# 归一化处理
train_ds = train_ds.map(preprocess_image, num_parallel_calls=AUTOTUNE)
# 验证集不需要增强
val_ds = val_ds.map(preprocess_val_image, num_parallel_calls=AUTOTUNE)
train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)
- RandomFlip:随机翻转,可以水平翻转(horizontal)和垂直翻转(vertical);
- RandomRotation:随机旋转;
- RandomZoom:随机缩放;
- RandomContrast:随机对比度调整,增加或减少亮暗差异;
4、 显示数据增强后的数据
三、构建CNN网络
直接调用官方VGG16
from keras.applications import VGG16
model = VGG16(weights='imagenet')
model.summary()
四、编译
model.compile(optimizer="adam",
loss ='sparse_categorical_crossentropy',
metrics =['accuracy'])
五、训练模型
from tqdm import tqdm
import tensorflow.keras.backend as K
epochs = 10
lr = 1e-4
# 记录训练数据,方便后面的分析
history_train_loss = []
history_train_accuracy = []
history_val_loss = []
history_val_accuracy = []
for epoch in range(epochs):
train_total = len(train_ds)
val_total = len(val_ds)
"""
total:预期的迭代数目
ncols:控制进度条宽度
mininterval:进度更新最小间隔,以秒为单位(默认值:0.1)
"""
with tqdm(total=train_total, desc=f'Epoch {epoch + 1}/{epochs}',mininterval=1,ncols=100) as pbar:
lr = lr*0.92
K.set_value(model.optimizer.lr, lr)
train_loss = []
train_accuracy = []
for image,label in train_ds:
"""
训练模型,简单理解train_on_batch就是:它是比model.fit()更高级的一个用法
想详细了解 train_on_batch 的同学,
可以看看我的这篇文章:https://www.yuque.com/mingtian-fkmxf/hv4lcq/ztt4gy
"""
# 这里生成的是每一个batch的acc与loss
history = model.train_on_batch(image,label)
train_loss.append(history[0])
train_accuracy.append(history[1])
pbar.set_postfix({"train_loss": "%.4f"%history[0],
"train_acc":"%.4f"%history[1],
"lr": K.get_value(model.optimizer.lr)})
pbar.update(1)
history_train_loss.append(np.mean(train_loss))
history_train_accuracy.append(np.mean(train_accuracy))
print('开始验证!')
with tqdm(total=val_total, desc=f'Epoch {epoch + 1}/{epochs}',mininterval=0.3,ncols=100) as pbar:
val_loss = []
val_accuracy = []
for image,label in val_ds:
# 这里生成的是每一个batch的acc与loss
history = model.test_on_batch(image,label)
val_loss.append(history[0])
val_accuracy.append(history[1])
pbar.set_postfix({"val_loss": "%.4f"%history[0],
"val_acc":"%.4f"%history[1]})
pbar.update(1)
history_val_loss.append(np.mean(val_loss))
history_val_accuracy.append(np.mean(val_accuracy))
print('结束验证!')
print("验证loss为:%.4f"%np.mean(val_loss))
print("验证准确率为:%.4f"%np.mean(val_accuracy))
Epoch 1/20: 100%|███| 43/43 [00:56<00:00, 1.31s/it, train_loss=0.7041, train_acc=0.4531, lr=9.2e-5]
开始验证!
Epoch 1/20: 100%|██████████████████| 11/11 [00:02<00:00, 3.79it/s, val_loss=0.7103, val_acc=0.5000]
结束验证!
验证loss为:0.7073
验证准确率为:0.5085
Epoch 2/20: 100%|██| 43/43 [00:10<00:00, 4.30it/s, train_loss=0.6984, train_acc=0.5312, lr=8.46e-5]
开始验证!
Epoch 2/20: 100%|██████████████████| 11/11 [00:01<00:00, 8.82it/s, val_loss=0.6984, val_acc=0.5000]
结束验证!
验证loss为:0.6955
验证准确率为:0.5085
Epoch 3/20: 100%|██| 43/43 [00:09<00:00, 4.48it/s, train_loss=0.6942, train_acc=0.4688, lr=7.79e-5]
开始验证!
Epoch 3/20: 100%|██████████████████| 11/11 [00:01<00:00, 8.85it/s, val_loss=0.6934, val_acc=0.5000]
结束验证!
验证loss为:0.6942
验证准确率为:0.4915
Epoch 4/20: 100%|██| 43/43 [00:09<00:00, 4.33it/s, train_loss=0.6976, train_acc=0.4531, lr=7.16e-5]
开始验证!
Epoch 4/20: 100%|██████████████████| 11/11 [00:01<00:00, 9.86it/s, val_loss=0.6944, val_acc=0.5000]
结束验证!
验证loss为:0.6959
验证准确率为:0.5043
....................................................................................................
Epoch 20/20: 100%|█| 43/43 [00:09<00:00, 4.49it/s, train_loss=0.3254, train_acc=0.7656, lr=1.89e-5]
开始验证!
Epoch 20/20: 100%|█████████████████| 11/11 [00:01<00:00, 9.75it/s, val_loss=0.3012, val_acc=0.9500]
结束验证!
验证loss为:0.1548
验证准确率为:0.9670
六、模型评估
epochs_range = range(epochs)
plt.figure(figsize=(14, 4))
plt.subplot(1, 2, 1)
plt.plot(epochs_range, history_train_accuracy, label='Training Accuracy')
plt.plot(epochs_range, history_val_accuracy, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')
plt.subplot(1, 2, 2)
plt.plot(epochs_range, history_train_loss, label='Training Loss')
plt.plot(epochs_range, history_val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.show()
七、预测
import numpy as np
# 采用加载的模型(new_model)来看预测结果
plt.figure(figsize=(18, 3)) # 图形的宽为18高为5
plt.suptitle("预测结果展示")
for images, labels in val_ds.take(1):
for i in range(8):
ax = plt.subplot(1,8, i + 1)
# 显示图片
plt.imshow(images[i].numpy())
# 需要给图片增加一个维度
img_array = tf.expand_dims(images[i], 0)
# 使用模型预测图片中的人物
predictions = model.predict(img_array)
plt.title(class_names[np.argmax(predictions)])
plt.axis("off")
输出:
1/1 [==============================] - 0s 303ms/step
1/1 [==============================] - 0s 26ms/step
1/1 [==============================] - 0s 19ms/step
1/1 [==============================] - 0s 19ms/step
1/1 [==============================] - 0s 20ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 20ms/step
1/1 [==============================] - 0s 21ms/step
八、总结
本次使用了自定义数据增强方式,对dataset进行操作,也可以使用数据增强生成器ImageDataGenerator进行数据增强。