参考部署教程,在趋动云,跑图像识别,并进行离线训练。教程第二部分
- 环境准备
- 镜像:TensorFlow 2.10.1
- 数据集: DogsVsCats
- 显存:GPU 24G,B1 large
- 代码:
猫狗识别代码DogsVsCats.py,上传到驱动云代码路径,code下。
import argparse
import tensorflow as tf
import os
parser = argparse.ArgumentParser(description='Process some integers')
parser.add_argument('--mode', default='train', help='train or test')
parser.add_argument("--num_epochs", default=5, type=int)
parser.add_argument("--batch_size", default=32, type=int)
parser.add_argument("--learning_rate", default=0.001)
parser.add_argument("--data_dir", default="/gemini/data-1")
parser.add_argument("--train_dir", default="/gemini/output")
args = parser.parse_args()
def _decode_and_resize(filename, label):
image_string = tf.io.read_file(filename)
image_decoded = tf.image.decode_jpeg(image_string, channels=3)
image_resized = tf.image.resize(image_decoded, [150, 150]) / 255.0
return image_resized, label
if __name__ == "__main__":
train_dir = args.data_dir + "/train"
cats = []
dogs = []
for file in os.listdir(train_dir):
if file.startswith("dog"):
dogs.append(train_dir + "/" + file)
else:
cats.append(train_dir + "/" + file)
print("dogSize:%d catSize:%d" % (len(cats), len(dogs)))
train_cat_filenames = tf.constant(cats[:10000])
train_dog_filenames = tf.constant(dogs[:10000])
train_filenames = tf.concat([train_cat_filenames, train_dog_filenames], axis=-1)
train_labels = tf.concat([
tf.zeros(train_cat_filenames.shape, dtype=tf.int32),
tf.ones(train_dog_filenames.shape, dtype=tf.int32)
], axis=-1)
train_dataset = tf.data.Dataset.from_tensor_slices((train_filenames, train_labels))
train_dataset = train_dataset.map(map_func=_decode_and_resize,
num_parallel_calls=tf.data.experimental.AUTOTUNE)
train_dataset = train_dataset.shuffle(buffer_size=20000)
train_dataset = train_dataset.batch(args.batch_size)
train_dataset = train_dataset.prefetch(tf.data.experimental.AUTOTUNE)
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(32, 3, activation="relu", input_shape=(150, 150, 3)),
tf.keras.layers.MaxPool2D(),
tf.keras.layers.Conv2D(64, 3, activation="relu"),
tf.keras.layers.MaxPool2D(),
tf.keras.layers.Conv2D(128, 3, activation="relu"),
tf.keras.layers.MaxPool2D(),
tf.keras.layers.Conv2D(128, 3, activation="relu"),
tf.keras.layers.MaxPool2D(),
tf.keras.layers.Flatten(),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(512, activation="relu"),
tf.keras.layers.Dense(2, activation="softmax")
])
model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate=args.learning_rate),
loss=tf.keras.losses.sparse_categorical_crossentropy,
metrics=[tf.keras.metrics.sparse_categorical_accuracy]
)
model.fit(train_dataset, epochs=args.num_epochs)
model.save(args.train_dir)
# 构建测试数据集
test_cat_filenames = tf.constant(cats[10000:])
test_dog_filenames = tf.constant(dogs[10000:])
test_filenames = tf.concat([test_cat_filenames, test_dog_filenames], axis=-1)
test_labels = tf.concat([
tf.zeros(test_cat_filenames.shape, dtype=tf.int32),
tf.ones(test_dog_filenames.shape, dtype=tf.int32)
], axis=-1)
test_dataset = tf.data.Dataset.from_tensor_slices((test_filenames, test_labels))
test_dataset = test_dataset.map(_decode_and_resize)
test_dataset = test_dataset.batch(args.batch_size)
sparse_categorical_accuracy = tf.keras.metrics.SparseCategoricalAccuracy()
for images, label in test_dataset:
y_pred = model.predict(images)
sparse_categorical_accuracy.update_state(y_true=label, y_pred=y_pred)
print("test accuracy:%f" % sparse_categorical_accuracy.result())
shell终端执行命令
python $GEMINI_RUN/DogsVsCats.py --num_epochs 5 --data_dir $GEMINI_DATA_IN1/DogsVsCats/ --train_dir $GEMINI_DATA_OUT
- 把第44行注释删掉, train_dataset = train_dataset.shuffle(buffer_size=20000) ,重新执行命令
python $GEMINI_RUN/DogsVsCats.py --num_epochs 5 --data_dir $GEMINI_DATA_IN1/DogsVsCats/ --train_dir $GEMINI_DATA_OUT
提交离线训练
- 点击右上方提交训练任务,代码选择新建1.0.1
- 点击左边训练,点击提交训练任务
- 选择单机任务,GPU24,数据集DogsVsCats,代码版本1.0.1
点击确定
离线任务完成后,会自动关闭,我们可以监控到GPU,IO资源的使用情况
支持日志导出,查看运行结果,我们看到最终的识别是0.84的正确率
- 左侧结果,右上角导出模型。
点击导出模型后,输入模型名称,选择文件,然后公开,最后点击创建
最终我们有了一个自己的模型