TensorFlow 2.3 模型评估model.evaluate()时accuracy丢失的坑

问题描述

对手写数据集 MNIST 进行简单的模型训练并评估

import tensorflow as tf

(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()
train_images = train_images / 255.0
test_images = test_images / 255.0
model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10)
])
model.compile(
    optimizer='adam',
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy']
)
model.fit(train_images, train_labels, epochs=2)
loss, accuracy = model.evaluate(test_images, test_labels, verbose=2)
print('loss: {:.2f} accuracy: {:.2f}% '.format(loss, accuracy * 100))
model.save('model.h5')
# Epoch 1/2
# 1875/1875 [==============================] - 2s 1ms/step - loss: 0.2618 - accuracy: 0.9251
# Epoch 2/2
# 1875/1875 [==============================] - 2s 1000us/step - loss: 0.1149 - accuracy: 0.9657
# 313/313 - 0s - loss: 0.0977 - accuracy: 0.9694
# loss: 0.10 accuracy: 96.94%

加载模型再评估

import tensorflow as tf

(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()
test_images = test_images / 255.0
model = tf.keras.models.load_model('model.h5')
loss, accuracy = model.evaluate(test_images, test_labels, verbose=2)
print('loss: {:.2f} accuracy: {:.2f}% '.format(loss, accuracy * 100))
# loss: 0.10 accuracy: 9.89%

惊奇地发现 loss 是正确的，但 accuracy 丢失了

这是一个 BUG ，当损失函数使用 sparse_categorical_crossentropy ，而度量指标是 accuracy 时出现

训练时把 accuracy 改为 sparse_categorical_accuracy

import tensorflow as tf

(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()
train_images = train_images / 255.0
test_images = test_images / 255.0
model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10)
])
model.compile(
    optimizer='adam',
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['sparse_categorical_accuracy']
)
model.fit(train_images, train_labels, epochs=2)
loss, accuracy = model.evaluate(test_images, test_labels, verbose=2)
print('loss: {:.2f} accuracy: {:.2f}% '.format(loss, accuracy * 100))
model.save('model.h5')
# Epoch 1/2
# 1875/1875 [==============================] - 1s 654us/step - loss: 0.2642 - sparse_categorical_accuracy: 0.9239
# Epoch 2/2
# 1875/1875 [==============================] - 2s 848us/step - loss: 0.1159 - sparse_categorical_accuracy: 0.9657
# 313/313 - 0s - loss: 0.1132 - sparse_categorical_accuracy: 0.9663
# loss: 0.11 accuracy: 96.63%

再评估就正确了

import tensorflow as tf

(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()
test_images = test_images / 255.0
model = tf.keras.models.load_model('model.h5')
loss, accuracy = model.evaluate(test_images, test_labels, verbose=2)
print('loss: {:.2f} accuracy: {:.2f}% '.format(loss, accuracy * 100))

使用 TensorFlow 2.4 或 新版本的 tf-nightly

pip install tensorflow-gpu==2.4.0

参考文献

1. Accuracy is lost after save/load
2. TF website tutorial “Save and load” fails on Google Colab for entire models