与前文Ai challenger 场景分类: train softmax using tfrecord的区别见代码前面的changes说明。
目前tfrecord坑很多,参见 [Enhancement] Redesigning TensorFlow’s input pipelines #7951
目前赤裸的softmax过拟合严重:0.7 vs 0.18
# -*- coding: utf-8 -*-
"""
Created on Wed Sep 20 16:05:02 2017
@author: wayne
FEELINGS
目前tfrecord的坑还是挺多的,未来的1.4版本和2版本特性参见
https://github.com/tensorflow/tensorflow/issues/7902
和
https://github.com/tensorflow/tensorflow/issues/7951
CHANGES
- 训练和测试的一体化,以方便加入统一的数据预处理:注意目前是直接将验证集作为测试集来使用!!!注意数据增强只在训练时使用。
train_flag = False (测试模式)
- 将测试集的结果写入提交格式submit.json,供官方提供的scene_eval.py 使用:
https://github.com/AIChallenger/AI_Challenger/tree/master/AI_Challenger_eval_public
- image = tf.image.per_image_standardization(image) 修改到tf.image.resize_images后
- 其他小细节的改进
TODO
【看着很复杂,分解后逐步实现比较容易(注意需要尽可能考虑程序未来的可扩展性,以降低重构的工作量),最后可以再考虑进一步优化程序的架构等等,先跑通必要的功能】
- NEXT (train_flag = True): 增加每训练一段时间显示一次验证准确率,即train_flag = True时需要load train和val.
https://stackoverflow.com/questions/44270198/when-using-tfrecord-how-can-i-run-intermediate-validation-check-a-better-way
https://github.com/tensorflow/tensorflow/issues/7902
训练结束显示整个训练集上的准确率?
- NEXT: finetune基于imagenet的inception-resnet v2, senet等
- NEXT: 调参和数据增强,模型复杂度, use log file, use input args 模块化等
REFERENCES
输入数据
https://stackoverflow.com/questions/44054656/creating-tfrecords-from-a-list-of-strings-and-feeding-a-graph-in-tensorflow-afte
https://indico.io/blog/tensorflow-data-inputs-part1-placeholders-protobufs-queues/
https://indico.io/blog/tensorflow-data-input-part2-extensions/
整个架构
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/how_tos/reading_data/fully_connected_reader.py
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/udacity/2_fullyconnected.ipynb
模型的存储和调用
http://blog.csdn.net/u014595019/article/details/53912710
http://blog.csdn.net/u012436149/article/details/52883747 (restore变量的子集)
https://github.com/SymphonyPy/Valified_Code_Classify/tree/master/Classified
"""
from PIL import Image
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import time
import json
def read_and_decode(tfrecords_file, batch_size, num_epochs):
filename_queue = tf.train.string_input_producer([tfrecord_file], num_epochs = num_epochs)
reader = tf.TFRecordReader()
_, serialized_example = reader.read(filename_queue)
img_features = tf.parse_single_example(
serialized_example,
features={
'label': tf.FixedLenFeature([], tf.int64),
'h': tf.FixedLenFeature([], tf.int64),
'w': tf.FixedLenFeature([], tf.int64),
'c': tf.FixedLenFeature([], tf.int64),
'image': tf.FixedLenFeature([], tf.string),
})
h = tf.cast(img_features['h'], tf.int32)
w = tf.cast(img_features['w'], tf.int32)
c = tf.cast(img_features['c'], tf.int32)
image = tf.decode_raw(img_features['image'], tf.uint8)
image = tf.reshape(image, [h, w, c])
label = tf.cast(img_features['label'],tf.int32)
#label = tf.reshape(label, [1])
##########################################################
'''data augmentation here'''
# distorted_image = tf.random_crop(images, [530, 530, img_channel])
# distorted_image = tf.image.random_flip_left_right(distorted_image)
# distorted_image = tf.image.random_brightness(distorted_image, max_delta=63)
# distorted_image = tf.image.random_contrast(distorted_image, lower=0.2, upper=1.8)
image = tf.image.resize_images(image, (image_size,image_size))
image = tf.image.per_image_standardization(image)
image = tf.reshape(image, [image_size * image_size * 3])
#image, label = tf.train.batch([image, label], batch_size= batch_size)