TF1 RandomShuffleQueue ‘_3_shuffle_batch/random_shuffle_queue‘ is closed and has insufficient ...

今天在跑TF1code的时候遇到这样一个问题：

OutOfRangeError (see above for traceback): RandomShuffleQueue '_3_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
	 [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

详细信息：

Caused by op u'shuffle_batch', defined at:
  File "/home/weiyuhua/Medical_Image_Segmentation/Robust-Mseg/main.py", line 428, in <module>
    main()
  File "/home/weiyuhua/Medical_Image_Segmentation/Robust-Mseg/main.py", line 424, in main
    multimodal_model.train()
  File "/home/weiyuhua/Medical_Image_Segmentation/Robust-Mseg/main.py", line 256, in train
    data, brainmask, label = data_loader.load_data(train_list_pth, train_data_pth, modality_list, BATCH_SIZE, gt_flag=True, crop_size=CROP_SIZE, num_cls=NUM_CLS)
  File "/home/weiyuhua/Medical_Image_Segmentation/Robust-Mseg/data_loader.py", line 121, in load_data
    data_vol_batch, brainmask_batch, label_batch = tf.train.shuffle_batch([data_vol, brainmask, label], batch_size, 50, 10, num_threads=1)
  File "/home/weiyuhua/Medical_Image_Segmentation/envs/py27/local/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 1300, in shuffle_batch
    name=name)
  File "/home/weiyuhua/Medical_Image_Segmentation/envs/py27/local/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 846, in _shuffle_batch
    dequeued = queue.dequeue_many(batch_size, name=name)
  File "/home/weiyuhua/Medical_Image_Segmentation/envs/py27/local/lib/python2.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 476, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "/home/weiyuhua/Medical_Image_Segmentation/envs/py27/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3480, in queue_dequeue_many_v2
    component_types=component_types, timeout_ms=timeout_ms, name=name)
  File "/home/weiyuhua/Medical_Image_Segmentation/envs/py27/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/weiyuhua/Medical_Image_Segmentation/envs/py27/local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
    return func(*args, **kwargs)
  File "/home/weiyuhua/Medical_Image_Segmentation/envs/py27/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
    op_def=op_def)
  File "/home/weiyuhua/Medical_Image_Segmentation/envs/py27/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
    self._traceback = tf_stack.extract_stack()

OutOfRangeError (see above for traceback): RandomShuffleQueue '_3_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
	 [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

debug思路与解决方法：https://stackoverflow.com/questions/46099109/outofrangeerror-randomshufflequeue-2-shuffle-batch-random-shuffle-queue-is-c

1. 检查读取TF records的路径
2. 数据形状不匹配
3. 数据格式不匹配
   
   依照上面的提示检查，路径没问题；然后检查数据形状是否有问题（因为有reshape操作），发现确实是数据形状匹配的时候错了，如下：

InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 17856000 values, but the requested shape has 8928000
	 [[Node: Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](DecodeRaw/_25, Reshape/shape)]]

因此定位到保存TF records的代码，将

data_arr = np.zeros((dsize_dim0_val, dsize_dim1_val, dsize_dim2_val))

（type为float64，因此按float32的大小做reshape的时候，会出现以上错误）

改为

data_arr = np.zeros((dsize_dim0_val, dsize_dim1_val, dsize_dim2_val))
data_arr = np.float32(data_arr)

（一定要注意细节！！）