TensorFlow中踩的坑 : RandomShuffleQueue '***' is closed and has insufficient elements

Traceback (most recent call last):
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
    return fn(*args)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insuff
icient elements (requested 16, current size 10)
         [[{{node shuffle_batch}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\supervisor.py", line 994, in managed_session
    yield sess
  File "main.py", line 359, in <module>
    results = sess.run(fetches)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
    run_metadata_ptr)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
    run_metadata)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insuff
icient elements (requested 16, current size 10)
         [[node shuffle_batch (defined at G:\SuperResolution\5.SRGAN\tensorflow\SRGAN-tensorflow-master\lib\model.py:119) ]]

Caused by op 'shuffle_batch', defined at:
  File "main.py", line 226, in <module>
    data = data_loader(FLAGS)
  File "G:\SuperResolution\5.SRGAN\tensorflow\SRGAN-tensorflow-master\lib\model.py", line 119, in data_loader
    min_after_dequeue=FLAGS.image_queue_capacity, num_threads=FLAGS.queue_thread)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\input.py", line 1346, in shuffle_batch
    name=name)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\input.py", line 873, in _shuffle_batch
    dequeued = queue.dequeue_many(batch_size, name=name)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\ops\data_flow_ops.py", line 488, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 3645, in queue_dequeue_many_v2
    timeout_ms=timeout_ms, name=name)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
    op_def=op_def)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient element
s (requested 16, current size 10)
         [[node shuffle_batch (defined at G:\SuperResolution\5.SRGAN\tensorflow\SRGAN-tensorflow-master\lib\model.py:119) ]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 388, in <module>
    print('Optimization done!!!!!!!!!!!!')
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\supervisor.py", line 1004, in managed_session
    self.stop(close_summary_writer=close_summary_writer)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\supervisor.py", line 832, in stop
    ignore_live_threads=ignore_live_threads)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\queue_runner_impl.py", line 257, in _run
    enqueue_callable()
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1257, in _single_operation_run
    self._call_tf_sessionrun(None, {}, [], target_list, None)
  File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[985,652,3] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
         [[{{node load_image/convert_image_1/Cast}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allo
cation info.

以上是我跑的出错结果，跑的是修改后的SRGAN tensorflow代码SRGAN_tensorflow代码，我电脑上的配置是 16G内存+RTX2080显卡+i5 8400的CPU，系统装在固态硬盘，程序和代码存在机械硬盘，在程序运行时机械硬盘读写一直都占满，查了下网上对于

tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insuff
icient elements (requested 16, current size 10)

的说明，此文觉得可能是因为文件输入来不及读取完全的缘故，解决方法是把图片裁小。而对于OOM的问题

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[985,652,3] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu

此文认为是batch_size大小的问题。但我程序和原来相比就只加深了网络，为什么到了后面才出错呢？

# 跑到这里才出错
progress  epoch 168  step 29  image/sec 49.2  remaining 80m
global_step 85199
PSNR 25.048532
content_loss 0.043942176
learning_rate 1e-04

放一下tensorboard上的图