Traceback (most recent call last):
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insuff
icient elements (requested 16, current size 10)
[[{{node shuffle_batch}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\supervisor.py", line 994, in managed_session
yield sess
File "main.py", line 359, in <module>
results = sess.run(fetches)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insuff
icient elements (requested 16, current size 10)
[[node shuffle_batch (defined at G:\SuperResolution\5.SRGAN\tensorflow\SRGAN-tensorflow-master\lib\model.py:119) ]]
Caused by op 'shuffle_batch', defined at:
File "main.py", line 226, in <module>
data = data_loader(FLAGS)
File "G:\SuperResolution\5.SRGAN\tensorflow\SRGAN-tensorflow-master\lib\model.py", line 119, in data_loader
min_after_dequeue=FLAGS.image_queue_capacity, num_threads=FLAGS.queue_thread)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\input.py", line 1346, in shuffle_batch
name=name)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\input.py", line 873, in _shuffle_batch
dequeued = queue.dequeue_many(batch_size, name=name)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\ops\data_flow_ops.py", line 488, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\ops\gen_data_flow_ops.py", line 3645, in queue_dequeue_many_v2
timeout_ms=timeout_ms, name=name)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
op_def=op_def)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in __init__
self._traceback = tf_stack.extract_stack()
OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient element
s (requested 16, current size 10)
[[node shuffle_batch (defined at G:\SuperResolution\5.SRGAN\tensorflow\SRGAN-tensorflow-master\lib\model.py:119) ]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 388, in <module>
print('Optimization done!!!!!!!!!!!!')
File "G:\Anaconda3\envs\tensorflow-gpu\lib\contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\supervisor.py", line 1004, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\supervisor.py", line 832, in stop
ignore_live_threads=ignore_live_threads)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\six.py", line 693, in reraise
raise value
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\training\queue_runner_impl.py", line 257, in _run
enqueue_callable()
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1257, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "G:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[985,652,3] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
[[{{node load_image/convert_image_1/Cast}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allo
cation info.
以上是我跑的出错结果,跑的是修改后的SRGAN tensorflow代码SRGAN_tensorflow代码,我电脑上的配置是 16G内存+RTX2080显卡+i5 8400的CPU,系统装在固态硬盘,程序和代码存在机械硬盘,在程序运行时机械硬盘读写一直都占满,查了下网上对于
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insuff
icient elements (requested 16, current size 10)
的说明,此文觉得可能是因为文件输入来不及读取完全的缘故,解决方法是把图片裁小。而对于OOM的问题
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[985,652,3] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
此文认为是batch_size大小的问题。但我程序和原来相比就只加深了网络,为什么到了后面才出错呢?
# 跑到这里才出错
progress epoch 168 step 29 image/sec 49.2 remaining 80m
global_step 85199
PSNR 25.048532
content_loss 0.043942176
learning_rate 1e-04
放一下tensorboard上的图