0.环境
Win10
python3.5
tensorflow-gpu==1.12.0
持续监控GPU使用情况:
watch -n 10 nvidia-smi
1.只使用CPU
在代码前加入一下代码:
import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
2.百分比使用GPU
在代码前加入一下代码:
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.5 # 占用GPU50%的显存
session = tf.Session(config=config)
3.最小量增长使用GPU
在代码前加入一下代码:
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)
4.指定多GPU中一个或者几个GPU
指定多GPU中的一个GPU时,在代码前加入一下代码(Pytorch也可以使用以下):
import os
os.environ['CUDA_VISIBLE_DEVICES']='3'
指定多GPU中的多个GPU时,在代码前加入一下代码:
import os
os.environ['CUDA_VISIBLE_DEVICES']='2,3'#假设指定第二和第三块GPU
如果设置多GPU还是出现
2020-06-07 14:39:52.003640: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 524288 totalling 512.0KiB
2020-06-07 14:39:52.003650: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 524544 totalling 512.2KiB
2020-06-07 14:39:52.003661: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 690944 totalling 674.8KiB
2020-06-07 14:39:52.003671: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 10.37MiB
2020-06-07 14:39:52.003684: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit: 11908877517
InUse: 10878208
MaxInUse: 6485541376
NumAllocs: 2717750
MaxAllocSize: 3227053056
2020-06-07 14:39:52.003740: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *___________________________________________________________________________________________________
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InternalError'>, Dst tensor is not initialized.
[[{{node clone_1/prefetch_queue_Dequeue/_3899}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_500_clone_1/prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
I0607 14:39:58.117163 139928698971904 tf_logging.py:115] Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InternalError'>, Dst tensor is not initialized.
[[{{node clone_1/prefetch_queue_Dequeue/_3899}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_500_clone_1/prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[{{node clone_1/prefetch_queue_Dequeue/_3899}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_500_clone_1/prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./train.py", line 188, in <module>
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 306, in new_func
return func(*args, **kwargs)
File "./train.py", line 184, in main
graph_hook_fn=graph_rewriter_fn)
File "/opt/SSD_Mobilenetv2/models-1.12.0/research/object_detection/legacy/trainer.py", line 415, in train
saver=saver)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 770, in train
sess, train_op, global_step, train_step_kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 487, in train_step
run_metadata=run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[{{node clone_1/prefetch_queue_Dequeue/_3899}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_500_clone_1/prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
Dst tensor is not initialized,GPU显存问题,我们尝试可以添加:
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.8) #0.333
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True, gpu_options=gpu_options))
参考