为tensorflow分配GPU或CPU

0.环境

Win10
python3.5
tensorflow-gpu==1.12.0

持续监控GPU使用情况：

watch -n 10 nvidia-smi

 

1.只使用CPU

  在代码前加入一下代码： 

import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

2.百分比使用GPU

  在代码前加入一下代码： 

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.5 # 占用GPU50%的显存
session = tf.Session(config=config)

3.最小量增长使用GPU

  在代码前加入一下代码： 

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)

4.指定多GPU中一个或者几个GPU

  指定多GPU中的一个GPU时，在代码前加入一下代码（Pytorch也可以使用以下）： 

import os
os.environ['CUDA_VISIBLE_DEVICES']='3'

  指定多GPU中的多个GPU时，在代码前加入一下代码： 

import os
os.environ['CUDA_VISIBLE_DEVICES']='2,3'#假设指定第二和第三块GPU

如果设置多GPU还是出现

2020-06-07 14:39:52.003640: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 524288 totalling 512.0KiB
2020-06-07 14:39:52.003650: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 524544 totalling 512.2KiB
2020-06-07 14:39:52.003661: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 690944 totalling 674.8KiB
2020-06-07 14:39:52.003671: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 10.37MiB
2020-06-07 14:39:52.003684: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                 11908877517
InUse:                    10878208
MaxInUse:               6485541376
NumAllocs:                 2717750
MaxAllocSize:           3227053056

2020-06-07 14:39:52.003740: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *___________________________________________________________________________________________________
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InternalError'>, Dst tensor is not initialized.
         [[{{node clone_1/prefetch_queue_Dequeue/_3899}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_500_clone_1/prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
I0607 14:39:58.117163 139928698971904 tf_logging.py:115] Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InternalError'>, Dst tensor is not initialized.
         [[{{node clone_1/prefetch_queue_Dequeue/_3899}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_500_clone_1/prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
         [[{{node clone_1/prefetch_queue_Dequeue/_3899}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_500_clone_1/prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./train.py", line 188, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 306, in new_func
    return func(*args, **kwargs)
  File "./train.py", line 184, in main
    graph_hook_fn=graph_rewriter_fn)
  File "/opt/SSD_Mobilenetv2/models-1.12.0/research/object_detection/legacy/trainer.py", line 415, in train
    saver=saver)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 770, in train
    sess, train_op, global_step, train_step_kwargs)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 487, in train_step
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
         [[{{node clone_1/prefetch_queue_Dequeue/_3899}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_500_clone_1/prefetch_queue_Dequeue", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]

Dst tensor is not initialized，GPU显存问题，我们尝试可以添加：

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.8) #0.333
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True, gpu_options=gpu_options))

 

 参考

   1.tensorflow在训练模型的时候如何指定GPU进行训练

   2.Tensorflow多GPU使用详解