2019-05-16 09:41:32.289878: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f178c

2019-05-16 13:04:19.139020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 63 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-05-16 13:04:19.139465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10470 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
2019-05-16 13:04:19.139961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10470 MB memory) -> physical GPU (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:82:00.0, compute capability: 6.1)
2019-05-16 13:04:19.140418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10470 MB memory) -> physical GPU (device: 3, name: GeForce GTX 1080 Ti, pci bus id: 0000:83:00.0, compute capability: 6.1)
2019-05-16 13:04:30.389514: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 167.53MiB.  Current allocation summary follows.
2019-05-16 13:04:30.389592: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.389632: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.389689: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):     Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2019-05-16 13:04:30.389722: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.389753: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.389809: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.389840: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.389889: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.389922: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.389971: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390017: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390049: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390103: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390149: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390186: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390219: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390252: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390290: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):     Total Chunks: 1, Chunks in use: 0. 63.50MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390322: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390355: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390390: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-05-16 13:04:30.390427: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 167.53MiB was 128.00MiB, Chunk State:
2019-05-16 13:04:30.390453: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f2818000000 of size 1280
2019-05-16 13:04:30.390480: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0x7f2818000500 of size 66583296
2019-05-16 13:04:30.390505: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-05-16 13:04:30.390530: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-05-16 13:04:30.390558: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 1.2KiB
2019-05-16 13:04:30.390585: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                    66584576
InUse:                        1280
MaxInUse:                     1280
NumAllocs:                       1
MaxAllocSize:                 1280

 

问题就是显存不足的问题.

报错原因：显卡内存不够；

解决办法：①、首先查看显卡占用情况，有可能是显卡内存被别的程序占用没有退出，可以后台查看；

                  ②、排除第一种情况之后，考虑：bath_size可能设置过大，可以适当调小。

问题解决：指定使用哪一块GPU:

• import os

• os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"

• os.environ["CUDA_VISIBLE_DEVICES"] = "1"