错误：W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at transpose_op.cc:199

2019-07-23 10:05:55.269273: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.16GiB.  Current allocation summary follows.
2019-07-23 10:05:55.269540: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): 	Total Chunks: 66, Chunks in use: 66. 16.5KiB allocated for chunks. 16.5KiB in use in bin. 352B client-requested in use in bin.
2019-07-23 10:05:55.269800: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): 	Total Chunks: 2, Chunks in use: 2. 1.0KiB allocated for chunks. 1.0KiB in use in bin. 800B client-requested in use in bin.
2019-07-23 10:05:55.270057: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): 	Total Chunks: 1, Chunks in use: 1. 1.3KiB allocated for chunks. 1.3KiB in use in bin. 1.0KiB client-requested in use in bin.
2019-07-23 10:05:55.270336: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): 	Total Chunks: 31, Chunks in use: 31. 116.3KiB allocated for chunks. 116.3KiB in use in bin. 109.0KiB client-requested in use in bin.
2019-07-23 10:05:55.270603: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-23 10:05:55.270846: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-23 10:05:55.271095: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): 	Total Chunks: 6, Chunks in use: 6. 127.5KiB allocated for chunks. 127.5KiB in use in bin. 126.6KiB client-requested in use in bin.
2019-07-23 10:05:55.271373: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-23 10:05:55.271625: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): 	Total Chunks: 6, Chunks in use: 6. 760.5KiB allocated for chunks. 760.5KiB in use in bin. 759.4KiB client-requested in use in bin.
2019-07-23 10:05:55.271897: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): 	Total Chunks: 6, Chunks in use: 6. 1.48MiB allocated for chunks. 1.48MiB in use in bin. 1.48MiB client-requested in use in bin.
2019-07-23 10:05:55.272166: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): 	Total Chunks: 12, Chunks in use: 12. 5.19MiB allocated for chunks. 5.19MiB in use in bin. 5.19MiB client-requested in use in bin.
2019-07-23 10:05:55.272434: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): 	Total Chunks: 12, Chunks in use: 12. 8.16MiB allocated for chunks. 8.16MiB in use in bin. 8.16MiB client-requested in use in bin.
2019-07-23 10:05:55.272698: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): 	Total Chunks: 1, Chunks in use: 0. 1.32MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-23 10:05:55.272946: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-23 10:05:55.273183: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): 	Total Chunks: 2, Chunks in use: 1. 10.58MiB allocated for chunks. 5.29MiB in use in bin. 5.29MiB client-requested in use in bin.
2019-07-23 10:05:55.273433: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-23 10:05:55.273660: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-23 10:05:55.273885: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-23 10:05:55.276650: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-23 10:05:55.276927: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-07-23 10:05:55.277246: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): 	Total Chunks: 8, Chunks in use: 6. 8.43GiB allocated for chunks. 6.93GiB in use in bin. 6.92GiB client-requested in use in bin.
2019-07-23 10:05:55.277529: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 1.16GiB was 256.00MiB, Chunk State: 
2019-07-23 10:05:55.277652: I tensorflow/core/common_runtime/bfc_allocator.cc:619]   Size: 368.86MiB | Requested Size: 181.49MiB | in_use: 0, prev:   Size: 1.16GiB | Requested Size: 1.16GiB | in_use: 1
2019-07-23 10:05:55.277839: I tensorflow/core/common_runtime/bfc_allocator.cc:619]   Size: 1.14GiB | Requested Size: 1.14GiB | in_use: 0, prev:   Size: 1.15GiB | Requested Size: 1.14GiB | in_use: 1, next:   Size: 1.16GiB | Requested Size: 1.16GiB | in_use: 1
2019-07-23 10:05:55.278075: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000000 of size 1280
2019-07-23 10:05:55.278402: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000500 of size 256
2019-07-23 10:05:55.278554: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000600 of size 256
2019-07-23 10:05:55.278718: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000700 of size 256
2019-07-23 10:05:55.278886: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000800 of size 256
2019-07-23 10:05:55.279063: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000900 of size 256
2019-07-23 10:05:55.279172: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000A00 of size 256
2019-07-23 10:05:55.279281: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000B00 of size 256
2019-07-23 10:05:55.279390: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000C00 of size 256
2019-07-23 10:05:55.279498: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000D00 of size 256
2019-07-23 10:05:55.279607: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000E00 of size 256
2019-07-23 10:05:55.279716: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311000F00 of size 256
2019-07-23 10:05:55.279828: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001000 of size 256
2019-07-23 10:05:55.279941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001100 of size 256
2019-07-23 10:05:55.280053: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001200 of size 256
2019-07-23 10:05:55.280478: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001300 of size 256
2019-07-23 10:05:55.280655: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001400 of size 256
2019-07-23 10:05:55.280808: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001500 of size 256
2019-07-23 10:05:55.280973: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001600 of size 256
2019-07-23 10:05:55.281153: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001700 of size 256
2019-07-23 10:05:55.281319: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001800 of size 256
2019-07-23 10:05:55.281471: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001900 of size 256
2019-07-23 10:05:55.281636: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001A00 of size 256
2019-07-23 10:05:55.281840: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311001B00 of size 129792
2019-07-23 10:05:55.282005: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311021600 of size 259328
2019-07-23 10:05:55.282227: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311060B00 of size 777728
2019-07-23 10:05:55.282423: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000131111E900 of size 21760
2019-07-23 10:05:55.282608: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311123E00 of size 388864
2019-07-23 10:05:55.282823: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311182D00 of size 256
2019-07-23 10:05:55.282998: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311182E00 of size 518400
2019-07-23 10:05:55.283237: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311201700 of size 648192
2019-07-23 10:05:55.283471: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000131129FB00 of size 3840
2019-07-23 10:05:55.283661: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013112A0A00 of size 256
2019-07-23 10:05:55.283848: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013112A0B00 of size 256
2019-07-23 10:05:55.284017: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013112A0C00 of size 256
2019-07-23 10:05:55.284224: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013112A0D00 of size 256
2019-07-23 10:05:55.284390: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013112A0E00 of size 256
2019-07-23 10:05:55.284560: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013112A0F00 of size 129792
2019-07-23 10:05:55.284755: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013112C0A00 of size 259328
2019-07-23 10:05:55.284941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013112FFF00 of size 388864
2019-07-23 10:05:55.285111: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000131135EE00 of size 518400
2019-07-23 10:05:55.285265: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013113DD700 of size 648192
2019-07-23 10:05:55.285379: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000131147BB00 of size 777728
2019-07-23 10:05:55.285498: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311539900 of size 21760
2019-07-23 10:05:55.285613: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000131153EE00 of size 129792
2019-07-23 10:05:55.285730: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000131155E900 of size 129792
2019-07-23 10:05:55.285847: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000131157E400 of size 259328
2019-07-23 10:05:55.285963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013115BD900 of size 259328
2019-07-23 10:05:55.286081: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013115FCE00 of size 777728
2019-07-23 10:05:55.286198: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013116BAC00 of size 777728
2019-07-23 10:05:55.286314: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311778A00 of size 21760
2019-07-23 10:05:55.286435: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000131177DF00 of size 21760
2019-07-23 10:05:55.286558: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311783400 of size 388864
2019-07-23 10:05:55.286684: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013117E2300 of size 388864
2019-07-23 10:05:55.286811: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841200 of size 256
2019-07-23 10:05:55.286933: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841300 of size 256
2019-07-23 10:05:55.287061: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841400 of size 256
2019-07-23 10:05:55.287621: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841500 of size 256
2019-07-23 10:05:55.287777: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841600 of size 256
2019-07-23 10:05:55.287889: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841700 of size 256
2019-07-23 10:05:55.288001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841800 of size 256
2019-07-23 10:05:55.288113: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841900 of size 256
2019-07-23 10:05:55.288224: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841A00 of size 256
2019-07-23 10:05:55.288335: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841B00 of size 256
2019-07-23 10:05:55.288446: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841C00 of size 256
2019-07-23 10:05:55.288558: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841D00 of size 256
2019-07-23 10:05:55.288669: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841E00 of size 256
2019-07-23 10:05:55.288780: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311841F00 of size 256
2019-07-23 10:05:55.288892: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311842000 of size 256
2019-07-23 10:05:55.289003: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311842100 of size 256
2019-07-23 10:05:55.289437: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311842200 of size 256
2019-07-23 10:05:55.289552: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311842300 of size 518400
2019-07-23 10:05:55.289668: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013118C0C00 of size 518400
2019-07-23 10:05:55.289785: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000131193F500 of size 648192
2019-07-23 10:05:55.289900: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013119DD900 of size 648192
2019-07-23 10:05:55.290016: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A7BD00 of size 3840
2019-07-23 10:05:55.290307: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A7CC00 of size 3840
2019-07-23 10:05:55.290490: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A7DB00 of size 3840
2019-07-23 10:05:55.290637: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A7EA00 of size 3840
2019-07-23 10:05:55.290749: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A7F900 of size 3840
2019-07-23 10:05:55.290862: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A80800 of size 3840
2019-07-23 10:05:55.290974: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A81700 of size 3840
2019-07-23 10:05:55.291087: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A82600 of size 3840
2019-07-23 10:05:55.291199: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A83500 of size 3840
2019-07-23 10:05:55.291311: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A84400 of size 3840
2019-07-23 10:05:55.291423: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A85300 of size 3840
2019-07-23 10:05:55.291535: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A86200 of size 3840
2019-07-23 10:05:55.291647: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A87100 of size 3840
2019-07-23 10:05:55.291760: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A88000 of size 3840
2019-07-23 10:05:55.291873: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A88F00 of size 3840
2019-07-23 10:05:55.291986: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A89E00 of size 3840
2019-07-23 10:05:55.292102: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8AD00 of size 3840
2019-07-23 10:05:55.292552: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8BC00 of size 3840
2019-07-23 10:05:55.292668: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8CB00 of size 256
2019-07-23 10:05:55.292780: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8CC00 of size 256
2019-07-23 10:05:55.292891: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8CD00 of size 256
2019-07-23 10:05:55.293002: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8CE00 of size 256
2019-07-23 10:05:55.293114: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8CF00 of size 256
2019-07-23 10:05:55.293226: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8D000 of size 256
2019-07-23 10:05:55.293337: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8D100 of size 256
2019-07-23 10:05:55.293448: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8D200 of size 256
2019-07-23 10:05:55.293559: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8D300 of size 256
2019-07-23 10:05:55.293671: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8D400 of size 256
2019-07-23 10:05:55.293783: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8D500 of size 256
2019-07-23 10:05:55.293895: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8D600 of size 256
2019-07-23 10:05:55.294008: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8D700 of size 256
2019-07-23 10:05:55.294500: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8D800 of size 256
2019-07-23 10:05:55.294617: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8D900 of size 512
2019-07-23 10:05:55.294729: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311A8DB00 of size 512
2019-07-23 10:05:55.294840: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0000001311A8DD00 of size 1387264
2019-07-23 10:05:55.294955: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311BE0800 of size 256
2019-07-23 10:05:55.295067: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311BE0900 of size 256
2019-07-23 10:05:55.295179: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311BE0A00 of size 3840
2019-07-23 10:05:55.295291: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311BE1900 of size 388864
2019-07-23 10:05:55.295404: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311C40800 of size 3840
2019-07-23 10:05:55.295517: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311C41700 of size 129792
2019-07-23 10:05:55.295630: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311C61200 of size 518400
2019-07-23 10:05:55.295745: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311CDFB00 of size 3840
2019-07-23 10:05:55.295858: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311CE0A00 of size 648192
2019-07-23 10:05:55.295973: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311D7EE00 of size 3840
2019-07-23 10:05:55.296089: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311D7FD00 of size 777728
2019-07-23 10:05:55.296611: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311E3DB00 of size 3840
2019-07-23 10:05:55.296791: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311E3EA00 of size 21760
2019-07-23 10:05:55.296976: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311E43F00 of size 256
2019-07-23 10:05:55.297165: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311E44000 of size 3840
2019-07-23 10:05:55.297352: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311E44F00 of size 259328
2019-07-23 10:05:55.297548: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311E84400 of size 256
2019-07-23 10:05:55.297702: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311E84500 of size 388864
2019-07-23 10:05:55.297901: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311EE3400 of size 3840
2019-07-23 10:05:55.298012: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311EE4300 of size 129792
2019-07-23 10:05:55.298127: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311F03E00 of size 518400
2019-07-23 10:05:55.298241: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311F82700 of size 3840
2019-07-23 10:05:55.298353: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001311F83600 of size 648192
2019-07-23 10:05:55.298467: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001312021A00 of size 3840
2019-07-23 10:05:55.298579: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001312022900 of size 777728
2019-07-23 10:05:55.298692: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013120E0700 of size 3840
2019-07-23 10:05:55.298805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013120E1600 of size 21760
2019-07-23 10:05:55.298918: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013120E6B00 of size 256
2019-07-23 10:05:55.299030: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013120E6C00 of size 3840
2019-07-23 10:05:55.299442: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013120E7B00 of size 259328
2019-07-23 10:05:55.299648: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001312127000 of size 3840
2019-07-23 10:05:55.299818: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001312127F00 of size 256
2019-07-23 10:05:55.299996: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001312128000 of size 256
2019-07-23 10:05:55.300136: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 0000001312128100 of size 5548800
2019-07-23 10:05:55.300251: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001312672C00 of size 5548800
2019-07-23 10:05:55.300367: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0000001312BBD700 of size 1245600000
2019-07-23 10:05:55.300484: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000135CFA3000 of size 1245600000
2019-07-23 10:05:55.300602: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013A7388900 of size 1239120128
2019-07-23 10:05:55.300720: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000013F1140200 of size 1235880192
2019-07-23 10:05:55.300838: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000143ABE0B00 of size 1232640000
2019-07-23 10:05:55.300956: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 000000148436A300 of size 1229400064
2019-07-23 10:05:55.301085: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 00000014CD7DCB00 of size 1242360064
2019-07-23 10:05:55.301203: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free  at 00000015178AB400 of size 386780672
2019-07-23 10:05:55.301320: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size: 
2019-07-23 10:05:55.301437: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 66 Chunks of size 256 totalling 16.5KiB
2019-07-23 10:05:55.301553: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 512 totalling 1.0KiB
2019-07-23 10:05:55.301668: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.3KiB
2019-07-23 10:05:55.301784: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 31 Chunks of size 3840 totalling 116.3KiB
2019-07-23 10:05:55.301906: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 21760 totalling 127.5KiB
2019-07-23 10:05:55.302043: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 129792 totalling 760.5KiB
2019-07-23 10:05:55.302637: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 259328 totalling 1.48MiB
2019-07-23 10:05:55.302756: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 388864 totalling 2.22MiB
2019-07-23 10:05:55.302872: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 518400 totalling 2.97MiB
2019-07-23 10:05:55.302988: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 648192 totalling 3.71MiB
2019-07-23 10:05:55.303104: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 6 Chunks of size 777728 totalling 4.45MiB
2019-07-23 10:05:55.303220: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 5548800 totalling 5.29MiB
2019-07-23 10:05:55.303337: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1232640000 totalling 1.15GiB
2019-07-23 10:05:55.303455: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1235880192 totalling 1.15GiB
2019-07-23 10:05:55.303574: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1239120128 totalling 1.15GiB
2019-07-23 10:05:55.303693: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1242360064 totalling 1.16GiB
2019-07-23 10:05:55.303812: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 1245600000 totalling 2.32GiB
2019-07-23 10:05:55.303931: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 6.95GiB
2019-07-23 10:05:55.304046: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: 
Limit:                  9086468751
InUse:                  7463351808
MaxInUse:               8883318272
NumAllocs:                     214
MaxAllocSize:           1245600000
2019-07-23 10:05:55.304603: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *********************************************************************_____________**************____
2019-07-23 10:05:55.304796: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at transpose_op.cc:199 : Resource exhausted: OOM when allocating tensor with shape[100,1,3451,900] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):

    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-4c179d8d96c8>", line 1, in <module>

这是tensorflow 一个经常性错误，错误的原因在于：显卡内存不够。
解决方法就是降低显卡的使用内存，途径有以下几种措施:
1 减少Batch 的大小

2 分析错误的位置，在哪一层出现显卡不够，比如在全连接层出现的，则降低全连接层的维度，把2048改成1042啥的

3 增加pool 层，降低整个网络的维度。

4 修改输入大小