self._traceback = tf_stack.extract_stack()报错

最新推荐文章于 2022-04-28 10:07:28 发布
小粽子~
最新推荐文章于 2022-04-28 10:07:28 发布
阅读量2.1k
点赞数
分类专栏： cudnn conda环境文章标签： tensorflow gpu
本文链接：https://blog.csdn.net/qq_38846606/article/details/111499733
版权
conda环境同时被 2 个专栏收录
4 篇文章 0 订阅
订阅专栏
cudnn
2 篇文章 0 订阅
订阅专栏
self._traceback = tf_stack.extract_stack()报错

WARNING:tensorflow:From main.py:62: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From main.py:51: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

W1221 20:56:15.050470 140262780077824 deprecation_wrapper.py:119] From main.py:51: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-12-21 20:56:15.069244: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-12-21 20:56:15.180222: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:3f:00.0
2020-12-21 20:56:15.180715: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-12-21 20:56:15.184348: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-12-21 20:56:15.188109: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-12-21 20:56:15.188686: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-12-21 20:56:15.192638: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-12-21 20:56:15.194724: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-12-21 20:56:15.202523: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-12-21 20:56:15.207166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-12-21 20:56:15.207819: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2020-12-21 20:56:15.225886: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2100000000 Hz
2020-12-21 20:56:15.231150: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55792517a2a0 executing computations on platform Host. Devices:
2020-12-21 20:56:15.231204: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2020-12-21 20:56:15.233391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:3f:00.0
2020-12-21 20:56:15.233488: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-12-21 20:56:15.233514: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-12-21 20:56:15.233549: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-12-21 20:56:15.233582: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-12-21 20:56:15.233602: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-12-21 20:56:15.233637: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-12-21 20:56:15.233658: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-12-21 20:56:15.237529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-12-21 20:56:15.237603: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-12-21 20:56:15.456264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-12-21 20:56:15.456310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2020-12-21 20:56:15.456326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2020-12-21 20:56:15.461050: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3251 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:3f:00.0, compute capability: 7.5)
2020-12-21 20:56:15.463363: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5579259d2c40 executing computations on platform CUDA. Devices:
2020-12-21 20:56:15.463390: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/ops.py:12: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W1221 20:56:15.469286 140262780077824 deprecation_wrapper.py:119] From /root/userfolder/pix2pix-tensorflow-master-ssim2/ops.py:12: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:69: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W1221 20:56:15.471386 140262780077824 deprecation_wrapper.py:119] From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:69: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/ops.py:48: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W1221 20:56:15.478073 140262780077824 deprecation_wrapper.py:119] From /root/userfolder/pix2pix-tensorflow-master-ssim2/ops.py:48: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W1221 20:56:15.849536 140262780077824 lazy_loader.py:50] 
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:253: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
W1221 20:56:16.182949 140262780077824 deprecation.py:506] From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:253: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:213: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.

W1221 20:56:16.434811 140262780077824 deprecation_wrapper.py:119] From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:213: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.

WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:86: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

W1221 20:56:16.836539 140262780077824 deprecation_wrapper.py:119] From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:86: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:88: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.

W1221 20:56:16.839233 140262780077824 deprecation_wrapper.py:119] From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:88: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.

WARNING:tensorflow:From /root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W1221 20:56:16.843780 140262780077824 deprecation.py:323] From /root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
(1,)
Tensor("Reshape_12:0", shape=(), dtype=float32)
WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:103: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W1221 20:56:16.987187 140262780077824 deprecation_wrapper.py:119] From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:103: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:116: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W1221 20:56:16.992135 140262780077824 deprecation_wrapper.py:119] From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:116: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:141: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W1221 20:56:17.085684 140262780077824 deprecation_wrapper.py:119] From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:141: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:149: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.

W1221 20:56:23.869896 140262780077824 deprecation_wrapper.py:119] From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:149: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.

WARNING:tensorflow:From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:152: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

W1221 20:56:23.876255 140262780077824 deprecation_wrapper.py:119] From /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:152: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

 [*] Reading checkpoint...
WARNING:tensorflow:From /root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W1221 20:56:24.478442 140262780077824 deprecation.py:323] From /root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from ./checkpoint/facades_1_256/pix2pix.model-1111502
I1221 20:56:24.481139 140262780077824 saver.py:1280] Restoring parameters from ./checkpoint/facades_1_256/pix2pix.model-1111502
 [*] Load SUCCESS
2020-12-21 20:56:26.734962: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-12-21 20:56:27.420447: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-12-21 20:56:28.673295: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-12-21 20:56:28.754756: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-12-21 20:56:28.785498: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-12-21 20:56:28.787434: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node generator/g_e1_conv/Conv2D}}]]
	 [[discriminator/Sigmoid/_223]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node generator/g_e1_conv/Conv2D}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 62, in <module>
    tf.app.run()
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "main.py", line 57, in main
    model.train(args)
  File "/root/userfolder/pix2pix-tensorflow-master-ssim2/model.py", line 177, in train
    feed_dict={ self.real_data: batch_images })
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node generator/g_e1_conv/Conv2D (defined at /root/userfolder/pix2pix-tensorflow-master-ssim2/ops.py:50) ]]
	 [[discriminator/Sigmoid/_223]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node generator/g_e1_conv/Conv2D (defined at /root/userfolder/pix2pix-tensorflow-master-ssim2/ops.py:50) ]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node generator/g_e1_conv/Conv2D:
 strided_slice_1 (defined at /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:75)	
 generator/g_e1_conv/w/read (defined at /root/userfolder/pix2pix-tensorflow-master-ssim2/ops.py:49)

Input Source operations connected to node generator/g_e1_conv/Conv2D:
 strided_slice_1 (defined at /root/userfolder/pix2pix-tensorflow-master-ssim2/model.py:75)	
 generator/g_e1_conv/w/read (defined at /root/userfolder/pix2pix-tensorflow-master-ssim2/ops.py:49)

Original stack trace for 'generator/g_e1_conv/Conv2D':
  File "main.py", line 62, in <module>
    tf.app.run()
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "main.py", line 54, in main
    checkpoint_dir=args.checkpoint_dir, sample_dir=args.sample_dir)
  File "/root/userfolder/pix2pix-tensorflow-master-ssim2/model.py", line 66, in __init__
    self.build_model()
  File "/root/userfolder/pix2pix-tensorflow-master-ssim2/model.py", line 77, in build_model
    self.fake_B = self.generator(self.real_A)
  File "/root/userfolder/pix2pix-tensorflow-master-ssim2/model.py", line 234, in generator
    e1 = conv2d(image, self.gf_dim, name='g_e1_conv')
  File "/root/userfolder/pix2pix-tensorflow-master-ssim2/ops.py", line 50, in conv2d
    conv = tf.nn.conv2d(input_, w, strides=[1, d_h, d_w, 1], padding='SAME')
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1953, in conv2d
    name=name)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1071, in conv2d
    data_format=data_format, dilations=dilations, name=name)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/root/anaconda3/envs/pix2pix-tensorflow/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()
原因：
1、为了tensorboard能够运转，我卸载了tensorboard-plugin-wit，只要重新pip install tensorboard-plugin-wit==1.6.0.post3就能运行了。
2、GPU内存溢出了。。。。关闭其他占用GPU代码或程序就能解决。