分布式计算学习第二坑 cudnn

最新推荐文章于 2021-09-12 20:33:48 发布
最新推荐文章于 2021-09-12 20:33:48 发布
阅读量653 收藏
点赞数
分类专栏： tensorflow on spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_45939774/article/details/106350214
版权
tensorflow 同时被 3 个专栏收录
1 篇文章 0 订阅
订阅专栏
on
1 篇文章 0 订阅
订阅专栏
spark
1 篇文章 0 订阅
订阅专栏 
andrew@1manjaro:~/mount/arch/TensorFlowOnSpark/examples/resnet#export TF_CONFIG='{"cluster": { "chief": ["localhost:2222"], "worker": ["localhost:2223"]}, "task": {"type": "chief", "index": 0}}'
python resnet_cifar_main.py --data_dir=${CIFAR_DATA} --num_gpus=1 --ds=multi_worker_mirrored --train_epochs=100
2020-05-26 10:43:02.240568: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-05-26 10:43:03.329536: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-26 10:43:03.332103: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.332337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:1c:00.0 name: GeForce RTX 2070 computeCapability: 7.5
coreClock: 1.725GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-26 10:43:03.332359: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-05-26 10:43:03.333363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 10:43:03.334473: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 10:43:03.334654: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 10:43:03.335746: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 10:43:03.336382: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 10:43:03.338684: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 10:43:03.338811: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.339096: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.339297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-26 10:43:03.339909: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
2020-05-26 10:43:03.360884: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3701195000 Hz
2020-05-26 10:43:03.361383: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5586666e7300 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-26 10:43:03.361399: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-26 10:43:03.682688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.682977: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558665a43420 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-26 10:43:03.683005: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 2070, Compute Capability 7.5
2020-05-26 10:43:03.683211: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.683432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:1c:00.0 name: GeForce RTX 2070 computeCapability: 7.5
coreClock: 1.725GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-26 10:43:03.683458: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-05-26 10:43:03.683481: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 10:43:03.683491: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 10:43:03.683500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 10:43:03.683510: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 10:43:03.683519: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 10:43:03.683528: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 10:43:03.683571: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.683789: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.683974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-26 10:43:03.683995: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-05-26 10:43:03.959819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-26 10:43:03.959852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2020-05-26 10:43:03.959860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2020-05-26 10:43:03.960066: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.960320: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.960529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 35 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070, pci bus id: 0000:1c:00.0, compute capability: 7.5)
2020-05-26 10:43:03.960938: I tensorflow/core/common_runtime/process_util.cc:147] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2020-05-26 10:43:03.961343: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.961546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:1c:00.0 name: GeForce RTX 2070 computeCapability: 7.5
coreClock: 1.725GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-26 10:43:03.961570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-05-26 10:43:03.961593: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 10:43:03.961604: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 10:43:03.961614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 10:43:03.961624: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 10:43:03.961634: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 10:43:03.961644: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 10:43:03.961686: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.961902: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.962084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-26 10:43:03.962100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-26 10:43:03.962106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2020-05-26 10:43:03.962111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2020-05-26 10:43:03.962169: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.962391: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.962579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:chief/replica:0/task:0/device:GPU:0 with 35 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070, pci bus id: 0000:1c:00.0, compute capability: 7.5)
2020-05-26 10:43:03.964986: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job chief -> {0 -> localhost:2222}
2020-05-26 10:43:03.965006: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> localhost:2223}
2020-05-26 10:43:03.965380: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:390] Started server with target: grpc://localhost:2222
INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:chief/replica:0/task:0/device:CPU:0', '/job:chief/replica:0/task:0/device:XLA_CPU:0', '/job:chief/replica:0/task:0/device:XLA_GPU:0', '/job:chief/replica:0/task:0/device:GPU:0']
I0526 10:43:03.965880 140302475241280 collective_all_reduce_strategy.py:303] Enabled multi-worker collective ops with available devices: ['/job:chief/replica:0/task:0/device:CPU:0', '/job:chief/replica:0/task:0/device:XLA_CPU:0', '/job:chief/replica:0/task:0/device:XLA_GPU:0', '/job:chief/replica:0/task:0/device:GPU:0']
INFO:tensorflow:Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
I0526 10:43:03.966263 140302475241280 mirrored_strategy.py:500] Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
I0526 10:43:03.966428 140302475241280 collective_all_reduce_strategy.py:344] MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.433851 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.437643 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.444913 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.447288 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.500257 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.502656 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.508196 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.510025 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.543776 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.545943 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Running Distribute Coordinator with mode = 'independent_worker', cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, environment = None, rpc_layer = 'grpc'
I0526 10:43:06.832544 140302475241280 distribute_coordinator.py:773] Running Distribute Coordinator with mode = 'independent_worker', cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, environment = None, rpc_layer = 'grpc'
WARNING:tensorflow:`eval_fn` is not passed in. The `worker_fn` will be used if an "evaluator" task exists in the cluster.
W0526 10:43:06.832681 140302475241280 distribute_coordinator.py:825] `eval_fn` is not passed in. The `worker_fn` will be used if an "evaluator" task exists in the cluster.
WARNING:tensorflow:`eval_strategy` is not passed in. No distribution strategy will be used for evaluation.
W0526 10:43:06.832730 140302475241280 distribute_coordinator.py:829] `eval_strategy` is not passed in. No distribution strategy will be used for evaluation.
INFO:tensorflow:Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
I0526 10:43:06.833080 140302475241280 mirrored_strategy.py:500] Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
I0526 10:43:06.833179 140302475241280 collective_all_reduce_strategy.py:344] MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
INFO:tensorflow:Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
I0526 10:43:06.833521 140302475241280 mirrored_strategy.py:500] Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
I0526 10:43:06.833611 140302475241280 collective_all_reduce_strategy.py:344] MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
Epoch 1/100
INFO:tensorflow:Collective batch_all_reduce: 176 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:08.031161 140302475241280 cross_device_ops.py:1054] Collective batch_all_reduce: 176 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 176 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:11.108297 140302475241280 cross_device_ops.py:1054] Collective batch_all_reduce: 176 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
2020-05-26 10:43:15.515986: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 10:43:15.864391: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 10:43:16.010903: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-05-26 10:43:16.021040: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
  File "resnet_cifar_main.py", line 288, in <module>
    app.run(main)
  File "/usr/lib/python3.8/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/lib/python3.8/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "resnet_cifar_main.py", line 282, in main
    return run(flags.FLAGS)
  File "resnet_cifar_main.py", line 251, in run
    history = model.fit(train_input_dataset,
  File "/usr/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 72, in _method_wrapper
    return dc.run_distribute_coordinator(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_coordinator.py", line 852, in run_distribute_coordinator
    return _run_single_worker(worker_fn, strategy, cluster_spec, task_type,
  File "/usr/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_coordinator.py", line 360, in _run_single_worker
    return worker_fn(strategy)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 73, in <lambda>
    lambda _: method(self, *args, **kwargs),
  File "/usr/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 848, in fit
    tmp_logs = train_function(iterator)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 644, in _call
    return self._stateless_fn(*args, **kwds)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2420, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call
    return self._call_flat(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 593, in call
    outputs = execute.execute(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node resnet56/conv1/Conv2D (defined at /threading.py:932) ]]
	 [[GroupCrossDeviceControlEdges_0/Identity_2/_35]]
  (1) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node resnet56/conv1/Conv2D (defined at /threading.py:932) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_19079]

Function call stack:
train_function -> train_function

2020-05-26 10:43:16.566656: W tensorflow/core/common_runtime/eager/context.cc:447] Unable to destroy server_ object, so releasing instead. Servers don't support clean shutdown.

是显存不够用导致的

import tensorflow as tf
import keras

config = tf.compat.v1.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.4
tf.compat.v1.keras.backend.set_session(tf.compat.v1.Session(config=config))

在这里插入图片描述

weixin_45939774
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
tensorflow系列(3)分布式tensorflow
EternalMind
08-14 1193
多机如何分布式运行tensorflow模型？
Colab tqdm_progress_bar.py error
linyuanthocr的博客
04-11 443
Colab tqdm_process_bar.py 报错解决代码：错误提示解决方法 代码： import numpy as np import tensorflow as tf import tensorflow_addons as tfa x = np.random.random((5,1,5)) y = np.random.random((5,1,5)) inputs = tf.keras...
卷积神经网络的分布式训练（一）ResNet
qq_36484003的博客
09-18 1363
卷积神经网络的分布式训练（一）ResNet 该专题是我本科毕设的各个知识点介绍和分析，以及一些实验结果展示。本科毕业了，研究生开学，又进入了一个新的学习阶段，那么趁着有时间就总结一下这个课题，分享给大家~ 如果你觉得对你有帮助，给我点赞吧！小女子能力有限，也欢迎大家批评指正。 目录卷积神经网络的分布式训练（一）ResNet一、ResNet 介绍基本原理层次种类二、网络模型结构设计1、预处理模块2、基本单元3、池化层4、全连接层参考文献 一、ResNet 介绍 ResNet（Residual Neural N
python学习之多任务②
qq_39377119的博客
01-21 111
线程共享全局变量 import threading # 定义全局变量 globals_num globals_num = 0 def test1(num): global globals_num for i in range(num): globals_num += 1 print("test1当前变量值为:%d" % globals_num) d...
python学习（2）— 就业岗位准备
Hubert的博客
08-26 221
一、岗位表 职位 技能 Python后端开发工程师 Python基础Django / Flask / Tornado / SanicRESTful / 接口文档撰写MySQL / Redis / MongoDB / ElasticSearchLinux / Git / Scrum / PyCharm python量化交易开发工程师 Python基础数据结构 / 算法 /...
热门深度学习工具排行：Pylearn2 居首，Caffe第三1
08-08
DL4J的目标是使深度学习技术更易于在企业环境中部署，支持大规模数据处理和分布式计算。 比较这些工具，每个都有其独特的优点和适用场景。Caffe以其速度见长，而Torch在灵活性和易用性上得分较高。Theano和Pylearn2...
深度学习框架(TensorFlow)基础教程——第2章：TensorFlow环境搭建
10-09
它支持分布式计算，可以在CPU和GPU上运行，并且提供了丰富的API用于数据预处理、模型构建、训练和评估。 **二、安装TensorFlow** 1. **系统需求**：确保你的计算机满足TensorFlow的基本运行条件，包括兼容的操作...
cudnn-8.0-linux-x64-v5.1
05-14
CUDA深度神经网络库（CuDNN）是NVIDIA公司推出的一个加速深度学习计算的库，主要针对GPU进行优化。在本压缩包“cudnn-8.0-linux-x64-v5.1”中，我们关注的是CuDNN的第五个主要版本（v5.1），适用于Linux操作系统，...
DeepLearning深度学习教程_第十五章_异构运算、GPU及框架选型.pdf
08-23
在第十五章中，我们探讨了异构计算、GPU以及深度学习框架的选择，这些都是提高计算效率的关键因素。 1. **异构计算**：它是指利用不同类型的处理器（如CPU、GPU、FPGA等）协同工作以解决计算问题。异构计算的优势...
清华大学精品AI人工智能课程深度学习全套课程PPT课件含习题（33页） 第6章 深度学习开源框架.pptx
05-29
【深度学习开源框架】是深度学习领域中至关重要的一部分，它为开发者提供了实现和训练深度学习模型的工具。本节主要介绍了三个重要的深度学习框架：Caffe、TensorFlow以及其它一些开源框架。 **Caffe...
深度学习库安装填坑
weixin_43888760的博客
12-13 1571
1.tensorflow2.3版本安装 用pip 和 conda 都失败，怎么办？——>下载miniconda3.5-3.7版本，然后采用miniconda中的conda 或 pip命令安装即可。亲测成功！ cpu版本的tensorflow安装： pip install tensorflow==2.3.0 -i https://pypi.douban.com/simple/ gpu版本的tensorflow安装： 对于NVIDIA显卡，要求其CUDA Compute Capability 须不低于3.
Tensorflow:out of memory 错误解决方法
weixin_42084032的博客
09-12 1365
>>> print(tf.test.is_gpu_available()) 2021-09-12 20:17:56.416492: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUM.
Keras_ocr遇到的问题
baiyang10010的博客
05-21 2332
1 显示内存不够 Looking for C:\Users\有对象真好\.keras-ocr\crnn_kurapan.h5 2020-05-21 14:00:01.016118: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 3019898880 exceeds 10% of free system memory. 2020-05-21 14:00:03.402349: W tensorflow/core/frame
分布式系统原理介绍_分布式系统的全面介绍
cumian9828的博客
07-25 4170
分布式系统原理介绍by Stanislav Kozlovski 斯坦尼斯拉夫·科兹洛夫斯基(Stanislav Kozlovski) 分布式系统的全面介绍 (A Thorough Introduction to Distributed Systems) 什么是分布式系统，为什么这么复杂？ (What is a Distributed System and why is it so compl...
解决Tensorflow使用CPU报错[ /job:localhost/replica:0/task:0/device:CPU:0 ]
热门推荐
梦醒时分
11-21 2万+
最近在跟着&lt;21个项目玩转深度学习&gt;中第三章的模型,但是遇见一个问题,我认为应该是CPU和GPU的使用问题. 解决方法是将slim文件夹中的train_image_classifier.py文件中的 tf.app.flags.DEFINE_boolean('clone_on_cpu', Flase, 'Use CPUs to...
离线安装keras linux gpu,使用GPU而不是CPU与Keras搭配使用Tensorflow后端的Linux
weixin_31201481的博客
05-17 294
我无法使Keras使用GPU版本的Tensorflow而不是CPU。每次我进口keras它只是说：使用GPU而不是CPU与Keras搭配使用Tensorflow后端的Linux>>> import kerasUsing TensorFlow backend......这意味着它的工作，但在CPU上，而不是GPU。 我安装CUDA和cuDNN和使用环境：conda create -...
error
qq_25368961的博客
06-12 557
error 2017-11-28 23:25:44.991463: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so...
第四届工业大数据创新大赛代码1（自写）
wojiaoawenlong的博客
09-28 527
import tensorflow as tf import matplotlib.pyplot as plt %matplotlib inline import pandas as pd import numpy as np # 这是个回归问题，参考波士顿房价预测 # 数据预处理 x_train = np.zeros((16600, 39)) y_train = np.zeros((16600, 3)) x_test = np.zeros((3953, 39)) df_x
分布式计算(第二版).pdf
最新发布
01-28
总的来说，《分布式计算(第二版).pdf》是一本全面系统地介绍了分布式计算概念和技术的书籍，适合对分布式计算感兴趣的读者阅读学习。通过阅读该书，读者可以对分布式计算有一个深入的了解，掌握分布式计算的核心知识...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值