分布式计算学习第二坑 cudnn

最新推荐文章于 2021-09-12 20:33:48 发布
最新推荐文章于 2021-09-12 20:33:48 发布
阅读量653 收藏
点赞数
分类专栏： tensorflow on spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_45939774/article/details/106350214
版权
tensorflow 同时被 3 个专栏收录
1 篇文章 0 订阅
订阅专栏
on
1 篇文章 0 订阅
订阅专栏
spark
1 篇文章 0 订阅
订阅专栏 
andrew@1manjaro:~/mount/arch/TensorFlowOnSpark/examples/resnet#export TF_CONFIG='{"cluster": { "chief": ["localhost:2222"], "worker": ["localhost:2223"]}, "task": {"type": "chief", "index": 0}}'
python resnet_cifar_main.py --data_dir=${CIFAR_DATA} --num_gpus=1 --ds=multi_worker_mirrored --train_epochs=100
2020-05-26 10:43:02.240568: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-05-26 10:43:03.329536: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-26 10:43:03.332103: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.332337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:1c:00.0 name: GeForce RTX 2070 computeCapability: 7.5
coreClock: 1.725GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-26 10:43:03.332359: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-05-26 10:43:03.333363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 10:43:03.334473: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 10:43:03.334654: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 10:43:03.335746: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 10:43:03.336382: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 10:43:03.338684: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 10:43:03.338811: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.339096: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.339297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-26 10:43:03.339909: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
2020-05-26 10:43:03.360884: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3701195000 Hz
2020-05-26 10:43:03.361383: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5586666e7300 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-26 10:43:03.361399: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-26 10:43:03.682688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.682977: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558665a43420 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-26 10:43:03.683005: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 2070, Compute Capability 7.5
2020-05-26 10:43:03.683211: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.683432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:1c:00.0 name: GeForce RTX 2070 computeCapability: 7.5
coreClock: 1.725GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-26 10:43:03.683458: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-05-26 10:43:03.683481: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 10:43:03.683491: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 10:43:03.683500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 10:43:03.683510: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 10:43:03.683519: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 10:43:03.683528: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 10:43:03.683571: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.683789: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.683974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-26 10:43:03.683995: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-05-26 10:43:03.959819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-26 10:43:03.959852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2020-05-26 10:43:03.959860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2020-05-26 10:43:03.960066: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.960320: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.960529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 35 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070, pci bus id: 0000:1c:00.0, compute capability: 7.5)
2020-05-26 10:43:03.960938: I tensorflow/core/common_runtime/process_util.cc:147] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2020-05-26 10:43:03.961343: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.961546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:1c:00.0 name: GeForce RTX 2070 computeCapability: 7.5
coreClock: 1.725GHz coreCount: 36 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-26 10:43:03.961570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
2020-05-26 10:43:03.961593: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 10:43:03.961604: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-26 10:43:03.961614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-26 10:43:03.961624: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-26 10:43:03.961634: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-26 10:43:03.961644: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 10:43:03.961686: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.961902: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.962084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-05-26 10:43:03.962100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-26 10:43:03.962106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2020-05-26 10:43:03.962111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2020-05-26 10:43:03.962169: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.962391: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-26 10:43:03.962579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:chief/replica:0/task:0/device:GPU:0 with 35 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070, pci bus id: 0000:1c:00.0, compute capability: 7.5)
2020-05-26 10:43:03.964986: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job chief -> {0 -> localhost:2222}
2020-05-26 10:43:03.965006: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> localhost:2223}
2020-05-26 10:43:03.965380: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:390] Started server with target: grpc://localhost:2222
INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:chief/replica:0/task:0/device:CPU:0', '/job:chief/replica:0/task:0/device:XLA_CPU:0', '/job:chief/replica:0/task:0/device:XLA_GPU:0', '/job:chief/replica:0/task:0/device:GPU:0']
I0526 10:43:03.965880 140302475241280 collective_all_reduce_strategy.py:303] Enabled multi-worker collective ops with available devices: ['/job:chief/replica:0/task:0/device:CPU:0', '/job:chief/replica:0/task:0/device:XLA_CPU:0', '/job:chief/replica:0/task:0/device:XLA_GPU:0', '/job:chief/replica:0/task:0/device:GPU:0']
INFO:tensorflow:Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
I0526 10:43:03.966263 140302475241280 mirrored_strategy.py:500] Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
I0526 10:43:03.966428 140302475241280 collective_all_reduce_strategy.py:344] MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.433851 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.437643 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.444913 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.447288 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.500257 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.502656 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.508196 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.510025 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.543776 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:04.545943 140302475241280 cross_device_ops.py:1059] Collective batch_all_reduce: 1 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Running Distribute Coordinator with mode = 'independent_worker', cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, environment = None, rpc_layer = 'grpc'
I0526 10:43:06.832544 140302475241280 distribute_coordinator.py:773] Running Distribute Coordinator with mode = 'independent_worker', cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, environment = None, rpc_layer = 'grpc'
WARNING:tensorflow:`eval_fn` is not passed in. The `worker_fn` will be used if an "evaluator" task exists in the cluster.
W0526 10:43:06.832681 140302475241280 distribute_coordinator.py:825] `eval_fn` is not passed in. The `worker_fn` will be used if an "evaluator" task exists in the cluster.
WARNING:tensorflow:`eval_strategy` is not passed in. No distribution strategy will be used for evaluation.
W0526 10:43:06.832730 140302475241280 distribute_coordinator.py:829] `eval_strategy` is not passed in. No distribution strategy will be used for evaluation.
INFO:tensorflow:Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
I0526 10:43:06.833080 140302475241280 mirrored_strategy.py:500] Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
I0526 10:43:06.833179 140302475241280 collective_all_reduce_strategy.py:344] MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
INFO:tensorflow:Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
I0526 10:43:06.833521 140302475241280 mirrored_strategy.py:500] Using MirroredStrategy with devices ('/job:chief/task:0/device:GPU:0',)
INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
I0526 10:43:06.833611 140302475241280 collective_all_reduce_strategy.py:344] MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:2222'], 'worker': ['localhost:2223']}, task_type = 'chief', task_id = 0, num_workers = 2, local_devices = ('/job:chief/task:0/device:GPU:0',), communication = CollectiveCommunication.AUTO
Epoch 1/100
INFO:tensorflow:Collective batch_all_reduce: 176 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:08.031161 140302475241280 cross_device_ops.py:1054] Collective batch_all_reduce: 176 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
INFO:tensorflow:Collective batch_all_reduce: 176 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
I0526 10:43:11.108297 140302475241280 cross_device_ops.py:1054] Collective batch_all_reduce: 176 all-reduces, num_workers = 2, communication_hint = AUTO, num_packs = 1
2020-05-26 10:43:15.515986: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-26 10:43:15.864391: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-26 10:43:16.010903: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-05-26 10:43:16.021040: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
  File "resnet_cifar_main.py", line 288, in <module>
    app.run(main)
  File "/usr/lib/python3.8/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/lib/python3.8/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "resnet_cifar_main.py", line 282, in main
    return run(flags.FLAGS)
  File "resnet_cifar_main.py", line 251, in run
    history = model.fit(train_input_dataset,
  File "/usr/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 72, in _method_wrapper
    return dc.run_distribute_coordinator(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_coordinator.py", line 852, in run_distribute_coordinator
    return _run_single_worker(worker_fn, strategy, cluster_spec, task_type,
  File "/usr/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_coordinator.py", line 360, in _run_single_worker
    return worker_fn(strategy)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 73, in <lambda>
    lambda _: method(self, *args, **kwargs),
  File "/usr/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 848, in fit
    tmp_logs = train_function(iterator)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 644, in _call
    return self._stateless_fn(*args, **kwds)
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2420, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call
    return self._call_flat(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 593, in call
    outputs = execute.execute(
  File "/usr/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node resnet56/conv1/Conv2D (defined at /threading.py:932) ]]
	 [[GroupCrossDeviceControlEdges_0/Identity_2/_35]]
  (1) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node resnet56/conv1/Conv2D (defined at /threading.py:932) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_19079]

Function call stack:
train_function -> train_function

2020-05-26 10:43:16.566656: W tensorflow/core/common_runtime/eager/context.cc:447] Unable to destroy server_ object, so releasing instead. Servers don't support clean shutdown.

是显存不够用导致的

import tensorflow as tf
import keras

config = tf.compat.v1.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.4
tf.compat.v1.keras.backend.set_session(tf.compat.v1.Session(config=config))

在这里插入图片描述

weixin_45939774
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
tensorflow系列(3)分布式tensorflow
EternalMind
08-14 1193
多机如何分布式运行tensorflow模型？
Colab tqdm_progress_bar.py error
linyuanthocr的博客
04-11 443
Colab tqdm_process_bar.py 报错解决代码：错误提示解决方法 代码： import numpy as np import tensorflow as tf import tensorflow_addons as tfa x = np.random.random((5,1,5)) y = np.random.random((5,1,5)) inputs = tf.keras...
卷积神经网络的分布式训练（一）ResNet
qq_36484003的博客
09-18 1363
卷积神经网络的分布式训练（一）ResNet 该专题是我本科毕设的各个知识点介绍和分析，以及一些实验结果展示。本科毕业了，研究生开学，又进入了一个新的学习阶段，那么趁着有时间就总结一下这个课题，分享给大家~ 如果你觉得对你有帮助，给我点赞吧！小女子能力有限，也欢迎大家批评指正。 目录卷积神经网络的分布式训练（一）ResNet一、ResNet 介绍基本原理层次种类二、网络模型结构设计1、预处理模块2、基本单元3、池化层4、全连接层参考文献 一、ResNet 介绍 ResNet（Residual Neural N
python学习之多任务②
qq_39377119的博客
01-21 111
线程共享全局变量 import threading # 定义全局变量 globals_num globals_num = 0 def test1(num): global globals_num for i in range(num): globals_num += 1 print("test1当前变量值为:%d" % globals_num) d...
python学习（2）— 就业岗位准备
Hubert的博客
08-26 221
一、岗位表 职位 技能 Python后端开发工程师 Python基础Django / Flask / Tornado / SanicRESTful / 接口文档撰写MySQL / Redis / MongoDB / ElasticSearchLinux / Git / Scrum / PyCharm python量化交易开发工程师 Python基础数据结构 / 算法 /...
热门深度学习工具排行：Pylearn2 居首，Caffe第三1
08-08
DL4J的目标是使深度学习技术更易于在企业环境中部署，支持大规模数据处理和分布式计算。 比较这些工具，每个都有其独特的优点和适用场景。Caffe以其速度见长，而Torch在灵活性和易用性上得分较高。Theano和Pylearn2...
深度学习框架(TensorFlow)基础教程——第2章：TensorFlow环境搭建
10-09
它支持分布式计算，可以在CPU和GPU上运行，并且提供了丰富的API用于数据预处理、模型构建、训练和评估。 **二、安装TensorFlow** 1. **系统需求**：确保你的计算机满足TensorFlow的基本运行条件，包括兼容的操作...
cudnn-8.0-linux-x64-v5.1
05-14
CUDA深度神经网络库（CuDNN）是NVIDIA公司推出的一个加速深度学习计算的库，主要针对GPU进行优化。在本压缩包“cudnn-8.0-linux-x64-v5.1”中，我们关注的是CuDNN的第五个主要版本（v5.1），适用于Linux操作系统，...
DeepLearning深度学习教程_第十五章_异构运算、GPU及框架选型.pdf
08-23
在第十五章中，我们探讨了异构计算、GPU以及深度学习框架的选择，这些都是提高计算效率的关键因素。 1. **异构计算**：它是指利用不同类型的处理器（如CPU、GPU、FPGA等）协同工作以解决计算问题。异构计算的优势...
清华大学精品AI人工智能课程深度学习全套课程PPT课件含习题（33页） 第6章 深度学习开源框架.pptx
05-29
【深度学习开源框架】是深度学习领域中至关重要的一部分，它为开发者提供了实现和训练深度学习模型的工具。本节主要介绍了三个重要的深度学习框架：Caffe、TensorFlow以及其它一些开源框架。 **Caffe...
深度学习库安装填坑
weixin_43888760的博客
12-13 1571
1.tensorflow2.3版本安装 用pip 和 conda 都失败，怎么办？——>下载miniconda3.5-3.7版本，然后采用miniconda中的conda 或 pip命令安装即可。亲测成功！ cpu版本的tensorflow安装： pip install tensorflow==2.3.0 -i https://pypi.douban.com/simple/ gpu版本的tensorflow安装： 对于NVIDIA显卡，要求其CUDA Compute Capability 须不低于3.
Tensorflow:out of memory 错误解决方法
weixin_42084032的博客
09-12 1367
>>> print(tf.test.is_gpu_available()) 2021-09-12 20:17:56.416492: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUM.
Keras_ocr遇到的问题
baiyang10010的博客
05-21 2332
1 显示内存不够 Looking for C:\Users\有对象真好\.keras-ocr\crnn_kurapan.h5 2020-05-21 14:00:01.016118: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 3019898880 exceeds 10% of free system memory. 2020-05-21 14:00:03.402349: W tensorflow/core/frame
分布式系统原理介绍_分布式系统的全面介绍
cumian9828的博客
07-25 4170
分布式系统原理介绍by Stanislav Kozlovski 斯坦尼斯拉夫·科兹洛夫斯基(Stanislav Kozlovski) 分布式系统的全面介绍 (A Thorough Introduction to Distributed Systems) 什么是分布式系统，为什么这么复杂？ (What is a Distributed System and why is it so compl...
解决Tensorflow使用CPU报错[ /job:localhost/replica:0/task:0/device:CPU:0 ]
热门推荐
梦醒时分
11-21 2万+
最近在跟着&lt;21个项目玩转深度学习&gt;中第三章的模型,但是遇见一个问题,我认为应该是CPU和GPU的使用问题. 解决方法是将slim文件夹中的train_image_classifier.py文件中的 tf.app.flags.DEFINE_boolean('clone_on_cpu', Flase, 'Use CPUs to...
离线安装keras linux gpu,使用GPU而不是CPU与Keras搭配使用Tensorflow后端的Linux
weixin_31201481的博客
05-17 294
我无法使Keras使用GPU版本的Tensorflow而不是CPU。每次我进口keras它只是说：使用GPU而不是CPU与Keras搭配使用Tensorflow后端的Linux>>> import kerasUsing TensorFlow backend......这意味着它的工作，但在CPU上，而不是GPU。 我安装CUDA和cuDNN和使用环境：conda create -...
error
qq_25368961的博客
06-12 557
error 2017-11-28 23:25:44.991463: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so...
第四届工业大数据创新大赛代码1（自写）
wojiaoawenlong的博客
09-28 527
import tensorflow as tf import matplotlib.pyplot as plt %matplotlib inline import pandas as pd import numpy as np # 这是个回归问题，参考波士顿房价预测 # 数据预处理 x_train = np.zeros((16600, 39)) y_train = np.zeros((16600, 3)) x_test = np.zeros((3953, 39)) df_x
分布式计算(第二版).pdf
最新发布
01-28
《分布式计算(第二版).pdf》是一本关于分布式计算概念和技术的书籍，内容涵盖了分布式计算的基本原理、系统结构、算法设计、性能优化等方面的内容。该书在第二版中对一些新的技术发展和应用进行了更新和扩充，使得读者可以更全面地了解和掌握分布式计算的知识。 该书首先介绍了分布式计算的基本概念和发展历程，让读者对分布式计算有一个整体的认识。然后详细介绍了分布式系统的基本组成和架构设计，包括并行计算、集群系统、网格计算等内容，读者可以通过学习这些知识了解到不同类型的分布式系统的组成和工作原理。 在算法设计和性能优化方面，该书着重介绍了分布式计算中的一些经典算法和优化策略，帮助读者深入理解分布式计算问题的本质，并学习如何应用合适的算法来解决实际的分布式计算任务。此外，该书还介绍了一些分布式计算的应用领域，如大数据处理、云计算、物联网等，帮助读者了解分布式计算在不同领域的应用场景和发展趋势。 总的来说，《分布式计算(第二版).pdf》是一本全面系统地介绍了分布式计算概念和技术的书籍，适合对分布式计算感兴趣的读者阅读学习。通过阅读该书，读者可以对分布式计算有一个深入的了解，掌握分布式计算的核心知识和技术，为后续的研究和实践工作打下良好的基础。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值