linux节点状态 slurm,linux – 为什么slurm中的作业在TensorFlow脚本中无限期冻结？

最新推荐文章于 2024-05-04 22:11:59 发布

18992697150

最新推荐文章于 2024-05-04 22:11:59 发布

阅读量148

点赞数

文章标签： linux节点状态 slurm

我使用slurm(

http://slurm.schedmd.com/)工作负载管理器时遇到此错误.当我运行一些tensorflow python脚本时,有时会导致错误(附加).它似乎无法找到安装的cuda库,但我正在运行不需要GPU的脚本.因此,我发现为什么cuda会成为一个问题非常令人困惑.如果我不需要,为什么cuda安装会出现问题？

我从slurm-job_id文件中获得的唯一有用信息如下：

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally

I tensorflow/stream_executor/dso_loader.cc:102] Couldn't open CUDA library libcudnn.so. LD_LIBRARY_PATH: /cm/shared/openmind/cuda/7.5/lib64:/cm/shared/openmind/cuda/7.5/lib

I tensorflow/stream_executor/cuda/cuda_dnn.cc:2092] Unable to load cuDNN DSO

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so locally

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally

E tensorflow/stream_executor/cuda/cuda_driver.cc:491] Failed call to cuInit: CUDA_ERROR_NO_DEVICE

I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:153] retrieving CUDA diagnostic information for host: node047

I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:160] hostname: node047

I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:185] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program

I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:347] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module 352.63 Sat Nov 7 21:25:42 PST 2015

GCC version: gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)

"""

I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] kernel reported version is: 352.63.0

I tensorflow/core/common_runtime/gpu/gpu_init.cc:81] No GPU devices available on machine.

我一直以为张量流不需要GPU.所以我假设最后一个错误说没有GPU不会导致错误(如果我错了,请纠正我).

我不明白为什么我需要CUDA库.我正在尝试用GPU运行我的工作,如果我的工作是cpu工作,为什么我需要cuda库？