最近要给一台共用的ubuntu18.04配置tf2.0 gpu环境,原机器上是tf1.x版本+CUDA10.1,gpu可以正常使用和运行,但是虚拟环境安装tf2.0-gpu后,执行tf.test.is_gpu_available()出现了
tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libcudart.so.10.0’; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
…
tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices…
这样的错误,可以看到,大概的意思就是找不到 libcudart.so.10.0(等)文件,在官方的CUDA与Tensorflow支持列表中:
https://tensorflow.google.cn/install/source
是不显示tf2.0-gpu对应的CUDA版本的,网上搜索后,说tf2.0必须要用CUDA10.0的版本,不能用CUDA10.1的版本,但是由于服务器是共用的,如果更改CUDA版本可能会带来一些其他问题,所以无奈,只能在网上寻找解决方法。
相关问题:
https://github.com/tensorflow/tensorflow/issues/32731
https://github.com/tensorflow/tensorflow/issues/26289
https://github.com/tensorflow/tensorflow/issues/26182
刚开始,我在 https://github.com/tensorflow/tensorflow/issues/26289 中,找到了@selvamshan对问题的评论,
Hi working for me after i did some sym link,
in /usr/local/cuda-10.1/lib64
libcublas.so -> /usr/local/cuda-10.1/lib64/libcublas.so.10.1
libcublas.so.10.0 -> /usr/local/cuda-10.1/lib64/libcublas.so.10.1
libcusolver.so.10.0 -> /usr/lib/x86_64-linux-gnu/libcusolver.so.10
libcurand.so.10.0 -> /usr/lib/x86_64-linux-gnu/libcurand.so.10
libcufft.so.10.0 -> /usr/lib/x86_64-linux-gnu/libcufft.so.10
libcudart.so.10.0 -> /usr/lib/x86_64-linux-gnu/libcudart.so.10.1
libcusparse.so.10.0 -> /usr/lib/x86_64-linux-gnu/libcusparse.so.10
大概意思就是,似乎库文件夹中缺少确切的库名称,需要手动建立软链接来帮助tf找到他们,但是事实上,我发现我的/usr/local/cuda-10.1/lib64/下已经存在了这些软链接,但是为什么tf2.0没有找到呢,后来在
https://github.com/tensorflow/tensorflow/issues/26182
@kunrenzhilu 的评论中:
Exactly.
find your CUDA install path, in my case it is /usr/local/cuda
export LD_LIBRARY_PATH=/usr/local/cuda/lib64
Then TF follow LD_LIBRARY_PATH to locate libcublas.so.10.0
想到了可能是环境变量的问题,我们需要添加名为LD_LIBRARY_PATH的环境变量,但是事实上,我的目录并不是/usr/local/cuda/lib64,虽然我也有这个目录,但是看起来,CUDA10.1已经将这个目录下的一些‘文件’转移至了/usr/local/cuda-10.1/lib64/,
于是,我执行了:
export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64/
这样,
import tensorflow as tf
print(tf.test.is_gpu_available())
print(tf.test.is_built_with_cuda())
输出了
2019-12-05 11:30:49.597177: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-12-05 11:30:49.634375: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2100000000 Hz
2019-12-05 11:30:49.636410: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e45b5dc880 executing computations on platform Host. Devices:
2019-12-05 11:30:49.636471: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
2019-12-05 11:30:49.642753: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-12-05 11:30:49.856177: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e45b6860c0 executing computations on platform CUDA. Devices:
2019-12-05 11:30:49.856275: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-12-05 11:30:49.859084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:b3:00.0
2019-12-05 11:30:49.859573: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 11:30:49.864154: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-05 11:30:49.867689: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-05 11:30:49.868285: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-05 11:30:49.872370: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-05 11:30:49.874863: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-05 11:30:49.881286: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-05 11:30:49.884138: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-05 11:30:49.884191: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 11:30:49.885885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-05 11:30:49.885899: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2019-12-05 11:30:49.885907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2019-12-05 11:30:49.887912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 10281 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5)
True
True
问题解决。