【踩坑日记2】OSError: libcudart.so.12: cannot open shared object file: No such file or directory

问题描述

Downloading deepspeed-0.13.4.tar.gz (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 519.3 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      Traceback (most recent call last):
        File "/mnt/lab/XXX/anaconda3/envs/promptist/lib/python3.9/site-packages/torch/__init__.py", line 176, in _load_global_deps
          ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
        File "/mnt/lab/XXX/anaconda3/envs/promptist/lib/python3.9/ctypes/__init__.py", line 374, in __init__
          self._handle = _dlopen(self._name, mode)
      OSError: libcudart.so.12: cannot open shared object file: No such file or directory
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-j60edzt8/deepspeed_5c3d7e5ebeb94217ab178e179251405b/setup.py", line 31, in <module>
          import torch
        File "/mnt/lab/XXX/anaconda3/envs/promptist/lib/python3.9/site-packages/torch/__init__.py", line 236, in <module>
          _load_global_deps()
        File "/mnt/lab/XXX/anaconda3/envs/promptist/lib/python3.9/site-packages/torch/__init__.py", line 197, in _load_global_deps
          _preload_cuda_deps(lib_folder, lib_name)
        File "/mnt/lab/XXX/anaconda3/envs/promptist/lib/python3.9/site-packages/torch/__init__.py", line 163, in _preload_cuda_deps
          ctypes.CDLL(lib_path)
        File "/mnt/lab/XXX/anaconda3/envs/promptist/lib/python3.9/ctypes/__init__.py", line 374, in __init__
          self._handle = _dlopen(self._name, mode)
      OSError: libnvJitLink.so.12: cannot open shared object file: No such file or directory
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

问题分析

是包之间版本的相互依赖，其他版本包阻止DeepSpeed的安装，尝试先安装DeepSpeed，再安装其他包。

解决办法

1. 重新建立新的环境
2. 安装pytorch
   安装DeepSpeed，要求pytorch已经安装。
3. 安装包DeepSpeed
4. 安装其他需要的包