RuntimeError: Failed to infer device type

最新推荐文章于 2025-04-11 17:47:44 发布

Beason、

最新推荐文章于 2025-04-11 17:47:44 发布

阅读量1.4k

点赞数 1

文章标签： docker 语言模型

本文链接：https://blog.csdn.net/dhzfjnkl/article/details/145922109

版权

在使用vllm启动部署大模型时，出现了：

Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 368, in run_mp_engine
    raise e
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 114, in from_engine_args
    engine_config = engine_args.create_engine_config(usage_context)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1026, in create_engine_config
    device_config = DeviceConfig(device=self.device)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 1433, in __init__
    raise RuntimeError("Failed to infer device type")
RuntimeError: Failed to infer device type

解决方法：

sudo vim /etc/nvidia-container-runtime/config.toml

找到 no-cgroups 将其改为 false, 重新启动docker容器

sudo systemctl restart docker