vllm 加速模型推理报错

1. 使用目的与报错信息

使用VLLM加速大模型推理时，启动模型，报错信息如下：

INFO 04-16 15:10:08 llm_engine.py:70] Initializing an LLM engine with config: model='/home/wzqi/GPT/models/vicuna-13b-v1.5', tokenizer='/home/wzqi/GPT/models/vicuna-13b-v1.5', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=3, quantization=None, enforce_eager=False, seed=0)
2024-04-16 15:10:08 vm2255 dbgpt.model.cluster.worker.manager[15008] ERROR Error starting worker manager: model vicuna-13b-v1.5@vllm(172.1.1.15:7860) start failed, Traceback (most recent call last):
  File "/home/wzqi/GPT/dbgpt/model/cluster/worker/manager.py", line 506, in _start_worker
    await self.run_blocking_func(
  File "/home/wzqi/GPT/dbgpt/model/cluster/worker/manager.py", line 104, in run_blocking_func
    return await loop.run_in_executor(self.executor, func, *args)
  File "/home/wzqi/local/anaconda3/envs/py310_dbgpt/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/wzqi/GPT/dbgpt/model/cluster/worker/default_worker.py", line 116, in start
    self.model, self.tokenizer = self.ml.loader_with_params(
  File "/home/wzqi/GPT/dbgpt/model/adapter/loader.py", line 132, in loader_with_params
    return llm_adapter.load_from_params(model_params)
  File "/home/wzqi/GPT/dbgpt/model/adapter/vllm_adapter.py", line 78, in load_from_params
    engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/wzqi/local/anaconda3/envs/py310_dbgpt/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 500, in from_engine_args
    engine = cls(parallel_config.worker_use_ray,
  File "/home/wzqi/local/anaconda3/envs/py310_dbgpt/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 273, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/home/wzqi/local/anaconda3/envs/py310_dbgpt/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 318, in _init_engine
    return engine_class(*args, **kwargs)
  File "/home/wzqi/local/anaconda3/envs/py310_dbgpt/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 93, in __init__
    self._verify_args()
  File "/home/wzqi/local/anaconda3/envs/py310_dbgpt/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 256, in _verify_args
    self.model_config.verify_with_parallel_config(self.parallel_config)
  File "/home/wzqi/local/anaconda3/envs/py310_dbgpt/lib/python3.10/site-packages/vllm/config.py", line 192, in verify_with_parallel_config
    raise ValueError(
ValueError: Total number of attention heads (40) must be divisible by tensor parallel size (3).

INFO:     Shutting down
INFO:     Waiting for application shutdown.

2 分析报错信息

tensor_parallel_size=3

Total number of attention heads (40) must be divisible by tensor parallel size (3)

结合环境是三个显卡，但是一般会使用 2/4/8/16 个数用于模型，所以在配置文件里面修改CUDA的数量，指定合理的显卡进行模型加速，

CUDA_VISIBLE_DEVICES="0,1"

我这里总共三个显卡，我使用序号为 0 和 1 的显卡，大模型启动成功。