代码
在使用如下代码,进行大模型推理时,出现了报错,无法从本地路径导入大模型;
import os
import json
import pickle
import pandas as pd
from vllm import LLM, SamplingParams
sampling_params = SamplingParams(temperature=0, n=1, max_tokens=128)
# 大模型名字或本地路径
model_path = '../../../models/summary_cls'
# model_path = "/mnt/workspace/LLaMA-Factory/models/summary_cls"
llm = LLM(
model=model_path,
trust_remote_code=True,
tokenizer=model_path,
tokenizer_mode='auto',
tensor_parallel_size=1
)
model_path
是本地经过lora权重合并后的大模型路径,无论是使用相对路径还是绝对路径都会报错;
使用相对路径报了如下错误:
return complexjson.loads(self.text, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/simplejson/__init__.py", line 514, in loads
return _default_decoder.decode(s)
File "/opt/conda/lib/python3.10/site-packages/simplejson/decoder.py", line 386, in decode
obj, end = self.raw_decode(s)
File "/opt/conda/lib/python3.10/site-packages/simplejson/decoder.py", line 416, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/workspace/LLaMA-Factory/demos/guihua_classify/scripts/test.py", line 13, in <module>
llm = LLM(
File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 109, in __init__
self.llm_engine = LLMEngine.from_engine_args(engine_args)
File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 351, in from_engine_args
engine_configs = engine_args.create_engine_configs()
File "/opt/conda/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 272, in create_engine_configs
model_config = ModelConfig(self.model, self.tokenizer,
File "/opt/conda/lib/python3.10/site-packages/vllm/config.py", line 96, in __init__
model_path = snapshot_download(model_id=model,
File "/opt/conda/lib/python3.10/site-packages/modelscope/hub/snapshot_download.py", line 98, in snapshot_download
revision_detail = _api.get_valid_revision_detail(
File "/opt/conda/lib/python3.10/site-packages/modelscope/hub/api.py", line 497, in get_valid_revision_detail
all_branches_detail, all_tags_detail = self.get_model_branches_and_tags_details(
File "/opt/conda/lib/python3.10/site-packages/modelscope/hub/api.py", line 578, in get_model_branches_and_tags_details
d = r.json()
File "/opt/conda/lib/python3.10/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
我当前的 vllm 版本如下:
Name: vllm
Version: 0.3.0
Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
Home-page: https://github.com/vllm-project/vllm
Author: vLLM Team
Author-email:
License: Apache 2.0
Location: /opt/conda/lib/python3.10/site-packages
Requires: aioprometheus, fastapi, ninja, numpy, psutil, pydantic, pynvml, ray, sentencepiece, torch, transformers, uvicorn, xformers
解决方法
pip install vllm==0.3.3
升级一下 vllm 的版本就好了