背景:
在服务端,使用VLLM启动LLM后
python -O -u -m vllm.entrypoints.openai.api_server --model /Qwen2-72B-Instruct-GPTQ-Int4 --host 172.18.25.21 --port 9000 --served-model-name Qwen2-72B-Instruct-GPTQ-Int4 --tokenizer Qwen2-72B-Instruct-GPTQ-Int4 --trust-remote-code --tensor-parallel-size 2 --api-key token-abc123
在客户端使用curl测试
curl http://172.18.25.21:9000/v1/models
{"error":"Unauthorized"}
解决方法:
加上选项-H指定授权
curl http://172.18.25.21:9000/v1/models -H "Authorization: Bearer token-abc123"
{"object":"list","data":[{"id":"Qwen2-72B-Instruct-GPTQ-Int4","object":"model","created":1722667279,"owned_by":"vllm","root":"Qwen2-72B-Instruct-GPTQ-Int4","parent":null,"max_model_len":32768,"permission":[{"id":"modelperm-6ebab1da7a734301b5b26650aae38688","object":"model_permission","created":1722667279,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}