LMDeploy 部署 基础

教程：Tutorial/docs/L2/LMDeploy/readme.md at camp3 · InternLM/Tutorial · GitHub

里面的计算讲解清晰

搭建环境

conda create -n lmdeploy  python=3.10 -y
conda activate lmdeploy
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia -y
pip install timm==1.0.8 openai==1.40.3 lmdeploy[all]==0.5.3

创建软链接

mkdir /root/models
ln -s /root/share/new_models//Shanghai_AI_Laboratory/internlm2_5-7b-chat /root/models
ln -s /root/share/new_models/OpenGVLab/InternVL2-26B /root/models

LMdeploy 命令行启动internlm2.5 7B

lmdeploy chat /root/models/internlm2_5-7b-chat

提问，要双回车

LMdeploy API server启动internlm2.5 7B

lmdeploy serve api_server \
    /root/models/internlm2_5-7b-chat \
    --model-format hf \
    --quant-policy 0 \
    --server-name 0.0.0.0 \
    --server-port 23333 \
    --tp 1

本地端口映射23333

本地打开localhost:23333

另开终端窗口，API到localhost:23333

conda activate lmdeploy
lmdeploy serve api_client http://localhost:23333

gradio链接API server

lmdeploy serve gradio http://localhost:23333 \
    --server-name 0.0.0.0 \
    --server-port 6006

本地映射6006端口

打开localhost:6006