LMDeploy 量化部署 LLM 实践（作业）

最新推荐文章于 2024-10-03 18:47:41 发布

DizzyOK

最新推荐文章于 2024-10-03 18:47:41 发布

阅读量259

点赞数 2

文章标签：笔记开源 pytorch

本文链接：https://blog.csdn.net/DizzyOK/article/details/139691702

版权

基础作业（结营必做）

完成以下任务，并将实现过程记录截图：

配置 LMDeploy 运行环境
以命令行方式与 InternLM2-Chat-1.8B 模型对话

创建conda环境

studio-conda -t lmdeploy -o pytorch-2.1.2
conda activate lmdeploy

安装LMDeploy

pip install lmdeploy[all]==0.3.0

下载模型

ln -s /root/share/new_models/Shanghai_AI_Laboratory/internlm2-chat-1_8b /root/
# cp -r /root/share/new_models/Shanghai_AI_Laboratory/internlm2-chat-1_8b /root/

使用Transformer运行模型

使用vscode打开terminal界面
新建文件

touch /root/pipeline_transformer.py

写入代码

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("/root/internlm2-chat-1_8b", trust_remote_code=True)

# Set `torch_dtype=torch.float16` to load model in float16, otherwise it will be loaded as float32 and cause OOM Error.
model = AutoModelForCausalLM.from_pretrained("/root/internlm2-chat-1_8b", torch_dtype=torch.float16, trust_remote_code=True).cuda()
model = model.eval()

inp = "hello"
print("[INPUT]", inp)
response, history = model.chat(tokenizer, inp, history=[])
print("[OUTPUT]", response)

inp = "please provide three suggestions about time management"
print("[INPUT]", inp)
response, history = model.chat(tokenizer, inp, history=history)
print("[OUTPUT]", response)