本地部署千文2多模态大模型Qwen2-VL-7B-Instruct-GPTQ-Int4

二毛宝宝吖

已于 2024-09-25 15:37:05 修改

阅读量1.8k

点赞数 4

文章标签：人工智能

于 2024-09-24 11:01:15 首次发布

本文链接：https://blog.csdn.net/yylhm1125/article/details/142460061

版权

本地部署千文2多模态大模型Qwen2-VL-7B-Instruct-GPTQ-Int4

第一部分准备环境
1、创建环境
conda create --name qwen2vl python=3.9
2、激活环境
conda activate qwen2vl
3、安装modelscope（下载方式为魔塔社区命令行下载）
pip install modelscope
第二部下载模型
1、下载命令
modelscope download --model qwen/Qwen2-VL-7B-Instruct-GPTQ-Int4 --cache_dir /home/ubuntu/ChatPro
第三部分编写python代码加载模型并推理
1、编写代码

from PIL import Image
import requests
import torch
from torchvision import io
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor


model_dir = "qwen/Qwen2-VL-7B-Instruct-GPTQ-Int4"
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_dir, torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_dir)
# Image
url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
image = Image.open(requests.get(url, stream=True).raw)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]


# Preprocess the inputs
text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(
    text=[text_prompt], images=[image], padding=True, return_tensors="pt"
)
inputs = inputs.to("cuda")

# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [
    output_ids[len(input_ids) :]
    for input_ids, output_ids in zip(inputs.input_ids, output_ids)
]
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
)
print(output_text)

2、安装代码运行环境
pip install opencv-python
pip install uvicorn
pip install fastapi
pip install git+https://github.com/huggingface/transformers
pip install qwen-vl-utils
pip install torchvision
pip install python-multipart
pip install ‘accelerate>=0.26.0’
pip install optimum（安装这个模块的时候会卸载transformers，所以如果是先安装的transformers按完这个需要再次安装transformers，或者后安装transformers）
pip install git+https://github.com/huggingface/transformers
pip install auto-gptq

第四部分可能出现的问题（有待补充）
1、报错信息：The checkpoint you are trying to load has model type starcoder2 but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
解决办法：卸载了之前的transformer重新安装transformer（记得重新安装的时候一定要看官方文档推荐你怎么安或者推荐安装哪个版本，这一点很重要很重要很重要！！！，transformer的版本装不对，会进行不下去）
2、报错信息：ModuleNotFoundError: No module named ‘optimum’
解决办法：pip install optimum
3、报错信息：importlib.metadata.PackageNotFoundError: No package metadata was found for auto-gptq
解决办法：pip install auto-gptq
4、报错信息：torch.cuda.OutOfMemoryError: CUDA out of memory
解决办法：看你的显卡内存够不够

附：Qwen2-VL-7B-Instruct-GPTQ-Int4官方文档地址