Qwen2模型量化时关于bitsandbytes安装的问题

Qwen2模型量化时关于bitsandbytes安装的问题


问题描述:

from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig,BitsAndBytesConfig
CUDA_DEVICE = "cuda:0"
model_name_or_path = '/qwen2-1.5b-instruct'
Tokenizer = AutoTokenizer.from_pretrained(model_name_or_path,use_fast=False)
bnb_config = BitsAndBytesConfig(
                    load_in_4bit=True,
                    bnb_4bit_use_double_quant=True,
                    bnb_4bit_quant_type="nf4",
                    bnb_4bit_compute_dtype=torch.bfloat16
                    )
Model = AutoModelForCausalLM.from_pretrained(model_name_or_path, 
												  device_map="auto",#CUDA_DEVICE, 
												  #load_in_8bit=True,
												  quantization_config=bnb_config,
												  #torch_dtype="auto"
												  )
messages = [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": prompt}
        ]

# "genarate chat" 
input_ids = Tokenizer.apply_chat_template(messages,tokenize=False,add_generation_prompt=True)
model_inputs = Tokenizer([input_ids], return_tensors="pt").to(CUDA_DEVICE)
generated_ids = Model.generate(model_inputs.input_ids,top_p=0.2,max_new_tokens=512)
generated_ids = [
	output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = self.Tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

在模型量化时遇到如下问题:

RuntimeError:
        CUDA Setup failed despite GPU being available. Please run the following command to get mor                                                                                   e information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need                                                                                    to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m                                                                                    bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

解决方法1:

可以卸载重新安装bitsandbytes,
step1:pip uninstall bitsandbytes
step2:pip install bitsandbytes

python3 -m bitsandbytes

解决方法2:

Step1:确认系统查找动态库libcudart.so的路径,将查找到的路径添加进去便解决这个问题
find / -name “libcudart.so*” #我的libcudart.so在/usr/local/cuda-11.4中
export LD_LIBRARY_PATH=“/usr/local/cuda-11.4/bin:$LD_LIBRARY_PATH”
python3 -m bitsandbytes

参考资料:

《Ubuntu18.04+CUDA11安装bitsandbytes出现的问题》
https://blog.csdn.net/steptoward/article/details/135507131
《大模型训练时关于bitsandbytes安装的问题》
https://gitcode.csdn.net/662f78a69ab37021bfb31bf0.html
《bitsandbytes 报错》
https://blog.csdn.net/weixin_43967256/article/details/134006312

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

wang151038606

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值