Qwen2模型量化时关于bitsandbytes安装的问题

Qwen2模型量化时关于bitsandbytes安装的问题

---

问题描述：

from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig,BitsAndBytesConfig
CUDA_DEVICE = "cuda:0"
model_name_or_path = '/qwen2-1.5b-instruct'
Tokenizer = AutoTokenizer.from_pretrained(model_name_or_path,use_fast=False)
bnb_config = BitsAndBytesConfig(
                    load_in_4bit=True,
                    bnb_4bit_use_double_quant=True,
                    bnb_4bit_quant_type="nf4",
                    bnb_4bit_compute_dtype=torch.bfloat16
                    )
Model = AutoModelForCausalLM.from_pretrained(model_name_or_path, 
												  device_map="auto",#CUDA_DEVICE, 
												  #load_in_8bit=True,
												  quantization_config=bnb_config,
												  #torch_dtype="auto"
												  )
messages = [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": prompt}
        ]

# "genarate chat" 
input_ids = Tokenizer.apply_chat_template(messages,tokenize=False,add_generation_prompt=True)
model_inputs = Tokenizer([input_ids], return_tensors="pt").to(CUDA_DEVICE)
generated_ids = Model.generate(model_inputs.input_ids,top_p=0.2,max_new_tokens=512)
generated_ids = [
	output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = self.Tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

在模型量化时遇到如下问题：

RuntimeError:
        CUDA Setup failed despite GPU being available. Please run the following command to get mor                                                                                   e information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need                                                                                    to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m                                                                                    bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

解决方法1：

可以卸载重新安装bitsandbytes，
step1：pip uninstall bitsandbytes
step2：pip install bitsandbytes

python3 -m bitsandbytes

解决方法2：

Step1:确认系统查找动态库libcudart.so的路径,将查找到的路径添加进去便解决这个问题
find / -name “libcudart.so*” #我的libcudart.so在/usr/local/cuda-11.4中
export LD_LIBRARY_PATH=“/usr/local/cuda-11.4/bin:$LD_LIBRARY_PATH”
python3 -m bitsandbytes

参考资料：

《Ubuntu18.04+CUDA11安装bitsandbytes出现的问题》
https://blog.csdn.net/steptoward/article/details/135507131
《大模型训练时关于bitsandbytes安装的问题》
https://gitcode.csdn.net/662f78a69ab37021bfb31bf0.html
《bitsandbytes 报错》
https://blog.csdn.net/weixin_43967256/article/details/134006312