【个人开发】baichuan部署踩坑（一）——Error：self.quant_state[0] = self.quant_state[0].cuda(device)

背景

踩完llama2的坑，来踩下baichuan的坑。按照官网下载模型使用Baichuan2-13B-Chat-4bits。

启动官方给的案例：

from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("/data/model/baichuan-inc/Baichuan2-13B-Chat-4bits", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("/data/model/baichuan-inc/Baichuan2-13B-Chat-4bits", device_map="auto", trust_remote_code=True)
inputs = tokenizer('登鹳雀楼->王之涣\n夜雨寄北->', return_tensors='pt')
inputs = inputs.to('cuda:1')
pred = model.generate(**inputs, max_new_tokens=64, repetition_penalty=1.1)
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

没想到，报错！！

找了一圈，还得是github的issueTypeError: ‘NoneType’ object is not subscriptable,使用baichuan-incBaichuan2-13B-Chat-4bits报错了 #52

对齐包版本：

pip install bitsandbytes==0.41.1 
pip install accelerate==0.25.0 

解决！