目录
简介,linux安装 使用方法
bitsandbytes是对CUDA自定义函数的轻量级封装,特别是针对8位优化器、矩阵乘法(LLM.int8())和量化函数。
GitHub:https://github.com/TimDettmers/bitsandbytes
文档:Module tree: — bitsandbytes v0.0.24 documentation
论文:https://arxiv.org/abs/2110.02861
1、bitsandbytes的特点
混合精度分解的8位矩阵乘法
LLM.int8()推断
8位优化器:Adam、AdamW、RMSProp、LARS、LAMB、Lion(节省75%的内存)
稳定的嵌入层:通过更好的初始化和归一化改进稳定性
8位量化:分位数、线性和动态量化
快速分位数估计:比其他算法快100倍
参考:
Py之bitsandbytes:bitsandbytes的简介、安装、使用方法之详细攻略-CSDN博客
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
'decapoda-research/llama-7b-hf,
device_map='auto',
load_in_8bit=True,
max_memory=f'{int(torch.cuda.mem_get_info()[0]/1024**3)-2}GB')
win11系统安装:
bitsandbytes 安装排错
报错:
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:
python -m bitsandbytes
Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
解决方法:
1.bitsandbytes-windows 没成功:
pip install bitsandbytes-windows
2.重装系统,重装cuda,没有必要
[踩坑记] CUDA环境下bitsandbytes安装报错/异常解决_cuda setup failed despite gpu being available. ple-CSDN博客
3.正确方法:
下载whl安装文件,pip install 就可以了。
https://github.com/jllllll/bitsandbytes-windows-webui/releases/tag/wheels