本文探讨了8位浮点数在深度学习中的应用，特别是FP8格式的E4M3和E5M2，适合前向和反向计算。介绍了FP4、QLoRA和双量化技术，以及在GPU上的4位量化训练。强调了PEFT在4/8位模型训练中的作用，并展望了在RLHF中使用量化模型的潜力。
https://huggingface.co/blog/4bit-transformers-bitsandbytes
1. 8 位float
The FP8 (floating point 8) format has been first introduced in the paper “FP8 for Deep Learning” with two different FP8 encodings: E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa).
The potential floating points that can be represented in the E4M3 format are in the range -448 to 448, whereas in the E5M2 format, as the number of bits of the e
订阅专栏 解锁全文
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
