[论文笔记] pai-megatron-patch 环境安装

Installation — Transformer Engine 1.8.0 documentation

除了官方repo中的requirement.txt

icetk
ftfy
hjson
ninja
tokenizers
transformers>=4.38.2
accelerate>=0.23.0
datasets==2.16.1
wcwidth==0.2.12
pyarrow==12.0.1
jieba
sentencepiece
rjieba
sqlitedict
sacrebleu
datasets
tensorboard
lm_dataformat
tiktoken

还需要安装 flash-attn2.5、transformer-engine

pip install flash-attn -i https://pypi.tuna.tsinghua.edu.cn/simple --no-build-isolation

pip install git+https://github.com/NVIDIA/TransformerEngine.git@stable