[论文笔记] pai-megatron-patch Qwen2-57B(A14B)-CT 后预训练 踩坑记录

Qwen2-57B(A14B)-CT 脚本

长文本开不到8K，目前能开1K，具体能开多少需要测试。

context parallel需要测试。

#!/bin/bash
## 测试对齐样例：at least 4nodes
# pip install  transformers==4.40  -i https://mirrors.aliyun.com/pypi/simple/
# cd /mnt/cpfs/kexin/dlc_code/qwen2/moe/Pai-Megatron-Patch/examples/qwen2
# sh run_pretrain_qwen.sh dlc A14B 1 8 1e-5 1e-6 128 128 bf16 4 1 4 sel true true true true 100000 /mnt/cpfs/kexin/dlc_code/qwen2/Pai-Megatron-Patch/kx_data/wudao_qwenbpe_text_document /mnt/nas/kexin/model_weights/Qwen2-57B-A14B-hf-to-mcore-te-tp4-pp1-ep4 100000000 10000 /mnt/nas/kexin/output_mcore_qwen
############################### Miao Miao ####################################
# set -e
# [Converting]
# cd /mnt/cpfs/kexin/dlc_code/qwen2/moe/Pai-Megatron-Patch
# sh hf2mcore_qwen2_convertor.sh A14B /mnt/nas/shanzhi/eval_models/Qwen2-57B-A14B/ /mnt/nas/kexin/model_weights/Qwen2-57B-A14B-hf-to-mcore-te-tp8-pp1-ep4 8 1 4 fp32 true false
# [Traini