[论文笔记] pai-megatron-patch megatron 模型接着训练 继续训练

继续训练的话，不要用megatron的检查点转换回hf模型。

1、要直接 load megatron模型。load $checkpoint_path

2、其中 checkpoint_path 中的 last_iteration_txt (好像是这个名字) 要改成自己需要的。

3、需要在checkpoints文件夹里面加config.json、vocab.json，merges.txt等。

#!/bin/bash
set +x
export NCCL_DEBUG=WARN
export NCCL_IB_QPS_PER_CONNECTION=32
# ^(=.=)^
PYTHON=$(which python)
sed -i "s|barrier_timeout: float = 300|barrier_timeout: float = 1800|g" $PYTHO/site-packages/torch/distributed/elastic/utils/store.py
pip install transformers==4.37.0 -i https://pypi.doubanio.com/simple/
# 
EPOCH=2
SEQUENCE_LENGTH=4096

WARMUP_TOKENS=0
MICRO_BATCH_SIZE=1
GLOBAL_BATCH_SIZE=1024
MODEL_SIZE=72B
TP_SIZE=8
PP_SIZE=4
PAD_LEN=$SEQUENCE_LENGTH
SAVE_INTERVAL=1000

DIR_NAME="qwen1.5_72B"
DATA_OUTPUT_PATH="/mnt/nas/pretrain/runs/qwen1.5-72B-22lan-ct-06111345"
CHECKPOINT_PATH="$DATA_OUTPUT_PATH/checkpoint