1 系统环境
硬件环境(Ascend/GPU/CPU): Ascend
MindSpore版本: 2.2.0
执行模式(PyNative/ Graph): 不限
2 报错信息
2.1 问题描述
模型跑4层网络,设置并行策略为dp:mp:pp=1:1:8,出现报错
2.2 报错信息
Traceback (most recent call last):
File "wizardcoder/run_wizardcode r.py", line 148, in <module>
device_id=args.device_id)
File "wizardcoder/run_wizardcoder.py", line 90, in main
task. finetune(finetune_checkpoint=config.load_checkpoint • auto_trans_ckpt=config .auto_trans_ckpt, resume=resume)
File "/home/wizardcoder/1_wizardcoder-mindformers-916/mindformers/trainer/trainer.py", Tine 522, in finetune
is_full_config=True, **kwargs
File"/home/wizardcoder/1_wizardcoder-mindformers-916/mindformers/traine r/caus al_language_modeling/caus al_language_modeling.py", line 106, in train
**kwargs)
File "/home/wizardcoder/1_wizardcoder-mindformers-916/mindformers/traine r/base_trainer.py", line 616, in training_process
transform_and_load_checkpoint (config, model, network, dataset)
File "/home/wizardcoder/1_wizardc oder-mindformers-916/mindforme rs/trainer/utils.py", line 300, in transform_ and_load_checkpoint
build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict)
File "/home/wizardcoder/1_wizardcoder-mindformers-916/mindformers/trainer/utils.py", line 330, in build_model
sink_size=config.runner_config.sink_size)
File "/root/miniconda3/envs/wiz ardcoder/lib/python3.7/site-packages/mindspore/train/model .py", line 1263, in build
self._init(train_dataset , valid_dataset, sink_siz e, epoch)
File "/root/miniconda3/envs/wiza rdcoder/lib/python3.1/site-packages /mindspore/train/model.py", line 524, in _init
train_network.compile( *inputs)
File "/root/miniconda3/envs/wizardcoder/lib/python3 .7/site-packages/mindspore/nn/cell .py", line 939, in compile
jit_config_dict=self._jit_config_dict, *compile_args, **kwargs)
File n/root/miniconda3/envs7wiza rdcoder/lib/python3.7/site-packages /mindspore/common/api.py", line 1623, in compile
result = self. araoh executor.comoilelobi. aras. kwaras, phase, self. use vm mode()
RuntimeError: Stage num is 8 is not equal to stage used: 5
复制
3 根因分析
这是因为模型层数只有4层,无法进行pipeline=8的分层切割。
4 解决方案
需要满足pipeline_stage小于等于num_layers这一条件。