https://gitee.com/mindspore/mindspore/tree/r1.3/model_zoo/official/cv/resnet_thor
【操作步骤&问题现象】
1、拉取mindspore 1.3的镜像:
docker pull swr.cn-south-1.myhuaweicloud.com/mindspore/mindspore-gpu:1.3.0
2、运行镜像
docker run -it -v /dev/shm:/dev/shm -v /path/to/dataset:/path/to/dataset --runtime=nvidia --privileged=true swr.cn-south-1.myhuaweicloud.com/mindspore/mindspore-gpu:1.3.0
3、在8卡v100机器上训练
sh run_distribute_train_gpu.sh /path/to/dataset 8
【日志信息】(可选,上传日志内容或者附件)
epoch: 32 step: 5004, loss is 2.2936077
epoch: 32 step: 5004, loss is 2.403049
epoch: 32 step: 5004, loss is 1.8597395
epoch: 32 step: 5004, loss is 2.1784513
epoch: 32 step: 5004, loss is 2.1430013
epoch: 32 step: 5004, loss is 1.8287767
epoch: 32 step: 5004, loss is 1.927578
epoch: 32 step: 5004, loss is 1.8246645
epoch time: 5372604.995 ms, per step time: 1073.662 ms
epoch time: 5372619.719 ms, per step time: 1073.665 ms
epoch time: 5372814.251 ms, per step time: 1073.704 ms
epoch time: 5372513.237 ms, per step time: 1073.644 ms
epoch time: 5372558.263 ms, per step time: 1073.653 ms
epoch time: 5372484.428 ms, per step time: 1073.638 ms
epoch time: 5372676.158 ms, per step time: 1073.676 ms
epoch time: 5372801.586 ms, per step time: 1073.701 ms
可以看到一个epoch需要一个多小时,非常慢
解答:
目前发现原因是数据集放在NAS中导致数据IO时间异常缓慢,将数据放到本地后训练速度正常。
epoch time: 578507.771 ms, per step time: 115.609 ms
epoch time: 578507.287 ms, per step time: 115.609 ms
epoch time: 578551.361 ms, per step time: 115.618 ms
epoch time: 578576.957 ms, per step time: 115.623 ms
epoch time: 578575.861 ms, per step time: 115.623 ms
epoch time: 578589.717 ms, per step time: 115.625 ms
epoch time: 578743.663 ms, per step time: 115.656 ms
epoch time: 578714.027 ms, per step time: 115.650 ms