相同版本的mindspore 1.6 相同的代码 相同的参数 在GPU上训练可以稳定收敛但在Ascend上loss大幅度震荡

完整代码位于aaaaahmjz/MIMO-Net - MIMO-Net - OpenI - 启智AI开源社区提供免费算力！

该代码在GPU 上训练可以稳定收敛，但在Ascend上训练loss大幅度震荡

GPU上的训练日志：

epoch1/1200 train loss:0.09618113
epoch2/1200 train loss:0.09475256
epoch3/1200 train loss:0.09482105
epoch4/1200 train loss:0.092700526
epoch5/1200 train loss:0.091152735
epoch6/1200 train loss:0.09077169
epoch7/1200 train loss:0.08915941
epoch8/1200 train loss:0.089081906
epoch9/1200 train loss:0.088791214
epoch10/1200 train loss:0.08740218
epoch11/1200 train loss:0.085918084
epoch12/1200 train loss:0.084289424
epoch13/1200 train loss:0.08336934
epoch14/1200 train loss:0.08237906
epoch15/1200 train loss:0.085033596
epoch16/1200 train loss:0.08358286
epoch17/1200 train loss:0.08077453
epoch18/1200 train loss:0.08015344
epoch19/1200 train loss:0.081314534
epoch20/1200 train loss:0.0802836
epoch21/1200 train loss:0.07918551
epoch22/1200 train loss:0.078385964
epoch23/1200 train loss:0.077790335
epoch24/1200 train loss:0.07726628
epoch25/1200 train loss:0.07705663
epoch26/1200 train loss:0.076115556
epoch27/1200 train loss:0.07579533
epoch28/1200 train loss:0.07339258
epoch29/1200 train loss:0.07480494
epoch30/1200 train loss:0.07413012
epoch31/1200 train loss:0.07439254
epoch32/1200 train loss:0.07392034
epoch33/1200 train loss:0.072549134
epoch34/1200 train loss:0.0732402
epoch35/1200 train loss:0.07388319
epoch36/1200 train loss:0.070965484
epoch37/1200 train loss:0.070704296
epoch38/1200 train loss:0.07271889
epoch39/1200 train loss:0.071915485
epoch40/1200 train loss:0.07153733
epoch41/1200 train loss:0.07091346
epoch42/1200 train loss:0.069559954
epoch43/1200 train loss:0.0703414
epoch44/1200 train loss:0.07100435
epoch45/1200 train loss:0.06868055
epoch46/1200 train loss:0.06927643
epoch47/1200 train loss:0.067962065
epoch48/1200 train loss:0.068191744
epoch49/1200 train loss:0.06834627
epoch50/1200 train loss:0.06858324
epoch50/1200 train loss:0.06858324
epoch51/1200 train loss:0.06708284
epoch52/1200 train loss:0.0687678
epoch53/1200 train loss:0.06864959
epoch54/1200 train loss:0.068560846
epoch55/1200 train loss:0.06725955
epoch56/1200 train loss:0.067026466
epoch57/1200 train loss:0.06651329
epoch58/1200 train loss:0.066440865
epoch59/1200 train loss:0.06573461
epoch60/1200 train loss:0.066976674
epoch61/1200 train loss:0.06692096
epoch62/1200 train loss:0.066341795
epoch63/1200 train loss:0.065649554
epoch64/1200 train loss:0.06518027
epoch65/1200 train loss:0.066249125
epoch66/1200 train loss:0.06591753
epoch67/1200 train loss:0.06461888
epoch68/1200 train loss:0.063435026
epoch69/1200 train loss:0.066237845
epoch70/1200 train loss:0.06500661
epoch71/1200 train loss:0.0643674
epoch72/1200 train loss:0.0641
epoch73/1200 train loss:0.06358909
epoch74/1200 train loss:0.064878136
epoch75/1200 train loss:0.06357353
epoch76/1200 train loss:0.06486606
epoch77/1200 train loss:0.06374107
epoch78/1200 train loss:0.06285781
epoch79/1200 train loss:0.06425326
epoch80/1200 train loss:0.06261319
epoch81/1200 train loss:0.06338666
epoch82/1200 train loss:0.06434809
epoch83/1200 train loss:0.061426703
epoch84/1200 train loss:0.062140156
epoch85/1200 train loss:0.0628395
epoch86/1200 train loss:0.06324419
epoch87/1200 train loss:0.06272308
epoch88/1200 train loss:0.062274076
epoch89/1200 train loss:0.06277734

在Ascend 910上的训练日志：

epoch1/3000 train loss:0.09737113 
epoch2/3000 train loss:0.096444555 
epoch3/3000 train loss:0.09381536 
epoch4/3000 train loss:0.095001 
epoch5/3000 train loss:0.093053386 
epoch6/3000 train loss:0.09273996 
epoch7/3000 train loss:0.091802895 
epoch8/3000 train loss:0.09110157 
epoch9/3000 train loss:0.09081672 
epoch10/3000 train loss:0.0970297 
epoch11/3000 train loss:0.12774853 
epoch12/3000 train loss:0.70554173 
epoch13/3000 train loss:0.13691531 
epoch14/3000 train loss:0.8906944 
epoch15/3000 train loss:14.624553 
epoch16/3000 train loss:341.9412 
epoch17/3000 train loss:9.878007

并且每次训练开始时Ascend都会报出如下警告：

[WARNING] DEVICE(13563,ffff9fec1840,python):2022-09-02-19:03:38.206.396 [mindspore/ccsrc/runtime/hardware/ascend/ascend_graph_optimization.cc:259] SelectKernel] There are 797 node/nodes used reduce precision to selected the kernel!

我猜可能是类似于【求助】基于Ascend910的MindSpore训练无法复现GPU上的模型效果_MindSpore_昇腾_华为云论坛 (huaweicloud.com)，于是我手动设置了整个网络的精度为fp32, 类似于这样：

net=MIMOUNetPlus().to_float(mstype.float32)

但是依然会报上述警告，loss也仍然震荡。请问这是为什么呢

****************************************************解答*****************************************************

Ascend和GPU上算子计算结果有误差，对于loss震荡的情况，建议还是调一下参数