1、请问一下，在我执行脚本训练的时提示这个异常，
[EXCEPTION] PYNATIVE(2834706,7f17151ac700,python):2021-11-12-10:30:18.738.479 [mindspore/ccsrc/pipeline/pynative/pynative_execute.cc:1116] GetOpOutput] : The pointer[cnode] is null.
[ERROR] MD(2834706,7f17151ac700,python):2021-11-12-10:30:20.968.034 [mindspore/ccsrc/minddata/dataset/util/task.cc:67] operator()] Task: GeneratorOp(ID:4) - thread(139737115051776) is terminated with err msg: Exception thrown from PyFunc. RuntimeError: mindspore/ccsrc/pipeline/pynative/pynative_execute.cc:1116 GetOpOutput] : The pointer[cnode] is null.
但脚本可以继续执行下去，训练一个迭代后，会再提示这个异常，然后程序就停止了。
【日志信息】（可选，上传日志内容或者附件）
(mind) user@gpu007:~/mind/GCM$ python pgcn_train.py thumos14 -b 2 --lr 0.0001 --snapshot_pref exp/test > log.txt
[WARNING] ME(2834706:139745305834304,MainProcess):2021-11-12-10:28:36.741.951 [mindspore/run_check/_check_version.py:181] Cuda ['10.1', '11.1'] version(need by mindspore-gpu) is not found, please confirm that the path of cuda is set to the env LD_LIBRARY_PATH, please refer to the installation guidelines: https://www.mindspore.cn/install
[WARNING] ME(2834706:139745305834304,MainProcess):2021-11-12-10:28:36.742.278 [mindspore/run_check/_check_version.py:181] Cuda ['10.1', '11.1'] version(need by mindspore-gpu) is not found, please confirm that the path of cuda is set to the env LD_LIBRARY_PATH, please refer to the installation guidelines: https://www.mindspore.cn/install
[WARNING] ME(2834706:139745305834304,MainProcess):2021-11-12-10:28:36.742.386 [mindspore/run_check/_check_version.py:181] Cuda ['10.1', '11.1'] version(need by mindspore-gpu) is not found, please confirm that the path of cuda is set to the env LD_LIBRARY_PATH, please refer to the installation guidelines: https://www.mindspore.cn/install
[WARNING] ME(2834706:139745305834304,MainProcess):2021-11-12-10:28:36.742.473 [mindspore/run_check/_check_version.py:181] Cuda ['10.1', '11.1'] version(need by mindspore-gpu) is not found, please confirm that the path of cuda is set to the env LD_LIBRARY_PATH, please refer to the installation guidelines: https://www.mindspore.cn/install
[WARNING] ME(2834706:139745305834304,MainProcess):2021-11-12-10:28:36.742.585 [mindspore/run_check/_check_version.py:181] Cuda ['10.1', '11.1'] version(need by mindspore-gpu) is not found, please confirm that the path of cuda is set to the env LD_LIBRARY_PATH, please refer to the installation guidelines: https://www.mindspore.cn/install
[WARNING] ME(2834706:139745305834304,MainProcess):2021-11-12-10:28:36.742.664 [mindspore/run_check/_check_version.py:181] Cuda ['10.1', '11.1'] version(need by mindspore-gpu) is not found, please confirm that the path of cuda is set to the env LD_LIBRARY_PATH, please refer to the installation guidelines: https://www.mindspore.cn/install
[WARNING] ME(2834706:139745305834304,MainProcess):2021-11-12-10:28:36.742.737 [mindspore/run_check/_check_version.py:181] Cuda ['10.1', '11.1'] version(need by mindspore-gpu) is not found, please confirm that the path of cuda is set to the env LD_LIBRARY_PATH, please refer to the installation guidelines: https://www.mindspore.cn/install
[WARNING] ME(2834706:139745305834304,MainProcess):2021-11-12-10:28:36.759.987 [mindspore/run_check/_check_version.py:181] Cuda ['10.1', '11.1'] version(need by mindspore-gpu) is not found, please confirm that the path of cuda is set to the env LD_LIBRARY_PATH, please refer to the installation guidelines: https://www.mindspore.cn/install
[WARNING] ME(2834706:139745305834304,MainProcess):2021-11-12-10:28:36.766.784 [mindspore/run_check/_check_version.py:130] Can not found cuda libs, please confirm that the correct cuda version has been installed, you can refer to the installation guidelines: https://www.mindspore.cn/install
[EXCEPTION] PYNATIVE(2834706,7f17151ac700,python):2021-11-12-10:30:18.738.479 [mindspore/ccsrc/pipeline/pynative/pynative_execute.cc:1116] GetOpOutput] : The pointer[cnode] is null.
[ERROR] MD(2834706,7f17151ac700,python):2021-11-12-10:30:20.968.034 [mindspore/ccsrc/minddata/dataset/util/task.cc:67] operator()] Task: GeneratorOp(ID:4) - thread(139737115051776) is terminated with err msg: Exception thrown from PyFunc. RuntimeError: mindspore/ccsrc/pipeline/pynative/pynative_execute.cc:1116 GetOpOutput] : The pointer[cnode] is null.
#
At:
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/ops/primitive.py(682): _run_op
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/common/api.py(80): wrapper
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/ops/primitive.py(247): __call__
/home/user/mind/GCM/ops/I3D_Pooling.py(32): I3D_Pooling
/home/user/mind/GCM/dataset.py(455): get_training_data
/home/user/mind/GCM/dataset.py(472): __getitem__
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/dataset/engine/datasets.py(3687): _cpp_sampler_fn
Line of code : 195
File : /home/jenkins/agent-working-dir/workspace/Compile_GPU_X86_CentOS_Cuda10_PY39/mindspore/mindspore/ccsrc/minddata/dataset/engine/datasetops/source/generator_op.cc
[ERROR] MD(2834706,7f17151ac700,python):2021-11-12-10:30:20.968.108 [mindspore/ccsrc/minddata/dataset/util/task_manager.cc:217] InterruptMaster] Task is terminated with err msg(more detail in info level log):Exception thrown from PyFunc. RuntimeError: mindspore/ccsrc/pipeline/pynative/pynative_execute.cc:1116 GetOpOutput] : The pointer[cnode] is null.
#
At:
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/ops/primitive.py(682): _run_op
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/common/api.py(80): wrapper
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/ops/primitive.py(247): __call__
/home/user/mind/GCM/ops/I3D_Pooling.py(32): I3D_Pooling
/home/user/mind/GCM/dataset.py(455): get_training_data
/home/user/mind/GCM/dataset.py(472): __getitem__
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/dataset/engine/datasets.py(3687): _cpp_sampler_fn
Line of code : 195
File : /home/jenkins/agent-working-dir/workspace/Compile_GPU_X86_CentOS_Cuda10_PY39/mindspore/mindspore/ccsrc/minddata/dataset/engine/datasetops/source/generator_op.cc
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.076.221 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [TensorScatterUpdate] does not support int64, cast input 1 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.076.360 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [TensorScatterUpdate] input 1
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.089.006 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [TopK] does not support int64, cast input 1 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.089.115 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [TopK] input 1
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.132.226 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [TensorScatterUpdate] does not support int64, cast input 1 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.132.356 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [TensorScatterUpdate] input 1
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.147.790 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [TopK] does not support int64, cast input 1 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.147.946 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [TopK] input 1
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.292.688 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [UniformInt] does not support int64, cast input 0 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.292.816 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [UniformInt] input 0
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.302.092 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [UniformInt] does not support int64, cast input 0 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.302.217 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [UniformInt] input 0
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.304.708 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Concat] does not support int64, cast input 0 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.304.737 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Concat] does not support int64, cast input 1 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.304.750 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:104] ReducePrecision] Kernel [Concat] does not support int64, cast output 0 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.304.854 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Concat] input 0
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.304.924 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Concat] input 1
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.306.795 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Concat] does not support int64, cast input 0 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.306.824 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Concat] does not support int64, cast input 1 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.306.837 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:104] ReducePrecision] Kernel [Concat] does not support int64, cast output 0 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.306.938 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Concat] input 0
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.307.008 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Concat] input 1
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.309.339 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Gather] does not support int64, cast input 1 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.309.367 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Gather] does not support int64, cast input 0 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.309.380 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:104] ReducePrecision] Kernel [Gather] does not support int64, cast output 0 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.309.485 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Gather] input 0
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.309.551 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Gather] input 1
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.311.555 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Gather] does not support int64, cast input 1 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.311.574 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Gather] does not support int64, cast input 0 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.311.601 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:104] ReducePrecision] Kernel [Gather] does not support int64, cast output 0 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.311.671 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Gather] input 0
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.311.718 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Gather] input 1
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.313.313 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Gather] does not support int64, cast input 1 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.313.347 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Gather] does not support int64, cast input 0 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:21.313.361 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:104] ReducePrecision] Kernel [Gather] does not support int64, cast output 0 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.313.477 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Gather] input 0
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:21.313.532 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Gather] input 1
[WARNING] OPTIMIZER(2834706,7f18fd503340,python):2021-11-12-10:30:22.095.731 [mindspore/ccsrc/frontend/optimizer/ad/kprim.cc:287] GetBprop] Fail to find bprop function for UniformInt. fn: None
[WARNING] OPTIMIZER(2834706,7f18fd503340,python):2021-11-12-10:30:22.097.735 [mindspore/ccsrc/frontend/optimizer/ad/kprim.cc:287] GetBprop] Fail to find bprop function for UniformInt. fn: None
[WARNING] OPTIMIZER(2834706,7f18fd503340,python):2021-11-12-10:30:22.105.634 [mindspore/ccsrc/frontend/optimizer/ad/kprim.cc:287] GetBprop] Fail to find bprop function for UniformInt. fn: None
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:22.790.790 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [TopK] does not support int64, cast input 1 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:22.790.909 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [TopK] input 1
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:22.807.404 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Gather] does not support int64, cast input 1 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:22.807.439 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [Gather] does not support int64, cast input 0 to int32.
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:22.807.448 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:104] ReducePrecision] Kernel [Gather] does not support int64, cast output 0 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:22.807.521 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Gather] input 0
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:22.807.566 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [Gather] input 1
[WARNING] KERNEL(2834706,7f18fd503340,python):2021-11-12-10:30:22.809.235 [mindspore/ccsrc/backend/kernel_compiler/gpu/gpu_kernel_factory.cc:96] ReducePrecision] Kernel [TopK] does not support int64, cast input 1 to int32.
[WARNING] PRE_ACT(2834706,7f18fd503340,python):2021-11-12-10:30:22.809.336 [mindspore/ccsrc/backend/optimizer/gpu/reduce_precision_fusion.cc:83] Run] Reduce precision for [TopK] input 1
2021-11-12 10:30:22,842 INFO Epoch: [0][0/100], lr: 0.00100 Time 93.641 (93.641) Data 89.534 (89.534) Loss 10.4698 (10.4698) Act. Loss 5.676 ( 5.676) Comp. Loss 0.008 ( 0.008) Reg. Loss 9.580 (9.580)
Act. FG 12.31 (12.31) Act. BG 0.00 (0.00)
Traceback (most recent call last):
File "/home/user/mind/GCM/pgcn_train.py", line 532, in
main()
File "/home/user/mind/GCM/pgcn_train.py", line 167, in main
train(train_loader, model, activity_criterion, completeness_criterion, regression_criterion, optimizer, epoch)
File "/home/user/mind/GCM/pgcn_train.py", line 205, in train
for i, (batch_act_fts, batch_comp_fts, batch_props_type,
File "/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/dataset/engine/iterators.py", line 134, in __next__
data = self._get_next()
File "/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/dataset/engine/iterators.py", line 207, in _get_next
print(self._iterator.GetNextAsList())
RuntimeError: Exception thrown from PyFunc. RuntimeError: mindspore/ccsrc/pipeline/pynative/pynative_execute.cc:1116 GetOpOutput] : The pointer[cnode] is null.
#
At:
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/ops/primitive.py(682): _run_op
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/common/api.py(80): wrapper
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/ops/primitive.py(247): __call__
/home/user/mind/GCM/ops/I3D_Pooling.py(32): I3D_Pooling
/home/user/mind/GCM/dataset.py(455): get_training_data
/home/user/mind/GCM/dataset.py(472): __getitem__
/home/user/anaconda3/envs/mind/lib/python3.9/site-packages/mindspore/dataset/engine/datasets.py(3687): _cpp_sampler_fn
Line of code : 195
File : /home/jenkins/agent-working-dir/workspace/Compile_GPU_X86_CentOS_Cuda10_PY39/mindspore/mindspore/ccsrc/minddata/dataset/engine/datasetops/source/generator_op.cc
pynative模式下generatordataset的source里面用了计算侧的算子，在GeneratorDataset的source函数中，替换mindspore的ops算子为对应的numpy算子或其他第三方算子，不替换就是错误用法，没法跑起来的。请检查脚本，然后把这些算子改成对应numpy算子吧。
322
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
