nn.parallel.DistributedDataParallel报错，一块GPU时不报错

Warning: Traceback of forward call that caused the error:
  File "train_YnetV2_dist.py", line 273, in <module>
    pred_waypoint_map,pred_traj_map, pred_traj, pred_goal = model(feature_input,V_obs,A_obs,gt_waypoint_map)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 307, in forward
    inter_feat, adjA = self.inter_features(V_obs, A_obs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 342, in inter_features
    inter_represent = self.stGcn(x, A)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 170, in forward
    z = self.tcn(y).permute(0, 2, 1, 3)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 106, in forward
    exponential_average_factor, self.eps)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/functional.py", line 1685, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
 (print_stack at /opt/conda/conda-bld/pytorch_1582013237873/work/torch/csrc/autograd/python_anomaly_mode.cpp:58)
Traceback (most recent call last):
  File "train_YnetV2_dist.py", line 289, in <module>
    loss.backward(retain_graph=True)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [5]] is at version 10; expected version 9 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Warning: Traceback of forward call that caused the error:
  File "train_YnetV2_dist.py", line 273, in <module>
    pred_waypoint_map,pred_traj_map, pred_traj, pred_goal = model(feature_input,V_obs,A_obs,gt_waypoint_map)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 307, in forward
    inter_feat, adjA = self.inter_features(V_obs, A_obs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 342, in inter_features
    inter_represent = self.stGcn(x, A)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 170, in forward
    z = self.tcn(y).permute(0, 2, 1, 3)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 106, in forward
    exponential_average_factor, self.eps)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/functional.py", line 1685, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
 (print_stack at /opt/conda/conda-bld/pytorch_1582013237873/work/torch/csrc/autograd/python_anomaly_mode.cpp:58)
Traceback (most recent call last):
  File "train_YnetV2_dist.py", line 289, in <module>
    loss.backward(retain_graph=True)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [5]] is at version 10; expected version 9 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Warning: Traceback of forward call that caused the error:
  File "train_YnetV2_dist.py", line 273, in <module>
    pred_waypoint_map,pred_traj_map, pred_traj, pred_goal = model(feature_input,V_obs,A_obs,gt_waypoint_map)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 307, in forward
    inter_feat, adjA = self.inter_features(V_obs, A_obs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 342, in inter_features
    inter_represent = self.stGcn(x, A)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 170, in forward
    z = self.tcn(y).permute(0, 2, 1, 3)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 106, in forward
    exponential_average_factor, self.eps)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/functional.py", line 1685, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
 (print_stack at /opt/conda/conda-bld/pytorch_1582013237873/work/torch/csrc/autograd/python_anomaly_mode.cpp:58)
Traceback (most recent call last):
  File "train_YnetV2_dist.py", line 289, in <module>
    loss.backward(retain_graph=True)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [5]] is at version 10; expected version 9 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Warning: Traceback of forward call that caused the error:
  File "train_YnetV2_dist.py", line 273, in <module>
    pred_waypoint_map,pred_traj_map, pred_traj, pred_goal = model(feature_input,V_obs,A_obs,gt_waypoint_map)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 307, in forward
    inter_feat, adjA = self.inter_features(V_obs, A_obs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 342, in inter_features
    inter_represent = self.stGcn(x, A)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/Multimodal1229_noaug/model/model_heatmapv2_dist.py", line 170, in forward
    z = self.tcn(y).permute(0, 2, 1, 3)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 106, in forward
    exponential_average_factor, self.eps)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/nn/functional.py", line 1685, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
 (print_stack at /opt/conda/conda-bld/pytorch_1582013237873/work/torch/csrc/autograd/python_anomaly_mode.cpp:58)
Traceback (most recent call last):
  File "train_YnetV2_dist.py", line 289, in <module>
    loss.backward(retain_graph=True)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [5]] is at version 10; expected version 9 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Traceback (most recent call last):
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/gs/home/wangrp/anaconda/envs/w01/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/gs/home/wangrp/anaconda/envs/w01/bin/python', '-u', 'train_YnetV2_dist.py', '--local_rank=3']' returned non-zero exit status 1.

解决方案：
设置broadcast_buffers。原因不明

nn.parallel.DistributedDataParallel(model,device_ids=[device_id],output_device=device_id,\
        broadcast_buffers=False,find_unused_parameters=True)

添加链接描述