NCCL Error 2: unhandled system error | data parallel on GPU A100(多卡DP训练报错)

单卡训练没有问题，多卡DP训练的时候报错

Traceback (most recent call last):
  File "run.py", line 602, in <module>
    main()
  File "run.py", line 392, in main
    loss.backward()
  File "/project//anaconda3/envs/evil-env/lib/python3.7/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/project//anaconda3/envs/evil-env/lib/python3.7/site-packages/torch/autograd/__init__.py", line 175, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
  File "/project//anaconda3/envs/evil-env/lib/python3.7/site-packages/torch/autograd/function.py", line 253, in apply
    return user_fn(self, *args)
  File "/project//anaconda3/envs/evil-env/lib/python3.7/site-packages/torch/nn/parallel/_functions.py", line 34, in backward
    return (None,) + ReduceAddCoalesced.apply(ctx.input_device, ctx.num_inputs, *grad_outputs)
  File "/project//anaconda3/envs/evil-env/lib/python3.7/site-packages/torch/nn/parallel/_functions.py", line 45, in forward
    return comm.reduce_add_coalesced(grads_, destination)
  File "/project//anaconda3/envs/evil-env/lib/python3.7/site-packages/torch/nn/parallel/comm.py", line 143, in reduce_add_coalesced
    flat_result = reduce_add(flat_tensors, destination)
  File "/project//anaconda3/envs/evil-env/lib/python3.7/site-packages/torch/nn/parallel/comm.py", line 96, in reduce_add
    nccl.reduce(inputs, output=result, root=root_index)
  File "/project//anaconda3/envs/evil-env/lib/python3.7/site-packages/torch/cuda/nccl.py", line 93, in reduce
    torch._C._nccl_reduce(inputs, _output, root, op, streams, comms)
RuntimeError: NCCL Error 2: unhandled system error

根据https://github.com/Lightning-AI/lightning/issues/4420

在命令行输入

export NCCL_IB_DISABLE=1
export NCCL_P2P_DISABLE=1

解决问题~

相关链接
Can we do distributed data parallel on GPU A100？