RuntimeError: Expected to mark a variable ready only once.

在进行深度学习模型训练时，我们可能会遇到以下错误：

RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.

 这个错误的主要原因可能是，我们的模型在分布式训练过程中，对某个网络层计算了两次loss. 想要找到问题出在哪一层，可以用以下代码调试：

TORCH_DISTRIBUTED_DEBUG=DETAIL python xxx.py

这是可以看到出错的网络层：

 但，即使找到这个网络层，其实也比较难解决该问题。

简单粗暴的解决方案有以下两种：

方法一：

torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu], find_unused_parameters=False)

方法二：

ddp_model = DistributedDataParallel(model) 

ddp_model._set_static_graph()

方法二一般可以解决该问题。