connect [127.0.1.1]:926: Connection refused

问题描述：

尝试DGL基于CPU的分布式训练

miniconda虚拟环境：

python：3.6.13

pytorch：1.10.1 

dgl：1.1.0

三台主机搭建分布式集群，一台NFS服务，两台Client，集群之间可以SSH免密登录 

报错信息：

Traceback (most recent call last):
  File "main.py", line 16, in <module>
    th.distributed.init_process_group(backend='gloo')
  File "/home/hadoop/miniconda3/envs/p112p376/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 610, in init_process_group
    timeout=timeout,
  File "/home/hadoop/miniconda3/envs/p112p376/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 703, in _new_process_group_helper
    pg = ProcessGroupGloo(prefix_store, rank, world_size, timeout=timeout)
Client[1] in group[0] is exiting...
RuntimeError: [/opt/conda/conda-bld/pytorch_1656352405797/work/third_party/gloo/gloo/transport/tcp/pair.cc:799] connect [127.0.1.1]:926: Connection refused 

解决方法：

报错原因：

无法建立gloo进程组，IP的主机名指向127.0.1.1。因此，gloo将主机名解析为本地主机并连接到自身

解决方法：

修改主机/etc/hosts文件，将主机ip与主机名对应且不重复，如下：

ip_addr1 server
ip_addr2 node1
ip_addr3 node2

参考：

Failed to create Gloo new group after initialized with NCCL · Issue #68726 · pytorch/pytorch (github.com)