基本内容:
图卷积网络(GCNs)由于其在图上的表示学习能力而引起了广泛的关注。与卷积神经网络(CNN)不同,卷积神经网络能够堆叠非常深的层,GCN在深入时存在梯度消失、过度平滑和过度拟合问题。这些挑战限制了GCN在大规模图上的表示能力。如何能够成功且可靠地训练非常深的GCN的DeepGCN。
1、定义可微广义聚合函数,以统一不同的消息聚合操作(例如平均值、最大值)。
2、提出了一个新的规范化层,即MsgNorm和GCN残差连接的预激活版本。
算法原理:
The GENeralized Graph Convolution (GENConv) from the “DeeperGCN: All You Need to Train Deeper GCNs” paper. Supports SoftMax & PowerMean aggregation. The message construction is:
x
i
′
=
M
L
P
(
x
i
+
A
G
G
(
{
R
e
L
U
(
x
j
+
e
j
i
)
+
ϵ
,
j
∈
N
(
i
)
}
)
)
\mathbf{x}_i^{\prime} = \mathrm{MLP} \left( \mathbf{x}_i + \mathrm{AGG} \left( \left\{ \mathrm{ReLU} \left( \mathbf{x}_j + \mathbf{e_{ji}} \right) +\epsilon ,j \in \mathcal{N}(i) \right\} \right) \right)
xi′=MLP(xi+AGG({ReLU(xj+eji)+ϵ,j∈N(i)}))
算法核心实现阅读和分析:
pytorch_geometric/ogbn_proteins_deepgcn.py at master · pyg-team/pytorch_geometric · GitHub
def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
edge_attr: OptTensor = None, size: Size = None) -> Tensor:
""""""
if isinstance(x, Tensor):
x: OptPairTensor = (x, x)
if hasattr(self, 'lin_src'):
x = (self.lin_src(x[0]), x[1])
if isinstance(edge_index, SparseTensor):
edge_attr = edge_index.storage.value()
if edge_attr is not None and hasattr(self, 'lin_edge'):
edge_attr = self.lin_edge(edge_attr)
# Node and edge feature dimensionalites need to match.
if edge_attr is not None:
assert x[0].size(-1) == edge_attr.size(-1)
# propagate_type: (x: OptPairTensor, edge_attr: OptTensor)
# 经过传播进行聚合;
out = self.propagate(edge_index, x=x, edge_attr=edge_attr, size=size)
# 聚合;
if hasattr(self, 'lin_aggr_out'):
out = self.lin_aggr_out(out)
# 归一化步骤;
if hasattr(self, 'msg_norm'):
out = self.msg_norm(x[1] if x[1] is not None else x[0], out)
# 这个是公式中的x_i;
x_dst = x[1]
if x_dst is not None:
if hasattr(self, 'lin_dst'):
x_dst = self.lin_dst(x_dst)
out = out + x_dst
# 计算公式MLP后面的部分;
return self.mlp(out)
def message(self, x_j: Tensor, edge_attr: OptTensor) -> Tensor:
# 计算公式中AGG括号内的部分;
msg = x_j if edge_attr is None else x_j + edge_attr
return msg.relu() + self.eps
参考:
1、torch_geometric.nn — pytorch_geometric documentation (pytorch-geometric.readthedocs.io)
2、[2006.07739] DeeperGCN: All You Need to Train Deeper GCNs (arxiv.org)
3、pytorch_geometric/ogbn_proteins_deepgcn.py at master · pyg-team/pytorch_geometric · GitHub