What is the expressive power
(ability to distinguish different graph structures) of these GNN models?
graph包括2部分:1. node feature;2. graph structure
接下来先讨论graph structure,假设node feature都一样(是否会讨论node featue 不知道,接着看)
接下来讨论的问题是:How well can a GNN distinguish different graph structures?
Key question: Can GNN node embeddings distinguish different node’s local neighborhood structures?
§ If so, when? If not, when will a GNN fail?
node embedding由其computational graph决定
computational graph is defined by the neighborhood
下图中,在node feature都一样(且不包含node id)的设定下,node 1 和node 2 的计算图就是完全一样的,是无法被区分开的
在node feature都一样(且不包含node id)的设定下,只要computational graph 结构是一样的,获得的embedding就是一样的
Computational graphs are identical to rooted subtree structures around each node. 意思是node 的Computational graphs是由其rooted subtree structures唯一确定的
Most expressive GNN maps different rooted subtrees into different node embeddings
GNN‘s node embeddings capture rooted subtree structures.
所要做的是maps different rooted subtrees into different node embeddings,无法处理有相同rooted subtree structures的node
接下来要做的是确认GNN是否能maps different rooted subtrees into different node embeddings
If each step of GNN’s aggregation can fully retain the neighboring information, the generated node embeddings can distinguish different rooted subtrees
most expressive GNN would use an injective neighbor aggregation function at each step.
which means Maps different neighbors to different embeddings.
结论:能将不同数量的neighbor映射成不同embedding的aggragation 的 expressive power 最强
Injective function
Function 𝑓: 𝑋 → Y is injective if it maps different elements into different outputs.
Intuition: 𝑓 retains all the information about input.
expressive power analysis
analyse the expressive power of different aggragation function:
上图中的repeating elements可以是不同的,例如一共有2种elements,黄色和蓝色,一个multi-set里可能2种element都有多个
multi-set function需要把各种elements分别有多少个的信息都记录下来,这样才能最大化expressive power
各种GNN expressive power分析
GCN (mean-pool)
当不同multi-sets中各种elements的比例相同时,mean-pool无法区分
为什么?
计算mean时,分子上各个neighbor的和,右边是左边的两倍;分母是neighbor 数量,右边的也是左边的两倍;分子除以分母,2倍被约掉以后,就是完全一样的。如下图
GraphSAGE (max-pool) (element-wise max)
当两个multi-set是由相同类型的element构成时,无法区分
element-wise max: 在neighbor的embedding的各个维度上取最大值形成新的向量,而不是取最大的整个向量。
因为只需要选择最大的,所以有多少个同类元素没有任何影响,有影响的只是有多少种元素,以及各种元素的embedding的各个维度上最大的值。
Therefore, GCN and GraphSAGE are not maximally powerful GNNs
Design maximally powerful GNNs
how to design
需要design injective neighbor aggregation function over multisets
Theorem: Any injective multi-set function can be expressed as:
Φ ( ∑ x ∈ S f ( x ) ) \Phi( \sum\limits_{x \in S} f(x)) Φ(x∈S∑f(x))
其中 Φ ( ) \Phi() Φ()和 f ( ) f() f()都nonlinear function, f ( ) f() f()作用于set中每个元素, S S S为multi-set的所有元素
问题:
Φ
(
)
\Phi()
Φ()和
f
(
)
f()
f()都必须是injective function吗?
回答,是的,如果有一个不是,那就会损失信息
Proof Intuition:
f
(
)
f()
f()为将node映射为one-hot vector的function,one-hot相加表示对各类元素做计数,没有损失信息
MLP
Universal Approximation Theorem:
1-hidden-layer MLP with sufficiently-large hidden dimensionality and appropriate non-linearity
σ
(
⋅
)
\sigma(⋅)
σ(⋅) (including ReLU and sigmoid) can approximate any continuous function to an arbitrary accuracy
意思是一层足够大的MLP可以学习到任意的injective function
所以aggregation function over multisets可以定义为:
M
L
P
Φ
(
∑
x
∈
S
M
L
P
f
(
x
)
)
MLP_{\Phi}( \sum\limits_{x \in S} MLP_{f}(x))
MLPΦ(x∈S∑MLPf(x))
In practice, MLP hidden dimensionality of 100 to 500 is sufficient.
GIN
上式就是Graph Isomorphism Network (GIN),GIN is THE most expressive GNN in the class of message-passing GNNs!
The Complete GIN
h
v
(
k
+
1
)
=
M
L
P
Φ
(
(
1
+
ϵ
)
⋅
M
L
P
f
(
h
v
(
k
)
)
+
∑
u
∈
N
v
M
L
P
f
(
h
u
(
k
)
)
)
h^{(k+1)}_v =MLP_{\Phi}( (1+\epsilon) \cdot MLP_f(h^{(k)}_v) + \sum\limits_{u \in N_v} MLP_{f}(h^{(k)}_u))
hv(k+1)=MLPΦ((1+ϵ)⋅MLPf(hv(k))+u∈Nv∑MLPf(hu(k)))
ϵ \epsilon ϵ是一个可以学习的参数
问题: ϵ \epsilon ϵ起什么作用?怎么初始化?
用来neighbor和自己上一层的MLP是同一个吗?
问题:有可能学到任何injective function就意味着会学习到期待的function吗?
WL kernel has been both theoretically and empirically shown to distinguish most of the realworld graphs [Cai et al. 1992].
Hence, GIN is also powerful enough to distinguish most of the real graphs!