GNN-CS224W: 9 Theory of Graph Neural Networks

What is the expressive power

(ability to distinguish different graph structures) of these GNN models?

graph包括2部分：1. node feature；2. graph structure
接下来先讨论graph structure，假设node feature都一样（是否会讨论node featue 不知道，接着看）

接下来讨论的问题是：How well can a GNN distinguish different graph structures?

Key question: Can GNN node embeddings distinguish different node’s local neighborhood structures?
§ If so, when? If not, when will a GNN fail?

---

node embedding由其computational graph决定

computational graph is defined by the neighborhood

下图中，在node feature都一样(且不包含node id)的设定下，node 1 和node 2 的计算图就是完全一样的，是无法被区分开的

在node feature都一样(且不包含node id)的设定下，只要computational graph 结构是一样的，获得的embedding就是一样的

Computational graphs are identical to rooted subtree structures around each node. 意思是node 的Computational graphs是由其rooted subtree structures唯一确定的

---

Most expressive GNN maps different rooted subtrees into different node embeddings

GNN‘s node embeddings capture rooted subtree structures.

所要做的是maps different rooted subtrees into different node embeddings，无法处理有相同rooted subtree structures的node

接下来要做的是确认GNN是否能maps different rooted subtrees into different node embeddings

If each step of GNN’s aggregation can fully retain the neighboring information, the generated node embeddings can distinguish different rooted subtrees

most expressive GNN would use an injective neighbor aggregation function at each step.

which means Maps different neighbors to different embeddings.

结论：能将不同数量的neighbor映射成不同embedding的aggragation 的 expressive power 最强

Injective function

Function 𝑓: 𝑋 → Y is injective if it maps different elements into different outputs.

Intuition: 𝑓 retains all the information about input.

---

expressive power analysis

analyse the expressive power of different aggragation function:

上图中的repeating elements可以是不同的，例如一共有2种elements，黄色和蓝色，一个multi-set里可能2种element都有多个

multi-set function需要把各种elements分别有多少个的信息都记录下来，这样才能最大化expressive power

各种GNN expressive power分析

GCN (mean-pool)

当不同multi-sets中各种elements的比例相同时，mean-pool无法区分

为什么？
计算mean时，分子上各个neighbor的和，右边是左边的两倍；分母是neighbor 数量，右边的也是左边的两倍；分子除以分母，2倍被约掉以后，就是完全一样的。如下图

GraphSAGE (max-pool) (element-wise max)

当两个multi-set是由相同类型的element构成时，无法区分

element-wise max: 在neighbor的embedding的各个维度上取最大值形成新的向量，而不是取最大的整个向量。

因为只需要选择最大的，所以有多少个同类元素没有任何影响，有影响的只是有多少种元素，以及各种元素的embedding的各个维度上最大的值。

Therefore, GCN and GraphSAGE are not maximally powerful GNNs

---

Design maximally powerful GNNs

how to design

需要design injective neighbor aggregation function over multisets

Theorem: Any injective multi-set function can be expressed as:

$$\Phi (\sum _{x\in S}f(x))$$

其中$\Phi ()$和$f()$都nonlinear function，$f()$作用于set中每个元素，$S$为multi-set的所有元素

问题：$\Phi ()$和$f()$都必须是injective function吗？
回答，是的，如果有一个不是，那就会损失信息

Proof Intuition:

$f()$为将node映射为one-hot vector的function，one-hot相加表示对各类元素做计数，没有损失信息

MLP

Universal Approximation Theorem:
1-hidden-layer MLP with sufficiently-large hidden dimensionality and appropriate non-linearity $\sigma (\cdot )$ (including ReLU and sigmoid) can approximate any continuous function to an arbitrary accuracy

意思是一层足够大的MLP可以学习到任意的injective function

所以aggregation function over multisets可以定义为：
$$ML{P}_{\Phi }(\sum _{x\in S}ML{P}_f(x))$$

In practice, MLP hidden dimensionality of 100 to 500 is sufficient.

GIN

上式就是Graph Isomorphism Network (GIN)，GIN is THE most expressive GNN in the class of message-passing GNNs!

The Complete GIN
$$h_v^{(k+1)}=ML{P}_{\Phi }((1+ϵ)\cdot ML{P}_f(h_v^{(k)})+\sum _{u\in N_v}ML{P}_f(h_u^{(k)}))$$

$ϵ$是一个可以学习的参数

问题：$ϵ$起什么作用？怎么初始化？

用来neighbor和自己上一层的MLP是同一个吗？

问题：有可能学到任何injective function就意味着会学习到期待的function吗？

WL kernel has been both theoretically and empirically shown to distinguish most of the realworld graphs [Cai et al. 1992].
Hence, GIN is also powerful enough to distinguish most of the real graphs!

---

The key is to use element-wise sum pooling, instead of mean-/max-pooling