我向OGB排行榜提交代码的经历

Our paper: https://arxiv.org/pdf/2105.08330.pdf.
Our code: https://github.com/ytchx1999/PyG-OGB-Tricks/tree/main

---

ogbn_arxiv_GCN_res

This is an improvement of baesline on the ogbn-arxiv dataset.
我的代码：https://github.com/ytchx1999/ogbn_arxiv_GCN_res

ogbn-arxiv

The framework is shown in the figure.

Improvement Strategy：

• add skip-connection inspired by DeeperGCN
  按比例把前一层的结果加到当前层的结果上，既可以加速模型收敛速度，又可以缓解过平滑问题。
• add initial residual connection inspired by GCNII
  主要是借鉴了初始剩余连接的思路，按照比例在后面层的结果中加上了$X^{(0)}$，可以缓解模型过平滑问题，不并且能够小幅提升模型的acc。
• add jumping knowledge inspired by JKNet
  类似JKNet，先保存每一层的结果，然后经过softmax后进行sum，得到最终的节点特征表示，可以有效缓解过平滑问题。

Experiment Setup：

The model is 8 layers, and runs 500 epochs.

python ogbn_gcn_res.py

Detailed Hyperparameter:

num_layers = 8
hidden_dim = 128
dropout = 0.5
lr = 0.01
runs = 10
epochs = 500
alpha = 0.2
beta = 0.5

Result:

All runs:
Highest Train: 77.94 ± 0.50
Highest Valid: 73.69 ± 0.21
  Final Train: 77.72 ± 0.46
   Final Test: 72.62 ± 0.37

Model	Test Accuracy	Valid Accuracy	Parameters	Hardware
GCN_res	0.7262 ± 0.0037	0.7369 ± 0.0021	155824	Tesla T4（16GB）

已经向OGB排行榜提交了申请，OGB团队正在验证我的model，看看能不能选上吧。

---

2021.2.22更新

OGB团队接受了我的代码，暂时位列19，开森。
https://ogb.stanford.edu/docs/leader_nodeprop/#ogbn-arxiv

已经超过了GCN、GraphSAGE等baseline的表现，算是毕设的目标初步达成？
欢迎各位大佬批评指正！

---

2021.2.25更新

后面发现，我的model标准差std稍微有点大，所以采用了FLAG方法进行对抗性数据增广，以稳定模型并小幅提升准确率。

GCN_res-FLAG

This is an improvement of the (GCN_res + 8 layers) model, using the FLAG method.
我的代码：https://github.com/ytchx1999/GCN_res-FLAG

ogbn-arxiv

• Check out the model：(GCN_res + 8 layers)

• Check out the FLAG method：FLAG

Improvement Strategy：

• add FLAG method

Environmental Requirements

• pytorch == 1.7.1
• pytorch_geometric == 1.6.3
• ogb == 1.2.4

Experiment Setup：

The model is 8 layers, 10 runs which conclude 500 epochs.

python ogbn_gcn_res_flag.py

Detailed Hyperparameter:

num_layers = 8
hidden_dim = 128
dropout = 0.5
lr = 0.01
runs = 10
epochs = 500
alpha = 0.2
beta = 0.7

Result:

All runs:
Highest Train: 78.61 ± 0.49
Highest Valid: 73.89 ± 0.12
  Final Train: 78.44 ± 0.46
   Final Test: 72.76 ± 0.24

Model	Test Accuracy	Valid Accuracy	Parameters	Hardware
GCN_res + FLAG	0.7276 ± 0.0024	0.7389 ± 0.0012	155824	Tesla T4（16GB）

可以发现，模型准确率有了小幅度的提升，并且标准差也降下来了，算是达到了预期的目标。

向OGB排行榜提交了代码之后，等了大约1天吧，代码被团队接受。表现超过了GCNII，位列18。
https://ogb.stanford.edu/docs/leader_nodeprop/#ogbn-arxiv

---

2021.3.25更新

将CorrectAndSmooth（C&S）方法应用到了我的模型GCN_res当中，经过了一番机械调参后，模型取得了不错的效果！

我的代码：https://github.com/ytchx1999/PyG-GCN_res-CS

PyG-GCN_res-CS

This is an improvement of the (GCN_res + 8 layers) model, using the C&S method.

ogbn-arxiv

• Check out the model：(GCN_res + 8 layers)

• Check out the C&S method：C&S

Improvement Strategy：

• add C&S method

Environmental Requirements

• pytorch == 1.7.1
• pytorch_geometric == 1.6.3
• ogb == 1.2.4

Experiment Setup：

The model is 8 layers, 10 runs which conclude 500 epochs.

python gcn_res_cs.py

Detailed Hyperparameter:

num_layers = 8
hidden_dim = 128
dropout = 0.5
lr = 0.01
runs = 10
epochs = 500
alpha = 0.2
beta = 0.7
num_correction_layers = 50
correction_alpha = 0.8
num_smoothing_layers = 50
smoothing_alpha = 0.8
scale = 1.
A1 = 'DAD'
A2 = 'DAD'

Result:

All runs:
Highest Train: 95.40 ± 0.02
Highest Valid: 74.23 ± 0.14
  Final Train: 95.40 ± 0.02
   Final Test: 72.97 ± 0.22

Model	Test Accuracy	Valid Accuracy	Parameters	Hardware
GCN_res + C&S	0.7297 ± 0.0022	0.7423 ± 0.0014	155824	Tesla T4（16GB）

代码很快就被接受了，连1个小时都不到，这速度真实杠杠的！这样的话我的排名又往前提升了几个位次，GCN_res + C&S暂时位列第16（19–>16）。

每次进步一点点吧。

---

2021.3.27更新

昨天去所里把服务器的环境配好了，然后用一块Tesla V100对GCN_res + C&S模型的C&S部分进行了机械调参（白嫖的快乐），然后成功获得了更好性能的v2版本（GCN_res + C&S_v2），之后便提交了代码。

代码：https://github.com/ytchx1999/GCN_res-CS-v2

相比之前的排名又有所进步，目前为止排名13，已经达到了GCN内核目前为止的最好性能了！后面估计要针对GAT或UniMP这种注意力机制的模型进行改进了。

毕设主要是针对GCN内核做出的改进，论文也是基于这个写的，感觉只看毕设的话可能已经到头了。

---

毕设差不多可以了，新开一个副本，另起炉灶。。后面向ogbn-products、ogbn-mag数据集提交代码。当然，就是练练手，学习学习别人的方法和代码，做实验的同时顺便提交一下代码。

2021.4.5更新

用GAT + C&S跑了一下ogbn-products数据集，当然，C&S是我自己手动加上去的，并没有找到直接可以跑的版本。

GAT w/NS + C&S

This is an improvement of the (GAT with NeighborSampling) model, using the C&S method.

我的代码：https://github.com/ytchx1999/PyG-ogbn-products/tree/main/gat

ogbn-products

• Check out the model： (GAT with NeighborSampling)

• Check out the C&S method：C&S

Improvement Strategy：

• add C&S method
• add BatchNorm

Environmental Requirements

• pytorch == 1.8.1
• pytorch_geometric == 1.6.3
• ogb == 1.3.0

Experiment Setup：

• Let the program run in the foreground.

python gat_cs_mini.py

• Or let the program run in the background and save the results to a log file.

nohup python gat_cs_mini.py > ./gat_cs_mini.log 2>&1 &

Detailed Hyperparameter:

num_layers = 3
hidden_dim = 128
heads = 4
dropout = 0.5
lr = 0.001
batch_size = 512
sizes = [10, 10, 10]
runs = 10
epochs = 100
num_correction_layers = 100
correction_alpha = 0.8
num_smoothing_layers = 100
smoothing_alpha = 0.8
scale = 10.
A1 = 'DAD'
A2 = 'DAD'

Result:

All runs:
Highest Train: 97.28 ± 0.06
Highest Valid: 92.63 ± 0.08
  Final Train: 97.28 ± 0.06
   Final Test: 80.92 ± 0.37

Model	Test Accuracy	Valid Accuracy	Parameters	Hardware
GAT w/NS + C&S	0.8092 ± 0.0037	0.9263 ± 0.0008	753622	Tesla V100 (32GB)

暂时位列第8名。MLP+C&S的表现过于的好，感觉模型越简单，C&S发挥的作用就越大。

同一天再更新一波吧。有用GraphSAGE做了实验，虽然比不上GAT，但是相对来说排名的提高已经很多了。

我的代码：https://github.com/ytchx1999/PyG-ogbn-products/tree/main/sage

---

2021.4.7更新

把GraphSAGE + C&S模型加了node2vec的嵌入表示，又向ogbn-products排行榜提交了一下。

我的代码：https://github.com/ytchx1999/PyG-ogbn-products/tree/main/sage%2Bnode2vec

GraphSAGE w/NS + C&S + node2vec

This is an improvement of the (NeighborSampling (SAGE aggr)) model, using the C&S method and node2vec embedding.

ogbn-products

• Check out the model： (NeighborSampling (SAGE aggr))
• Check out the C&S method：C&S
• Check out node2vec model：node2vec

Improvement Strategy：

• add C&S method
• add BatchNorm
• add node2vec embedding

Environmental Requirements

• pytorch == 1.8.1
• pytorch_geometric == 1.6.3
• ogb == 1.3.0

Experiment Setup：

1. Generate node2vec embeddings, which save in embedding.pt

   python node2vec_products.py
   

2. Run the real model

   • Let the program run in the foreground.

   python sage_cs_em.py
   

   • Or let the program run in the background and save the results to a log file.

   nohup python sage_cs_em.py > ./sage_cs_em.log 2>&1 &
   

Detailed Hyperparameter:

num_layers = 3
hidden_dim = 256
dropout = 0.5
lr = 0.003
batch_size = 1024
sizes = [15, 10, 5]
runs = 10
epochs = 20
num_correction_layers = 100
correction_alpha = 0.8
num_smoothing_layers = 100
smoothing_alpha = 0.8
scale = 10.
A1 = 'DAD'
A2 = 'DAD'

Result:

All runs:
Highest Train: 97.13 ± 0.07
Highest Valid: 92.38 ± 0.06
  Final Train: 97.13 ± 0.07
   Final Test: 81.54 ± 0.50

Model	Test Accuracy	Valid Accuracy	Parameters	Hardware
GraphSAGE w/NS + C&S + node2vec	0.8154 ± 0.0050	0.9238 ± 0.0006	103983	Tesla V100 (32GB)

最终结果排名第7，超过了GAT + C&S的表现。

---

2021.4.10更新

又换了一个数据集：ogbn-mag。这是一个异构图，十分的复杂。

我的代码：https://github.com/ytchx1999/PyG-ogbn-mag/tree/main/saint%2Bmetapath2vec

GraphSAINT + metapath2vec

This is an improvement of the (GraphSAINT (R-GCN aggr)) model, using metapath2vec embedding.

ogbn-products

• Check out the model： (GraphSAINT (R-GCN aggr))
• Check out metapath2vec model：metapath2vec

Improvement Strategy：

• adjust hidden_dim
• add metapath2vec embedding

Environmental Requirements

• pytorch == 1.8.1
• pytorch_geometric == 1.6.3
• ogb == 1.3.0

Experiment Setup：

1. Generate metapath2vec embeddings, which save in mag_embedding.pt

   python metapath2vec.py
   

2. Run the real model

   python rgcn_saint.py
   

Detailed Hyperparameter:

GrapgSAINT:

num_layers = 2
hidden_dim = 256
dropout = 0.5
lr = 0.005
batch_size = 20000
walk_length = 2
runs = 10
epochs = 30
num_steps = 30

Metapath2vec:

embedding_dim = 128
lr = 0.01
batch_size = 20000
walk_length = 64
epochs = 5

Result:

All runs:
Highest Train: 84.01 ± 2.72
Highest Valid: 50.66 ± 0.17
  Final Train: 84.01 ± 2.72
   Final Test: 49.66 ± 0.22

Model	Test Accuracy	Valid Accuracy	Parameters	Hardware
GraphSAINT + metapath2vec	0.4966 ± 0.0022	0.5066 ± 0.0017	309764724	Tesla V100 (32GB)

对GraphSAINT模型进行了改进，目前排名第5。

---

2021.5.24更新

给R-GSN模型增加了metapath2vec嵌入表示，成功冲到了ogbn-mag榜的第二名。

---

2021.6.8更新

又冲了一下ogbn-proteins榜。
代码：https://github.com/ytchx1999/PyG-OGB-Tricks/tree/main/DGL-ogbn-proteins

GAT + labels + node2vec

This is an improvement of the GAT model by Wang (DGL), using node2vec embedding.

Our paper is available at https://arxiv.org/pdf/2105.08330.pdf.

ogbn-proteins

Improvement Strategy：

• adjust hidden and embedding dim.
• add node2vec embedding ---- the usage of node2vec greatly accelerates the convergence of GAT.

Environmental Requirements

• dgl >= 0.5.0
• torch >= 1.6.0
• torch_geometric >= 1.6.0
• ogb == 1.3.0

Experiment Setup：

1. Generate node2vec embeddings, which save in proteins_embedding.pt

   python node2vec_proteins.py
   

2. Run the real model

   • Let the program run in the foreground.

   python gat.py --use-labels
   

   • Or let the program run in the background and save the results to a log file.

   nohup python gat.py --use-labels > ./gat.log 2>&1 &
   

Detailed Hyperparameter:

GAT:

Namespace(attn_drop=0.0, cpu=False, dropout=0.25, edge_drop=0.1, eval_every=5, gpu=0, input_drop=0.1, log_every=5, lr=0.01, n_epochs=1200, n_heads=6, n_hidden=128, n_layers=6, n_runs=10, no_attn_dst=False, plot_curves=False, save_pred=False, seed=0, use_embed=True, use_labels=True, wd=0)

--n-runs N_RUNS         running times (default: 10)
--n-epochs N_EPOCHS     number of epochs (default: 1200)
--use-labels            Use labels in the training set as input features. (default: False)
--lr LR                 learning rate (default: 0.01)
--n-layers N_LAYERS     number of layers (default: 6)
--n-heads N_HEADS       number of heads (default: 6)
--n-hidden N_HIDDEN     number of hidden units (default: 128)
--dropout DROPOUT       dropout rate (default: 0.25)
--input-drop INPUT_DROP input drop rate (default: 0.1)

node2vec:

embedding_dim = 16
lr = 0.01
batch_size = 256
walk_length = 80
epochs = 5

Result:

Val scores: [0.9229285934246892, 0.9211608885028892, 0.9213509308888836, 0.9219311666881109, 0.922188157691978, 0.9233155178378067, 0.9226761093114175, 0.9207967425451954, 0.9192225312946334, 0.9216411187053957]
Test scores: [0.8705177963169082, 0.8718678325708628, 0.871026339976343, 0.8713582109483052, 0.8706036035560922, 0.8709027982169764, 0.8704158483168263, 0.8704708862546975, 0.8713362807645616, 0.8726814140948117]

Average val score: 0.9217211756890998 ± 0.0011282315196969204
Average test score: 0.8711181011016385 ± 0.0006857984340481437

Model	Test Accuracy	Valid Accuracy	Parameters	Hardware
GAT + labels + node2vec	0.8711 ± 0.0007	0.9217 ± 0.0011	6360470	Tesla V100 (32GB)

---

2021.7.27更新

用PyG重写了MLP+C&S，好像还比原版还一点点