学习课程:gitee_Datawhale_GNN
学习论坛:Datawhale CLUB
公众号:Datawhale
本次学习的内容是有关于超大图的,具体的论文是Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks。根据论文标题猜测,使用聚类的算法将图分成小块,步步为营的解决超大图的问题。我们知道,图与传统机器学习最大的不同就是边的存在,将图一块块扯开来就像是藕断丝连一样,是很难做到分的清楚的。那么论文里是怎么做的?如何图分开来计算而兼顾图的关联性,且不损失精确性。
其实我现在也不是非常清楚为什么可以这样,并且对作者的做法还是有些疑问的,但是作为学习还是先一步步来。
1.首先作者比较了现有的三种图学习方法
Full-batch gradient descent:[memory: bad; time per epoch: good; convergence: bad]
Mini-batch SGD:[memory: good; time per epoch: bad; convergence: good]
** VR-GCN**:[memory: bad; time per epoch: good; convergence: good]
2.介绍图神经网络的基础数学模型,我的理解就是比传统的深度学习模型多了邻接矩阵A
3.对比分析mini-batch和cluster-gcn
左边是mini-batch,可以直观的看到mini-batch计算的时候会粘连其他区域的信息,导致剪不断理还乱的情况,但是cluster-gcn就显得泾渭分明
数学模型上也很直观,把图分为c个子图,将邻接矩阵A以簇的形式概括的写出,然后对角线邻接矩阵去近似邻接矩阵。这么做的原因是默认聚类分的很好,簇和簇之间没有太大关联性。这也是我最纳闷的地方,为什么可以这么做?原因是什么?聚类认为没有价值的边就没有价值了吗?不过工程的应用很多时候就是提效和优化,解决主要矛盾,圆周率还只是个近似呢。
分开的cluster使得小内存算大图成为现实。
如下的图想说明随机分区效果不如聚类,聚类之后一个cluster也能训练得到比较好的效果。
4.计算方法也是很直接,先聚类再按照图神经网路计算。
5.在不同数据集上验证,可以看到Cluster-GCN的效果完全不逊色于VRGCN,并且层数多的时候效果可能还更好。所以数据验证了既然能被聚类分离,那么说明这关系无关紧要。
仰望高端玩家,我加载不了论文里的数据集,所以我就拿Planetoid的数据集试了一试,结果非常纳闷了。
我用的是阿里云服务器,主要本地电脑加载了下Reddit之后觉得应该让它休息下。
阿里云服务器2G内存,40GSSD(简直是低端玩家中的青铜),但是可以云端一直跑啊哈哈哈。
于是使用的Planetoid的Cora,惊讶的发现改变cluster_num也就是聚类得到的簇的数量对实验基本没有影响,不论是精确度还是耗时还是cpu内存这些消耗情况。更改min-batch对计算时间影响很大,但是对其他也没有什么影响。
也许是因为数据集太小了,cluster-gcn并未能展现优势,不过话说回来,如果持平的话,那么说明cluster-gcn是一种不坑的方法,在更合适它的场景应该可以大放异彩。
运行的代码如下:
import time
from tqdm import tqdm
import matplotlib.pyplot as plt
import torch
import torch.nn.functional as F
from torch.nn import ModuleList
from torch_geometric.data import ClusterData, ClusterLoader, NeighborSampler
from torch_geometric.nn import SAGEConv
from torch_geometric.transforms import NormalizeFeatures
from torch_geometric.datasets import Planetoid
# dataset = Planetoid(root='data/Planetoid', name="Cora", transform=NormalizeFeatures())
dataset = Planetoid(root='data/Planetoid', name="PubMed", transform=NormalizeFeatures())
data = dataset[0]
num_parts = 1
epoch_num = 10
cluster_data = ClusterData(data, num_parts=num_parts, recursive=False,save_dir=dataset.processed_dir)
train_loader = ClusterLoader(cluster_data, batch_size=10, shuffle=True,num_workers=1)
subgraph_loader = NeighborSampler(data.edge_index, sizes=[-1], batch_size=10,shuffle=False, num_workers=1)
class Net(torch.nn.Module):
def __init__(self, in_channels, out_channels):
super(Net, self).__init__()
self.convs = ModuleList(
[SAGEConv(in_channels, 128),
SAGEConv(128, out_channels)])
def forward(self, x, edge_index):
for i, conv in enumerate(self.convs):
x = conv(x, edge_index)
if i != len(self.convs) - 1:
x = F.relu(x)
x = F.dropout(x, p=0.5, training=self.training)
return F.log_softmax(x, dim=-1)
def inference(self, x_all):
pbar = tqdm(total=x_all.size(0) * len(self.convs))
pbar.set_description('Evaluating')
# Compute representations of nodes layer by layer, using *all*
# available edges. This leads to faster computation in contrast to
# immediately computing the final representations of each batch.
for i, conv in enumerate(self.convs):
xs = []
for batch_size, n_id, adj in subgraph_loader:
edge_index, _, size = adj.to(device)
x = x_all[n_id].to(device)
x_target = x[:size[1]]
x = conv((x, x_target), edge_index)
if i != len(self.convs) - 1:
x = F.relu(x)
xs.append(x.cpu())
pbar.update(batch_size)
x_all = torch.cat(xs, dim=0)
pbar.close()
return x_all
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(dataset.num_features, dataset.num_classes).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.005)
def train():
model.train()
total_loss = total_nodes = 0
for batch in train_loader:
batch = batch.to(device)
optimizer.zero_grad()
out = model(batch.x, batch.edge_index)
loss = F.nll_loss(out[batch.train_mask], batch.y[batch.train_mask])
loss.backward()
optimizer.step()
nodes = batch.train_mask.sum().item()
total_loss += loss.item() * nodes
total_nodes += nodes
return total_loss / total_nodes
@torch.no_grad()
def test(): # Inference should be performed on the full graph.
model.eval()
out = model.inference(data.x)
y_pred = out.argmax(dim=-1)
accs = []
for mask in [data.train_mask, data.val_mask, data.test_mask]:
correct = y_pred[mask].eq(data.y[mask]).sum().item()
accs.append(correct / mask.sum().item())
return accs
loss_list = []
train_acc_list = []
val_acc_list = []
test_acc_list = []
T1 = time.time()
for epoch in range(1, epoch_num):
loss = train()
train_acc, val_acc, test_acc = test()
loss_list.append(loss)
train_acc_list.append(train_acc)
val_acc_list.append(val_acc)
test_acc_list.append(test_acc)
T2 = time.time()
plot_title = f'cluster_num:{num_parts}'+f', epoch_num:{epoch_num}'+f', time_period:{(T2-T1):.1f}s'
fig = plt.figure(figsize=(10,8))
ax1 = fig.add_subplot(211)
ax1.plot(range(1,epoch_num),loss_list,linewidth=3,label='loss')
ax1.set_title(plot_title)
ax1.legend(loc='lower left')
ax2 = fig.add_subplot(234)
ax2.plot(range(1,epoch_num),train_acc_list,color='green',alpha=0.5,label='train_acc',linewidth=3)
ax2.set_title('train_acc')
ax2.set_ylim(0,1.1)
ax2.legend(loc='lower left')
ax3 = fig.add_subplot(235)
ax3.plot(range(1,epoch_num),val_acc_list,color='orange',alpha=0.5,label='val_acc',linewidth=3)
ax3.set_title('val_acc')
ax3.set_ylim(0,1.1)
ax3.legend(loc='lower left')
ax4 = fig.add_subplot(236)
ax4.plot(range(1,epoch_num),test_acc_list,color='red',alpha=1,label='test_acc',linewidth=5)
ax4.set_title('test_acc')
ax4.set_ylim(0,1.1)
ax4.legend(loc='lower left')
plt.show()