CS224W: Machine Learning with Graphs - 02 Traditional Feature-based Methods

Traditional Feature-based Methods

1. Traditional ML Pipeline

• Design features for nodes/links/graphs
• Obtain features for all training data

2. Design Choices

• Features: $d$-dimensional vectors
• Objects: Nodes, edges, set of nodes, entire graphs
• Objective functions: What task are we aiming to solve?

3. Node-level Tasks

1). Node-level features: Overview

Goal: characterize the structure and position of a node in the network

• Node degree
• Node centrality
• Clustering coefficient
• Graphlets

2). Node Degree

• The degree $k_v$ of node $v$ is the number of edges (neighboring nodes) the node has
• Treats all neighboring nodes equally

3). Node Centrality

• Node degree counts the neighboring nodes without capturing their importance.
• Node centrality $c_v$ takes the node importance in a graph into account
• Different ways to model importance: eigenvector centrality, betweenness centrality, closeness centrality and so on.

a). Eigenvector centrality

• A node $v$ is important if surrounded by important neighboring nodes $u\in N(v)$.
• $c_v=\frac{1}{\lambda }{\sum }_{u\in N(v)}{c}_u$, $\lambda$ is some positive constant.
• $\lambda c=Ac$, $A$ is the adjacency matrix $A_{uv}=1$ if $u\in N(v)$ and $c$ is centrality vector
• The largest eigenvector ${\lambda }_{max}$ is always positive and unique
• The leading eigenvector $c_{max}$ is used for centrality

b). Betweenness centrality

• A node is important if it lies on many shortest paths between other nodes
  $$c_v=\sum _{s\ne v\ne t}\frac{\#(\text{shortest paths between s and t that contain v})}{\#(\text{shortest paths between s and t})}$$

c). Closeness centrality

• A node is important if it has small shortest path lengths to all other nodes
  $$c_v=\sum _{u\ne v}\frac{1}{\#(\text{shortest paths length between u and v})}$$

4). Node Features: Clustering Coefficient

• Measures how connected the neighboring nodes of $v$ are
  $$e_v=\sum _{u\ne v}\frac{\#(\text{edges among neighboring nodes})}{{(}_2^{k_v})}\in [0,1]$$
  where ${(}_2^{k_v})$ is the number of node pairs among $k_v$ neighboring nodes

5). Node Features: Graphlets

Observation: clustering coefficient counts the #(triangles) in the ego-network
Idea: we can generalize the above by counting #(pre-specified subgraphs, i.e., graphlets)

• Graphlets: rooted connected non-isomorphic subgraphs.
• Graphlet Degree Vector (GDV): Graphlet-based features for nodes

Compare with node degree and clustering coefficient
Degree counts #(edges) that a node touches
Clustering coefficient counts #(triangles) that a node touches
GDV counts #(graphlets) that a node touches

• Graphlet Degree Vector (GDV): a count vector of graphlets rooted at a given node.
• Considering graphlets on 2 to 5 nodes we get vector of 73 coordinates is a signature of a node that describes the topology of its neighborhood and captures its interconnectivities out to a distance of 4 hops
• GDV provides a more detailed measure of a node’s local network topology than node degrees or clustering coefficient

4. Link-level Tasks

The task is to predict new links based on existing links.
When testing, all node pairs (no existing links) are ranked, and top $K$ node pairs are predicted.
The key is to design features for a pair of nodes
Two formulations

• Links missing at random: remove a random set of links and then aim to predict them
• Links over time: given $G[t_0,t_0^{{}^{\prime }}]$, a graph on edges up to time $t_0^{{}^{\prime }}$, output a ranked list $L$ of links (not in $G[t_0,t_0^{{}^{\prime }}]$) that are predicted to appear in $G[t_1,t_1^{{}^{\prime }}]$

Methodology: compute score $c(x,y)$ for each pair of nodes $(x,y)$, sort pairs $(x,y)$ by the decreasing score $c(x,y)$, predict top $n$ pairs as new links, see which of these links actually appear in $G[t_1,t_1^{{}^{\prime }}]$

1). Link-level Features: Overview

• Distance-based feature
• Local neighborhood overlap
• Global neighborhood overlap

2). Distance-based Feature

Shortest-path distance between two nodes. However, this does not capture the degree of neighborhood overlap.

3). Local Neighborhood Overlap

Captures the number of neighboring nodes shared between two nodes $v_1$ and $v_2$

• Common neighbors: $|N(v_1)\cap N(v_1)|$
• Jaccard’s coefficient: $\frac{|N(v_1)\cap N(v_1)|}{|N(v_1)\cup N(v_1)|}$
• Adamic-Adar index: ${\sum }_{u\in N(v_1)\cap N(v_1)}\frac{1}{log(k_u)}$
  Limitation: metric is always zero if the two nodes do not have any neighbors in common. However, the two nodes may still potentially be connected in the future.

4). Global Neighborhood Overlap

• Power of adjacency matrix: let $P_{uv}^{(K)}=$ #(paths of lenght $K$ between $u$ and $v$), then $P^{(K)}=A^K$
• Katz index: count the number of paths of all lengths between a given pair of nodes
  $$S_{uv}=\sum _{l=1}^{\infty }{\beta }^l{A}_{uv}^l$$
  $$S=\sum _{l=1}^{\infty }{\beta }^l{A}^l=(I-\beta A{)}^{-1}-I$$

5. Graph-level Features

1). Kernel Methods

Idea: design kernels instead of feature vectors

• Kernel $k(G,G^{{}^{\prime }})\in \mathtt{R}$ measures similarity between data
• Kernel matrix $\mathtt{K}=(k(G,G^{{}^{\prime }}){)}_{G,G^{{}^{\prime }}}$ must always be positive semidefinite (i.e., has positive eigenvalues)
• There exists a feature representation $\varphi$ such that $k(G,G^{{}^{\prime }})=\varphi (G{)}^T\varphi (G^{{}^{\prime }})$
• Once the kernel is defined, off-the-shelf ML model, such as kernel SVM, can be used to make predictions.

Graph kernels
Goal: design graph feature vector $\varphi (G)$
Key idea: Bag-of-Words (BoW)* for a graph (regard nodes as words)
*: BoW simply uses the word counts as features for documents

• Graphlet kernel
• Weisfeiler-Lehman kernel
• random-walk kernel, shortest-path graph kernel and many more

2). Graphlet Features

Key idea: count the number of graphlets in a graph
Note: Definition of graphlets here is different from node-level features.

• Nodes in graphlets here do not need to be connected
• The graphlets here are not rooted

Given a graph $G$ and a graphlet list $G_k=(g_1,g_2,...,g_n)$, define the graphlet count vector $f_G$ as
$$(f_G{)}_i=\#(g_i\subseteq G)\text{for i=1,2,...,n}$$
Given two graphs $G$ and $G^{{}^{\prime }}$, graphlet kernel is
$$k(G,G^{{}^{\prime }})=h_G^T{h}_G^{{}^{\prime }}$$
where $h_G=\frac{f_G}{sum(f_G)}$
Limitations: counting graphlets is expensive.

• Counting size-$k$ graphlets for a graph with size $n$ by enumeration takes $n^k$
• This is unavoidable in the worst-case since subgraph isomorphism test is NP-hard
• If a graph’s node degree is bounded by $d$, an $O(n{d}^{k-1})$ algorithm exists to count all the graphlets of size $k$

3). Weisfeiler-Lehman Kernel

Goal: design an efficient graph feature descriptor $\varphi (G)$
Idea: use neighborhood structure to iteratively enrich node vocabulary
Color refinement: given a graph $G$ with a set of nodes $V$, assign an initial color $c^0(v)$ to each ndoe $v$. Then iteratively refine node colors by
c k + 1 ( v ) = HASH ( { c k ( v ) , { c k ( u ) } u ∈ N ( v ) } ) c^{k+1}(v)=\text{HASH}(\{c^k(v), \{c^k(u)\}_{u\in N(v)}\}) ck+1(v)=HASH({ck(v),{ck(u)}u∈N(v)​})
where HASH maps different inputs to different colors. After $K$ steps of color refinement, $c^k(v)$ summarizes the structure of $K$-hop neighborhood
WL is computationally efficient
The time complexity for color reinement at each step is linear in #(degs), since it involves aggregating neighboring colors.