GraphX之Connected Components

在Spark Graphx的org.apache.spark.graphx.lib包中有一些常用的图算法，其中一个就是Connected Components，本文将会介绍此算法的使用方法，下面是spark 1.6.3源码中对这个算法的注释：

Compute the connected component membership of each vertex and return a graph with the vertex value containing the lowest vertex id in the connected component containing that vertex.

Demo

首先准备数据源

links.csv

1,2,friend
1,3,sister
2,4,brother
3,2,boss
4,5,client
1,9,friend
6,7,cousin
7,9,coworker
8,9,father
10,11,colleague
10,12,colleague
11,12,colleague

people.csv

4,Dave,25
6,Faith,21
8,Harvey,47
2,Bob,18
1,Alice,20
3,Charlie,30
7,George,34
9,Ivy,21
5,Eve,30
10,Lily,35
11,Helen,35
12,Ann,35

图结构

样例

package graphx
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.graphx._
import org.apache.spark.rdd.RDD
object ConnectedComponentsDemo {
  case class Person(name:String,age:Int)
  def main(args: Array[String]): Unit = {
    val conf: SparkConf = new SparkConf().setAppName("SparkCoreTest").setMaster("local[*]")
    val sc: SparkContext = new SparkContext(conf)

    val peopleRDD: RDD[(VertexId, Person)] = sc.textFile("in/people.csv")
      .map(_.split(","))
      .map(x => (x(0).toLong, Person(x(1), x(2).toInt)))

    val linksRDD: RDD[Edge[String]] = sc.textFile("in/links.csv")
      .map(_.split(","))
      .map(x => Edge(x(0).toLong, x(1).toLong, x(2)))

    val tinySocial: Graph[Person, String] = Graph(peopleRDD,linksRDD)

    val cc: Graph[VertexId, String] = tinySocial.connectedComponents()

    cc.vertices.collect.foreach(println)

  }
}

输出的结果：

(4,1)
(6,1)
(8,1)
(12,10)
(10,10)
(2,1)
(11,10)
(1,1)
(3,1)
(7,1)
(9,1)
(5,1)

从结果中可以看到通过计算之后的图，每个顶点多了一个属性，这个属性表示的就是这个顶点所在的连通图中的最小顶点id。例如顶点11所在的连通图中的最小顶点id是10，顶点4所在的连通图中的最小顶点id是1。

扩展

经过connectedComponents得到的结果，可以知道哪些顶点在一个连通图中，这样就可以将一个大图拆分成若干个连通子图。

package graphx
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.graphx._
import org.apache.spark.rdd.RDD
object ConnectedComponentsDemo {
  case class Person(name:String,age:Int)
  def main(args: Array[String]): Unit = {
    val conf: SparkConf = new SparkConf().setAppName("SparkCoreTest").setMaster("local[*]")
    val sc: SparkContext = new SparkContext(conf)

    val peopleRDD: RDD[(VertexId, Person)] = sc.textFile("in/people.csv")
      .map(_.split(","))
      .map(x => (x(0).toLong, Person(x(1), x(2).toInt)))

    val linksRDD: RDD[Edge[String]] = sc.textFile("in/links.csv")
      .map(_.split(","))
      .map(x => Edge(x(0).toLong, x(1).toLong, x(2)))

    val tinySocial: Graph[Person, String] = Graph(peopleRDD,linksRDD)

    val cc: Graph[VertexId, String] = tinySocial.connectedComponents()

    val newGraph: Graph[(VertexId, String, PartitionID), String] = cc.outerJoinVertices(peopleRDD)((id,mincc,people)=>(mincc,people.get.name,people.get.age))

    cc.vertices.map(_._2).collect.distinct.foreach(id=>{
      val sub=newGraph.subgraph(vpred=(id1,attr)=>attr._1==id)
      sub.triplets.collect.foreach(println)
      println()
    })
  }
}

结果得到了两个子图，输出为：

((1,(1,Alice,20)),(2,(1,Bob,18)),friend)
((1,(1,Alice,20)),(3,(1,Charlie,30)),sister)
((1,(1,Alice,20)),(9,(1,Ivy,21)),friend)
((2,(1,Bob,18)),(4,(1,Dave,25)),brother)
((3,(1,Charlie,30)),(2,(1,Bob,18)),boss)
((4,(1,Dave,25)),(5,(1,Eve,30)),client)
((6,(1,Faith,21)),(7,(1,George,34)),cousin)
((7,(1,George,34)),(9,(1,Ivy,21)),coworker)
((8,(1,Harvey,47)),(9,(1,Ivy,21)),father)

((10,(10,Lily,35)),(11,(10,Helen,35)),colleague)
((10,(10,Lily,35)),(12,(10,Ann,35)),colleague)
((11,(10,Helen,35)),(12,(10,Ann,35)),colleague)

分析

1. 通过connectedComponents得到的新图的顶点属性已经没有了原始的那些信息，所以需要和原始信息作一个join，例如val newGraph = cc.outerJoinVertices(peopleRDD)((id, cc, p)=>(cc,p.get.name,p.get.age))
2. cc.vertices.map(_._2).collect.distinct会得到所有连通图中id最小的顶点编号
3. 通过连通图中最小顶点编号，使用subgraph方法得到每个连通子图