在Spark Graphx的org.apache.spark.graphx.lib
包中有一些常用的图算法,其中一个就是Connected Components,本文将会介绍此算法的使用方法,下面是spark 1.6.3源码中对这个算法的注释:
Compute the connected component membership of each vertex and return a graph with the vertex value containing the lowest vertex id in the connected component containing that vertex.
Demo
首先准备数据源
links.csv
1,2,friend
1,3,sister
2,4,brother
3,2,boss
4,5,client
1,9,friend
6,7,cousin
7,9,coworker
8,9,father
10,11,colleague
10,12,colleague
11,12,colleague
people.csv
4,Dave,25
6,Faith,21
8,Harvey,47
2,Bob,18
1,Alice,20
3,Charlie,30
7,George,34
9,Ivy,21
5,Eve,30
10,Lily,35
11,Helen,35
12,Ann,35
图结构
样例
package graphx
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.graphx._
import org.apache.spark.rdd.RDD
object ConnectedComponentsDemo {
case class Person(name:String,age:Int)
def main(args: Array[String]): Unit = {
val conf: SparkConf = new SparkConf().setAppName("SparkCoreTest").setMaster("local[*]")
val sc: SparkContext = new SparkContext(conf)
val peopleRDD: RDD[(VertexId, Person)] = sc.textFile("in/people.csv")
.map(_.split(","))
.map(x => (x(0).toLong, Person(x(1), x(2).toInt)))
val linksRDD: RDD[Edge[String]] = sc.textFile("in/links.csv")
.map(_.split(","))
.map(x => Edge(x(0).toLong, x(1).toLong, x(2)))
val tinySocial: Graph[Person, String] = Graph(peopleRDD,linksRDD)
val cc: Graph[VertexId, String] = tinySocial.connectedComponents()
cc.vertices.collect.foreach(println)
}
}
输出的结果:
(4,1)
(6,1)
(8,1)
(12,10)
(10,10)
(2,1)
(11,10)
(1,1)
(3,1)
(7,1)
(9,1)
(5,1)
从结果中可以看到通过计算之后的图,每个顶点多了一个属性,这个属性表示的就是这个顶点所在的连通图中的最小顶点id。例如顶点11所在的连通图中的最小顶点id是10,顶点4所在的连通图中的最小顶点id是1。
扩展
经过connectedComponents
得到的结果,可以知道哪些顶点在一个连通图中,这样就可以将一个大图拆分成若干个连通子图。
package graphx
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.graphx._
import org.apache.spark.rdd.RDD
object ConnectedComponentsDemo {
case class Person(name:String,age:Int)
def main(args: Array[String]): Unit = {
val conf: SparkConf = new SparkConf().setAppName("SparkCoreTest").setMaster("local[*]")
val sc: SparkContext = new SparkContext(conf)
val peopleRDD: RDD[(VertexId, Person)] = sc.textFile("in/people.csv")
.map(_.split(","))
.map(x => (x(0).toLong, Person(x(1), x(2).toInt)))
val linksRDD: RDD[Edge[String]] = sc.textFile("in/links.csv")
.map(_.split(","))
.map(x => Edge(x(0).toLong, x(1).toLong, x(2)))
val tinySocial: Graph[Person, String] = Graph(peopleRDD,linksRDD)
val cc: Graph[VertexId, String] = tinySocial.connectedComponents()
val newGraph: Graph[(VertexId, String, PartitionID), String] = cc.outerJoinVertices(peopleRDD)((id,mincc,people)=>(mincc,people.get.name,people.get.age))
cc.vertices.map(_._2).collect.distinct.foreach(id=>{
val sub=newGraph.subgraph(vpred=(id1,attr)=>attr._1==id)
sub.triplets.collect.foreach(println)
println()
})
}
}
结果得到了两个子图,输出为:
((1,(1,Alice,20)),(2,(1,Bob,18)),friend)
((1,(1,Alice,20)),(3,(1,Charlie,30)),sister)
((1,(1,Alice,20)),(9,(1,Ivy,21)),friend)
((2,(1,Bob,18)),(4,(1,Dave,25)),brother)
((3,(1,Charlie,30)),(2,(1,Bob,18)),boss)
((4,(1,Dave,25)),(5,(1,Eve,30)),client)
((6,(1,Faith,21)),(7,(1,George,34)),cousin)
((7,(1,George,34)),(9,(1,Ivy,21)),coworker)
((8,(1,Harvey,47)),(9,(1,Ivy,21)),father)
((10,(10,Lily,35)),(11,(10,Helen,35)),colleague)
((10,(10,Lily,35)),(12,(10,Ann,35)),colleague)
((11,(10,Helen,35)),(12,(10,Ann,35)),colleague)
分析
- 通过
connectedComponents
得到的新图的顶点属性已经没有了原始的那些信息,所以需要和原始信息作一个join
,例如val newGraph = cc.outerJoinVertices(peopleRDD)((id, cc, p)=>(cc,p.get.name,p.get.age))
cc.vertices.map(_._2).collect.distinct
会得到所有连通图中id最小的顶点编号- 通过连通图中最小顶点编号,使用
subgraph
方法得到每个连通子图