Spark组件之GraphX学习8--邻居集合

最新推荐文章于 2022-12-23 10:27:58 发布

KeepLearningBigData

最新推荐文章于 2022-12-23 10:27:58 发布

阅读量2.8k

点赞数

分类专栏： spark 文章标签：邻居集合

本文链接：https://blog.csdn.net/xubo245/article/details/51308337

版权

spark 专栏收录该内容

70 篇文章 27 订阅

订阅专栏

更多代码请见：https://github.com/xubo245/SparkLearning

1解释

收集每个顶点的邻居顶点

2.代码：

/**
 * @author xubo
 * ref http://spark.apache.org/docs/1.5.2/graphx-programming-guide.html
 * time 20160503
 */

package org.apache.spark.graphx.learning

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.graphx.Graph
import org.apache.spark.graphx.VertexId
import org.apache.spark.graphx.VertexRDD
import org.apache.spark.graphx.util.GraphGenerators
import org.jets3t.apps.synchronize.Synchronize
import breeze.linalg.reverse
import breeze.linalg.reverse
import org.apache.spark.graphx.EdgeDirection

object CollectingNeighbors {

  val K = 3
  var arr = new Array[(Int, Int)](K)
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("CollectingNeighbors").setMaster("local[4]")
    // Assume the SparkContext has already been constructed
    val sc = new SparkContext(conf)

    // Import random graph generation library
    // Create a graph with "age" as the vertex property.  Here we use a random graph for simplicity.
    val graph: Graph[Double, Int] =
      GraphGenerators.logNormalGraph(sc, numVertices = 6).mapVertices((id, _) => id.toDouble)
    // Compute the number of older followers and their total age

    println("Graph:");
    println("sc.defaultParallelism:" + sc.defaultParallelism);
    println("vertices:");
    graph.vertices.collect.foreach(println(_))
    println("edges:");
    graph.edges.collect.foreach(println(_))
    println("count:" + graph.edges.count);
    println("\ninDegrees");
    graph.inDegrees.foreach(println)

    println("\nneighbors0:");
    val neighbors0 = graph.collectNeighborIds(EdgeDirection.Out)
    neighbors0.foreach(println)
    neighbors0.collect.foreach { a =>
      {
        println(a._1 + ":")
        a._2.foreach(b => print(b + " "))
        println();
      }
    }

  }
}

代码解释：

每个顶点的出边对应的顶点

入度类推

3.结果：

Graph:
sc.defaultParallelism:4
vertices:
(4,4.0)
(0,0.0)
(1,1.0)
(5,5.0)
(2,2.0)
(3,3.0)
edges:
Edge(0,0,1)
Edge(0,3,1)
Edge(1,2,1)
Edge(1,3,1)
Edge(1,4,1)
Edge(1,4,1)
Edge(1,5,1)
Edge(2,1,1)
Edge(2,5,1)
Edge(3,1,1)
Edge(3,4,1)
Edge(4,2,1)
Edge(4,3,1)
Edge(4,4,1)
Edge(5,0,1)
Edge(5,2,1)
Edge(5,4,1)
Edge(5,4,1)
Edge(5,5,1)
count:19

inDegrees
(3,3)
(4,6)
(0,2)
(2,3)
(1,2)
(5,3)

neighbors0:
(1,[J@781c1417)
(5,[J@3a02a86a)
(2,[J@7b6b2f9b)
(3,[J@4fba6bc2)
(4,[J@42e18b85)
(0,[J@658eeb10)
4:
2 3 4 
0:
0 3 
1:
2 3 4 4 5 
5:
0 2 4 4 5 
2:
1 5 
3:
1 4

参考

【1】 http://spark.apache.org/docs/1.5.2/graphx-programming-guide.html

【2】https://github.com/xubo245/SparkLearning