Spark算子：RDD基本转换操作(3)–randomSplit、glom

2019独角兽企业重金招聘Python工程师标准>>>

randomSplit

def randomSplit(weights: Array[Double], seed: Long = Utils.random.nextLong): Array[RDD[T]]

该函数根据weights权重，将一个RDD切分成多个RDD。

该权重参数为一个Double数组

第二个参数为random的种子，基本可忽略。

def main(args: Array[String]): Unit = {
  //默认分区12个
  val sc = new SparkContext(new SparkConf().setMaster("local").setAppName("test").set("spark.default.parallelism", "12"))
  var rdd1 = sc.makeRDD(1 to 10, 2)
  var rdd2 = rdd1.randomSplit(Array(1,2,3))
  println("RDD size:" + rdd2.size)
  rdd2.foreach(_.collect.foreach(println(_)))
}

16/12/20 10:51:49 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.56.1, 63792)
RDD size:3
16/12/20 10:51:50 INFO SparkContext: Starting job: collect at ShellTest.scala:26
16/12/20 10:51:50 INFO DAGScheduler: Got job 0 (collect at ShellTest.scala:26) with 2 output partitions

... ...

2
6
10
16/12/20 10:51:50 INFO SparkContext: Starting job: collect at ShellTest.scala:26
... ...
1
9
16/12/20 10:51:51 INFO SparkContext: Starting job: collect at ShellTest.scala:26
.... ....
3
4
5
7
8
16/12/20 10:51:51 INFO SparkContext: Invoking stop() from shutdown hook
16/12/20 10:51:51 INFO SparkUI: Stopped Spark web UI at http://192.168.56.1:4040

 

glom

def glom(): RDD[Array[T]]

该函数是将RDD中每一个分区中类型为T的元素转换成Array[T]，这样每一个分区就只有一个数组元素。

1. scala> var rdd = sc.makeRDD(1 to 10,3)
2. rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[38] at makeRDD at :21
3. scala> rdd.partitions.size
4. res33: Int = 3 //该RDD有3个分区
5. scala> rdd.glom().collect
6. res35: Array[Array[Int]] = Array(Array(1, 2, 3), Array(4, 5, 6), Array(7, 8, 9, 10))
7. //glom将每个分区中的元素放到一个数组中，这样，结果就变成了3个数组
8.  

 

转载于:https://my.oschina.net/chensanti234/blog/808545