randomSplit
def randomSplit(weights: Array[Double], seed: Long = Utils.random.nextLong): Array[RDD[T]]
该函数根据weights权重,将一个RDD切分成多个RDD。
该权重参数为一个Double数组
第二个参数为random的种子,基本可忽略。
def main(args: Array[String]): Unit = { //默认分区12个 val sc = new SparkContext(new SparkConf().setMaster("local").setAppName("test").set("spark.default.parallelism", "12")) var rdd1 = sc.makeRDD(1 to 10, 2) var rdd2 = rdd1.randomSplit(Array(1,2,3)) println("RDD size:" + rdd2.size) rdd2.foreach(_.collect.foreach(println(_))) }
16/12/20 10:51:49 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.56.1, 63792)
RDD size:3
16/12/20 10:51:50 INFO SparkContext: Starting job: collect at ShellTest.scala:26
16/12/20 10:51:50 INFO DAGScheduler: Got job 0 (collect at ShellTest.scala:26) with 2 output partitions
... ...
2
6
10
16/12/20 10:51:50 INFO SparkContext: Starting job: collect at ShellTest.scala:26
... ...
1
9
16/12/20 10:51:51 INFO SparkContext: Starting job: collect at ShellTest.scala:26
.... ....
3
4
5
7
8
16/12/20 10:51:51 INFO SparkContext: Invoking stop() from shutdown hook
16/12/20 10:51:51 INFO SparkUI: Stopped Spark web UI at http://192.168.56.1:4040
glom
def glom(): RDD[Array[T]]
该函数是将RDD中每一个分区中类型为T的元素转换成Array[T],这样每一个分区就只有一个数组元素。
- scala> var rdd = sc.makeRDD(1 to 10,3)
- rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[38] at makeRDD at :21
- scala> rdd.partitions.size
- res33: Int = 3 //该RDD有3个分区
- scala> rdd.glom().collect
- res35: Array[Array[Int]] = Array(Array(1, 2, 3), Array(4, 5, 6), Array(7, 8, 9, 10))
- //glom将每个分区中的元素放到一个数组中,这样,结果就变成了3个数组