Spark函数讲解：coalesce

函数原型

1	def coalesce(numPartitions: Int, shuffle: Boolean = false)

2	(implicit ord: Ordering[T] = null): RDD[T]

　　返回一个新的RDD，且该RDD的分区个数等于numPartitions个数。如果shuffle设置为true，则会进行shuffle。

实例

01

/**

02	* User: 过往记忆

03	* Date: 15-03-09

04	* Time: 上午06:30

05	* bolg: http://www.iteblog.com

06	* 本文地址：http://www.iteblog.com/archives/1279

07	* 过往记忆博客，专注于hadoop、hive、spark、shark、flume的技术博客，大量的干货

08	* 过往记忆博客微信公共帐号：iteblog_hadoop

09

*/

10	scala> var data = sc.parallelize(List(1,2,3,4))

11	data: org.apache.spark.rdd.RDD[Int] =

12	ParallelCollectionRDD[45] at parallelize at <console>:12

13

14	scala> data.partitions.length

15	res68: Int = 30

16

17	scala> val result = data.coalesce(2, false)

18	result: org.apache.spark.rdd.RDD[Int] = CoalescedRDD[57] at coalesce at <console>:14

19

20	scala> result.partitions.length

21	res77: Int = 2

22

23	scala> result.toDebugString

24	res75: String =

25	(2) CoalescedRDD[57] at coalesce at <console>:14 []

26	|  ParallelCollectionRDD[45] at parallelize at <console>:12 []

27

28	scala> val result1 = data.coalesce(2, true)

29	result1: org.apache.spark.rdd.RDD[Int] = MappedRDD[61] at coalesce at <console>:14

30

31	scala> result1.toDebugString

32	res76: String =

33	(2) MappedRDD[61] at coalesce at <console>:14 []

34	|  CoalescedRDD[60] at coalesce at <console>:14 []

35	|  ShuffledRDD[59] at coalesce at <console>:14 []

36	+-(30) MapPartitionsRDD[58] at coalesce at <console>:14 []

37	|   ParallelCollectionRDD[45] at parallelize at <console>:12 []

　　从上面可以看出shuffle为false的时候并不进行shuffle操作；而为true的时候会进行shuffle操作。RDD.partitions.length可以获取相关RDD的分区数。