val conf = new SparkConf().setMaster("local[*]").setAppName("colseTest")
val sc = new SparkContext(conf)
val rdd1 = sc.parallelize(Seq("1", "2", "3"))
//如果重分区的数目大于原来的分区数,那么必须指定shuffle参数为tru
//从源码可以看出,repartition()方法就是coalesce()方法shuffle为true的情况。
val rdd2 = rdd1.coalesce(8, true)
val rdd3 = rdd1.repartition(8)
println("RDD2:" + rdd2.partitions.size)
println("RDD3:" + rdd3.partitions.size)
打印结果:
RDD2:8
RDD3:8
另外:
coalesce 在spark SQL function中有类似 oracle中nvl2点作用,如下解释
* For example, `coalesce(a, b, c)` will return a if a is not null,
* or b if a is null and b is not null, or c if both a and b are null but c is not null.
如 coalesce(null,null,4) 结果返回4
val df = Seq(
("1", "ll"),
("2", "ll"),
(null, "lk")
).toDF("id", "name")
res.select(coalesce($"id", lit(9))).show() //动态SQL表达
res.selectExpr("coalesce(id,'9')").show() //静态SQL表达