1.take
获取前num条记录。
def take(num: Int): Array[T] = withScope {
if (num == 0) {
new Array[T](0) } else {
val buf = newArrayBuffer[T] val totalParts = this.partitions.length var partsScanned = 0 while (buf.size < num && partsScanned < totalParts) {
// The number of partitions to try in this iteration. It is ok for this number to be // greater than totalParts because we actually cap it at totalParts in runJob. var numPartsToTry =1 if (partsScanned > 0) {
// If we didn't find any rows after the previous iteration, quadruple and retry. // Otherwise, interpolate the number of partitions we need to try, but overestimate // it by 50%. We also cap the estimation in the end. if (buf.size ==0) {//截止目前为止buf为空的话,则扩大4倍范围 numPartsToTry = partsScanned * 4 } else {
//截止目前为止还有部分值没取到的话,则扩大至Math.max((1.5 * num * partsScanned / buf.size).toInt - partsScanned, 1),但是不超过当前已扫描过分区的4倍 // the left side of max is >=1 whenever partsScanned >= 2 numPartsToTry = Math.max((1.5* num * partsScanned / buf.size).toInt - partsScanned, 1) numPartsToTry = Math.min(numPartsToTry, partsScanned * 4) } } val left = num - buf.size val p = partsScanned until math.min(partsScanned + numPartsToTry, totalParts) val res = sc.runJob(this, (it:Iterator[T]) => it.take(left).toArray, p, allowLocal =true) res.foreach(buf ++= _.take(num - buf.size)) partsScanned += numPartsToTry } buf.toArray } } |
首先关注下sc.runJob函数的传参:
/** * Run a job on a given set of partitions of an RDD, but take a function of type * `Iterator[T] => U` instead of `(TaskContext, Iterator[T]) => U`. */ def runJob[T,U: ClassTag]( rdd: RDD[T], func: Iterator[T] =>U, partitions: Seq[Int], allowLocal: Boolean ): Array[U] = {
val cleanedFunc = clean(func) runJob(rdd, (ctx: TaskContext, it: Iterator[T]) => cleanedFunc(it), partitions, allowLocal) } |
其中partitions: Seq[Int]代表需要计算的分区,可以计算某个分区,也可以计算多个分区,是待计算的分区集合。
其次先看第一次循环,其partsScanned为0,numPartsToTry为1,因此先计算第一个分区的结果,如果第一次计算可以取得满足条件的num个值,则循环结束,如果取不到满足条件的num个值,则扩大第二次计算的分区范围,很可能一下子扫多个分区。
其执行过程见下图:
![](https://img-blog.csdn.net/20170302094825177?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvd2wwNDQwOTA0MzI=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center)
Take可以避免全量计算,执行时间比较短。但可能会多次触发action。
2.first
取RDD的第一个元素
/** * Return the first element in this RDD. */ def first(): T = withScope {
take(1) match {
case Array(t) => t case _ => throw newUnsupportedOperationException("empty collection") } } |
其实就是调用take来完成的,take的流程可以查阅take函数详解
3.sortByKey
def sortByKey(ascending: Boolean =true, numPartitions: Int = self.partitions.length) : RDD[(K, V)] = self.withScope {
val part = newRangePartitioner(numPartitions, self, ascending) new ShuffledRDD[K,V, V](self, part) .setKeyOrdering(if (ascending)ordering else |