Spark算子执行流程详解之二

4.count

def count(): Long = sc.runJob(this, Utils.getIteratorSize_).sum

计算数据总量，每个分区各自计算自己的总数，然后汇总到driver端，driver端再把每个分区的总数相加统计出对应rdd的数据量，其流程如下：

 

5.countApprox

在一定的超时时间之内返回rdd元素的个数，其rdd元素的总数分布符合正态分布，其分布因子为confidence，当超过timeout时，返回一个未完成的结果。

/**  * :: Experimental ::  * Approximate version of count() that returns a potentially incomplete result  * within a timeout, even if not all tasks have finished.  */ @Experimental def countApprox(     timeout: Long,     confidence: Double = 0.95): PartialResult[BoundedDouble] = withScope { //定义在excutor端计算总数的函数   val countElements: (TaskContext, Iterator[T]) => Long = { (ctx, iter) =>     var result = 0L     while (iter.hasNext) {       result += 1L       iter.next()     }     result   } //定义在driver端的一个监听回调函数，当task完成的时候，会触发里面的merge操作，当超时时间到之后或者任务提前完成的话，会取里面的当前状态，即currentResult   val evaluator = newCountEvaluator(partitions.length, confidence) //提交任务   sc.runApproximateJob(this, countElements, evaluator, timeout) }

继续往下看，看看evaluator是如何执行的：

def runApproximateJob[T,U, R](     rdd: RDD[T],     func: (TaskContext, Iterator[T]) =>U,     evaluator: ApproximateEvaluator[U, R],     timeout: Long): PartialResult[R] = {   assertNotStopped()   val callSite = getCallSite   logInfo("Starting job: " + callSite.shortForm)   val start = System.nanoTime   val cleanedFunc = clean(func) // cleanedFunc就是countElements，evaluator就是CountEvaluator，超时时间为timeout   val result = dagScheduler.runApproximateJob(rdd, cleanedFunc, evaluator, callSite, timeout,     localProperties.get)   logInfo(     "Job finished: " + callSite.shortForm +", took " + (System.nanoTime- start) / 1e9 + " s")   result }

继续看runApproximateJob的实现：

def runApproximateJob[T,U,