以wordCount程序分析job触发流程原理和源码

先粘上wordCount代码：

val lines = sc.textFile()
val words = lines.flatMap(line => line.split(" "))
val pairs = words.map(word => (word, 1))
val counts = pairs.reduceByKey(_ + _)

counts.foreach(count => println(count._1 + ": " + count._2))

一、textFile()

sparkContext.scala：

/**
 * Read a text file from HDFS, a local file system (available on all nodes), or any
 * Hadoop-supported file system URI, and return it as an RDD of Strings.
 */
def textFile(
    path: String,
    minPartitions: Int = defaultMinPartitions): RDD[String] = withScope {
  assertNotStopped()
  hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], classOf[Text],
    minPartitions).map(pair => pair._2.toString).setName(path)
}

def hadoopFile[K, V](
    path: String,
    inputFormatClass: Class[_ <: InputFormat[K, V]],
    keyClass: Class[K],
    valueClass: Class[V],
    minPartitions: Int = defaultMinPartitions): RDD[(K, V)] = withScope {
  assertNotStopped()

  // This is a hack to enforce loading hdfs-site.xml.
  // See SPARK-11227 for details.
  FileSystem.getLocal(hadoopConfiguration)

  // A Hadoop configuration can be about 10 KB, which is pretty big, so broadcast it.
  //对这个Hadoop配置信息进行广播
  val confBroadcast = broadcast(new SerializableConfiguration(hadoopConfiguration))
  val setInputPathsFunc = (jobConf: JobConf) => FileInputFormat.setInputPaths(jobConf, path)
  new HadoopRDD(
    this,
    confBroadcast,
    Some(setInputPathsFunc),
    inputFormatClass,
    keyClass,
    valueClass,
    minPartitions).setName(path)
}

首先，hadoopFile方法的调用，会new 一个HadoopRDD，其中的元素是（key，value）pair，key是hdfs或文本文件的每一行的offset，value是文本行。然后对HadoopRDD调用了map方法（这个方法是在RDD.scala里面），剔除key，最终返回一个MapPartitionsRDD

为什么会说每个partition是hdfs的block，因为从HadoopRDD的getPartitions方法中可以看出，底层是调用的hadoop的inputFormat的getSplits方法

每个RDD最重要的就是compute（）方法，通过这个方法拿到每个partition的元素

二、val words = lines.flatMap(line => line.split(" "))

flatMap方法又是创建一个MapPartitionsRDD

三、val pairs = words.map(word => (word, 1))

map方法也是创建一个MapPartitionsRDD

四、val counts = pairs.reduceByKey(_ + _)

在RDD中其实是找不到reduceByKey这个方法的，RDD中导入了隐式转换，implicit def

implicit def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
  (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null): PairRDDFunctions[K, V] = {
  new PairRDDFunctions(rdd)
}

其实RDD里是没有reduceByKey的，因此对RDD调用reduceByKey()方法的时候，会触发scala的隐式转换；此时就会在作用域内，寻找隐式转换，会在RDD中找到rddToPairRDDFunctions()隐式转换，然后将RDD转换为PairRDDFunctions。

接着会调用PairRDDFunctions中的reduceByKey()方法

这里会返回一个kv的普通RDD

五、counts.foreach(count => println(count._1 + ": " + count._2))

之前一直说一个action会触发一个job，看完这个代码就明白了。

foreach会调用runJob方法

def runJob[T, U: ClassTag](
    rdd: RDD[T],
    func: (TaskContext, Iterator[T]) => U,
    partitions: Seq[Int],
    resultHandler: (Int, U) => Unit): Unit = {
  if (stopped.get()) {
    throw new IllegalStateException("SparkContext has been shutdown")
  }
  val callSite = getCallSite
  val cleanedFunc = clean(func)
  logInfo("Starting job: " + callSite.shortForm)
  if (conf.getBoolean("spark.logLineage", false)) {
    logInfo("RDD's recursive dependencies:\n" + rdd.toDebugString)
  }
  dagScheduler.runJob(rdd, cleanedFunc, partitions, callSite, resultHandler, localProperties.get)
  progressBar.foreach(_.finishAll())
  rdd.doCheckpoint()
}

在这个方法中调用了DAGScheduler的runJob方法。

后面再继续学习DAGScheduler的源码和运行机制。