spark2.11-2.3源码7_job工作原理

最新推荐文章于 2021-08-26 10:52:08 发布

caibaoli

最新推荐文章于 2021-08-26 10:52:08 发布

阅读量94

点赞数

分类专栏：实习第4天实习第四天之spark源码学习文章标签： spark job wordcount 源码 textFile

本文链接：https://blog.csdn.net/u011607686/article/details/86556656

版权

实习第4天同时被 2 个专栏收录

89 篇文章 0 订阅

订阅专栏

实习第四天之spark源码学习

13 篇文章 0 订阅

订阅专栏

例如wordcount程序SparkContent中textFile(生成rdd开始)

//方法的调用会创建一个会创建hadoopRDD，其中的元素，其实是(key,value) pair
//key是hdfs或文本文件的每一行的offset，value就是文本行，然后对HadoopRDD调用map()方法，会剔除key,只保留value,然后会获得一个MapPartitionsRDD,MapPartitionsRDD内部的元素，其实就是一行一行的文本行
def textFile(
      path: String,
      minPartitions: Int = defaultMinPartitions): RDD[String] = withScope {
    assertNotStopped()
    //创建hadoopRDD再map(去掉hadopp本身带的行号)
    hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], classOf[Text],
      minPartitions).map(pair => pair._2.toString).setName(path)
  }

hadoopFile

def hadoopFile[K, V](
      path: String,
      inputFormatClass: Class[_ <: InputFormat[K, V]],
      keyClass: Class[K],
      valueClass: Class[V],
      minPartitions: Int = defaultMinPartitions): RDD[(K, V)] = withScope {
    assertNotStopped()

    // This is a hack to enforce loading hdfs-site.xml.
    // See SPARK-11227 for details.
    FileSystem.getLocal(hadoopConfiguration)

    // A Hadoop configuration can be about 10 KB, which is pretty big, so broadcast it.
    //序列化hadoop配置文件再广播
    val confBroadcast = broadcast(new SerializableConfiguration(hadoopConfiguration))
    //输入格式化
    val setInputPathsFunc = (jobConf: JobConf) => FileInputFormat.setInputPaths(jobConf, path)
    new HadoopRDD(
      this,
      confBroadcast,
      Some(setInputPathsFunc),
      inputFormatClass,
      keyClass,
      valueClass,
      minPartitions).setName(path)
  }

注：RDD中没有reduceByKey，因此对RDD调用reduceByKey()方法的时候，会触发scala的隐式转换；此时就会在作用域内，寻找隐式转换。例如会在RDD中找到rddToPairRDDFunctions()隐式转换，然后将RDD转换为PairRDDFunctions,接着会调用PairRDDFunctions中的reduceByKey()方法。

runJob

def runJob[T, U: ClassTag](
      rdd: RDD[T],
      func: (TaskContext, Iterator[T]) => U,
      partitions: Seq[Int],
      resultHandler: (Int, U) => Unit): Unit = {
    if (stopped.get()) {
      throw new IllegalStateException("SparkContext has been shutdown")
    }
    val callSite = getCallSite
    val cleanedFunc = clean(func)
    logInfo("Starting job: " + callSite.shortForm)
    if (conf.getBoolean("spark.logLineage", false)) {
      logInfo("RDD's recursive dependencies:\n" + rdd.toDebugString)
    }
    //调用SparkContext,之前初始化时创建的DAGScheduler的runJob()方法
    //rdd最后一个rdd，进行action操作
    dagScheduler.runJob(rdd, cleanedFunc, partitions, callSite, resultHandler, localProperties.get)
    progressBar.foreach(_.finishAll())
    rdd.doCheckpoint()
  }