Spark SQL 程序调度执行框架（一）

通过 spark-sqlshell脚本执行 SparkSubmit 主函数

// Spark 程序启动
object SparkSubmit : main(args) 
    0 = "--class"
    1 = "org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver"
    2 = "spark-internal"
class SparkSubmit : doSubmit(args)  
class SparkSubmit : def submit()
class SparkSubmit : def doRunMain() 
class SparkSubmit : private def runMain(...) 
class JavaMainApplication : def start() --> public static void org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(java.lang.String[])

// Spark SQL 程序启动
main:274, SparkSQLCLIDriver$
processLine:376, CliDriver
processCmd:371, SparkSQLCLIDriver
run:62, SparkSQLDriver

SQL 语句的解析和执行

// sql 解析处理
markInAnalyzer:201, AnalysisHelper$ (org.apache.spark.sql.catalyst.plans.logical)
executeAndCheck:105, Analyzer (org.apache.spark.sql.catalyst.analysis)
analyzed$lzycompute:57, QueryExecution (org.apache.spark.sql.execution)
analyzed:55, QueryExecution (org.apache.spark.sql.execution)
assertAnalyzed:47, QueryExecution (org.apache.spark.sql.execution)
ofRows:78, Dataset$ (org.apache.spark.sql)
sql:642, SparkSession (org.apache.spark.sql)
sql:694, SQLContext (org.apache.spark.sql)

QueryPlan 翻译为RDD 执行过程

  // 1. 通过 (XX)Exec的 execute() 方法获取RDD，包装了函数执行的真正入口 doExecute()
  /**
   * Returns the result of this query as an RDD[InternalRow] by delegating to `doExecute` after
   * preparations.
   *
   * Concrete implementations of SparkPlan should override `doExecute`.
   */
  final def execute(): RDD[InternalRow] = executeQuery {
    if (isCanonicalizedPlan) {
      throw new IllegalStateException("A canonicalized plan is not supposed to be executed.")
    }
    doExecute()
  }

  // case class ShuffleExchangeExec 
  // 2. 以 ShuffleExchangeExec 为例，doExecute() 方法也只是获取一个 cachedShuffleRDD 的依赖关系而已
  // 在 ShuffleExchangeExec 中 cachedShuffleRDD
  protected override def doExecute(): RDD[InternalRow] = attachTree(this, "execute") {
    // Returns the same ShuffleRowRDD if this plan is used by multiple plans.
    if (cachedShuffleRDD == null) {
      val shuffleDependency = prepareShuffleDependency()
      cachedShuffleRDD = preparePostShuffleRDD(shuffleDependency)
    }
    cachedShuffleRDD
  }

  // 
  /**
   * Returns a [[ShuffleDependency]] that will partition rows of its child based on
   * the partitioning scheme defined in `newPartitioning`. Those partitions of
   * the returned ShuffleDependency will be the input of shuffle.
   */
  private[exchange] def prepareShuffleDependency()
    : ShuffleDependency[Int, InternalRow, InternalRow] = {
    ShuffleExchangeExec.prepareShuffleDependency(
      child.execute(), child.output, newPartitioning, serializer)
  }



  // case class ProjectExec(projectList: Seq[NamedExpression], child: SparkPlan)
  //  extends UnaryExecNode with CodegenSupport {
  override def inputRDDs(): Seq[RDD[InternalRow]] = {
    child.asInstanceOf[CodegenSupport].inputRDDs()
  }

  // case class InputAdapter(child: SparkPlan) extends UnaryExecNode with CodegenSupport
  override def inputRDDs(): Seq[RDD[InternalRow]] = {
    child.execute() :: Nil
  }

Job 和 Stage 提交执行过程

NodeManager 启动Executor

Shell启动脚本

NodeManager
  default_container_executor.sh  
    bash -c 'java *.*.CoarseGrainedExecutorBackend' --> 启动Executor 接收task计算任务
    bash -c 'java *.*.ExecutorLauncher' --> 这里应该是直接奔着启动 ApplicationMaster 去了

启动脚本样例：

启动脚本：/yarn/nm/usercache/hadoop/appcache/application_1557744110775_5172/container_e06_1557744110775_5172_01_000003/launch_container.sh

启动Java 进程：
exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms4096m -Xmx4096m '-Xdebug' '-Xnoagent' '-Djava.compiler=NONE' '-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=2346' -Djava.io.tmpdir=$PWD/tmp '-Dspark.authenticate.enableSaslEncryption=false' '-Dspark.authenticate=false' '-Dspark.driver.port=39563' '-Dspark.shuffle.service.port=7337' -Dspark.yarn.app.container.log.dir=/yarn/container-logs/application_1557744110775_5172/container_e06_1557744110775_5172_01_000003 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.59.34.203:39563 --executor-id 2 --hostname host-10-59-34-204 --cores 1 --app-id application_1557744110775_5172 --user-class-path file:$PWD/__app__.jar 1>/yarn/container-logs/application_1557744110775_5172/container_e06_1557744110775_5172_01_000003/stdout 2>/yarn/container-logs/application_1557744110775_5172/container_e06_1557744110775_5172_01_000003/stderr"

CoarseGrainedExecutorBackend 启动 RPC EndPoint

    // CoarseGrainedExecutorBackend
    
    // main()
    // run() 启动executor rpc Endpint
      env.rpcEnv.setupEndpoint("Executor", new CoarseGrainedExecutorBackend(
        env.rpcEnv, driverUrl, executorId, sparkHostPort, cores, userClassPath, env))

  // CoarseGrainedExecutorBackend 通过rpc接收任务管理的调用
  override def receive: PartialFunction[Any, Unit] = {
    case RegisteredExecutor(hostname) =>
      logInfo("Successfully registered with driver")
      try {
        executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)
      } catch {
        case NonFatal(e) =>
          exitExecutor(1, "Unable to create executor due to " + e.getMessage, e)
      }

    case RegisterExecutorFailed(message) =>
      exitExecutor(1, "Slave registration failed: " + message)

    case LaunchTask(data) =>
      if (executor == null) {
        exitExecutor(1, "Received LaunchTask command but executor was null")
      } else {
        val taskDesc = ser.deserialize[TaskDescription](data.value)
        logInfo("Got assigned task " + taskDesc.taskId)
        // 这里根据接收到的参数，实例化一个TaskRunner(Runnable)对象，再使用线程池提交执行
        executor.launchTask(this, taskId = taskDesc.taskId, attemptNumber = taskDesc.attemptNumber,
          taskDesc.name, taskDesc.serializedTask)
      }

    case KillTask(taskId, _, interruptThread) =>
      if (executor == null) {
        exitExecutor(1, "Received KillTask command but executor was null")
      } else {
        executor.killTask(taskId, interruptThread)
      }

    case StopExecutor =>
      stopping.set(true)
      logInfo("Driver commanded a shutdown")
      // Cannot shutdown here because an ack may need to be sent back to the caller. So send
      // a message to self to actually do the shutdown.
      self.send(Shutdown)

    case Shutdown =>
      stopping.set(true)
      executor.stop()
      stop()
      rpcEnv.shutdown()
  }

在 Executor 中，

• TaskRunner.run(Executor.scala:242)
• Task.run()
• Task.runTask(context) 执行的实现类为 ResultTask

Task的反序列化和执行

  override def runTask(context: TaskContext): U = {
    // Deserialize the RDD and the func using the broadcast variables.
    val deserializeStartTime = System.currentTimeMillis()
    val ser = SparkEnv.get.closureSerializer.newInstance()
    // 补充
    val (rdd, func) = ser.deserialize[(RDD[T], (TaskContext, Iterator[T]) => U)](
      ByteBuffer.wrap(taskBinary.value), Thread.currentThread.getContextClassLoader)
    _executorDeserializeTime = System.currentTimeMillis() - deserializeStartTime

    metrics = Some(context.taskMetrics)
    func(context, rdd.iterator(partition, context))
  }

先看下 taskBinary 的注释说明 : broadcasted version of the serialized RDD and the function to apply on each partition of the given RDD. Once deserialized, the type should be (RDD[T], (TaskContext, Iterator[T]) => U).

• rdd
  看一下 ser.deserialize() 返回的结果(rdd,func)
• rdd : 比较好理解，就是要计算集合的抽象
• rdd.iterator(partition, context) : 返回当前rdd在当前partition上的迭代器，如果依赖的rdd不存在，需要级联 compute() 调用
• func : 应该是先调用迭代器取值，再依次调用func函数进行计算，具体Func函数内容应该是map，groupByKey 这些函数… // TODO

参考文档

• Spark 任务调度之启动CoarseGrainedExecutorBackend