spark启动流程

最新推荐文章于 2024-08-28 11:10:11 发布

原创最新推荐文章于 2024-08-28 11:10:11 发布 · 978 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #大数据 #hadoop

spark 专栏收录该内容

1 篇文章

订阅专栏

本文详细介绍了Spark应用程序在YARN-cluster模式下的启动流程，从用户提交应用程序开始，涵盖SparkContext的初始化、executor资源申请、executor的启动与注册、任务调度等关键步骤，旨在帮助读者深入理解Spark在大数据环境中的工作原理。

spark概览

Spark应用程序在群集上作为独立的进程集运行,可以部署在不同环境的集群上，包括(yarn,mesos,standalone,etc…).

由driver中的SparkContext对象协调,这涉及到大概以下环节

在这里插入图片描述

用户client端提交应用程序，根据不同环境启动不同的client端。
Driver端执行用户代码,sparkContext初始化
用户代码中job提交，RDD划分statge，生成task
executor资源申请
executor拉起,并向Driver端注册
Driver端收到注册后任务下发
executor端执行

yarn-cluster模式下流程概览(概览~~ )

用户client端提交应用程序

示例代码

#示例代码
/bin/spark-submit --class org.apache.spark.examples.SparkPi \
    --master yarn \
    --deploy-mode cluster \
    --driver-memory 4g \
    --executor-memory 2g \
    --executor-cores 1 \
    --queue thequeue \
    examples/jars/spark-examples*.jar \
    10

spark-submit脚本中，直接拉起SparkSubmit程序。SparkSubmit主要是将参数解析，决定具体launch environment。

//参数解析
val (childArgs, childClasspath, sysProps, childMainClass) = prepareSubmitEnvironment(args)

//call launch environment
runMain(childArgs, childClasspath, sysProps, childMainClass, args.verbose)

yarn-cliuster模式下childMainClass如下

if (isYarnCluster) {
      childMainClass = "org.apache.spark.deploy.yarn.Client"
}

yarn.Client通过yarn API形式提交程序(Driver在AM中)。其主要是做了构建客户端，生成AMContainerSpec，提交，监控及report。在生成AMContainerSpec的代码里，可以看到AM mainclass为ApplicationMaster(省略部分代码)

private def createContainerLaunchContext(newAppResponse: GetNewApplicationResponse)
    : ContainerLaunchContext = {
    logInfo("Setting up container launch context for our AM")
    //略
    val amClass =
      if (isClusterMode) {
        Utils.classForName("org.apache.spark.deploy.yarn.ApplicationMaster").getName
      } else {
        Utils.classForName("org.apache.spark.deploy.yarn.ExecutorLauncher").getName
      }
    //略
  }

sparkContext初始化

yarn-cluster模式下，driver端在AM中初始化。直接看ApplicationMaster的rundriver(省略部分代码)

startUserApplication即是执行用户提交的代码，初始化sparkcontext，runjob.

private def runDriver(securityMgr: SecurityManager): Unit = {
    addAmIpFilter()
    userClassThread = startUserApplication()

    // This a bit hacky, but we need to wait until the spark.driver.port property has
    // been set by the Thread executing the user class.
    logInfo("Waiting for spark context initialization...")
    val totalWaitTime = sparkConf.get(AM_MAX_WAIT_TIME)
    try {
      val sc = ThreadUtils.awaitResult(sparkContextPromise.future,
        Duration(totalWaitTime, TimeUnit.MILLISECONDS))
      if (sc != null) {
        rpcEnv = sc.env.rpcEnv
        val driverRef = runAMEndpoint(
          sc.getConf.get("spark.driver.host"),
          sc.getConf.get("spark.driver.port"),
          isClusterMode = true)
        registerAM(sc.getConf, rpcEnv, driverRef, sc.ui.map(_.appUIAddress).getOrElse(""),
          securityMgr)
      }
      userClassThread.join()
    }
  }

上一步sparkcontext初始化完成，运行了执行代码，但是在yarn上是如何分配资源并launch executor呢。这主要是由不同的TaskScheduler和SchedulerBackend去实现的。在yarn-cluster中，TaskScheduler为YarnClusterScheduler，SchedulerBackend为YarnClusterSchedulerBackend

override def createTaskScheduler(sc: SparkContext, masterURL: String): TaskScheduler = {
    sc.deployMode match {
      case "cluster" => new YarnClusterScheduler(sc)
      case "client" => new YarnScheduler(sc)
      case _ => throw new SparkException(s"Unknown deploy mode '${sc.deployMode}' for Yarn")
    }
  }

  override def createSchedulerBackend(sc: SparkContext,
      masterURL: String,
      scheduler: TaskScheduler): SchedulerBackend = {
    sc.deployMode match {
      case "cluster" =>
        new YarnClusterSchedulerBackend(scheduler.asInstanceOf[TaskSchedulerImpl], sc)
      case "client" =>
        new YarnClientSchedulerBackend(scheduler.asInstanceOf[TaskSchedulerImpl], sc)
      case  _ =>
        throw new SparkException(s"Unknown deploy mode '${sc.deployMode}' for Yarn")
    }
  }

资源申请

yarn-cluster总的资源是预先分配好的.资源申请及executor的launch由driver端sparkContext初始化完成后registerAM实现。

private def registerAM(
      _sparkConf: SparkConf,
      _rpcEnv: RpcEnv,
      driverRef: RpcEndpointRef,
      uiAddress: String,
      securityMgr: SecurityManager) = {
    //省略部分代码
    allocator = client.register(driverUrl,
      driverRef,
      yarnConf,
      _sparkConf,
      uiAddress,
      historyAddress,
      securityMgr,
      localResources)

    allocator.allocateResources()//申请资源，并launch
    reporterThread = launchReporterThread()
}

allocateResources执行了:构建resourcerequest，申请container，launch executor

def allocateResources(): Unit = synchronized {
    //省略部分代码
    updateResourceRequests()

    val progressIndicator = 0.1f
    val allocateResponse = amClient.allocate(progressIndicator)

    val allocatedContainers = allocateResponse.getAllocatedContainers()
    handleAllocatedContainers(allocatedContainers.asScala)
    }
  }

//最终会执行，ExecutorRunnable::run 拉起container
//在executor container执行的主类是CoarseGrainedExecutorBackend
private def prepareCommand(): List[String] = {
val commands = prefixEnv ++ Seq(
      YarnSparkHadoopUtil.expandEnvironment(Environment.JAVA_HOME) + "/bin/java",
      "-server") ++
      javaOpts ++
      Seq("org.apache.spark.executor.CoarseGrainedExecutorBackend",
        "--driver-url", masterAddress,
        "--executor-id", executorId,
        "--hostname", hostname,
        "--cores", executorCores.toString,
        "--app-id", appId) ++
      userClassPath ++
      Seq(
        s"1>${ApplicationConstants.LOG_DIR_EXPANSION_VAR}/stdout",
        s"2>${ApplicationConstants.LOG_DIR_EXPANSION_VAR}/stderr")GrainedExecutorBackend
}

executor注册

executor启动后，会向driver注册RegisterExecutor消息

override def onStart() {
    logInfo("Connecting to driver: " + driverUrl)
    rpcEnv.asyncSetupEndpointRefByURI(driverUrl).flatMap { ref =>
      // This is a very fast action so we can use "ThreadUtils.sameThread"
      driver = Some(ref)
      ref.ask[Boolean](RegisterExecutor(executorId, self, hostname, cores, extractLogUrls))
    }(ThreadUtils.sameThread).onComplete {
      // This is a very fast action so we can use "ThreadUtils.sameThread"
      case Success(msg) =>
        // Always receive `true`. Just ignore it
      case Failure(e) =>
        exitExecutor(1, s"Cannot register with driver: $driverUrl", e, notifyDriver = false)
    }(ThreadUtils.sameThread)
  }

driver端CoarseGrainedSchedulerBackend收到注册后进行makeoffer派发任务。

override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {

  case RegisterExecutor(executorId, executorRef, hostname, cores, logUrls) =>
      //代码略
      logInfo(s"Registered executor $executorRef ($executorAddress) with ID $executorId")
      addressToExecutorId(executorAddress) = executorId
      totalCoreCount.addAndGet(cores)
      totalRegisteredExecutors.addAndGet(1)
      val data = new ExecutorData(executorRef, executorRef.address, hostname,
        cores, cores, logUrls)
      // This must be synchronized because variables mutated
      // in this block are read when requesting executors
      CoarseGrainedSchedulerBackend.this.synchronized {
        executorDataMap.put(executorId, data)
        if (currentExecutorIdCounter < executorId.toInt) {
          currentExecutorIdCounter = executorId.toInt
        }
        if (numPendingExecutors > 0) {
          numPendingExecutors -= 1
          logDebug(s"Decremented number of pending executors ($numPendingExecutors left)")
        }
 
      executorRef.send(RegisteredExecutor)
      // Note: some tests expect the reply to come after we put the executor in the map
      context.reply(true)
      listenerBus.post(
        SparkListenerExecutorAdded(System.currentTimeMillis(), executorId, data))
      makeOffers() //task分发执行
    }

Executor端执行

override def receive: PartialFunction[Any, Unit] = {
    case LaunchTask(data) =>
      if (executor == null) {
        exitExecutor(1, "Received LaunchTask command but executor was null")
      } else {
        val taskDesc = ser.deserialize[TaskDescription](data.value)
        logInfo("Got assigned task " + taskDesc.taskId)
        executor.launchTask(this, taskId = taskDesc.taskId, attemptNumber = taskDesc.attemptNumber,
          taskDesc.name, taskDesc.serializedTask)
      }
}