spark yarn-cluster 提交流程记录

最新推荐文章于 2021-12-07 16:03:36 发布

beetle_lzk

最新推荐文章于 2021-12-07 16:03:36 发布

阅读量394

点赞数

分类专栏： spark源码学习记录

本文链接：https://blog.csdn.net/lixiaoksi/article/details/114091281

版权

spark源码学习记录专栏收录该内容

3 篇文章 0 订阅

订阅专栏

spark yarn-cluster应用提交过程跟踪

闲扯

依稀还记得当初被领导问着想做实时，还是做离线，我毅然选择了这个名字自带闪光点的spark。
后来出差做了实时项目，也算是入了spark的门，带着无限的骄傲和自豪能对做离线的同事指手画脚，现在看来，当初的做法未免有点幼稚，不过在那storm盛行flink还未兴起的时代，spark独霸实时离线处理两大领域，能够用它做项目那简直是技术人员口中茶余饭后吹牛皮耀眼的谈资。
时过境迁，spark已不再独领风骚一枝独秀，flink的崛起撼动了spark的地位，但是正因为这样的更新换代，技术才不会停滞不前，思想才不会落后。
最近又捞起放下已久的spark，一方面是为了学习，另一方面也是为了找工作能够和技术官多扯两句，证明工作几年来还是有点沉淀，同时也践行着一边输出，一边学习的学习方法，输出便是最好的总结。
加油吧，技术人。

首先贴出 spark-submit 执行命令：

spark-submit --master yarn \
			 --deploy-mode cluster \
			 --executor-memory 1G \
			 --executor-cores 1 \
			 --class org.apache.spark.examples.SparkPi \
			 /opt/module/spark/examples/jars/spark-examples_2.11-2.2.0.jar 10000

一、spark执行入口类

1. spark-submit

话不多说，直入主题。

首先来看提交命令，该命令执行了spark-sumbit，找到bin/目录下的spark-submit
最后一段代码执行了 :
exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"
意思是将 org.apache.spark.deploy.SparkSubmit 和 “$@” 传给了 spark-class 这个脚本
其中 $@ 表示在执行 spark-submit 时传入的所有参数

2. spark-class

接着我们去bin目录下找 spark-class，一大段的命令解释略过，挑主要的内容看
exec "${CMD[@]}"

过程不再分析，将该命令打印出来以后为以下信息

/opt/module/jdk1.8.0_144/bin/java -cp /opt/module/spark/conf/:/opt/module/spark/jars/*:/opt/module/hadoop-2.7.7/etc/hadoop/ 
org.apache.spark.deploy.SparkSubmit 
	--master yarn 
	--deploy-mode cluster 
	--executor-memory 1G
	--executor-cores 1 
	--class org.apache.spark.examples.SparkPi 
	/opt/module/spark/examples/jars/spark-examples_2.11-2.2.0.jar 10000

可以看出最终执行了 org.apache.spark.deploy.SparkSubmit 这个类，既然知道了该类为入口类，double shift 在idea中搜索该类

此时，spark执行流程跟踪已告别了简单的脚本加工环节，进入到真正的源码环节了。

二、主要执行类

1. org.apache.spark.deploy.SparkSubmit

来到 SparkSubmit 类中，寻找main方法，

def main() ·····> case SparkSubmitAction.SUBMIT => submit(appArgs)
def private def runMain() ·····> mainMethod.invoke(null, childArgs.toArray)

该反射执行的是 org.apache.spark.deploy.yarn.Client 这个类

模式匹配进入到 submit(appArgs) 方法中，在该方法中，归根结底地调用了 doRunMain() 这个方法，这个方法里又归根结底地调用了 runMain() 这个方法，方法太长就不贴详细代码了，只贴出关键部分信息，如下：

private def runMain(）
	·····> mainClass = Utils.classForName(childMainClass)
	·····> val mainMethod = mainClass.getMethod("main", new Array[String](0).getClass)
	·····> mainMethod.invoke(null, childArgs.toArray)

依旧使用反射来执行 childMainClass 这个类，但是这个参数又是什么呢？
继续查看代码，发现该参数是从 prepareSubmitEnvironment(args) 这个方法中获取到的。
因为我们使用的是 yarn-cluster 模式提交的任务，自然 childMainClass = "org.apache.spark.deploy.yarn.Client"

    if (isYarnCluster) {
      childMainClass = "org.apache.spark.deploy.yarn.Client"
      if (args.isPython) {
        childArgs += ("--primary-py-file", args.primaryResource)
        childArgs += ("--class", "org.apache.spark.deploy.PythonRunner")
      } else if (args.isR) {
        val mainFile = new Path(args.primaryResource).getName
        childArgs += ("--primary-r-file", mainFile)
        childArgs += ("--class", "org.apache.spark.deploy.RRunner")
      } else {
        if (args.primaryResource != SparkLauncher.NO_RESOURCE) {
          childArgs += ("--jar", args.primaryResource)
        }
        childArgs += ("--class", args.mainClass)
      }
      if (args.childArgs != null) {
        args.childArgs.foreach { arg => childArgs += ("--arg", arg) }
      }
    }

流程似乎变得清晰起来，我们跳到 org.apache.spark.deploy.yarn.Client 这个类中。double shift搜索不到该类，这时需要引入 spark-yarn 的pom依赖并下载源码才能查看的到。

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-yarn_2.11</artifactId>
      <version>2.2.0</version>
    </dependency>

2. org.apache.spark.deploy.yarn.Client

通过包名可以看出，这部分内容应该跟yarn扯上了关系，至于是什么关系，我们接着看代码

  def main(argStrings: Array[String]) {
    if (!sys.props.contains("SPARK_SUBMIT")) {
      logWarning("WARNING: This client is deprecated and will be removed in a " +
        "future version of Spark. Use ./bin/spark-submit with \"--master yarn\"")
    }

    // Set an env variable indicating we are running in YARN mode.
    // Note that any env variable with the SPARK_ prefix gets propagated to all (remote) processes
    System.setProperty("SPARK_YARN_MODE", "true")
    val sparkConf = new SparkConf
    // SparkSubmit would use yarn cache to distribute files & jars in yarn mode,
    // so remove them from sparkConf here for yarn mode.
    //因为使用到了yarn模式，使用yarn本身的资源文件
    sparkConf.remove("spark.jars")
    sparkConf.remove("spark.files")
    val args = new ClientArguments(argStrings)
    new Client(args, sparkConf).run()
  }

代码最后一段 new 了一个 Client 对象，并调用了 run 方法。这里用到了yarn，yarn的简介见文末附录
在run方法中

def run() ·····> this.appId = submitApplication()

进入到 submitApplication() 方法中

  def submitApplication(): ApplicationId = {
    var appId: ApplicationId = null
    try {
      launcherBackend.connect()
      // Setup the credentials before doing anything else,
      // so we have don't have issues at any point.
      setupCredentials()
      yarnClient.init(yarnConf)
      yarnClient.start()

      logInfo("Requesting a new application from cluster with %d NodeManagers"
        .format(yarnClient.getYarnClusterMetrics.getNumNodeManagers))

      // 通过向ResourceManager提交执行任务申请，返回了一个应用封装，包含了appId
      // 获取到这个appId以后，然后才能真正的执行应用
      val newApp = yarnClient.createApplication()
      val newAppResponse = newApp.getNewApplicationResponse()
      appId = newAppResponse.getApplicationId()
      
	  //设置与hadoop通讯的客户端
      new CallerContext("CLIENT", sparkConf.get(APP_CALLER_CONTEXT),
        Option(appId.toString)).setCurrentContext()

      // 验证集群是否有足够的资源供我们使用
      verifyClusterResources(newAppResponse)

      // 为ApplicationMaster的运行设置上下文环境
      val containerContext = createContainerLaunchContext(newAppResponse)
      val appContext = createApplicationSubmissionContext(newApp, containerContext)

      logInfo(s"Submitting application $appId to ResourceManager")
      yarnClient.submitApplication(appContext)
      launcherBackend.setAppId(appId.toString)
      reportLauncherState(SparkAppHandle.State.SUBMITTED)

      appId
    } catch {
      case e: Throwable =>
        if (appId != null) {
          cleanupStagingDir(appId)
        }
        throw e
    }
  }

当前代码执行过程中，以 YarnClient 的身份向 ResourceManager 申请提交任务。在createContainerLaunchContext(newAppResponse) 方法中,command 参数中封装了所要执行的命令，并将这个 command 设置到 amContainer: ContainerLaunchContext 中并返回该Container，这个Container即是启动 ApplicationMaster 的 Container。但此时并未启动amContainer。

private def createContainerLaunchContext(newAppResponse: GetNewApplicationResponse){
......
	val amClass =
	      if (isClusterMode) {
	        Utils.classForName("org.apache.spark.deploy.yarn.ApplicationMaster").getName
	      } else {
	        Utils.classForName("org.apache.spark.deploy.yarn.ExecutorLauncher").getName
	      }
		......
	val commands = prefixEnv ++
	      Seq(Environment.JAVA_HOME.$$() + "/bin/java", "-server") ++
	      javaOpts ++ amArgs ++
	      Seq(
	        "1>", ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout",
	        "2>", ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr")
	
	    // TODO: it would be nicer to just make sure there are no null commands here
	    val printableCommands = commands.map(s => if (s == null) "null" else s).toList
	    amContainer.setCommands(printableCommands.asJava)
	    ......
	        val securityManager = new SecurityManager(sparkConf)
	    amContainer.setApplicationACLs(
	      YarnSparkHadoopUtil.getApplicationAclsForYarn(securityManager).asJava)
	    setupSecurityToken(amContainer)
	    amContainer
}

可以看到，针对不同的模式，amClass启动执行的类也可以不同。这里 amClass = “org.apache.spark.deploy.yarn.ApplicationMaster”。
既然设置好了提交启动 ApplicationMaster的Container 的上下文，这时候应该设置创建ApplicationMaster 的上下文对象了

def submitApplication() ·····> def createApplicationSubmissionContext()

参数和环境都准备好了，紧接着提交该请求

def submitApplication() ·····> yarnClient.submitApplication(appContext)

在这里发现又执行了提交 application 的动作，回顾之前的那次提交和这一次的提交。大概情形如下：
第一次是 yarnClient 向 ResourceManager 提出申请，告诉它我有任务要执行啦。
ResourceManager 收到申请以后回复说，好嘞，我给你个号码(applicationId)，你先拿着这个号码规划着你需要用到哪些东西，完了以后把这些东西统一set到 amContainer 里，接下来就不用管啦。
一切准备妥当以后，yarnClient 就执行了第二次真正的任务提交。

yarnClient 向 ResourceManager 提交请求，ResourceManager 启动一个Container来启动ApplicationMaster。只有当成功提交到yarn上之后才会返回之前已经拿到的appId，否则会抛出异常。此时，提交任务已创建，接下来怎么执行，我们就需要切入到 org.apache.spark.deploy.yarn.ApplicationMaster 这个类中去查看了

3. org.apache.spark.deploy.yarn.ApplicationMaster

老规矩，main方法走起。

def main(args: Array[String]) 
	·····> master = new ApplicationMaster(amArgs, new YarnRMClient)
	·····> System.exit(master.run())

在main方法中new了一个 ApplicationMaster 对象，并执行了run方法，具体内容我们进入到run方法中

final def run()
......
      if (isClusterMode) {
        runDriver(securityMgr)
      } else {
        runExecutorLauncher(securityMgr)
      }
......

可以看到，根据不同的模式此处的ApplicationMaster执行了不同的方法，也意味着启动了不同的服务，顾名思义 runDriver() ，好像表达的意思是运行Driver，那这里所说的 Driver 是不是跟我们经常说的Driver、Executor是一样的吗，这里的话我们来看一下 runDriver() 这个方法

  private def runDriver(securityMgr: SecurityManager): Unit = {
    addAmIpFilter()
    //启动用户线程(Driver)，startUserApplication()里面已经调用了start方法
    userClassThread = startUserApplication()

    // This a bit hacky, but we need to wait until the spark.driver.port property has
    // been set by the Thread executing the user class.
    logInfo("Waiting for spark context initialization...")
    val totalWaitTime = sparkConf.get(AM_MAX_WAIT_TIME)
    try {
      //等待获取 Driver 中创建的 sparkContext对象
      val sc = ThreadUtils.awaitResult(sparkContextPromise.future,
        Duration(totalWaitTime, TimeUnit.MILLISECONDS))
      //如果返回的sparkContext对象不为空，则向ApplicationMaster注册  
      if (sc != null) {
        rpcEnv = sc.env.rpcEnv
        val driverRef = runAMEndpoint(
          sc.getConf.get("spark.driver.host"),
          sc.getConf.get("spark.driver.port"),
          isClusterMode = true)
        registerAM(sc.getConf, rpcEnv, driverRef, sc.ui.map(_.webUrl), securityMgr)
      } else {
        // Sanity check; should never happen in normal operation, since sc should only be null
        // if the user app did not create a SparkContext.
        if (!finished) {
          throw new IllegalStateException("SparkContext is null but app is still running!")
        }
      }
      userClassThread.join()
    } catch {
      case e: SparkException if e.getCause().isInstanceOf[TimeoutException] =>
        logError(
          s"SparkContext did not initialize after waiting for $totalWaitTime ms. " +
           "Please check earlier log output for errors. Failing the application.")
        finish(FinalApplicationStatus.FAILED,
          ApplicationMaster.EXIT_SC_NOT_INITED,
          "Timed out waiting for SparkContext.")
    }
  }

首先执行了 startUserApplication() 这个方法，紧接着执行了registerAM()方法，之后将执行userClassThread.join()这个方法，他的作用是阻塞主线程并等待 userClassThread 这个线程执行完毕
来看 startUserApplication()方法

  private def startUserApplication(): Thread = {
    logInfo("Starting the user application in a separate Thread")

    val classpath = Client.getUserClasspath(sparkConf)
    val urls = classpath.map { entry =>
      new URL("file:" + new File(entry.getPath()).getAbsolutePath())
    }
    val userClassLoader =
      if (Client.isUserClassPathFirst(sparkConf, isDriver = true)) {
        new ChildFirstURLClassLoader(urls, Utils.getContextOrSparkClassLoader)
      } else {
        new MutableURLClassLoader(urls, Utils.getContextOrSparkClassLoader)
      }

    var userArgs = args.userArgs
    if (args.primaryPyFile != null && args.primaryPyFile.endsWith(".py")) {
      // When running pyspark, the app is run using PythonRunner. The second argument is the list
      // of files to add to PYTHONPATH, which Client.scala already handles, so it's empty.
      userArgs = Seq(args.primaryPyFile, "") ++ userArgs
    }
    if (args.primaryRFile != null && args.primaryRFile.endsWith(".R")) {
      // TODO(davies): add R dependencies here
    }
    val mainMethod = userClassLoader.loadClass(args.userClass)
      .getMethod("main", classOf[Array[String]])

    val userThread = new Thread {
      override def run() {
        try {
          mainMethod.invoke(null, userArgs.toArray)
          finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)
          logDebug("Done running users class")
        } catch {
          case e: InvocationTargetException =>
            e.getCause match {
              case _: InterruptedException =>
                // Reporter thread can interrupt to stop user class
              case SparkUserAppException(exitCode) =>
                val msg = s"User application exited with status $exitCode"
                logError(msg)
                finish(FinalApplicationStatus.FAILED, exitCode, msg)
              case cause: Throwable =>
                logError("User class threw exception: " + cause, cause)
                finish(FinalApplicationStatus.FAILED,
                  ApplicationMaster.EXIT_EXCEPTION_USER_CLASS,
                  "User class threw exception: " + cause)
            }
            sparkContextPromise.tryFailure(e.getCause())
        } finally {
          // Notify the thread waiting for the SparkContext, in case the application did not
          // instantiate one. This will do nothing when the user code instantiates a SparkContext
          // (with the correct master), or when the user code throws an exception (due to the
          // tryFailure above).
          sparkContextPromise.trySuccess(null)
        }
      }
    }
    userThread.setContextClassLoader(userClassLoader)
    userThread.setName("Driver")
    userThread.start()
    userThread
  }

通过反射的方式启动一个线程执行了 userClass 这个类，这个类就是我们的用户类spark任务 "org.apache.spark.examples.SparkPi" 并且可以看到，将该线程的name参数设置为"Driver"，是不是很熟悉。这也验证了 yarn-cluster 模式下在 Driver运行在 ApplicationMaster 中这一说法。
当然，光启动了Driver可不行，接着来看后续又做了什么操作

private def registerAM(sc.getConf, rpcEnv, driverRef, sc.ui.map(_.webUrl), securityMgr){
......
    allocator = client.register(driverUrl,
      driverRef,
      yarnConf,
      _sparkConf,
      uiAddress,
      historyAddress,
      securityMgr,
      localResources)

    allocator.allocateResources()
    reporterThread = launchReporterThread()
}

  def register(
      driverUrl: String,
      driverRef: RpcEndpointRef,
      conf: YarnConfiguration,
      sparkConf: SparkConf,
      uiAddress: Option[String],
      uiHistoryAddress: String,
      securityMgr: SecurityManager,
      localResources: Map[String, LocalResource]
    ): YarnAllocator = {
    amClient = AMRMClient.createAMRMClient()
    amClient.init(conf)
    amClient.start()
    this.uiHistoryAddress = uiHistoryAddress

    val trackingUrl = uiAddress.getOrElse {
      if (sparkConf.get(ALLOW_HISTORY_SERVER_TRACKING_URL)) uiHistoryAddress else ""
    }

    logInfo("Registering the ApplicationMaster")
    synchronized {
      amClient.registerApplicationMaster(Utils.localHostName(), 0, trackingUrl)
      registered = true
    }
    new YarnAllocator(driverUrl, driverRef, conf, sparkConf, amClient, getAttemptId(), securityMgr,
      localResources, new SparkRackResolver())
  }

当 Driver 启动完成之后，通过从 conf 中获取 Driver 的终端引用 driverRef ，并执行 registerAM() 方法
该方法主要有两个功能：
1：向 ResourceManger 注册 ApplicationMaster
2：获取 YarnAllocator 对象，负责向RM请求容器资源
在 allocator.allocateResources() 方法中，获取分配的容器列表
·····> val allocateResponse = amClient.allocate(progressIndicator)
·····> val allocatedContainers : List[Container] = allocateResponse.getAllocatedContainers()

4. org.apache.spark.deploy.yarn.YarnAllocator

def handleAllocatedContainers(allocatedContainers: Seq[Container])
此方法中实现了节点本地化，机架本地化策略
接着调用了 runAllocatedContainers(containersToUse)

  private def runAllocatedContainers(containersToUse: ArrayBuffer[Container]): Unit = {
    for (container <- containersToUse) {
      executorIdCounter += 1
      val executorHostname = container.getNodeId.getHost
      val containerId = container.getId
      val executorId = executorIdCounter.toString
      assert(container.getResource.getMemory >= resource.getMemory)
      logInfo(s"Launching container $containerId on host $executorHostname " +
        s"for executor with ID $executorId")

      def updateInternalState(): Unit = synchronized {
        numExecutorsRunning += 1
        executorIdToContainer(executorId) = container
        containerIdToExecutorId(container.getId) = executorId

        val containerSet = allocatedHostToContainersMap.getOrElseUpdate(executorHostname,
          new HashSet[ContainerId])
        containerSet += containerId
        allocatedContainerToHostMap.put(containerId, executorHostname)
      }

      if (numExecutorsRunning < targetNumExecutors) {
        if (launchContainers) {
          launcherPool.execute(new Runnable {
            override def run(): Unit = {
              try {
                new ExecutorRunnable(
                  Some(container),
                  conf,
                  sparkConf,
                  driverUrl,
                  executorId,
                  executorHostname,
                  executorMemory,
                  executorCores,
                  appAttemptId.getApplicationId.toString,
                  securityMgr,
                  localResources
                ).run()
                updateInternalState()
              } catch {
                case NonFatal(e) =>
                  logError(s"Failed to launch executor $executorId on container $containerId", e)
                  // Assigned container should be released immediately to avoid unnecessary resource
                  // occupation.
                  amClient.releaseAssignedContainer(containerId)
              }
            }
          })
        } else {
          // For test only
          updateInternalState()
        }
      } else {
        logInfo(("Skip launching executorRunnable as runnning Excecutors count: %d " +
          "reached target Executors count: %d.").format(numExecutorsRunning, targetNumExecutors))
      }
    }
  }

进入到run()方法中，有一个startContainer()方法，继续进入。

5. org.apache.spark.deploy.yarn.ExecutorRunnable

由类名可以知道，该类处理的是Executor相关的操作

  def run(): Unit = {
    logDebug("Starting Executor Container")
    nmClient = NMClient.createNMClient()
    nmClient.init(conf)
    nmClient.start()
    startContainer()
  }

  def startContainer(): java.util.Map[String, ByteBuffer] = {
	......
	 val commands = prepareCommand()
	 ctx.setCommands(commands.asJava)
	......
	nmClient.startContainer(container.get, ctx)
	......
 }

private def prepareCommand(): List[String] = {
	......
	val commands = prefixEnv ++
      Seq(Environment.JAVA_HOME.$$() + "/bin/java", "-server") ++
      javaOpts ++
      Seq("org.apache.spark.executor.CoarseGrainedExecutorBackend",
        "--driver-url", masterAddress,
        "--executor-id", executorId,
        "--hostname", hostname,
        "--cores", executorCores.toString,
        "--app-id", appId) ++
      userClassPath ++
      Seq(
        s"1>${ApplicationConstants.LOG_DIR_EXPANSION_VAR}/stdout",
        s"2>${ApplicationConstants.LOG_DIR_EXPANSION_VAR}/stderr")
        ......
}

经过一通组装，启动 executor 的方式和启动 ApplicationMaster 的方式如出一辙，都是通过封装command参数来执行目标类 "org.apache.spark.executor.CoarseGrainedExecutorBackend"。貌似好像大概似乎和executor有点关系了哈。
接下来我们进入到该类中继续跟踪。

6. org.apache.spark.executor.CoarseGrainedExecutorBackend

依然main方法走起

def main(args: Array[String]) {
......
run(driverUrl, executorId, hostname, cores, appId, workerUrl, userClassPath)
......
}

private def run(
      driverUrl: String,
      executorId: String,
      hostname: String,
      cores: Int,
      appId: String,
      workerUrl: Option[String],
      userClassPath: Seq[URL]) {
	......
	//创建Executor的执行环境
    val env = SparkEnv.createExecutorEnv(
      driverConf, executorId, hostname, port, cores, cfg.ioEncryptionKey, isLocal = false)
      env.rpcEnv.setupEndpoint("Executor", new CoarseGrainedExecutorBackend(
        env.rpcEnv, driverUrl, executorId, hostname, cores, userClassPath, env))
      workerUrl.foreach { url =>
        env.rpcEnv.setupEndpoint("WorkerWatcher", new WorkerWatcher(env.rpcEnv, url))
      }
      env.rpcEnv.awaitTermination()
	......
}

通过给定的一系列参数创建 ExecutorEnv，具体实现类是 NettyRpcEnv，并创建名为 Executor 的通信终端。
在 setupEndpoint() 方法中，会在 dispatcher 中注册 RpcEndpoint 。
Dispatcher是一个消息分发器，负责将消息分发给适合的Endpoint。

7. org.apache.spark.rpc.netty.Dispatcher

在 registerRpcEndpoint() 方法中，实例化 EndpointData 的过程中会维护一个 Inbox 用于接收消息，同时会放入第一个消息 OnStart 。从这也可以看出，只要注册了一个Endpoint，就会同时存在一个专门用来接收消息的Inbox。
org.apache.spark.rpc.netty.Inbox
inbox.synchronized {messages.add(OnStart) }

def registerRpcEndpoint(name: String, endpoint: RpcEndpoint): NettyRpcEndpointRef = {
    val addr = RpcEndpointAddress(nettyEnv.address, name)
    val endpointRef = new NettyRpcEndpointRef(nettyEnv.conf, addr, nettyEnv)
    synchronized {
      if (stopped) {
        throw new IllegalStateException("RpcEnv has been stopped")
      }
      if (endpoints.putIfAbsent(name, new EndpointData(name, endpoint, endpointRef)) != null) {
        throw new IllegalArgumentException(s"There is already an RpcEndpoint called $name")
      }
      val data = endpoints.get(name)
      endpointRefs.put(data.endpoint, data.ref)
      receivers.offer(data)  // for the OnStart message
    }
    endpointRef
  }

既然已经向 CoarseGrainedExecutorBackend 这个终端发送了消息，那我们来看看是怎么处理的把，ctrl + f 搜索到 OnStart 的用处，进入到 CoarseGrainedExecutorBackend 类中的重写方法 onStart() 中

  override def onStart() {
    logInfo("Connecting to driver: " + driverUrl)
    rpcEnv.asyncSetupEndpointRefByURI(driverUrl).flatMap { ref =>
      // This is a very fast action so we can use "ThreadUtils.sameThread"
      driver = Some(ref)
      ref.ask[Boolean](RegisterExecutor(executorId, self, hostname, cores, extractLogUrls))
    }(ThreadUtils.sameThread).onComplete {
      // This is a very fast action so we can use "ThreadUtils.sameThread"
      case Success(msg) =>
        // Always receive `true`. Just ignore it
      case Failure(e) =>
        exitExecutor(1, s"Cannot register with driver: $driverUrl", e, notifyDriver = false)
    }(ThreadUtils.sameThread)
  }

从代码可以看出，拿到了 Driver 的 RpcEndpointRef 后，会向 Driver 发送 RegisterExecutor 消息，这一步也被称之为反向注册，即 Executor 向 Driver 端注册。既然是通过DriverRef发送的信息，那么接收方肯定是 Driver 咯，在通信终端 DriverEndpoint 中寻找处理的消息的方法，可以看到在 org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend 文件中的内部类 DriverEndpoint 中的 receiveAndReply() 方法里

override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {

      case RegisterExecutor(executorId, executorRef, hostname, cores, logUrls) =>
        if (executorDataMap.contains(executorId)) {
          executorRef.send(RegisterExecutorFailed("Duplicate executor ID: " + executorId))
          context.reply(true)
        } else if (scheduler.nodeBlacklist != null &&
          scheduler.nodeBlacklist.contains(hostname)) {
          // If the cluster manager gives us an executor on a blacklisted node (because it
          // already started allocating those resources before we informed it of our blacklist,
          // or if it ignored our blacklist), then we reject that executor immediately.
          logInfo(s"Rejecting $executorId as it has been blacklisted.")
          executorRef.send(RegisterExecutorFailed(s"Executor is blacklisted: $executorId"))
          context.reply(true)
        } else {
          // If the executor's rpc env is not listening for incoming connections, `hostPort`
          // will be null, and the client connection should be used to contact the executor.
          val executorAddress = if (executorRef.address != null) {
              executorRef.address
            } else {
              context.senderAddress
            }
          logInfo(s"Registered executor $executorRef ($executorAddress) with ID $executorId")
          addressToExecutorId(executorAddress) = executorId
          totalCoreCount.addAndGet(cores)
          totalRegisteredExecutors.addAndGet(1)
          val data = new ExecutorData(executorRef, executorAddress, hostname,
            cores, cores, logUrls)
          // This must be synchronized because variables mutated
          // in this block are read when requesting executors
          CoarseGrainedSchedulerBackend.this.synchronized {
            executorDataMap.put(executorId, data)
            if (currentExecutorIdCounter < executorId.toInt) {
              currentExecutorIdCounter = executorId.toInt
            }
            if (numPendingExecutors > 0) {
              numPendingExecutors -= 1
              logDebug(s"Decremented number of pending executors ($numPendingExecutors left)")
            }
          }
          executorRef.send(RegisteredExecutor)
          // Note: some tests expect the reply to come after we put the executor in the map
          context.reply(true)
          listenerBus.post(
            SparkListenerExecutorAdded(System.currentTimeMillis(), executorId, data))
          makeOffers()
        }
        ......

DriverEndpoint 接收到消息以后，会判断Executor是否重复注册，如果重复注册，则发送注册失败消息。如果注册成功，则记录Executor的信息并累加资源和个数，初始化生成 ExecutorData ，通过 Executor 的引用 executorRef 向 CoarseGrainedExecutorBackend 发送 RegisteredExecutor 消息，CoarseGrainedExecutorBackend 接收到消息以后，会创建 Executor 对象。
executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)
到这里，创建 Executor 的过程结束。