Executor原理剖析与源码分析

版本：spark2.1.0

1、work中为application启动的executor,实际上是启动了CoarseGrainedExecutorBackend进程

private[spark] object CoarseGrainedExecutorBackend extends Logging {
  private def run(
      driverUrl: String,
      executorId: String,
      hostname: String,
      cores: Int,
      appId: String,
      workerUrl: Option[String],
      userClassPath: Seq[URL])

2、获取driver 的actor，向driver发送RegisterExecutor信息

override def onStart() {
    logInfo("Connecting to driver: " + driverUrl)
    rpcEnv.asyncSetupEndpointRefByURI(driverUrl).flatMap { ref =>
     //获取driver 的actor
      driver = Some(ref)
	  //向driver发送RegisterExecutor信息
      ref.ask[Boolean](RegisterExecutor(executorId, self, hostname, cores, extractLogUrls))
    }(ThreadUtils.sameThread).onComplete {
      // This is a very fast action so we can use "ThreadUtils.sameThread"
      case Success(msg) =>
        // Always receive `true`. Just ignore it
      case Failure(e) =>
        exitExecutor(1, s"Cannot register with driver: $driverUrl", e, notifyDriver = false)
    }(ThreadUtils.sameThread)
  }	

3、driver注册executor成功之后，会发送RegisteredExecutor消息

此时CoarseGrainedExecutorBackend会创建Executor执行句柄，大部分的功能都是通过Executor实现的

override def receive: PartialFunction[Any, Unit] = {
    case RegisteredExecutor =>
      logInfo("Successfully registered with driver")
      try {
        executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)
      } catch {
        case NonFatal(e) =>
          exitExecutor(1, "Unable to create executor due to " + e.getMessage, e)
      }

    case RegisterExecutorFailed(message) =>
      exitExecutor(1, "Slave registration failed: " + message)  

4、启动task，反序列化task，调用Executor执行器的launchTask（）启动一个task

//启动task
 case LaunchTask(data) =>
      if (executor == null) {
        exitExecutor(1, "Received LaunchTask command but executor was null")
      } else {
	  //反序列化task
        val taskDesc = ser.deserialize[TaskDescription](data.value)
        logInfo("Got assigned task " + taskDesc.taskId)
		调用Executor执行器的launchTask（）启动一个task
        executor.launchTask(this, taskId = taskDesc.taskId, attemptNumber = taskDesc.attemptNumber,
          taskDesc.name, taskDesc.serializedTask)
      }

5、对每个task都会创建一个TaskRunner，将TaskRunner放入内存缓存中

将task封装在一个线程中(TaskRunner),将线程丢入线程池中，然后执行

线程池是是自动实现了排队机制的

def launchTask(
      context: ExecutorBackend,
      taskId: Long,
      attemptNumber: Int,
      taskName: String,
      serializedTask: ByteBuffer): Unit = {
	  //对每个task都会创建一个TaskRunner
    val tr = new TaskRunner(context, taskId = taskId, attemptNumber = attemptNumber, taskName,
      serializedTask)
	  //将TaskRunner放入内存缓存中，
    runningTasks.put(taskId, tr)
	//将task封装在一个线程中(TaskRunner),将线程丢入线程池中，然后执行
	//线程池是是自动实现了排队机制的，
    threadPool.execute(tr)
  }	  

下节将分析task的具体被执行的原理