Spark Executor on YARN

YARN executor launch context

  env:
    CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
    SPARK_YARN_STAGING_DIR -> file:/Users/User/.sparkStaging/application_1479802900835_0001
    SPARK_USER -> User
    SPARK_YARN_MODE -> true

  command:
    {{JAVA_HOME}}/bin/java \
      -server \
      -Xmx2048m \
      -Djava.io.tmpdir={{PWD}}/tmp \
      -Dspark.yarn.app.container.log.dir=<LOG_DIR> \
      -XX:OnOutOfMemoryError='kill %p' \
      org.apache.spark.executor.CoarseGrainedExecutorBackend \
      --driver-url \
      spark://CoarseGrainedScheduler@172.16.61.169:53279 \
      --executor-id \
      <executorId> \
      --hostname \
      <hostname> \
      --cores \
      1 \
      --app-id \
      application_1479802900835_0001 \
      --user-class-path \
      file:$PWD/__app__.jar \
      1><LOG_DIR>/stdout \
      2><LOG_DIR>/stderr

Application Context

    if (isClusterMode) {
        runDriver(securityMgr)
      } else {
        runExecutorLauncher(securityMgr)
      }

• runDriver（Cluster模式)

  1. startUserApplication():
     在新线程中启动用户类
  2. runAMEndpoint:
     创建AM endpoint，返回driver endpoint的引用
  3. registerAM：
     
     • 在RM上注册AM，返回YarnAllocator
     • allocator.allocateResources()
  4. userClassThread.join()

• runExecutorLauncher(Client模式)

RPCEndpoint

• 生命周期：
  constructor -> onStart -> receive* -> onStop
  其中receive方法可以被并发调用。

DriverEndpoint

• onStart: send ReviveOffers，供后续工作执行
• receive:
  
  • StatusUpdate
  • ReviveOffers => Launch Task
  • KillTask

CoarseGrainedExecutorBackend

• onStart: 连接Driver
• receive：
  
  • RegisterExecutor: 在driver上注册executor
  • RegisterExecutorFailed：退出executor
  • LaunchTask：在executor中执行任务
  • KillTask: 在executor中杀死任务
  • StopExecutor：发送shutdown消息User
  • Shutdown：停止executor