Spark1.3.1源码(1)-Spark Submit任务提交以及Worker向Executor注册

最新推荐文章于 2022-11-10 12:30:20 发布

喜讯XiCent

最新推荐文章于 2022-11-10 12:30:20 发布

阅读量226

点赞数

分类专栏： spark 文章标签： spark

本文链接：https://blog.csdn.net/qq_42194171/article/details/107036579

版权

spark 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

当我们写好的Spark程序想要提交到集群运行时，我们通常是调用SparkSubmit脚本进行提交，那调用Spark Submit脚本是如何在集群中执行我们的Spark程序的，Master是如何进行调度的，Executor如何向Driver注册的等等，过程复杂，这篇文章将为你一一揭晓。

1.任务提交Executor的注册流程

执行命令bin/spark-submit，内部通过exec执行${SPARK_HOME}/bin/spark-class org.apache.spark.deploy.SparkSubmit。
通过spark-class脚本，最终执行的命令中，执行了程序的入口为org.apache.spark.deploy.SparkSubmit,类中的doRunMain中通过反射调用自定勒的main方法。
自定义main方法中构造了SparkContext执行其主构造器

1.1 调用createSparkEnv创建一个ActorSystem，这是akka用于通信的类

//TODO 该方法创建了一个ActorSystem
private[spark] def createSparkEnv(
    conf: SparkConf,
    isLocal: Boolean,
    listenerBus: LiveListenerBus): SparkEnv = {
  SparkEnv.createDriverEnv(conf, isLocal, listenerBus)
}
 
private[spark] val env = createSparkEnv(conf, isLocal, listenerBus)
SparkEnv.set(env)

------------------------

// Create the ActorSystem for Akka and get the port it binds to.
val (actorSystem, boundPort) = {
  val actorSystemName = if (isDriver) driverActorSystemName else executorActorSystemName
  //TODO 利用AkkaUtils这个工具类创建ActorSystem
  AkkaUtils.createActorSystem(actorSystemName, hostname, port, conf, securityManager)
}

1.2创建TaskScheduler->根据提交任务的URL进行匹配(Standalone)->创建TaskSchedulerImpl->创建SparkDeploySchedulerBackend(里面有两个Actor----ClientActor,DriverActor)

val SPARK_REGEX = """spark://(.*)""".r
------------
//TODO 创建一个TaskScheduler
private[spark] var (schedulerBackend, taskScheduler) =
  SparkContext.createTaskScheduler(this, master)
  
------------
 
  //TODO spark的StandAlone模式,根据URL匹配的Spark://
case SPARK_REGEX(sparkUrl) =>
  //TODO 创建了一个TaskSchedulerImpl
  val scheduler = new TaskSchedulerImpl(sc)
  val masterUrls = sparkUrl.split(",").map("spark://" + _)
  //TODO 创建了一个SparkDeploySchedulerBackend
  val backend = new SparkDeploySchedulerBackend(scheduler, sc, masterUrls)
  //TODO 调用initialize创建调度器
  scheduler.initialize(backend)
  (backend, scheduler)

1.3 通过ActorSystem创建了一个Actor，这个心跳是Executors和DriverActor的心跳

//TODO 通过ActorSystem创建了一个Actor，这个心跳是Executors和DriverActor的心跳
private val heartbeatReceiver = env.actorSystem.actorOf(
  Props(new HeartbeatReceiver(taskScheduler)), "HeartbeatReceiver")

1.4创建DAGScheduler，以后用来把DAG切分成Stage

this指的是SparkContext

//TODO 创建了一个DAGScheduler，以后用来把DAG切分成Stage
dagScheduler = new DAGScheduler(this)

DAGScheduler在初始化的时候，就会启动eventProcessLoop(FIFO)，DAGSchedulerEventProcessLoop继承了EventLoop，启动之后，会启动一个后台线程，从队列BlockingQueue里取各种event，然后根据取出的event，进行不同的处理。

1.5启动taskScheduler

SparkContext

//TODO 启动taskScheduler
taskScheduler.start()

TaskSchedulerImpl

//TODO 首先调用SparkDeploySchedulerBackend的start方法
backend.start()

SparkDeploySchedulerBackend

//TODO 首先调用父类的start方法来创建DriverActor
super.start()

CoarseGrainedSchedulerBackend

创建DriverActor，用于与Executor通信

 // TODO　通过ActorSystem创建DriverActor
  driverActor = actorSystem.actorOf(
    Props(new DriverActor(properties)), name = CoarseGrainedSchedulerBackend.ACTOR_NAME)
}

SparkDeploySchedulerBackend

回到SparkDeploySchedulerBackend调用父类start()方法之后

创建AppClient，并调用start方法，start方法中创建了ClientActor用于与Master通信。把应用信息封装在一个ApplicationDescription中发给Master

//TODO 准备一些参数，以后把这些参数封装到一个对象中，然后将该对象发送给Master
val driverUrl = AkkaUtils.address(
  AkkaUtils.protocol(actorSystem),
  SparkEnv.driverActorSystemName,
  conf.get("spark.driver.host"),
  conf.get("spark.driver.port"),
  CoarseGrainedSchedulerBackend.ACTOR_NAME)
  
//TODO 这个参数是以后Executor的实现类
val command = Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",
  args, sc.executorEnvs, classPathEntries ++ testingClassPath, libraryPathEntries, javaOpts)
val appUIAddress = sc.ui.map(_.appUIAddress).getOrElse("")
//TODO 把参数封装到ApplicationDescription
val appDesc = new ApplicationDescription(sc.appName, maxCores, sc.executorMemory, command,
  appUIAddress, sc.eventLogDir, sc.eventLogCodec)
//TODO 创建一个AppClient把ApplicationDescription通过主构造器传进去
client = new AppClient(sc.env.actorSystem, masters, appDesc, this, conf)
//TODO 然后调用AppClient的start方法，在start方法中创建了一个ClientActor用于与Master通信
client.start()
 
waitForRegistration()

AppClient

创建ClientActor，用于与Executor通信

def start() {
  // Just launch an actor; it will call back into the listener.
  //TODO 创建ClientActor调用主构造器 -》preStart -》 receive
  actor = actorSystem.actorOf(Props(new ClientActor))
}

生命周期方法(preStart())中向Master注册

//TODO ClientActor的生命周期方法
override def preStart() {
  context.system.eventStream.subscribe(self, classOf[RemotingLifecycleEvent])
  try {
    //TODO ClientActor向Master注册
    registerWithMaster()
  } catch {
    case e: Exception =>
      logWarning("Failed to connect to master", e)
      markDisconnected()
      context.stop(self)
  }
}

向Master注册成功后Master会返回case class RegisteredApplication

//TODO Master发送给ClientActor注册成功的消息
case RegisteredApplication(appId_, masterUrl) =>
  appId = appId_
  registered = true
  changeMaster(masterUrl)
  listener.connected(appId)

Master

接收到来自DriverActor的注册消息，保存信息，发送注册成功的消息，调用schedule()方法，开始调度资源，根据任务决定在哪些Worker上启动Executor

//TODO ClientActor发送过来的注册应用的消息
case RegisterApplication(description) => {
  if (state == RecoveryState.STANDBY) {
    // ignore, don't send response
  } else {
    logInfo("Registering app " + description.name)
    //TODO 首先把应用的信息放到内存中存储
    val app = createApplication(description, sender)
    registerApplication(app)
    logInfo("Registered app " + description.name + " with ID " + app.id)
    //TODO 利用持久化引擎保存
    persistenceEngine.addApplication(app)
    //TODO Master向ClientActor发送注册成功的消息
    sender ! RegisteredApplication(app.id, masterUrl)
    //TODO 重要：Master开始调度资源，其实就是把任务启动到哪些Worker上
    schedule()
  }
}

schedule()

Master调用该方法开始进行调度。在1.3中一个worker只能存在一个application的Executor，在1.6去掉了这种限制

有两种方式，尽量大散和尽量集中

尽量打散

过滤出所有符合memory的worker并按core的倒序排序
遍历worker，一次分配core，直到分完位置。core信息记录在一个assigned数组中
依次启动Executor

尽量集中

过滤出所有符合memory的worker
遍历worker，取worker剩余core和application剩余core两者的最小值(榨干)。在该worker中启动Executor。如果该任务还有core需要分配，则继续下一个worker。

Master

发送消息启动Executor，并告诉ClientActor Executor启动成功

//TODO Master发送消息让Worker启动Executor
launchExecutor(usableWorkers(pos), exec)

def launchExecutor(worker: WorkerInfo, exec: ExecutorDesc) {
  logInfo("Launching executor " + exec.fullId + " on worker " + worker.id)
  //TODO 记录该Worker使用的资源
  worker.addExecutor(exec)
  //TODO Master发送消息给Worker，把参数通过case class传递给Worker，让他启动Executor，
  worker.actor ! LaunchExecutor(masterUrl,
    exec.application.id, exec.id, exec.application.desc, exec.cores, exec.memory)
  //TODO Master向ClientActor发送消息，告诉它Executor已经启动了
  exec.application.driver ! ExecutorAdded(
    exec.id, worker.id, worker.hostPort, exec.cores, exec.memory)
}

Worker

接收到来自Master的消息，里面封装了以后要启动的Executor的信息

//TODO Master发送给Worker的消息，让Worker启动Executor。
//TODO LaunchExecutor是一个Case Class，里面封装了以后要启动的Executor的信息
case LaunchExecutor(masterUrl, appId, execId, appDesc, cores_, memory_) => ...........

创建一个ExecutorRunner，将参数都放到其中，然后在通过他启动Executor

//TODO 创建一个ExecutorRunner，将参数都放到其中，然后在通过他启动Executor
  val manager = new ExecutorRunner(
    appId,
    execId,
    appDesc.copy(command = Worker.maybeUpdateSSLSettings(appDesc.command, conf)),
    cores_,
    memory_,
    self,
    workerId,
    host,
    webUi.boundPort,
    publicAddress,
    sparkHome,
    executorDir,
    akkaUrl,
    conf,
    appLocalDirs, ExecutorState.LOADING)

保存Executor信息，并调用ExecutorRunner的start方法来启动Executor java子进程

  //TODO 把ExecutorID-> Executor放到一个Map中了
  executors(appId + "/" + execId) = manager
  //TODO 调用ExecutorRunner的start方法来启动Executor java子进程
  manager.start()
  coresUsed += cores_
  memoryUsed += memory_
  master ! ExecutorStateChanged(appId, execId, manager.state, None, None)
}

ExecutorRunner

创建一个线程，通过一个线程来启动一个Java子进程

def start() {
  //TODO 先创建一个线程对象，然后通过一个线程来启动一个Java子进程
  workerThread = new Thread("ExecutorRunner for " + fullId) {
    override def run() { fetchAndRunExecutor() }
  }
  //TODO 调用线程对象的start方法 -> 线程对象的run方法
  workerThread.start()
  // Shutdown hook that kills actors on shutdown.
  shutdownHook = new Thread() {
    override def run() {
      killProcess(Some("Worker shutting down"))
    }
  }
  Runtime.getRuntime.addShutdownHook(shutdownHook)
}

线程调用fetchAndRunExecutor()方法来启动子进程

//TODO 线程对象调用该方法启动Java子进程
  def fetchAndRunExecutor() {
    try {
      // Launch the process
      //TODO 启动子进程
      val builder = CommandUtils.buildProcessBuilder(appDesc.command, memory,
        sparkHome.getAbsolutePath, substituteVariables)
      val command = builder.command()
      logInfo("Launch command: " + command.mkString("\"", "\" \"", "\""))

      builder.directory(executorDir)
      builder.environment.put("SPARK_LOCAL_DIRS", appLocalDirs.mkString(","))
      // In case we are running this from within the Spark Shell, avoid creating a "scala"
      // parent process for the executor command
      builder.environment.put("SPARK_LAUNCH_WITH_SCALA", "0")

      // Add webUI log urls
      val baseUrl =
        s"http://$publicAddress:$webUiPort/logPage/?appId=$appId&executorId=$execId&logType="
      builder.environment.put("SPARK_LOG_URL_STDERR", s"${baseUrl}stderr")
      builder.environment.put("SPARK_LOG_URL_STDOUT", s"${baseUrl}stdout")

      //TODO 真正启动一个Java子进程 ->  CoarseGrainedExecutorBackend的main方法
      process = builder.start()
      val header = "Spark Executor Command: %s\n%s\n\n".format(
        command.mkString("\"", "\" \"", "\""), "=" * 40)

CoarseGrainedExecutorBackend

启动好一个Executor后，会调用Executor的生命周期方法，在生命周期方法中向Driver注册

//TODO CoarseGrainedExecutorBackend的生命周期方法
override def preStart() {
  logInfo("Connecting to driver: " + driverUrl)
  //TODO 跟Driver建立连接
  driver = context.actorSelection(driverUrl)
  //TODO Executor向DriverActor发送消息，来注册Exectuor
  driver ! RegisterExecutor(executorId, hostPort, cores, extractLogUrls)
  context.system.eventStream.subscribe(self, classOf[RemotingLifecycleEvent])
}

DriverActor

接收Executor向DriverActor发送的注册消息(RegisterExecutor)，并返回注册成功消息(RegisteredExecutor)

def receiveWithLogging = {
  //TODO Executor向DriverActor发送的消息
  case RegisterExecutor(executorId, hostPort, cores, logUrls) =>
    Utils.checkHostPort(hostPort, "Host port expected " + hostPort)
    if (executorDataMap.contains(executorId)) {
      sender ! RegisterExecutorFailed("Duplicate executor ID: " + executorId)
    } else {
      logInfo("Registered executor: " + sender + " with ID " + executorId)
      //TODO DriverActor向Executor发送消息，告诉Executor注册成功
      sender ! RegisteredExecutor

CoarseGrainedExecutorBackend

接收Driver发送来的注册成功消息，创建一个Executor实例

//TODO DirverActor发送给Executor的消息，告诉它已经注册成功
case RegisteredExecutor =>
  logInfo("Successfully registered with driver")
  val (hostname, _) = Utils.parseHostPort(hostPort)
  //TODO 创建了一个Executor实例，用来执行业务逻辑
  executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)

查看是否有任务需要提交

//TODO 查看是否有任务需要提交（DriverActor -> Executor）
makeOffers()

Executor

收到DriverActor的注册成功消息后，创建一个线程池用来执行task

 //TODO DirverActor发送给Executor的消息，告诉她已经注册成功
    case RegisteredExecutor =>
      logInfo("Successfully registered with driver")
      val (hostname, _) = Utils.parseHostPort(hostPort)
      //TODO 创建了一个Executor实例，用来执行业务逻辑
      executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)

//TODO 初始化线程池
val threadPool = Utils.newDaemonCachedThreadPool("Executor task launch worker")

向DriverActor发送心跳

//TODO Executor向DriverActor发送心跳
startDriverHeartbeater()

到这就完成Executor的注册工作啦，接下来就是Driver提交的job到Executor运行了

本篇先讲到这里，如果喜欢的话记得关注喔，公众号同名

喜讯XiCent

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Spark1.3.1源码(1)-Spark Submit任务提交以及Worker向Executor注册

当我们写好的Spark程序想要提交到集群运行时，我们通常是调用SparkSubmit脚本进行提交，那调用Spark Submit脚本是如何在集群中执行我们的Spark程序的，Master是如何进行调度的，Executor如何向Driver注册的等等，过程复杂，这篇文章将为你一一揭晓。1.任务提交Executor的注册流程执行命令bin/spark-submit，内部通过exec执行${SPARK_HOME}/bin/spark-class org.apache.spark.deploy.SparkS.
复制链接

扫一扫