Spark1.3从创建到提交：1)master和worker启动流程源码分析

1.Spark集群的Shell脚本启动

1.1start-all.sh

首先看下在master节点最简洁的集群启动方式:sbin/start-all.sh

# Load the Spark configuration
. "$sbin/spark-config.sh"
# Start Master
"$sbin"/start-master.sh $TACHYON_STR
# Start Workers
"$sbin"/start-slaves.sh $TACHYON_STR

从上面可以看到，读取完本机的配置文件后，sbin/start-all.sh->sbin/start-master.sh & start-slaves.sh

1.2start-master.sh

sbin/spark-daemon.sh start org.apache.spark.deploy.master.Master 1 --ip $SPARK_MASTER_IP --port $SPARK_MASTER_PORT --webui-port $SPARK_MASTER_WEBUI_PORT

读取本地变量后，sbin/start-master.sh->sbin/spark-daemon.sh

1.3spark-daemon.sh 

spark-daemon.sh的功能比较强大，包括master的start和stop以及任务的submit,这里只看master的start

 nohup nice -n $SPARK_NICENESS "$SPARK_PREFIX"/bin/spark-class $command "$@" >> "$log" 2>&1 < /dev/null

command=$1即org.apache.spark.deploy.master.Master，$@为之前start-master.sh调用spark-daemon.sh的所有参数。sbin/spark-daemon.sh->bin/spark-class

1.4spark-class

exec "$RUNNER" -cp "$CLASSPATH" $JAVA_OPTS "$@"

这里直接使用jar来执行了一个.class文件(org.apache.spark.deploy.master.Master)，从而启动master进程，slave的启动类似，只不过中间使用了ssh命令远程启动而已。

2.org.apache.spark.deploy.master.Master

首先看下Master类，它的伴随对象的main方法

 def main(argStrings: Array[String]) {
    SignalLogger.register(log)
    val conf = new SparkConf
    val args = new MasterArguments(argStrings, conf)
    val (actorSystem, _, _, _) = startSystemAndActor(args.host, args.port, args.webUiPort, conf)
    actorSystem.awaitTermination()
  }

其主要调用了startSystemAndActor方法，代码如下

def startSystemAndActor(
      host: String,
      port: Int,
      webUiPort: Int,
      conf: SparkConf): (ActorSystem, Int, Int, Option[Int]) = {
    val securityMgr = new SecurityManager(conf)
    val (actorSystem, boundPort) = AkkaUtils.createActorSystem(systemName, host, port, conf = conf,
      securityManager = securityMgr)
    val actor = actorSystem.actorOf(
      Props(classOf[Master], host, boundPort, webUiPort, securityMgr, conf), actorName)
    val timeout = AkkaUtils.askTimeout(conf)
    val portsRequest = actor.ask(BoundPortsRequest)(timeout)
    val portsResponse = Await.result(portsRequest, timeout).asInstanceOf[BoundPortsResponse]
    (actorSystem, boundPort, portsResponse.webUIPort, portsResponse.restPort)
  }

其中通过AkkaUtils.createActorSystem创建了一个ActorSystem对象，使用actorSystem的actorOf的方法创建了acotor(MasterActor),因为Master继承了Actor类，因此Master中的preStart方法将会被调用。而receiveWithLogging方法也会一直监听对应的消息，receiveWithLogging接受worker注册的消息代码如下：

case RegisterWorker(id, workerHost, workerPort, cores, memory, workerUiPort, publicAddress) =>
    {
      logInfo("Registering worker %s:%d with %d cores, %s RAM".format(
        workerHost, workerPort, cores, Utils.megabytesToString(memory)))
      if (state == RecoveryState.STANDBY) {
        // ignore, don't send response
      } else if (idToWorker.contains(id)) {
        sender ! RegisterWorkerFailed("Duplicate worker ID")
      } else {
        val worker = new WorkerInfo(id, workerHost, workerPort, cores, memory,
          sender, workerUiPort, publicAddress)
        if (registerWorker(worker)) {
          persistenceEngine.addWorker(worker)
          sender ! RegisteredWorker(masterUrl, masterWebUiUrl)
          schedule()
        } else {
          val workerAddress = worker.actor.path.address
          logWarning("Worker registration failed. Attempted to re-register worker at same " +
            "address: " + workerAddress)
          sender ! RegisterWorkerFailed("Attempted to re-register worker at same address: "
            + workerAddress)
        }
      }
    }

如果当前的mater不是standby的节点(高可靠性，双Master)，并且当期的worker没有注册，则把worker发送过来的信息封装成一个WorkerInfo对象，并给worker回复RegisteredWorker的信息。

小结下Master启动的流程：main->startSystemAndActor方法->创建actorSystem->创建MasterActor->preStart方法->接受worker的消息

3.org.apache.spark.deploy.worker.Worker

worker几乎和Master类似，比如下面的main方法

def main(argStrings: Array[String]) {
    SignalLogger.register(log)
    val conf = new SparkConf
    val args = new WorkerArguments(argStrings, conf)
    val (actorSystem, _) = startSystemAndActor(args.host, args.port, args.webUiPort, args.cores,
      args.memory, args.masters, args.workDir)
    actorSystem.awaitTermination()
  }

startSystemAndActor方法

def startSystemAndActor(
      host: String,
      port: Int,
      webUiPort: Int,
      cores: Int,
      memory: Int,
      masterUrls: Array[String],
      workDir: String,
      workerNumber: Option[Int] = None,
      conf: SparkConf = new SparkConf): (ActorSystem, Int) = {


    // The LocalSparkCluster runs multiple local sparkWorkerX actor systems
    val systemName = "sparkWorker" + workerNumber.map(_.toString).getOrElse("")
    val actorName = "Worker"
    val securityMgr = new SecurityManager(conf)
    val (actorSystem, boundPort) = AkkaUtils.createActorSystem(systemName, host, port,
      conf = conf, securityManager = securityMgr)
    val masterAkkaUrls = masterUrls.map(Master.toAkkaUrl(_, AkkaUtils.protocol(actorSystem)))
    actorSystem.actorOf(Props(classOf[Worker], host, boundPort, webUiPort, cores, memory,
      masterAkkaUrls, systemName, actorName,  workDir, conf, securityMgr), name = actorName)
    (actorSystem, boundPort)
  }

startSystemAndActor方法主要触发了Worker类下的PreStart方法，而它下面有个比较重要的方法registerWithMaster

def registerWithMaster() {
    // DisassociatedEvent may be triggered multiple times, so don't attempt registration
    // if there are outstanding registration attempts scheduled.
    registrationRetryTimer match {
      case None =>
        registered = false
        tryRegisterAllMasters()
        connectionAttemptCount = 0
        registrationRetryTimer = Some {
          context.system.scheduler.schedule(INITIAL_REGISTRATION_RETRY_INTERVAL,
            INITIAL_REGISTRATION_RETRY_INTERVAL, self, ReregisterWithMaster)
        }
      case Some(_) =>
        logInfo("Not spawning another attempt to register with the master, since there is an" +
          " attempt scheduled already.")
    }
  }

由于网络的原因，worker会连续注册几次，其中tryRegisterAllMasters比较核心，它通过context.actorSelection(masterAkkaUrl)可以获得masterActor对象的引用，并发送消息

private def tryRegisterAllMasters() {
    for (masterAkkaUrl <- masterAkkaUrls) {
      logInfo("Connecting to master " + masterAkkaUrl + "...")
      val actor = context.actorSelection(masterAkkaUrl)
      actor ! RegisterWorker(workerId, host, port, cores, memory, webUi.boundPort, publicAddress)
    }
  }

在看下比较核心的receive方法，里面有接收master发送回来的注册成功RegisteredWorker消息，和通过自身发往Master的心跳机制

 override def receiveWithLogging = {
    case RegisteredWorker(masterUrl, masterWebUiUrl) =>
      logInfo("Successfully registered with master " + masterUrl)
      registered = true
      changeMaster(masterUrl, masterWebUiUrl)
      context.system.scheduler.schedule(0 millis, HEARTBEAT_MILLIS millis, self, SendHeartbeat)
      if (CLEANUP_ENABLED) {
        logInfo(s"Worker cleanup enabled; old application directories will be deleted in: $workDir")
        context.system.scheduler.schedule(CLEANUP_INTERVAL_MILLIS millis,
          CLEANUP_INTERVAL_MILLIS millis, self, WorkDirCleanup)
      }

    case SendHeartbeat =>
      if (connected) { master ! Heartbeat(workerId) }
	// 其他的代码
 }

此时Master和Worker节点已经建立好通信机制，接下来就可以提交对应的任务了！

后话：之前总结过一篇Akka实现简单RPC通信，里面的内容和上面描述的类似，文章地址： Spark：Akka实现简单RPC通信