spark源码分析四(Driver启动流程)

最新推荐文章于 2023-01-02 17:43:52 发布

AngelaPotato

最新推荐文章于 2023-01-02 17:43:52 发布

阅读量455

点赞数

分类专栏： Spark 文章标签： Spark Driver spark- 源码

本文链接：https://blog.csdn.net/wangxilong1991/article/details/110115025

版权

本文深入剖析了Spark在Standalone集群模式下Driver的启动流程。从`spark-submit`开始，跟踪`SparkSubmit`类，重点解析`prepareSubmitEnvironment`方法中的StandaloneCluster分支，涉及`ClientApp`类的加载与启动。接着详细阐述了`DriverWrapper`如何作为Driver的运行容器，以及在Master和Worker节点间的消息传递过程，包括Worker节点上的`DriverRunner`如何启动并管理Driver。最终，Driver启动完成后，通过反射调用用户提交的逻辑代码，完成启动流程。

摘要由CSDN通过智能技术生成

spark-submit

源码版本2.4.7
spark-submit

exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"

主类:
org.apache.spark.deploy.SparkSubmit

spark-class 会启动 Moudle->launcher org.apache.spark.launcher.Main的jvm进程,构建一些JVM相关启动环境、参数

从main方法跟踪发现,最终会调用到

 /**
   * Run the main method of the child class using the submit arguments.
   *
   * This runs in two steps. First, we prepare the launch environment by setting up
   * the appropriate classpath, system properties, and application arguments for
   * running the child main class based on the cluster manager and the deploy mode.
   * Second, we use this launch environment to invoke the main method of the child
   * main class.
   *
   * Note that this main class will not be the one provided by the user if we're
   * running cluster deploy mode or python applications.
   */
  private def runMain(args: SparkSubmitArguments, uninitLog: Boolean): Unit = {
   
    val (childArgs, childClasspath, sparkConf, childMainClass) = prepareSubmitEnvironment(args)

进一步跟踪prepareSubmitEnvironment方法,其根据集群的模式组装环境、参数以及childMainClass,由于咱们分析的是StandaloneCluster集群部署方式,所以重点关注StandaloneCluster相关分支即可:

   // In standalone cluster mode, use the REST client to submit the application (Spark 1.3+).
    // All Spark parameters are expected to be passed to the client through system properties.
    if (args.isStandaloneCluster) {
   
      if (args.useRest) {
   
        childMainClass = REST_CLUSTER_SUBMIT_CLASS
        childArgs += (args.primaryResource, args.mainClass)
      } else {
   
        // In legacy standalone cluster mode, use Client as a wrapper around the user class
       //STANDALONE_CLUSTER_SUBMIT_CLASS = classOf[ClientApp].getName()
        childMainClass = STANDALONE_CLUSTER_SUBMIT_CLASS
        if (args.supervise) {
    childArgs += "--supervise" }
        Option(args.driverMemory).foreach {
    m => childArgs += ("--memory", m) }
        Option(args.driverCores).foreach {
    c => childArgs += ("--cores", c) }
        childArgs += "launch"
        childArgs += (args.master, args.primaryResource, args.mainClass)
      }
      if (args.childArgs != null) {
   
        childArgs ++= args.childArgs
      }
    }

STANDALONE_CLUSTER_SUBMIT_CLASS = classOf[ClientApp].getName()
childMainClass是一个ClientApp类

    try {
   
      mainClass = Utils.classForName(childMainClass)
    } catch {
   
      case e: ClassNotFoundException =>
        logWarning(s"Failed to load $childMainClass.", e)
        if (childMainClass.contains("thriftserver")) {
   
          logInfo(s"Failed to load main class $childMainClass.")
          logInfo("You need to build Spark with -Phive and -Phive-thriftserver.")
        }
        throw new SparkUserAppException(CLASS_NOT_FOUND_EXIT_STATUS)
      case e: NoClassDefFoundError =>
        logWarning(s"Failed to load $childMainClass: ${e.getMessage()}")
        if (e.getMessage.contains("org/apache/hadoop/hive")) {
   
          logInfo(s"Failed to load hive class.")
          logInfo("You need to build Spark with -Phive and -Phive-thriftserver.")
        }
        throw new SparkUserAppException(CLASS_NOT_FOUND_EXIT_STATUS)
    }

    val app: SparkApplication = if (classOf[SparkApplication].isAssignableFrom(mainClass)) {
   
      mainClass.newInstance().asInstanceOf[SparkApplication]
    } else {
   
      // SPARK-4170
      if (classOf[scala.App].isAssignableFrom(mainClass)) {
   
        logWarning("Subclasses of scala.App may not work correctly. Use a main() method instead.")
      }
      new JavaMainApplication(mainClass)