yarn源码分析之SparkSubmit

1. SparkSubmit

    // 启动进程

    -- main

        // 封装参数

        -- new SparkSubmitArguments

        // 提交

        -- submit

            // 准备提交环境

            -- prepareSubmitEnvironment

                // Cluster

                -- childMainClass = "org.apache.spark.deploy.yarn.Client"

                // Client

                -- childMainClass = args.mainClass (SparkPi)

            -- doRunMain (runMain)

                // 反射加载类

                -- Utils.classForName(childMainClass)

                // 查找main方法

                -- mainClass.getMethod("main", new Array[String](0).getClass)

                // 调用main方法

                -- mainMethod.invoke

2. Client（org.apache.spark.deploy.yarn.Client）

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-yarn_2.11</artifactId>
    <version>2.1.1</version>
</dependency>

导入此依赖才能找到org.apache.spark.deploy.yarn.Client这个类

    -- main

        -- new ClientArguments(argStrings)

        -- new Client

            -- yarnClient = YarnClient.createYarnClient

        -- client.run

                -- submitApplication

                    // 封装指令 command = bin/java org.apache.spark.deploy.yarn.ApplicationMaster (Cluster)

                    // command = bin/java org.apache.spark.deploy.yarn.ExecutorLauncher  (client)

                    -- createContainerLaunchContext

                    -- createApplicationSubmissionContext

                    // 向Yarn提交应用，提交指令

                    -- yarnClient.submitApplication(appContext)