关于spark作业提交：spark运行example为例

示例：yarn-cluster提交：

spark-submit --master yarn --deploy-mode cluster --executor-memory 2G --executor-cores 2 --queue root.helowin --class org.apache.spark.examples.SparkPi spark-examples-1.6.3-hadoop2.6.0.jar 1000

 

查看日志：

yarn logs -applicationId application_1536809934546_0008

 

spark提交作业分为多种，比较常用：

yarn 分2种，client、cluster

standalone分2种，client、cluster

还有不常用mesos等

参考：https://spark.apache.org/docs/2.2.0/submitting-applications.html

 

以SparkPi为例：

1.yarn-cluster模式提交：

spark-submit --master yarn \ 
--deploy-mode cluster \
--executor-memory 2G \
--executor-cores 2 \
--queue root.helowin \
--class org.apache.spark.examples.SparkPi \
spark-examples-1.6.3-hadoop2.6.0.jar \
1000

 

不指定deploy-mode，默认为client提交。

 

2.standalone-cluster提交：

这里测试时候发现无法提交，始终没有worker，后面发现，可能是集群资源分配问题，这里添加上driver给定资源，即可运行。client不存在该问题。

spark-submit \

--class org.apache.spark.examples.SparkPi \

--master spark://dc1:7077 \

--deploy-mode cluster \

--supervise \

--driver-cores 20 \

--driver-memory 100M \

--executor-memory 200M \

--total-executor-cores 2 \

spark-examples-1.6.3-hadoop2.6.0.jar \

1000

打印信息也短，因为日志直接写到节点上面，节点在集群当中，所以你本机只是作为一个提交作业的机器。只有client模式下，日志才会写回到本机，因为本机是作为driver了：

Running Spark using the REST application submission protocol.

18/09/17 17:57:15 INFO rest.RestSubmissionClient: Submitting a request to launch an application in spark://dc1:7077.

18/09/17 17:57:26 WARN rest.RestSubmissionClient: Unable to connect to server spark://dc1:7077.

Warning: Master endpoint spark://dc1:7077 was not a REST server. Falling back to legacy submission gateway instead.

以上提交有个warning，可以使用restfule形式来提交，具体端口可以在spark webUI上看到：

spark-submit --class org.apache.spark.examples.SparkPi \
--master spark://dc1:6066 \
--deploy-mode cluster \
--supervise \
--driver-cores 4 \
--driver-memory 200M \
--executor-memory 4G \
--total-executor-cores 6 \
spark-examples-1.6.3-hadoop2.6.0.jar 1000

打印信息：

Running Spark using the REST application submission protocol.

18/09/17 18:00:00 INFO rest.RestSubmissionClient: Submitting a request to launch an application in spark://dc1:6066.

18/09/17 18:00:01 INFO rest.RestSubmissionClient: Submission successfully created as driver-20180917180001-0015. Polling submission state...

18/09/17 18:00:01 INFO rest.RestSubmissionClient: Submitting a request for the status of submission driver-20180917180001-0015 in spark://dc1:6066.

18/09/17 18:00:01 INFO rest.RestSubmissionClient: State of driver driver-20180917180001-0015 is now RUNNING.

18/09/17 18:00:01 INFO rest.RestSubmissionClient: Driver is running on worker worker-20180913183011-192.168.9.168-7078 at 192.168.9.168:7078.

18/09/17 18:00:01 INFO rest.RestSubmissionClient: Server responded with CreateSubmissionResponse:

{

"action" : "CreateSubmissionResponse",

"message" : "Driver successfully submitted as driver-20180917180001-0015",

"serverSparkVersion" : "1.6.0",

"submissionId" : "driver-20180917180001-0015",

"success" : true

}

最后附带一张webUI上面的截图：

 

常见问题：

1.Warning: Master endpoint spark://dc1:7077 was not a REST server. Falling back to legacy submission gateway instead.

解决：提交master由默认7077端口换成rest server的端口：默认6066

2.standalone cluster提交后，貌似没有执行，但是yarn-cluster执行却正常。

解决：这可能是因为集群资源不够，常见在自己笔记本安装的虚拟机群。因为spark driver启动默认就需要vcpu 1，内存1G，造成机器资源不够。在web-ui上面可以看看是否被选中的Driver上分配到了core以及memory，如果有的话，再看看分配到worker是否也具有core和memory。如果都有，一般就没什么问题了，后台在运行了，查看节点上日志即可看到。