spark deploy_mode配置详解

spark的deploy_mode可以是client 或cluster

–deploy-mode: Whether to deploy your driver on the worker nodes 
(cluster) or locally as an external client (client) (default: client)

注意这里的client和cluster模式的区别在于driver的位置

A common deployment strategy is to submit your application from a 
gateway machine that is physically co-located with your worker 
machines (e.g. Master node in a standalone EC2 cluster). In this 
setup, client mode is appropriate. In client mode, the driver is 
launched directly within the spark-submit process which acts as a 
client to the cluster. The input and output of the application is 
attached to the console. Thus, this mode is especially suitable for 
applications that involve the REPL (e.g. Spark shell).

In client mode, the driver is launched in the same process as the 
client that submits the application. In cluster mode, however, the 
driver is launched from one of the Worker processes inside the 
cluster, and the client process exits as soon as it fulfills its 
responsibility of submitting the application without waiting for the 
application to finish.

大意就是，如果你进行spark-submit的机器和worker节点在地理上相近（比如同一个局域网），建议使用client模式，这时driver就在进行spark-submit的线程中。换句话说，进行spark-submit的机器没必要是集群里面的机器，如果你的笔记本安装了Spark环境，就可以提交任务到master节点，这个时候driver就在你的笔记本上运行。这时显然不推荐client模式，因为考虑到网络开销，最好使用cluster模式。使用cluster模式后，Spark就会根据调度算法在集群中选择一个机器作为driver执行的地方。

Master node根本不会执行用户提交的任务，它是负责整个调度的（运行cluster manager)。