spark提交任务的模板

./bin/spark-submit
–class
–master
–deploy-mode
–conf =
… # other options

[application-arguments]
举几个常用的用法例子：

Run application locally on 8 cores

./bin/spark-submit
–class org.apache.spark.examples.SparkPi
–master local[8]
/path/to/examples.jar
100

Run on a Spark Standalone cluster in client deploy mode

./bin/spark-submit
–class org.apache.spark.examples.SparkPi
–master spark://207.184.161.138:7077
–executor-memory 20G
–total-executor-cores 100
/path/to/examples.jar
1000

Run on a Spark Standalone cluster in cluster deploy mode with supervise

./bin/spark-submit
–class org.apache.spark.examples.SparkPi
–master spark://207.184.161.138:7077
–deploy-mode cluster
–supervise
–executor-memory 20G
–total-executor-cores 100
/path/to/examples.jar
1000

Run on a YARN cluster

export HADOOP_CONF_DIR=XXX
./bin/spark-submit
–class org.apache.spark.examples.SparkPi
–master yarn-cluster \ # can also be yarn-client for client mode
–executor-memory 20G
–num-executors 50
/path/to/examples.jar
1000

Run a Python application on a Spark Standalone cluster

./bin/spark-submit
–master spark://207.184.161.138:7077
examples/src/main/python/pi.py
1000