我的Spark学习笔记（七）

假设此刻，已完成基于YARN的高可用Spark集群配置

运行Hello World：计算PI的值

用YARN-Client方式提交

spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
--executor-memory 1G \
--total-executor-cores 3 \
/export/servers/spark/examples/jars/spark-examples_2.11-2.1.1.jar \
100

在命令行里就能看到日志输出：Pi is roughly 3.1412291141229116，但在Web UI页面里却看不到。

用YARN-Cluster方式提交

spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
--executor-memory 1G \
--total-executor-cores 3 \
/export/servers/spark/examples/jars/spark-examples_2.11-2.1.1.jar \
100

在Web UI页面里就能看到日志输出：Pi is roughly 3.1422959142295914，但在命令行里却看不到。

如果是基于Standalone的高可用Spark集群配置

用YARN-Client方式提交

spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://YOUR-MASTER-IP:7077 \
--deploy-mode client \
--executor-memory 1G \
--total-executor-cores 3 \
/export/servers/spark/examples/jars/spark-examples_2.11-2.1.1.jar \
100

用YARN-Cluster方式提交

spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://YOUR-MASTER-IP:7077 \
--deploy-mode cluster \
--executor-memory 1G \
--total-executor-cores 3 \
/export/servers/spark/examples/jars/spark-examples_2.11-2.1.1.jar \
100