Spark常用命令及端口

Spark
启动Spark
sbin/start-all.sh
停止Spark
sbin/stop-all.sh
单独启动:
sbin/start-master.sh
sbin/start-slaves.sh 所有slave
sbin/start-slave.sh 单台slave
启动spark-shell
/opt/module/spark/bin/spark-shell
–master spark://hadoop101:7077
启动spark–shell使用外部hive(如果spark/jars目录下有mysql驱动可以不写–jars驱动地址)
bin/spark-shell --master spark://hadoop101:7077 --jars jars/mysql-connector-java-5.1.27-bin.jar
启动Spark History Server
sbin/start-history-server.sh

提交jar(提交到spark)
bin/spark-submit
–class org.apache.spark.examples.SparkPi
–master spark://hadoop101:7077
–executor-memory 1G
–total-executor-cores 2
./examples/jars/spark-examples_2.11-2.1.1.jar
100
提交jar(提交到yarn) mode client/cluster
bin/spark-submit
–class org.apache.spark.examples.SparkPi
–master yarn
–deploy-mode client
./examples/jars/spark-examples_2.11-2.1.1.jar
100
web端口:
spark监控端口:http://hadoop101:8080/
历史服务端口http://hadoop101:4000/
Spark-shell端口:http://192.168.56.3:4040/