Spark Local模式
直接在 $SPARK_HOME/bin/spark-shell
中运行spark RDD程序,在UI界面http://nodename:4040/中可以看到jobs的执行情况
[elon@spark ~]$ spark-shell --master local
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/01/22 18:46:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://spark:4040
Spark context available as 'sc' (master = local, app id = local-1516618007091).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.2.1
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_152)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val source = sc.wholeTextFiles("/data")
source: org.apache.spark.rdd.RDD[(String, String)] = /data MapPartitionsRDD[1] at wholeTextFiles at <console>:24
scala> source.collect()
res0: Array[(String, String)] =
Array((hdfs://spark:9000/data/data1.txt,"Hello,I'm fine!
How old are you?
What's your name?
I'm a student and come from China.
GoodBye~
"), (hdfs://spark:9000/data/data2.txt,"192.168.1.101
192.168.1.202
Apache.org
Hadoop
Spark
"), (hdfs://spark:9000/data/data3.txt,"GoodBye My dear!
Nice to meet you next time
"))
Spark standalone模式
配置SPARK_MASTER_IP环境变量
首先在$SPARK_HOME/conf/spark-env.sh
中设置spark环境变量export SPARK_MASTER_IP=ipAddress,指定Master运行的主机IP地址,如果是在集群中执行时,有多个节点组件连接到Master时,具体配置看starting-a-cluster-manually
手动启动群集
你可以通过执行以下命令开启Master:
./$SPARK_HOME/sbin/start-master.sh
随之MasterUI界面也就启动了,具体 http://hadoop:8080/,在该页面可以查看到master-spark-URL
相应的,如果将其他从节点组件连接到Master时,可以在从节点主机上键入以下命令:
./$SPARK_HOME/sbin/start-slave.sh <master-spark-URL>
将应用程序连接到群集
因此,对集群运行交互式Spark shell时,通过以下命令行:
./$SPARK_HOME/bin/spark-shell --master spark://IP:PORT
此时可以查看Spark shell application UI http://hadoop:4041
● 详细配置,请移步到官方文档 spark-standalone
转载请注明出处:http://blog.csdn.net/coder__cs/article/details/79118590
本文出自【elon33的博客】