Spark-shell 是 Spark 给我们提供的交互式命令窗口,类似Linux的shell命令窗口。
启动spark
进入spark安装包,右键选择【在终端中打开】,使用的深度Linux系统。
输入启动命令:
./bin/spark-shell
可以通过IP:端口的形式查看spark web。
制作2个英文单词文件
我们就以spark启动的日志制作2个因为单词文件test1.txt于test2.txt。
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
20/09/01 21:25:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/09/01 21:25:23 WARN Utils: Your hostname, *******-PC resolves to a loopback address: 127.0.1.1; using 192.168.31.194 instead (on interface wlp1s0)
20/09/01 21:25:23 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
20/09/01 21:25:29 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
20/09/01 21:25:29 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
20/09/01 21:25:30 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.31.194:4040
Spark context available as 'sc' (master = local[*], app id = local-1598966724416).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.1
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_261)
Type in expressions to have them evaluated.
Type :help for more information.
运行spark统计单词
sc.textFile("input/").flatMap(_.split("\\W+")).map((_, 1)).reduceByKey(_ + _).collect
查看spark-jobs
命令解释
- textFile(“input”):读取本地文件input文件夹数据;
- flatMap(_.split(" ")):压平操作,按照空格分割符将一行数据映射成一个个单词;
- map((_,1)):对每一个元素操作,将单词映射为元组;
- reduceByKey(+):按照key将值进行聚合,相加;
- collect:将数据收集到Driver端展示。