搭建好Spark环境之后,简单实用一下:
代码:
val file = sc.textFile("file:///home/iie4bu/data/hello.txt")
val wordCounts = file.flatMap(line => line.split(",")).map((word => (word,1))).reduceByKey(_ + _)
wordCounts.collect
hello.txt
文件内容如下:
hello world welcome
hello welcome
运行shell:
./spark-shell --master spark://manager:7077