下载解压
下载地址:http://spark.apache.org/downloads.html
环境配置
先装好java, scala, HDFS
配置SPARK_HOME,Path
启动Spark-shell
启动时出错: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are:-------
解决方案:D:\hadoop-2.7.3\bin\winutils.exe chmod 777 D:\tmp\hive winutils可以到这里下载:https://github.com/steveloughran/winutils
启动成功可以看到:
编写单词计数程序
var file=sc.textFile("file:///D:/spark-2.2.0-bin-hadoop2.7/README.md")
var wordCount= file.flatMap(line=>line.split (" ")). map(word=>(word,1)).
reduceByKey((a,b) =>a+b). collect()
读取HDFS文件
var file=sc.textFile("hdfs://127.0.0.1:9000/files/README.txt")
var wordCount= file.flatMap(line=>line.split (" ")). map(word=>(word,1)).
reduceByKey((a,b) =>a+b). collect()