Spark学习笔记-windows环境搭建

下载解压

下载地址:http://spark.apache.org/downloads.html

环境配置

先装好java, scala, HDFS

配置SPARK_HOME,Path

启动Spark-shell

启动时出错： The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are:-------

解决方案：D:\hadoop-2.7.3\bin\winutils.exe chmod 777 D:\tmp\hive   winutils可以到这里下载：https://github.com/steveloughran/winutils

启动成功可以看到：

编写单词计数程序

var file=sc.textFile("file:///D:/spark-2.2.0-bin-hadoop2.7/README.md")
var wordCount= file.flatMap(line=>line.split (" ")). map(word=>(word,1)).
 reduceByKey((a,b) =>a+b). collect()

读取HDFS文件

var file=sc.textFile("hdfs://127.0.0.1:9000/files/README.txt")
var wordCount= file.flatMap(line=>line.split (" ")). map(word=>(word,1)). 
reduceByKey((a,b) =>a+b). collect()

界面查看任务

http://127.0.0.1:4040/jobs/

转载于:https://my.oschina.net/Endless2010/blog/1422896