spark环境安装http://www.jianshu.com/p/7b325155edab
Java环境搭建
JDK1.7下载,百度网盘 ,http://www.oracle.com/technetwork/java/javase/downloads/index.html
Scala下载安装 2.10.0
spark下载 1.6.2 sbt 0.13.8
IDEA下载安装
Scala插件下载 https://plugins.jetbrains.com/plugin/1347-scala
创建project
配置ssh无密登录
ssh-keygen
pwd
cd .ssh
touch authorized_keys
cat id_rsa.pub > authorized_keys
chmod 600 authorized_keys
ssh localhost 无密登录
Wordcount:
创建spark Context
加载数据
把每一行分割为单词
转换成pairs并且计数
create Scala class
object Wordcount{
}
打jar包,project structure,artifact,+,jar
启动集群:
启动master, ./sbin/start-master.sh
启动worker, ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://localhost.localdomain:7077
re -be 从本地上传文件 uers/项目名称/out/artifical/.jar
提交作业 ./bin/spark-submit --master spark://localhost.localdomain:7077 --class WordCount /home/maixia/soft/imoocpro.jar