spark 单机模式

一.安装scala

http://www.scala-lang.org/download/2.10.6.html

tar -zxvf scala-2.10.5.tar

二.安装spark

1.下载

http://spark.apache.org/downloads.html

选择：pre build for hadoop 2.6 later

 spark-1.5.2-bin-hadoop2.6.tgz

2.tar -zxvf spark-1.5.2-bin-hadoop2.6.tar

cd spark-1.5.2-bin-hadoop2.6/conf

3.mv spark-env.sh.template spark-env.sh

4.vi spark-env.sh

export SCALA_HOME=/Users/username/opt/scala-2.10.5
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home
export SPARK_MASTER_IP=localhost
export SPARK_WORKER_MEMORY=512m
export HADOOP_CONF_DIR=/Users/username/opt/hadoop-2.6.2/etc/hadoop

5.设置环境变量

export SCALA_HOME=/Users/username/opt/scala-2.10.5

export SPARK_HOME=/Users/username/opt/spark-1.5.2-bin-hadoop2.6

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$MAVEN_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME/bin

保存退出，刷新：souce ~/.bash_profile

6.启动

./spark-1.5.2-bin-hadoop2.6/sbin/start-all.sh 

http://localhost:4040

./spark-1.5.2-bin-hadoop2.6/sbin/start-history-server.sh 

http://localhost:18080

7.验证

usernamedeMacBook-Pro:opt username$ jps                                           
235 
768 SecondaryNameNode
1264 Jps
871 ResourceManager
601 NameNode
950 NodeManager
1178 Worker
674 DataNode
404 RemoteMavenServer
1061 Master

多两个线程

8.跑官网的例子

bin/run-example  org.apache.spark.examples.SparkPi

spark-submit --class org.apache.spark.examples.SparkPi  lib/spark-examples-1.5.2-hadoop2.6.0.jar 

å