Centos+spark2.1.0安装

本环境在hadoop完全分布式环境下搭建，在安装spark之前已完成hadoop完全分布式部署

1.下载spark2.1.0解压缩到/usr/local，重命名为spark

2.进入 /spark/conf下将spark-env.sh.template重命名为spark-env.sh，将slaves.template修改为slaves

mv spark-env.sh.template spark-env.sh
mv slaves.template slaves

3.修改spark-env.sh

export JAVA_HOME=/usr/local/jdk1.7.0_80/
export HADOOP_CONF_DIR=/usr/local/hadoop/
export SPARK_MASTER_HOST=hadoop01
export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=1g

 4.修改slaves 注意：hadoop01为master主机名，hadoop02,03为从主机名，并且已在/etc/hosts中配置

hadoop01
hadoop02
hadoop03

 4.1/etc/hosts文件内容

[root@hadoop01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.25.135 hadoop01 www.hadoop01.com
192.168.25.136 hadoop02 www.hadoop02.com
192.168.25.137 hadoop03 www.hadoop03.com

5.配置spark环境变量

vi /etc/profile

vi /etc/profile

#set java environment
JAVA_HOME=/usr/local/jdk1.7.0_80
CLASSPATH=.:$JAVA_HOME/lib.tools.jar
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH 
HADOOP_HOME=/usr/local/hadoop
ZK_HOME=/usr/local/zookeeper-3.4.9
HIVE_HOME=/usr/local/hive/
SQOOP_HOME=/usr/local/sqoop
SPARK_HOME=/usr/local/spark
PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZK_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin:$SPARK_HOME/bin:

6.重新加载配置文件

source /etc/profile

7.将配置好的环境变量分发到其他节点上 

scp -r spark hadoop02:/usr/local/
scp -r spark hadoop03:/usr/local/

 8.进入hadoop/sbin启动hadoop和spark/

 ./start-all.sh

 ./start-all.sh

9.查看jps

10.浏览器查看spark运行

 

11.启动spark-shell