Hadoop+Spark+Zookeeper 集群搭建

Hadoop version	2.6.0
Version	2.6.0, re3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
Compiled	2014-11-13T21:10Z by jenkins from (detached from e349649)

---

Spark RELEASE	1.3.1
Version	Spark 1.3.1 (git revision 908a0bf) built for Hadoop 2.6.0
Build flags	-Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -Pyarn -

---

ZooKeeper RELEASE	3.4.6
Version	3.4.6
Build flags	3.4.6-1569965

---

1、系统信息
[root@IC-1 ~]# cat /etc/issue
Red Hat Enterprise Linux Server release 6.4 (Santiago)
Kernel \r on an \m

[root@IC-1 ~]# uname -a
Linux IC-1 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@IC-1 ~]# hostname
IC-1

[root@IC-1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.1.9 IC-1
192.168.1.7 IC-2

[root@IC-1 ~]# ll /lib64/libc.so.6
lrwxrwxrwx. 1 root root 12 May 12 21:35 /lib64/libc.so.6 -> libc-2.12.so

2、准备工作

2.1、安装jdk
[root@IC-1 ~]# yum remove java
[root@IC-1 ~]# rpm -ivh jdk-8u45-linux-x64.rpm
Preparing… ########################################### [100%]
1:jdk ########################################### [100%]
Unpacking JAR files…
rt.jar…
jsse.jar…
charsets.jar…
tools.jar…
localedata.jar…

[root@lsn ~]# java -version
java version “1.8.0_45”
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

2.2、/etc/profile配置，增加：

export JAVA_HOME=/usr/java/jdk1.8.0_45
export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin

2.3、ssh免密码登陆（在root目录完成）
[root@IC-1 ~]# ssh-keygen -t rsa -P “”
Then 一直回车
[root@IC-1 ~]# cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
[root@IC-1 ~]# scp /root/.ssh/id_rsa.pub root@IC-2:/root/.ssh/id_rsa.pub
[root@IC-1 ~]# ssh IC-2
[root@IC-2 ~]# cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
[root@IC-2 ~]# chmod 600 /root/.ssh/authorized_keys
[root@IC-2 ~]# ssh IC-1
[root@IC-1 ~]# ssh IC-2

2) 安装ZooKeeper
tar zxf zookeeper-3.4.6.tar.gz

export ZOOKEEPER_HOME=/opt/develop_tools/zookeeper-3.4.6
export PATH=