hadoop的HA集群搭建
环境
centos7,jdk1.8,zookeeper-3.5.5,hadoop-2.7.2
1.配置hosts:vi /etc/hosts
192.168.221.100 hadoop100
192.168.221.101 hadoop101
192.168.221.102 hadoop102
192.168.221.103 hadoop103
192.168.221.104 hadoop104
zookeeper的集群安装
1.进入zookeeper的conf目录下进行操作
cp zoo_sample.cfg zoo.cfg
2.zoo.cfg的配置
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/usr/local/software/zookeeper/zookeeper-3.5.5/data
# the port at which the clients will connect
clientPort=2181
server.0=192.168.221.102:2888:3888
server.1=192.168.221.103:2888:3888
server.2=192.168.221.104:2888:3888
(2)配置参数解读
Server.A=B:C:D。
A是一个数字,表示这个是第几号服务器;
B是这个服务器的IP地址;
C是这个服务器与集群中的Leader服务器交换信息的端口;
D是万一集群中的Leader服务器挂了,需要一个端口来重新进行选举,选出一个新的Leader,而这个端口就是用来执行选举时服务器相互通信的端口。
集群模式下配置一个文件myid,这个文件在dataDir目录下,这个文件里面有一个数据就是A的值,Zookeeper启动时读取此文件,拿到里面的数据与zoo.cfg里面的配置信息比较从而判断到底是哪个server
3.在data(dataDir)的目录下创建文件;在文件中添加与server对应的编号:如下图
vi myid
4.配置环境变量
#set zookeeper environment
export ZK_HOME=/usr/local/software/zookeeper/zookeeper-3.5.5
export PATH=$PATH:$ZK_HOME/bin
hadoop的搭建
1.进入hadoop的etc的hadoop文件下
2.配置core-site.xml
<!-- 指定HDFS中NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop102:9000</value>
</property>
<!-- 指定Hadoop运行时产生文件的存储目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/software/hadoop-2.7.2/data/tmp</value>
</property>
<!-- 指定zookeeper地址 -->
<property