Docker环境下HA(高可用)-Hadoop集群

一、前置准备

Hadoop前置准备：Hadoop前置准备

Hadoop完全分布式集群环境搭建：Hadoop完全分布式集群环境搭建

Zookeeper集群环境搭建：Zookeeper集群环境搭建

Docker环境下Hadoop单机伪分布式：Docker环境下Hadoop单机伪分布式-视频教程

Docker环境下Hadoop完全分布式：Docker环境下Hadoop完全分布式-视频教程

Docker之镜像拉取、删除及重命名：Docker之镜像拉取、删除及重命名-视频教程

二、集群规划

三、搭建Zookeeper集群

3.1 创建、启动并进入容器

在这里大家需要将上一期的五个容器中的前三个提交成镜像，因为我们的端口映射需要做相应的更改（hadoop01、hadoop02和hadoop03）：

[xiaokang@hadoop ~]$ sudo docker run -d --name hadoop01 --hostname hadoop01 -P -p 50070:50070 -p 19888:19888 --net xiaokang-network --ip 172.24.0.2 --privileged 48733864a062 /usr/sbin/init
[xiaokang@hadoop ~]$ sudo docker run -d --name hadoop02 --hostname hadoop02 -P -p 50072:50070 -p 8088:8088 --net xiaokang-network --ip 172.24.0.3 --privileged a7e1b3514980 /usr/sbin/init
[xiaokang@hadoop ~]$ sudo docker run -d --name hadoop03 --hostname hadoop03 -P -p 8083:8088 --net xiaokang-network --ip 172.24.0.4 --privileged 91e46b86dbb1 /usr/sbin/init

[xiaokang@hadoop ~]$ sudo docker exec -ti 400942b927f6 /bin/bash
[root@hadoop01 ~]#

[xiaokang@hadoop ~]$ sudo docker exec -ti 98737bd9503d /bin/bash
[root@hadoop02 ~]#

[xiaokang@hadoop ~]$ sudo docker exec -ti c600aa6e9d1a /bin/bash
[root@hadoop03 ~]#

3.2 复制解压包到容器内(hadoop01)

此步骤需要在宿主机进行操作：

[xiaokang@hadoop ~]$ sudo docker cp /home/xiaokang/apache-zookeeper-3.5.7-bin.tar.gz 400942b927f6:/opt/software/

3.3 解压并配置

#解压
[xiaokang@hadoop01 ~]$ tar -zxvf /opt/software/apache-zookeeper-3.5.7-bin.tar.gz -C /opt/software/
[xiaokang@hadoop01 software]$ mv apache-zookeeper-3.5.7-bin/ zookeeper-3.5.7

先创建所需目录：

[xiaokang@hadoop01 ~]$ mkdir /opt/software/zookeeper-3.5.7/zoo_data
[xiaokang@hadoop01 ~]$ mkdir /opt/software/zookeeper-3.5.7/zoo_logs

修改其配置文件 zoo.cfg，内容如下。之后使用 scp 命令将安装包分发到另外的节点上：

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/software/zookeeper-3.5.7/zoo_data
dataLogDir=/opt/software/zookeeper-3.5.7/zoo_logs
clientPort=2181

# server.1 这个1是服务器的标识，可以是任意有效数字，标识这是第几个服务器节点，这个标识要写到dataDir目录下面myid文件里
# 指名集群间通讯端口和选举端口
server.1=hadoop01:2888:3888
server.2=hadoop02:2888:3888
server.3=hadoop03:2888:3888

分别在三台主机的 dataDir 目录下新建 myid 文件,并写入对应的节点标识。Zookeeper 集群通过 myid 文件识别集群节点，并通过上文配置的节点通信端口和选举端口来进行节点通信，选举出 Leader 节点。

[xiaokang@hadoop01 ~]$ echo 1 >> /opt/software/zookeeper-3.5.7/zoo_data/myid

3.4 分发安装包

[xiaokang@hadoop01 ~]$ scp -r /opt/software/zookeeper-3.5.7 xiaokang@hadoop02:/opt/software/
[xiaokang@hadoop01 ~]$ scp -r /opt/software/zookeeper-3.5.7 xiaokang@hadoop03:/opt/software/

修改hadoop02和hadoop03中的myid值

[xiaokang@hadoop02 ~]$ vim /opt/software/zookeeper-3.5.7/zoo_data/myid
[xiaokang@hadoop03 ~]$ vim /opt/software/zookeeper-3.5.7/zoo_data/myid

3.5 启动集群

#这里不配置环境变量的话需要到bin目录下执行
[xiaokang@hadoop01 bin]$ ./zkServer.sh start
[xiaokang@hadoop02 bin]$ ./zkServer.sh start
[xiaokang@hadoop03 bin]$ ./zkServer.sh start

#查看集群状态
[xiaokang@hadoop01 bin]$ ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/software/zookeeper-3.5.7/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
[xiaokang@hadoop02 bin]$ ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/software/zookeeper-3.5.7/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
[xiaokang@hadoop03 bin]$ ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/software/zookeeper-3.5.7/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader

四、Hadoop集群配置

先创建好所需目录：

#删除完全分布式创建的目录
[xiaokang@hadoop01 ~]$ rm -rf /opt/software/hadoop-2.7.7/tmp/ /opt/software/hadoop-2.7.7/dfs/ /opt/software/hadoop-2.7.7/checkpoint/ /opt/software/hadoop-2.7.7/logs/
[xiaokang@hadoop01 ~]$ mkdir -p /opt/software/hadoop-2.7.7/tmp
[xiaokang@hadoop01 ~]$ mkdir -p /opt/software/hadoop-2.7.7/dfs/journalnode_data
[xiaokang@hadoop01 ~]$ mkdir -p /opt/software/hadoop-2.7.7/dfs/edits
[xiaokang@hadoop01 ~]$ mkdir -p /opt/software/hadoop-2.7.7/dfs/datanode_data
[xiaokang@hadoop01 ~]$ mkdir -p /opt/software/hadoop-2.7.7/dfs/namenode_data

1. hadoop-env.sh

#25行 export JAVA_HOME
export JAVA_HOME=/opt/moudle/jdk1.8.0_191
#33行 export HADOOP_CONF_DIR
export HADOOP_CONF_DIR=/opt/software/hadoop-2.7.7/etc/hadoop

2. core-site.xml

<configuration>
	<property>
		<!--指定hadoop集群在zookeeper上注册的节点名-->
		<name>fs.defaultFS</name>
		<value>hdfs://hacluster</value>
	</property>  
	<property>
		<!--用来指定hadoop运行时产生文件的存放目录-->   
		<name>hadoop.tmp.dir</name>
		<value>/opt/software/hadoop-2.7.7/tmp</value>
	</property>
	<property>
			<!--设置缓存大小，默认4kb-->
	        <name