【若泽大数据高级第三天】--基于阿里云的Hadoop HA和YARN HA部署-zookeeper

Hadoop HA和YARN HA架构

基于阿里云的Hadoop HA架构
基于阿里云的YARN HA架构

进程部署情况

主机名	部署进程名称	进程名称	进程名称	进程名称	进程名称
Hadoop001	zookeeper	NN	ZKFC	JN	DN
Hadoop002	zookeeper	NN	ZKFC	JN	DN
Hadoop003	zookeeper	-	-	JN	DN
Hadoop001	zookeeper	RM(zkfc)	NM	-	-
Hadoop002	zookeeper	RM(zkfc)	NM	-	-
Hadoop003	zookeeper	-	NM	-	-

• JN进程组成的集群
  在任意时刻只能有一个NN active写，NN standby进行读
  部署进程数：2n+1
  至少向JN节点成功写入n/2 + 1 个节点
  读取时是任意选择一台进行读取

HDFS YARN HA部署

环境准备

配置hosts文件

[root@hadoop001 opt]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.130	hadoop001	hadoop001
192.168.137.131	hadoop002	hadoop002
192.168.137.132	hadoop003	hadoop003

[root@hadoop002 opt]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.130	hadoop001	hadoop001
192.168.137.131	hadoop002	hadoop002
192.168.137.132	hadoop003	hadoop003

[root@hadoop003 opt]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.130	hadoop001	hadoop001
192.168.137.131	hadoop002	hadoop002
192.168.137.132	hadoop003	hadoop003

创建hadoop集群管理用户–hadoop

[root@hadoop001 opt]# useradd zookeeper
[root@hadoop001 opt]# su - zookeeper
[zookeeper@hadoop001 ~]$ mkdir app opt
[zookeeper@hadoop001 ~]$ ll
total 8
drwxrwxr-x. 2 zookeeper zookeeper 4096 Apr 10 16:06 app
drwxrwxr-x. 2 zookeeper zookeeper 4096 Apr 10 16:06 opt

[root@hadoop002 opt]# useradd zookeeper
[root@hadoop002 opt]# su - zookeeper
[zookeeper@hadoop002 ~]$ mkdir app opt
[zookeeper@hadoop002 ~]$ ll
total 8
drwxrwxr-x. 2 zookeeper zookeeper 4096 Apr 10 16:06 app
drwxrwxr-x. 2 zookeeper zookeeper 4096 Apr 10 16:06 opt

[root@hadoop003 opt]# useradd zookeeper
[root@hadoop003 opt]# su - zookeeper
[zookeeper@hadoop003 ~]$ mkdir app opt
[zookeeper@hadoop003 ~]$ ll
total 8
drwxrwxr-x. 2 zookeeper zookeeper 4096 Apr 10 16:06 app
drwxrwxr-x. 2 zookeeper zookeeper 4096 Apr 10 16:06 opt

• 将准备好的安装包cp到app目录
  

配置用户ssh互信关系，支持多台机器间的无密码访问

【Linux基础】建立多服务器互信关系

部署JDK

【Linux基础】CentOS部署JDK

关闭防火墙

云主机防火墙默认是关闭状态

部署zookeeper

• 解压zookeeper安装包

tar -zxvf ./opt/zookeeper-3.4.6.tar.gz -C ./app/

• 创建目录软连接

[zookeeper@hadoop001 app]$ ln -s zookeeper-3.4.6/ zookeeper

• 配置zookeeper环境变量

[zookeeper@hadoop001 app]$ vi ~/.bash_profile
PATH=$PATH:$HOME/bin
export PATH

export ZOOKEEPER_HOME=/home/zookeeper/app/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH

• 在hadoop001中配置zookeeper配置文件

[hadoop@hadoop002 conf]$ cp zoo_sample.cfg zoo.cfg
[hadoop@hadoop002 conf]$ vi zoo.cfg 
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/home/hadoop/app/zookeeper/data				# 修改了目录 
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
## 新增指定服务器ID
server.1=hadoop001:2888:3888
server.2=hadoop002:2888:3888
server.3=hadoop003:2888:3888

—创建dataDir

mkdir /home/hadoop/app/zookeeper/data

• 创建指定的ID
  

• 将hadoop001配置好的zoo.cfg文件传到hadoop002、hadoop003

[hadoop@hadoop001 zookeeper]$ scp conf/zoo.cfg hadoop002:/home/hadoop/app/zookeeper/conf
zoo.cfg                                                    100% 1027     1.0KB/s   00:00    
[hadoop@hadoop001 zookeeper]$ scp conf/zoo.cfg hadoop003:/home/hadoop/app/zookeeper/conf
zoo.cfg                                                    100% 1027     1.0KB/s   00:00    
[hadoop@hadoop001 zookeeper]$ 

• 启动zookeeper
  
• 修改zookeeper.out文件输出路径

[hadoop@hadoop003 bin]$ cd /home/hadoop/app/zookeeper/bin
[hadoop@hadoop003 bin]$ vi zkEnv.sh
if [ "x${ZOO_LOG_DIR}" = "x" ]
then
    ZOO_LOG_DIR="/home/hadoop/app/zookeeper/log"
fi

• 重新启动zookeeper后查看日志