HDFS和YARN HA部署

一、版本

组件名	版本	备注及下载地址
Centos	7.2 64bit	lsb_release -a查看操作系统版本 file /bin/ls查看操作系统位数
JRE	java version 1.8.45	jdk-8u45-linux-x64.gz
Hadoop	hadoop-2.6.0-cdh5.15.1	hadoop-2.6.0-cdh5.15.1.tar.gz
Zookeeper	zookeeper-3.4.6	zookeeper-3.4.6.tar.gz

二、主机规划

Host	安装软件	进程
ruozedata001	Hadoop, Zookeeper	Name Node/DFSZKFailoverController/JournalNode/DataNode/ Resouce Manager/JobHistoryServer/NodeManager/QuorumPeerMain
ruozedata002	Hadoop, Zookeeper	Name Node/DFSZKFailoverController/JournalNode/DataNode/ Resouce Manager/JobHistoryServer/NodeManager/QuorumPeerMain
ruozedata003	Hadoop, Zookeeper	JournalNode/DataNode/NodeManager/QuorumPeerMain

三、目录规划

名称	路径	备注
$HADOOP_NAME	/opt/software/hadoop
Data	$HADOOP_NAME/data
Log	$HADOOP_NAME/logs
hadoop.tmp.dir	$HADOOP_NAME/tmp	需要手工创建，权限777,root:root
$ZOOKEEPER_HOME	/opt/software/zookeeper

四、常用脚本及命令

1.启动集群
[root@ruozedata001 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh start
[root@ruozedata002 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh start
[root@ruozedata003 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh start

[root@ruozedata001 ~]# $ZOOKEEPER_HOME/sbin/start-all.sh
[root@ruozedata002 ~]# $ZOOKEEPER_HOME/sbin/yarn-daemon.sh start resourcemananger

[root@ruozedata001 ~]# $ZOOKEEPER_HOME/sbin/mr-jobhistory-daemon.sh start historyserver

2.关闭集群
[root@ruozedata001 ~]# $ZOOKEEPER_HOME/sbin/mr-jobhistory-daemon.sh stop historyserver

[root@ruozedata002 ~]# $ZOOKEEPER_HOME/sbin/yarn-daemon.sh stop resourcemananger
[root@ruozedata001 ~]# $ZOOKEEPER_HOME/sbin/stop-all.sh

[root@ruozedata001 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh stop
[root@ruozedata002 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh stop
[root@ruozedata003 ~]# $ZOOKEEPER_HOME/bin/zkServer.sh stop

3.监控集群
hdfs dfsadmin -report
###web界面

HDFS:http://120.24.190.115:50070
HDFS:http://47.106.109.17:50070

ResourceManager(Active):http://120.24.190.115:8088
ResourceManager(Standby):http://47.106.109.17:8088/cluster/cluster

JobHistory:http://120.24.190.115:19888/jobhistory

4.单个进程关闭/启动
hadoop-daemon.sh start|stop namenode|datanode|journalnode|zkfc
yarn-daemon.sh start|stop resourcemanager|nodemanager

五、环境配置

1.现在在阿里云上买三台云主机（2 vCPU 8 GiB (I/O优化) 1Mbps），按量付费的。
https://blog.csdn.net/weixin_41158277/article/details/99900916

2.每台机器上增加hadoop用户

[root@ruozedata001 ~]# useradd hadoop
[root@ruozedata001 ~]# su - hadoop
[hadoop@ruozedata001 ~]$ mkdir app software source data lib script tmp maven_repos

3.上传hadoop-2.6.0-cdh5.15.1.tar.gz，jdk-8u45-linux-x64.gz，zookeeper-3.4.6.tar.gz到/software

[hadoop@ruozedata001 software]$ rz

4.IP与hostname绑定（3台）

[root@ruozedata001 ~]# vi /etc/hosts
172.18.98.137 ruozedata001
172.18.98.138 ruozedata002
172.18.98.136 ruozedata003

5.设置3台machines，SSH互相通信

开启发送健输入到所有会话
[hadoop@ruozedata001 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
7d:4c:45:03:98:87:f8:aa:f3:d8:7f:c4:f9:ad:74:b2 hadoop@ruozedata001
The key's randomart image is:
+--[ RSA 2048]----+
|         . +.o+  |
|        . + .. . |
|         . ..    |
|         ..o     |
|        S...o.   |
|        .  .+    |
|       .   . .o..|
|      oo    ...+.|
|      .oo...  E. |
+-----------------+
[hadoop@ruozedata001 ~]$ cd .ssh
[hadoop@ruozedata001 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1679 Aug 23 00:34 id_rsa
-rw-r--r-- 1 hadoop hadoop  401 Aug 23 00:34 id_rsa.pub

选取ruozedata001作为主节点，其他两台作为分节点

#相当于ruozedata001机器的公钥放在这个可信任的文件里面，然后把ruozedata002、ruozedata003的公钥文件传给ruozedata001

hadoop@ruozedata001 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

#scp id_rsa.pub root@172.18.98.137:/home/hadoop/.ssh/id_rsa2
172.18.98.137这个是ruozedata001的内网ip
ruozedata003也是一样。

[hadoop@ruozedata002 ~]$ cd .ssh
[hadoop@ruozedata002 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1679 Aug 23 00:34 id_rsa
-rw-r--r-- 1 hadoop hadoop  401 Aug 23 00:34 id_rsa.pub
[hadoop@ruozedata002 .ssh]$ scp id_rsa.pub root@172.18.98.137:/home/hadoop/.ssh/id_rsa2
The authenticity of host '172.18.98.137 (172.18.98.137)' can't be established.
ECDSA key fingerprint is 56:84:2b:95:01:36:b7:11:18:54:7e:49:f0:e0:91:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.18.98.137' (ECDSA) to the list of known hosts.
root@172.18.98.137's password: 
id_rsa.pub                                                                           100%  401     0.4KB/s   00:00    
[hadoop@ruozedata002 .ssh]$

[hadoop@ruozedata003 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Aug 23 00:34 id_rsa
-rw-r--r-- 1 hadoop hadoop  401 Aug 23 00:34 id_rsa.pub
[hadoop@ruozedata003 .ssh]$ scp id_rsa.pub root@172.18.98.137:/home/hadoop/.ssh/id_rsa3
The authenticity of host '172.18.98.137 (172.18.98.137)' can't be established.
ECDSA key fingerprint is 56:84:2b:95:01:36:b7:11:18:54:7e:49:f0:e0:91:5d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.18.98.137' (ECDSA) to the list of known hosts.
root@172.18.98.137's password: 
id_rsa.pub                                                                           100%  401     0.4KB/s   00:00    
[hadoop@ruozedata003 .ssh]$

[hadoop@ruozedata001 .ssh]$ ll
total 24
-rw------- 1 hadoop hadoop  401 Aug 23 00:41 authorized_keys
-rw------- 1 hadoop hadoop 1679 Aug 23 00:34 id_rsa
-rw-r--r-- 1 root   root    401 Aug 23 00:50 id_rsa2
-rw-r--r-- 1 root   root    401 Aug 23 00:54 id_rsa3
-rw-r--r-- 1 hadoop hadoop  401 Aug 23 00:34 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop  376 Aug 23 01:01 known_hosts
[hadoop@ruozedata001 .ssh]$ cat id_rsa2 >> authorized_keys
[hadoop@ruozedata001 .ssh]$ cat id_rsa3 >> authorized_keys
[hadoop@ruozedata001 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCy64olNdPTq20DgFil2WcWp4FraJHoKEyapmSNyI1bzZH78VulQsqGJZMluInoEon5nuacN0AN638MGy0jOZJ1V1eTaoNEKjmuO8jtKIyuvl8LWF+tKo/Lv4ATU5vfTNg96BMdDxjX5pJmGvWaT55Pi+4C3tI/etPox1luqQkZyMKdIM853KIwmUgrjhhyAUaO3k9YIRoGXO5sDJeh35UpwLPoR2ysT4YWUmSd0gVxW210ZgwQJU5mdSr8JpIJx9G3KAYVaU0+mY8FCs1jm4G/IXQsCwIjZilbwbDFMLj6RZDH5E/npXRNeTY+JBhMExNPso+gEUx1qYlXJkY2Phzd hadoop@ruozedata001
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/SJOwDpturrrxszJFBrqHkXfS5vg6U0EtA+w8bPuuNQnkQKWGivX8Bnhe5vHKsQ95pejfHHF5weXRgGotxRXwahaB/1CpJBilvRlNIRO5uGrlYMv0SoppfNAQhCtoy3rBiiTyXr5IeXyK5C9UxeZ62F6E7tE+idcsPGxDjHMrtKd9nV9gkMPULe8M0Y62SH9M/BDIhZBAchB+KEDfWu4nbWDSz0oSa2nDaNsyi6VOoQMFLF5e1dXkYIKWOpGmroxR1D5R5bVYwepab5jtq+IxyVL39b6MUs0mcPSMpWOFc0TssExhfcDMyz12WW8Pdw3sN8TbywEfyLkncZdhIIjv hadoop@ruozedata002
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYLPhxaFKjymbH54uy2FQ8czEUr1ufGXWZU/j5QcSmB4223BEiaZ2HlK7/3JKiM9vMg14jCuDk9YyT/a+m4OfPj7e7QsIQLNCG9vdNAlwzr+DhNPywsABuoc+dnc7V3Pef9aHFLusiZgzbACxKy0OZ6nMGx38nbdRVxY76ATFQg3Wv/p4zXh56xh5xVmhIWgQTKOsSwK11JErOKtQvKGktDC+qg0pMc3md6PgDn3fiSym4YP6th16sz8YD5ZSC0mAoJM2tL7SFnly0uTIF9fWiPXGxNMLLN3l4S3MBpYp+DmXZMnTggab5vN1BNmYq5oZTpVcWsZ7gXRG7IPwDX6LZ hadoop@ruozedata003
[hadoop@ruozedata001 .ssh]$

scp authorized_keys root@ruozedata002:/home/hadoop/.ssh/
scp authorized_keys root@ruozedata003:/home/hadoop/.ssh/

[hadoop@ruozedata001 .ssh]$ scp authorized_keys root@ruozedata002:/home/hadoop/.ssh/
root@ruozedata002's password: 
authorized_keys                                                                      100% 1203     1.2KB/s   00:00    
[hadoop@ruozedata001 .ssh]$ scp authorized_keys root@ruozedata003:/home/hadoop/.ssh/
The authenticity of host 'ruozedata003 (172.18.98.136)' can't be established.
ECDSA key fingerprint is 4c:50:25:e7:47:46:a2:31:7e:84:f8:d2:6a:88:3d:09.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ruozedata003,172.18.98.136' (ECDSA) to the list of known hosts.
root@ruozedata003's password: 
authorized_keys                                                                      100% 1203     1.2KB/s   00:00    
[hadoop@ruozedata001 .ssh]$ chmod 600 authorized_keys 


[hadoop@ruozedata002 .ssh]$ exit
logout
[root@ruozedata002 ~]# cd /home/hadoop/.ssh/
[root@ruozedata002 .ssh]# chmod 644 authorized_keys
[root@ruozedata002 .ssh]# ll
total 16
-rw-r--r-- 1 root   root   1203 Aug 23 01:14 authorized_keys
-rw------- 1 hadoop hadoop 1675 Aug 23 00:34 id_rsa
-rw-r--r-- 1 hadoop hadoop  401 Aug 23 00:34 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop  175 Aug 23 00:54 known_hosts


[hadoop@ruozedata003 .ssh]$ exit
logout
[root@ruozedata003 ~]# cd /home/hadoop/.ssh/
[root@ruozedata003 .ssh]# chmod 644 authorized_keys
[root@ruozedata003 .ssh]# ll
total 16
-rw-r--r-- 1 root   root   1203 Aug 23 01:14 authorized_keys
-rw------- 1 hadoop hadoop 1675 Aug 23 00:34 id_rsa
-rw-r--r-- 1 hadoop hadoop  401 Aug 23 00:34 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop  175 Aug 23 00:54 known_hosts

ssh ruozedata001 date
ssh ruozedata002 date
ssh ruozedata003 date

[hadoop@ruozedata001 .ssh]$ ssh ruozedata001 date
The authenticity of host 'ruozedata001 (172.18.98.137)' can't be established.
ECDSA key fingerpri