前置准备
CentOS7、jdk1.8、hadoop-2.7.7、zookeeper-3.5.7
想要完成本期视频中所有操作,需要以下准备:
大数据开发之Hadoop前置准备
Hadoop完全分布式集群环境搭建-视频教程
HA(高可用)-Hadoop集群环境搭建视频+图文教程
大数据常用shell脚本之分发脚本编写-视频教程
大数据常用shell脚本之ha-hadoop脚本编写-视频教程
一、集群规划
下面的步骤我们将实现把hadoop04动态上线,再将hadoop04动态下线
二、启动现有集群
[root@master ~]$ ha-hadoop.sh start
三、服役新节点(动态上线节点)
3.1 创建新的节点(实际场景下机器已备好)
克隆一个新节点,然后务必将之前集群的数据删除(配置文件中配置的存放数据的目录)。自定义用户、固定IP、主机名、IP与主机映射、自身免密登陆、防火墙都搞好。
[root@master ~]$ rm -rf /opt/software/hadoop-2.7.7/tmp/*
[root@master ~]$ rm -rf /opt/software/hadoop-2.7.7/dfs/journalnode_data/*
[root@master ~]$ rm -rf /opt/software/hadoop-2.7.7/dfs/edits/*
[root@master ~]$ rm -rf /opt/software/hadoop-2.7.7/dfs/datanode_data/*
[root@master ~]$ rm -rf /opt/software/hadoop-2.7.7/dfs/namenode_data/*
将hadoop04的公钥发给hadoop01和02
[root@master ~]$ ssh-copy-id hadoop01
[root@master ~]$ ssh-copy-id hadoop02
将hadoop01和02的公钥发给slave2
[root@masterback~]$ ssh-copy-id slave2
[root@masterback ~]$ ssh-copy-id slave2
3.2 创建dn-include.conf文件
在NameNode的hadoop安装目录的etc/hadoop目录下创建dn-include.conf文件(注意文件内一行一个主机名),内容如下
[xiaokang@hadoop01 ~]$ touch /opt/software/hadoop-2.7.7/etc/hadoop/dn-include.conf
hadoop01
hadoop02
hadoop03
hadoop04
3.3 修改hdfs-site.xml和slaves
hdfs-site.xml配置文件中增加dfs.hosts属性
<property>
<name>dfs.hosts</name>
<value>/opt/software/hadoop-2.7.7/etc/hadoop/dn-include.conf</value>
</property>
slaves
hadoop01
hadoop02
hadoop03
hadoop04
3.4 分发文件
[xiaokang@hadoop01 ~]$ distribution-dn.sh /opt/software/hadoop-2.7.7/etc/hadoop/dn-include.conf
[xiaokang@hadoop01 ~]$ distribution-dn.sh /opt/software/hadoop-2.7.7/etc/hadoop/hdfs-site.xml
[xiaokang@hadoop01 ~]$ distribution-dn.sh /opt/software/hadoop-2.7.7/etc/hadoop/slaves
3.5 启动DN和NM
[xiaokang@hadoop04 ~]$ hadoop-daemon.sh start datanode
[xiaokang@hadoop04 ~]$ yarn-daemon.sh start nodemanager
3.6 主NN和主RM进行刷新
[xiaokang@hadoop01 ~]$ hdfs dfsadmin -refreshNodes
[xiaokang@hadoop03 ~]$ yarn rmdmin -refreshNodes
3.7 开启数据均衡
[xiaokang@hadoop04 ~]$ start-balancer.sh -threshold 10
参数值10代表集群中各个节点的磁盘空间利用率相差不超过10%,可根据实际情况进行调整,值越低各节点越平衡,但这样消耗时间也更长
数据分散差不多后,这里一定要记得停止数据均衡,不然会一直检测各个节点的磁盘空间利用率,很耗费资源
[xiaokang@hadoop04 ~]$ stop-balancer.sh
四、退役旧节点(动态下线节点)
退役之前咱们先上传一个74M的jar包,hadoop04退役之后数据会转移到其它DN之上
[xiaokang@hadoop04 ~]$ hdfs dfs -put event_ingestion-0.0.1.jar /
4.1 创建dn-exclude.conf文件
在NameNode的hadoop安装目录的etc/hadoop目录下创建dn-exclude.conf文件(注意文件内一行一个主机名),将需要退役的节点的主机名加入其中
hadoop04
4.2 修改hdfs-site.xml
hdfs-site.xml配置文件中增加dfs.hosts.exclude属性
<property>
<name>dfs.hosts.exclude</name>
<value>/opt/software/hadoop-2.7.7/etc/hadoop/dn-exclude.conf</value>
</property>
4.3 分发文件
[xiaokang@hadoop01 ~]$ distribution-dn.sh /opt/software/hadoop-2.7.7/etc/hadoop/dn-exclude.conf
[xiaokang@hadoop01 ~]$ distribution-dn.sh /opt/software/hadoop-2.7.7/etc/hadoop/hdfs-site.xml
4.4 主NN和主RM进行刷新
[xiaokang@hadoop01 ~]$ hdfs dfsadmin -refreshNodes
[xiaokang@hadoop03 ~]$ yarn rmdmin -refreshNodes
通过WebUI可以看到退役节点的状态为Decommission In Progress(退役中),说明数据节点正在复制块到其它节点。
4.5 关闭DN和RM
等待退役节点状态为 Decommissioned (所有块已经复制完成),执行下面操作
[xiaokang@hadoop04 ~]$ hadoop-daemon.sh stop datanode
[xiaokang@hadoop04 ~]$ yarn-daemon.sh stop nodemanager
4.6 从slaves和dn-include.conf中剔除退役节点
1.将slaves和dn-include.conf中将hadoop04去除
2.修改好后进行分发
[xiaokang@hadoop01 hadoop]$ distribution-dn.sh /opt/software/hadoop-2.7.7/etc/hadoop/slaves
[xiaokang@hadoop01 hadoop]$ distribution-dn.sh /opt/software/hadoop-2.7.7/etc/hadoop/dn-include.conf
3.主NN和主RM进行刷新
[xiaokang@hadoop01 ~]$ hdfs dfsadmin -refreshNodes
[xiaokang@hadoop03 ~]$ yarn rmdmin -refreshNodes
最后大功告成,如下图所示