centos7中Hadoop集群动态添加datanode节点方法
功能:把datanode2数据节点动态添加到hadoop集群中
安装一台datanode的centos7操作系统datanode2
vi /etc/hostname
datanode2
ip静态配置为192.168.145.212
vi /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="yes"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="2a495777-1dc7-42b2-b378-30658fc0361e"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="192.168.145.212"
PREFIX="24"
GATEWAY="192.168.145.2"
DNS1="192.168.145.2"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes"
IPV6_PRIVACY="no"
vi /etc/hosts
127.0.0.1 core localhost.localdomain localhost
192.168.145.200 master
192.168.145.201 slave1
192.168.145.202 slave2
192.168.145.211 datanode1
192.168.145.212 datanode2
5台机子的/etc/hosts文件配置为一样
master主机中depoly.conf部署配置文件内容:
master,all,namenode,zookeeper,resourcemanager,
slave1,all,slave,namenode,zookeeper,resourcemanager,
slave2,all,slave,datanode,zookeeper,
datanode1,all,slave,datanode,zookeeper,nodemanager,
datanode2,all,slave,datanode,zookeeper,nodemanager,
在master主机中复制分发到其余4台机器中
deploy.sh /etc/hosts /etc/hosts slave
关闭防火墙:
systemctl -stop firewalld.service
永久关闭防火墙:
systemctl disable firewalld.service
永久关闭selinux:
vi /etc/selinux/config文件
设置“SELINUX=disabled”
免密设置
先在datanode2服务器上添加用户组hadoop和用户hadoop3,然后生成本地密码文件
groupadd hadoop
useradd -G hadoop hadoop3
给用户hadoop3设置密码
passwd hadoop3
1
生成密码文件并合并5台服务器免密文件
su hadoop3
cd /home/hadoop3
mkdir .ssh
ssh-keygen –t rsa
cd .ssh
ls
cat id_rsa.pub >>id_rsa212.pub
cd ..
chmod 700 .ssh
chmod 600 .ssh/*
scp id_rsa212.pub hadoop3@master:/home/hadoop3/.ssh/
ssh master
cd /home/hadoop3/.ssh
cat id_rsa212.pub >>authorized_keys
部署5台服务器免密操作
deploy.sh /home/hadoop3/.ssh/authorized_keys /home/hadoop3/.ssh/authorized_keys slave
拷贝master中hadoop目录到datanode2中
scp -r /home/hadoop3/app hadoop3@datanode2:/home/hadoop3/
scp -r /home/hadoop3/data hadoop3@datanode2:/home/hadoop3/
scp -r /home/hadoop3/tools hadoop3@datanode2:/home/hadoop3/
或者一次性用户目录全拷贝过去
scp -r /home/hadoop3/ hadoop3@datanode2:/home/
(在新节点安装好hadoop,并把namenode的有关配置文件复制到该节点)
修改namenode节点的masters和slaves文件,增加该节点
hadoop3.0版本修改workers文件
vi /home/hadoop3/app/hadoop/etc/hadoop/workers
master
slave1
slave2
datanode1
datanode2
单独启动该datanode2节点上的Datanode和Nodemanager
hadoop-daemon.sh start datanode
yarn-daemon.sh start nodemanager
运行start-balancer.sh进行数据负载均衡,
start-balancer.sh
作用:当节点出现故障,或新增加节点时,数据块分布可能不均匀,负载均衡可以重新平衡各个datanode上数据块的分布
master主元数据名称节点web界面显示结果: