Hadoop HA集群简单搭建

1、集群规划

2、服务器规划

3、安装文件准备

4、ha系统环境准备

4.1、操作系统安装，此环境采用centos minimal安装，系统版本CentOS Linux release 7.8.2003 (Core)。

mkdir /soft

上传安装文件到soft目录（hadoop-2.7.7-centos7.tar.gz 、jdk-8u151-linux-x64.tar.gz、 zookeeper-3.4.14.tar.gz）

4.2、关闭防火墙

systemctl stop firewalld.service && systemctl disable firewalld.service

4.3、修改selinux

sed -i 's/enforcing/disabled/g' /etc/selinux/config

4.4、Java安装

cd /soft

tar -zxvf jdk-8u151-linux-x64.tar.gz -C /usr/local/

修改环境变量

echo "export ZOOKEEPER_HOME=/opt/hadoop/zookeeper">> /etc/profile

echo "export HADOOP_HOME=/opt/hadoop/hadoop" >> /etc/profile

echo "export JAVA_HOME=/usr/local/jdk1.8.0_151" >> /etc/profile

echo "export JRE_HOME=/usr/local/jdk1.8.0_151/jre" >> /etc/profile

echo "export CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib" >> /etc/profile

echo "export PATH=/opt/hadoop/hadoop/bin:/opt/hadoop/hadoop/sbin:/opt/hadoop/zookeeper/bin:/usr/local/jdk1.8.0_151/bin:/usr/local/jdk1.8.0_151/jre/bin:$PATH" >> /etc/profile

source /etc/profile

4.5、添加hadoop用户

groupadd hadoop && useradd -g hadoop -d /home/hadoop hadoop

echo 'hadoop' | passwd hadoop --stdin

4.5、创建hadoop安装目录，并修改权限

mkdir -p /opt/hadoop && chown -R hadoop:hadoop /opt/hadoop

mkdir -p /opt/data/hadoop/hdfs && chown -R hadoop:hadoop /opt/data/hadoop/hdfs

mkdir -p /opt/data/hadoop/tmp && chown -R hadoop:hadoop /opt/data/hadoop/tmp

4.6、修改hosts配置文件

echo "192.168.32.11 node1" >>/etc/hosts

echo "192.168.32.12 node2" >>/etc/hosts

echo "192.168.32.13 node3" >>/etc/hosts

echo "192.168.32.14 node4" >>/etc/hosts

4.7、配置免密码登录(相关节点均执行)

ssh-keygen -t rsa

[root@node1 ~]# ssh-copy-id node1

[root@node1 ~]# ssh-copy-id node2

[root@node1 ~]# ssh-copy-id node3

[root@node1 ~]# ssh-copy-id node4

5、hadoop高可用安装

5.1、zookeeper安装

[root@node1 ~]# cd /soft/

tar -zxvf zookeeper-3.4.14.tar.gz -C /opt/hadoop/

mv zookeeper-3.4.14/ zookeeper

配置zoo.cfg配置文件，并创建相关目录

[root@node1 ~]# cd /opt/hadoop/zookeeper/conf

[root@node1 conf]# cp zoo_sample.cfg zoo.cfg

[root@node1 conf]# vim zoo.cfg

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/opt/hadoop/zookeeper/data

dataLogDir=/opt/hadoop/zookeeper/dataLog

clientPort=2181

server.1=node1:2888:3888

server.2=node2:2888:3888

server.3=node3:2888:3888

maxClientCnxns=60

[root@node1 ~]# mkdir -p /opt/hadoop/zookeeper/data

[root@node1 ~]# mkdir -p /opt/hadoop/zookeeper/dataLog

修改myid文件

[root@node1 conf]# cd /opt/hadoop/zookeeper/data