hadoop 分布式集群部署 一 zookeeper 集群部署

主机规划

IP	主机名	服务角色
192.168.36.131	namenode01.example.io	zookeeper、journalnode、namenode、resourcemanager
192.168.36.132	namenode02.example.io	zookeeper、journalnode、namenode、resourcemanager
192.168.36.133	namenode03.example.io	zookeeper、journalnode、namenode、hive metastore
192.168.36.134	datanode01.example.io	datanode、nodemanager
192.168.36.135	datanode02.example.io	datanode、nodemanager
192.168.36.136	datanode03.example.io	datanode、nodemanager
192.168.36.137	edge.example.io	mr-jobhistory、timeline server、hive metastore、client

主机基本配置

本文OS以CentOS7为例。主机名、网络配置不再具体描述

1、关闭防火墙

~]# systemctl disable firewalld.service
~]# systemctl stop firewalld.service

2、关闭 SELinux

~]# setenforce 0
~]# vi /etc/selinux/config
SELINUX=disabled

3、设置时间同步

~]# yum install chrony -y
# 修改时间源、如果公司内部有时间服务器、可以配置为公司内部时间服务器、这里演示使用的是阿里的时间服务器
~]# vi /etc/chrony.conf
server ntp.aliyun.com iburst

~]# systemctl --now enable chronyd.service 

# 确认时间同步状态
~]# chronyc sources                            
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 203.107.6.88                  2   7   377    82   +367us[ +581us] +/-   17ms

4、JDK配置

下载JDK并配置环境变量

~]# tar -xzf jdk-8u151-linux-x64.tar.gz -C /opt
~]# vi /etc/profile.d/jdk.sh 
export JAVA_HOME=/opt/jdk1.8.0_151
export PATH=$JAVA_HOME/bin:$PATH

配置zookeeper服务

1、下载并配置 zookeeper

3个zookeeper节点配置都一样，除了 myid 文件要和zoo.cfg.dynamic配置对应，每个节点都有不同的值

~]# tar -xzf apache-zookeeper-3.6.3-bin.tar.gz -C /opt
~]# cd /opt/apache-zookeeper-3.6.3-bin/
~]# mkdir /etc/zookeeper

# 复制一份配置文件模板
~]# cp -r /opt/apache-zookeeper-3.6.3-bin/conf/* /etc/zookeeper/

~]# vi /etc/zookeeper/zoo.cfg
initLimit=5
syncLimit=2
tickTime=2000
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpPort=7000
dataDir=/var/lib/zookeeper/data
admin.serverPort=8680
reconfigEnabled=true
dynamicConfigFile=/etc/zookeeper/zoo.cfg.dynamic.100000000

~]# vi /etc/zookeeper/zoo.cfg.dynamic
server.1=192.168.36.131:2888:3888:participant;2181
server.2=192.168.36.132:2888:3888:participant;2181
server.3=192.168.36.133:2888:3888:participant;2181

namenode01 ~]# cat /var/lib/zookeeper/data/myid 
1
namenode02 ~]# cat /var/lib/zookeeper/data/myid 
2
namenode03 ~]# cat /var/lib/zookeeper/data/myid 
3

~]# vi /etc/zookeeper/zookeeper-env.sh 
JAVA_HOME=/opt/jdk1.8.0_151
ZOO_LOG4J_PROP="INFO, ROLLINGFILE"
ZOO_LOG_DIR=/var/log/zookeeper
ZK_SERVER_HEAP=1024
SERVER_JVMFLAGS="-Dzookeeper.DigestAuthenticationProvider.superDigest=super:9G5bEp+Sy/dSJEa1bC0sHK/rpe0="

2、创建数据目录

3个zookeeper节点操作方式一样

~]# useradd zookeeper
~]# mkdir /var/lib/zookeeper/data
~]# chown -R zookeeper:zookeeper /var/lib/zookeeper/data

3、注册并启动服务

3个zookeeper节点操作方式一样

~]# vi /usr/lib/systemd/system/zookeeper.service
[Unit]
Description=Zookeeper Service
After=network.target

[Service]
User=zookeeper
Environment="ZOOCFGDIR=/etc/zookeeper"
ExecStart=/opt/apache-zookeeper-3.6.3-bin/bin/zkServer.sh start
ExecStop=/opt/apache-zookeeper-3.6.3-bin/bin/zkServer.sh stop
ExecReload=/opt/apache-zookeeper-3.6.3-bin/bin/zkServer.sh restart
PIDFile=/var/lib/zookeeper/data/zookeeper_server.pid
Restart=on-failure

[Install]
WantedBy=multi-user.target

~]# systemctl --now enable zookeeper.service 
~]# systemctl status zookeeper