hadoop 分布式集群部署 一 HDFS 高可用部署（Observer NameNode 进行读写分离）

Observer NameNode

在启用高可用的hadoop集群中，有一个 Active NameNode 和一个或多个Standby NameNode。Active NameNode负责为所有客户端请求提供服务，而Standby NameNode只是通过跟踪JournalNodes的 edit logs 以及通过接收所有DataNodes的块报告来保持有关命名空间的最新信息。这种体系结构的一个缺点是，Active NameNode可能是一个瓶颈，尤其是在繁忙的集群中。

在hadoop 3以后，引入一种称为Observer NameNode的新型NameNode来解决上述问题。与Standby NameNode类似，Observer NameNode使自己保持命名空间和块位置信息的最新状态，它能提供和Active NameNode一致读取的能力。由于一般环境中读取请求在占大多数，因此这有助于负载平衡NameNode流量并提高整体吞吐量。

Hadoop 安装配置

主机规划和基础配置 hadoop 分布式集群部署 一 zookeeper 集群部署

1、下载安装hadoop

~]# tar -xzf hadoop-3.3.6.tar.gz -C /opt
~]# mkdir /etc/hadoop
# 复制一份配置模板
~]# cp -r /opt/hadoop-3.3.6/etc/hadoop /etc/hadoop/conf

2、创建HDFS用户及相关目录

~]# useradd hdfs

# namenode 节点要创建 namenode 和 journalnode 数据目录
~]# mkdir -pv /data/dfs/nn
~]# mkdir -pv /data/dfs/jn
~]# chown -R hdfs:hdfs /data/dfs

# datanode 节点假如有4块数据盘，分别挂在 /data/1 /data/2 /data/3 /data/4 目录下
~]# mkdir -pv /data/{1,2,3,4}/dfs/dn
~]# chown -R hdfs:hdfs /data/{1,2,3,4}/dfs/dn

# 创建HDFS服务相关的日志文件目录和PID文件目录
~]# mkdir -pv /var/log/hdfs/  && chown hdfs:hdfs /var/log/hdfs
~]# mkdir -pv /opt/hadoop-3.3.6/run/hdfs && chown hdfs:hdfs /opt/hadoop-3.3.6/run/hdfs

# 在3个namenode上做hdfs用户的ssh免密
namenode01 ~]# su - hdfs
namenode01 ~]$ ssh-keygen -t rsa
namenode01 ~]$ vi .ssh/config   
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
namenode01 ~]$ scp -r .ssh namenode02:/home/hdfs/
namenode01 ~]$ scp -r .ssh namenode03:/home/hdfs/

3、配置core-site.xml

~]# cat /etc/hadoop/conf/core-site.xml 
<?xml version="1.0"?>
<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://cluster</value>
        </property>
        <property>
                <name>io.file.buffer.size</name>
                <value>131072</value>
        </property>
        <property>
                <name>fs.trash.interval</name>
                <value>10080</value>
        </property>
        <property>
                <name>fs.trash.checkpoint.interval</name>
                <value>1440</value>
        </property>
        <property>
                <name>ha.zookeeper.quorum</name>
                <value>192.168.36.131:2181,192.168.36.132:2181,192.168.36.133:2181</value>
        </property>
        <property>
                <name>hadoop.proxyuser.hive.hosts</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.hive.groups</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.hdfs.hosts</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.hdfs.groups</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.yarn.hosts</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.yarn.groups</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.mapred.hosts</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.mapred.groups</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.spark.hosts</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.spark.groups</name>
                <value>*</value>
        </property>
</configuration>

4、配置 hdfs-site.xml

~]# vi /etc/hadoop/conf/hdfs-site.xml 
<?xml version="1.0"?>
<configuration>
        <property>
                <name>dfs.nameservices</name>
                <value>cluster</value>
        </property>
        <property>
                <name>dfs.ha.namenodes.cluster</name>
                <value>nn1,nn2,nn3</value>
        </property>
        <property>
                <name>dfs.ha.automatic-failover.enabled</name>
                <value>true</value>
        </property>
        <property>
                <name>dfs.ha.fencing.methods</name>
                <value>sshfence(hdfs:22)</value>
        </property>
        <property>
                <name>dfs.ha.fencing.ssh.private-key-files</name>
                <value>/home/hdfs/.ssh/id_rsa</value>
        </property>
        <property>
                <name>dfs.ha.fencing.ssh.connect-timeout</name>
                <value>30000</value>
        </property>
        <property>
                <name>dfs.namenode.shared.edits.dir</name>
                <value>qjournal://192.168.36.131:8485;192.168.36.132:8485;192.168.36.133:8485/cluster</value>
        </property>
        <property>
                <name>dfs.journalnode.edits.dir</name>
                <value>/data/dfs/jn/edits</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>/data/dfs/nn</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>/data/1/dfs/dn,/data/2/dfs/dn,/data/3/dfs/dn,/data/4/dfs/dn</value>
        </property>
        <property>
                <name>dfs.blocksize</name>
                <value>268435456</value>
        </property>
        <property>
                <name>dfs.ha.nn.not-become-active-in-safemode</name>
                <value>true</value>
        </property>
        <property>
                <name>dfs.namenode.state.context.enabled</name>
                <value>true</value>
        </property>
        <property>
                <name>dfs.ha.tail-edits.in-progress</name>
                <value>true</value>
        </property>
        <property>
                <name>dfs.ha.tail-edits.period</name>
                <value>0ms</value>
        </property>
        <property>
                <name>dfs.ha.tail-edits.period.backoff-max</name>
                <value>10s</value>
        </property>
        <property>
                <name>dfs.journalnode.edit-cache-size.bytes</name>
                <value>1048576</value>
        </property>
        <property>
                <name>dfs.namenode.accesstime.precision</name>
                <value>0</value>
        </property>
        <property>
                <name>dfs.client.failover.proxy.provider.cluster</name>
                <value>org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider</value>
        </property>
        <property>
                <name>dfs.client.failover.observer.auto-msync-period.cluster</name>
                <value>500ms</value>
        </property>
        <property>
                <name>dfs.namenode.handler.count</name>
                <value>100</value>
        </property>
        <property>
                <name>dfs.namenode.service.handler.count</name>
                <value>100</value>
        </property>
        <property>
                <name>dfs.datanode.handler.count</name>
                <value>100</value>
        </property>
        <property>
                <name>dfs.namenode.avoid.read.stale.datanode</name>
                <value>true</value>
        </property>
        <property>
                <name>dfs.namenode.avoid.write.stale.datanode</name>
                <value>true</value>
        </property>
        <property>
                <name>dfs.datanode.du.reserved.pct</name>
                <value>5</value>
        </property>
        <property>
                <name>dfs.datanode.du.reserved.calculator</name>
                <value>org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReservedSpaceCalculator$ReservedSpaceCalculatorPercentage</value>
        </property>
        <property>
                <name>dfs.namenode.rpc-address.cluster.nn1</name>
                <value>192.168.36.131:8020</value>
        </property>
        <property>
                <name>dfs.namenode.servicerpc-address.cluster.nn1</name>
                <value>192.168.36.131:8010</value>
        </property>
        <property>
                <name>dfs.namenode.http-address.cluster.nn1</name>
                <value>192.168.36.131:9870</value>
        </property>
        <property>
                <name>dfs.namenode.rpc-address.cluster.nn2</name>
                <value>192.168.36.132:8020</value>
        </property>
        <property>
                <name>dfs.namenode.servicerpc-address.cluster.nn2</name>
                <value>192.168.36.132:8010</value>
        </property>
        <property>
                <name>dfs.namenode.http-address.cluster.nn2</name>
                <value>192.168.36.132:9870</value>
        </property>
        <property>
                <name>dfs.namenode.rpc-address.cluster.nn3</name>
                <value>192.168.36.133:8020</value>
        </property>
        <property>
                <name>dfs.namenode.servicerpc-address.cluster.nn3</name>
                <value>192.168.36.133:8010</value>
        </property>
        <property>
                <name>dfs.namenode.http-address.cluster.nn3</name>
                <value>192.168.36.133:9870</value>
        </property>

</configuration>

5、配置 hadoop-env.sh、hdfs-env.sh

~]# vi /etc/hadoop/conf/hadoop-env.sh
JAVA_HOME=/opt/jdk1.8.0_151
HADOOP_HOME=/opt/hadoop-3.3.6
HADOOP_CONF_DIR=/etc/hadoop/conf

~]# vi /etc/hadoop/conf/hdfs-env.sh 
export HADOOP_LOG_DIR=/var/log/hdfs
export HADOOP_PID_DIR=/opt/hadoop-3.3.6/run/hdfs

6、注册系统服务

~]# cat /usr/lib/systemd/system/journalnode.service;
[Unit]
Description=journalnode Service
After=network.target
[Service]
User=hdfs
EnvironmentFile=-/etc/hadoop/conf/hadoop-env.sh
ExecStart=/opt/hadoop-3.3.6/bin/hdfs --daemon start journalnode
PIDFile=/opt/hadoop-3.3.6/run/hdfs/hadoop-hdfs-journalnode.pid
Restart=on-failure
[Install]
WantedBy=multi-user.target


~]# cat /usr/lib/systemd/system/namenode.service 
[Unit]
Description=namenode Service
After=network.target
[Service]
User=hdfs
EnvironmentFile=-/etc/hadoop/conf/hadoop-env.sh
ExecStart=/opt/hadoop-3.3.6/bin/hdfs --daemon start namenode
PIDFile=/opt/hadoop-3.3.6/run/hdfs/hadoop-hdfs-namenode.pid
Restart=on-failure
[Install]
WantedBy=multi-user.target

~]# cat /usr/lib/systemd/system/zkfc.service 
[Unit]
Description=zkfc Service
After=network.target
[Service]
User=hdfs
EnvironmentFile=-/etc/hadoop/conf/hadoop-env.sh
ExecStart=/opt/hadoop-3.3.6/bin/hdfs --daemon start zkfc
PIDFile=/opt/hadoop-3.3.6/run/hdfs/hadoop-hdfs-zkfc.pid
Restart=on-failure
[Install]
WantedBy=multi-user.target

7、启动 journalnode 服务

~]# systemctl --now enable journalnode.service

8、格式化HDFS并启动namenode

# 在 namenode01
~]# su - hdfs
# 格式化 namenode
~]$ hdfs namenode -format
# 然后启动 namenode
~]# systemctl --now enable namenode.service

# 在 namenode02和namenode03
~]# su - hdfs
~]$ hdfs namenode -bootstrapStandby
~]# systemctl --now enable journalnode.service

# 在任意一个 namenode 节点上格式化 zkfc
~]# su - hdfs
~]$ hdfs zkfc -formatZK

# 在所有 namenode 节点上启动 zkfc
~]# systemctl --now enable zkfc.service

# 将 namenode03 切换为 observer 
~]$ hdfs haadmin -ns cluster -transitionToObserver nn3 --forcemanual

9、启动 datanode

在所有 datanode 节点上注册并启动 datanode 服务

~]# systemctl --now enable zkfc.service~]# vi /usr/lib/systemd/system/datanode.service
[Unit]
Description=datanode Service
After=network.target
[Service]
User=hdfs
EnvironmentFile=-/etc/hadoop/conf/hadoop-env.sh
ExecStart=/opt/hadoop-3.3.6/bin/hdfs --daemon start datanode
PIDFile=/opt/hadoop-3.3.6/run/hdfs/hadoop-hdfs-datanode.pid
Restart=on-failure
[Install]
WantedBy=multi-user.target

~]# systemctl --now enable datanode.service

验证服务状态