Centos7 zookeeper3.5.6 + hadoop3.2.1+Hbase2.2.2完全分布式+高可用（HA）图文教程

Centos7 zookeeper3.5.6 + hadoop3.2.1+Hbase2.2.2完全分布式+高可用（HA）图文教程

目录

1. 分布式大数据介绍 4
2. 版本说明 4
   2.1. 安装centos 4
   2.2. Hbase版本 4
   2.3. Hadoop版本 6
   2.4. zookeeper版本 7
3. 原理说明 7
4. 安装规划 8
5. 公共基础配置 8
   5.1. 配置IP 8
   5.2. 修改hostname 9
   5.3. 关闭防火墙 9
   5.4. 安装JDK 10
   5.5. 配置环境变量 10
   5.6. 配置hosts 10
   5.7. 设置用户参数 11
   5.8. 虚拟机拷贝 12
   5.9. 建立SSH信任 13
6. 配置zookeeper 14
7. 配置hadoop 15
   7.1. 配置hadoop-env.sh 15
   7.2. 配置core-site.xml 15
   7.3. 配置hdfs-site.xml： 16
   7.4. 配置mapred-site.xml： 19
   7.5. 配置yarn-site.xml： 19
   7.6. 配置workers 21
   7.7. 启动hadoop(初次) 21
   7.8. 验证： 22
   7.9. 启动hadoop(再次) 23
   7.10. 停止hadoop 24
   7.11. 常用命令 24
   7.11.1. 查看hadoop文件 24
   7.11.2. 创建文件夹 24
   7.11.3. 上传文件 24
   7.11.4. 测试高可用 25
   7.11.5. 主备切换 25
   7.11.6. 执行MR 25
8. 配置hbase 27
   8.1. 安装hbase 27
   8.2. 配置环境变量 27
   8.3. 配置hbase-env.sh 27
   8.4. 配置hbase-site.xml 27
   8.5. 配置regionservers 28
   8.6. 配置Hmaster高可用 28
   8.7. log4j冲突处理 29
   8.8. 启动hbase 29
   8.9. 停止hbase 29
   8.10. Hbase web 29
9. Hbase常用命令 30
   9.1. Common 30
   9.2. DDL 31
   9.3. DML 31
10. 参考资料 31

1. 分布式大数据介绍

Hadoop是Apache软件基金会旗下的一个开源分布式计算平台。以Hadoop分布式文件系统(HDFS，Hadoop Distributed Filesystem)和MapReduce(Google MapReduce的开源实现)为核心的Hadoop为用户提供了系统底层细节透明的分布式基础架构。HDFS的高容错性、高伸缩性等优点，允许用户将Hadoop部署在低廉的硬件上，轻松地组织计算机资源，从而搭建自己的分布式计算平台，并且可以充分利用集群的计算和存储能力，完成海量数据的处理。
Hadoop目前已经取得了非常突出的成绩。随着互联网的发展，新的业务模式还将不断涌现，Hadoop的应用也会从互联网领域向工业生产、电信、电子商务、银行、生物制药等领域拓展。
HBase 是一个开源的非关系(NoSQL)的可伸缩性分布式数据库。它是面向列的，并适合于存储超大型松散数据。HBase适合于实时，随机对Big数据进行读写操作的业务环境。
这些技术可以用来:存储海量数据、实时分析和计算、数据挖掘，国外使用这些技术的有雅虎、亚马逊和facebook等，国内使用该技术的有京东、百度、淘宝、阿里、华为和奇虎360等。

2. 版本说明

OS：centos7位（CentOS-7-x86_64-DVD-1804.iso）
JDK: jdk8
HADOOP:hadoop-3.2.1.tar
ZOOKEEPER: zookeeper-3.5.6.tar
HBASE: hbase-2.2.2-bin.tar

2.1. 安装centos

选择：gnome+x-window

2.2. Hbase版本

Hbase2.2.2下载地址：
http://mirror.bit.edu.cn/apache/hbase/2.2.2/hbase-2.2.2-bin.tar.gz
历史版及官方稳定版下载地址：
http://archive.apache.org/dist/hbase/stable/

以下安装在一个节点上完成，然后copy到其他节点上。

2.3. Hadoop版本

Hadoop稳定版：

hbase和hadoop匹配表：
hadoop版本下载地址：
http://archive.apache.org/dist/hadoop/core/

官方文档：
https://hadoop.apache.org/docs/r3.2.1/
华为云采用的版本：

2.4. zookeeper版本

http://apache.fayea.com/zookeeper/stable/

3. 原理说明

In a typical HA cluster, two separate machines are configured as NameNodes. At any point in time, exactly one of the NameNodes is in an Active state, and the other is in a Standby state. The Active NameNode is responsible for all client operations in the cluster, while the Standby is simply acting as a slave, maintaining enough state to provide a fast failover if necessary.
In order for the Standby node to keep its state synchronized with the Active node, both nodes communicate with a group of separate daemons called “JournalNodes” (JNs). When any namespace modification is performed by the Active node, it durably logs a record of the modification to a majority of these JNs. The Standby node is capable of reading the edits from the JNs, and is constantly watching them for changes to the edit log. As the Standby Node sees the edits, it applies them to its own namespace. In the event of a failover, the Standby will ensure that it has read all of the edits from the JounalNodes before promoting itself to the Active state. This ensures that the namespace state is fully synchronized before a failover occurs.
In order to provide a fast failover, it is also necessary that the Standby node have up-to-date information regarding the location of blocks in the cluster. In order to achieve this, the DataNodes are configured with the location of both NameNodes, and send block location information and heartbeats to both.
It is vital for the correct operation of an HA cluster that only one of the NameNodes be Active at a time. Otherwise, the namespace state would quickly diverge between the two, risking data loss or other incorrect results. In order to ensure this property and prevent the so-called “split-brain scenario,” the JournalNodes will only ever allow a single NameNode to be a writer at a time. During a failover, the NameNode which is to become active will simply take over the role of writing to the JournalNodes, which will effectively prevent the other NameNode from continuing in the Active state, allowing the new Active to safely proceed with failover.

4. 安装规划

Zookeeper进程：
Qp:QuorumPeerMain
Dfs进程：
Nn:NameNode
Dn:DataNode
Jn:JournalNode
Df :DFSZK Failover Controller
Yarn进程：
Rm: Resource Manager
Nm:NodeManager
Hbase进程：
Hm:HMaster
Hr:HRegionServe

5. 公共基础配置

用root用户完成本文操作，先统一进行公共基础配置

5.1. 配置IP

cd /etc/sysconfig/network-scripts
找到当前网卡名字
vim ifcfg-ens33
将ONBOOT=no改为yes，
将BOOTPROTO=dhcp改为static,
并在后面增加几行内容：

IPADDR=192.168.80.51
NETMASK=255.255.255.0
GATEWAY=192.168.80.2
DNS1=192.168.80.2

保存后退出，然后输入命令来重启网络服务：
systemctl restart network.service

5.2. 修改hostname

//永久性的修改主机名称，立即生效，重启后能保持有效。
hostnamectl set-hostname hmaster1
#Hostname

5.3. 关闭防火墙

查看防火墙状态sudo systemctl status firewalld
永久禁用防火墙sudo systemctl disable firewalld
关闭防火墙sudo systemctl stop firewalld
开启防火墙sudo systemctl start firewalld

5.4. 安装JDK

为了避免干扰，先卸载系统自带的open jkd
查看gcj的版本号：rpm -qa|grep jdk
卸载系统自带的open jkd：

rpm -e --nodeps `rpm -qa | grep java`

Oracle官网下载最新版jdk：jdk-8u131-linux-x64.tar，Oracle官网：
http://www.oracle.com/technetwork/java/javase/downloads
$ tar - zxvf jdk-8u131-linux-x64.tar
安装目录：/home/hadoop/jdk8

5.5. 配置环境变量

vim /etc/profile

HADOOP_HOME=/home/hadoop/hadoop
JAVA_HOME=/home/hadoop/jdk8
HIVE_HOME=/home/hadoop/hive-0.11.0
HBASE_HOME=/home/hadoop/hbase 
SCALA_HOME=/home/hadoop/scala
SPARK_HOME=/home/hadoop/spark
PATH=.:$SCALA_HOME/bin:$SPARK_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$J