1、概述
Zookeeper是一个开源的分布式的,为分布式应用提供协调服务的Apache项目。
2、Zookeeper工作机制
3、Zookeeper特点
4、数据结构
5、应用场景
提供的服务包括:统一命名服务、统一配置管理、统一集群管理、服务器节点动态上下线、软负载均衡等。
5.1 统一命名服务
5.2 统一配置管理
5.3 统一集群管理
5.4 服务器节点动态上下线
5.5 软负载均衡
下载地址
1.官网首页:
https://zookeeper.apache.org/
2.下载截图,如图5-5,5-6,5-7所示
Zookeeper安装
1、本地模式安装部署
1.安装前准备
(1)安装Jdk
(2)拷贝Zookeeper安装包到Linux系统下
(3)解压到指定目录
[root@hadoop100 zookeeper]# tar -zxvf zookeeper-3.4.5.tar.gz -c /usr/local/java/zookeeper
2.配置修改
(1)将/usr/local/java/zookeeper/zookeeper-3.4.10/conf这个路径下的zoo_sample.cfg修改为zoo.cfg;
(2)打开zoo.cfg文件,修改dataDir路径:
修改如下内容:
dataDir=/usr/local/java/zookeeper/zookeeper3.4.5/zkData
(3)在/usr/local/java/zookeeper/zookeeper-3.4.10/这个目录上创建zkData文件夹
[root@hadoop100 zookeeper3.4.5]# mkdir zkData
3、操作Zookeeper
(1)启动Zookeeper
[root@hadoop100 zookeeper3.4.5]# bin/zkServer.sh start
JMX enabled by default
Using config: /usr/local/java/zookeeper/zookeeper3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@hadoop100 zookeeper3.4.5]#
(2)查看进程是否启动
[root@hadoop100 zookeeper3.4.5]# jps
7713 QuorumPeerMain
7734 Jps
7662 NameNode
[root@hadoop100 zookeeper3.4.5]#
(3)查看状态:
[root@hadoop100 zookeeper3.4.5]# bin/zkServer.sh status
JMX enabled by default
Using config: /usr/local/java/zookeeper/zookeeper3.4.5/bin/../conf/zoo.cfg
Mode: standalone
[root@hadoop100 zookeeper3.4.5]#
(4)启动客户端:
[root@hadoop100 zookeeper3.4.5]# bin/zkCli.sh
Connecting to localhost:2181
2019-12-01 07:45:18,728 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2019-12-01 07:45:18,732 [myid:] - INFO [main:Environment@100] - Client environment:host.name=hadoop100
2019-12-01 07:45:18,732 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_221
2019-12-01 07:45:18,733 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-12-01 07:45:18,734 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/local/java/jdk/jdk1.8.0_221/jre
2019-12-01 07:45:18,734 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/local/java/zookeeper/zookeeper3.4.5/bin/../build/classes:/usr/local/java/zookeeper/zookeeper3.4.5/bin/../build/lib/*.jar:/usr/local/java/zookeeper/zookeeper3.4.5/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/java/zookeeper/zookeeper3.4.5/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/java/zookeeper/zookeeper3.4.5/bin/../lib/netty-3.2.2.Final.jar:/usr/local/java/zookeeper/zookeeper3.4.5/bin/../lib/log4j-1.2.15.jar:/usr/local/java/zookeeper/zookeeper3.4.5/bin/../lib/jline-0.9.94.jar:/usr/local/java/zookeeper/zookeeper3.4.5/bin/../zookeeper-3.4.5.jar:/usr/local/java/zookeeper/zookeeper3.4.5/bin/../src/java/lib/*.jar:/usr/local/java/zookeeper/zookeeper3.4.5/bin/../conf:.:/usr/local/java/jdk/jdk1.8.0_221/lib/dt.jar:/usr/local/java/jdk/jdk1.8.0_221/lib/tools.jar
2019-12-01 07:45:18,735 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-12-01 07:45:18,744 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-12-01 07:45:18,744 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-12-01 07:45:18,744 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-12-01 07:45:18,744 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-12-01 07:45:18,744 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-957.el7.x86_64
2019-12-01 07:45:18,745 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2019-12-01 07:45:18,746 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2019-12-01 07:45:18,747 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/usr/local/java/zookeeper/zookeeper3.4.5
2019-12-01 07:45:18,750 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@69d0a921
(5)退出客户端:
[zk: localhost:2181(CONNECTED) 0] quit
Quitting...
2019-12-01 07:46:20,749 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x16ec06a14e60000 closed
2019-12-01 07:46:20,750 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@509] - EventThread shut down
[root@hadoop100 zookeeper3.4.5]#
(6)停止Zookeeper
[root@hadoop100 zookeeper3.4.5]# bin/zkServer.sh stop
JMX enabled by default
Using config: /usr/local/java/zookeeper/zookeeper3.4.5/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
[root@hadoop100 zookeeper3.4.5]#
配置总结:
- 解压
- 把conf文件夹下配置文件改个名字
cp zoo_sample.cfg zoo.cfg - 编辑zoo.cfg,配置datadir
dataDir=/opt/module/zookeeper-3.4.10/zkData - 配置集群机器,每台机器分配一个不同的Serverid
server.0=hadoop102:2888:3888
server.1=hadoop103:2888:3888
server.2=hadoop104:2888:3888
以上配置0,1,2就是Serverid - 在zkData文件夹里新建一个myid文件,内容是本机的Serverid
[root@hadoop100 zkData]# touch myid
[root@hadoop100 zkData]#
[root@hadoop100 zkData]# vim myid
[root@hadoop100 zkData]#
0
- 配置Zookeeper的LogDIR:配置bin/zkEnv.sh文件
[root@hadoop100 zookeeper3.4.5]# cd bin/
[root@hadoop100 bin]# ls
README.txt zkCleanup.sh zkCli.cmd zkCli.sh zkEnv.cmd zkEnv.sh zkServer.cmd zkServer.sh zookeeper.out
[root@hadoop100 bin]# vim zkEnv.sh
ZOO_LOG_DIR="/usr/local/java/zookeeper/zookeeper3.4.5/logs"
export JAVA_HOME=/usr/local/java/jdk/jdk1.8.0_221
保存退出!
分发到别的机器:
[root@hadoop100 zookeeper]# scp -r /usr/local/java/zookeeper/zookeeper3.4.5/* root@hadoop101:/usr/local/java/zookeeper/zookeeper3.4.5/
[root@hadoop100 zookeeper]# scp -r /usr/local/java/zookeeper/zookeeper3.4.5/* root@hadoop102:/usr/local/java/zookeeper/zookeeper3.4.5/
修改hadoop101机器配置参数:
[root@hadoop101 zookeeper3.4.5]# cd zkData/
[root@hadoop101 zkData]# ls
myid version-2
[root@hadoop101 zkData]# echo 1 > myid
[root@hadoop101 zkData]# cat myid
1
[root@hadoop101 zkData]#
修改hadoop102机器配置参数:
[root@hadoop102 ~]# cd /usr/local/java/zookeeper/zookeeper3.4.5/zkData/
[root@hadoop102 zkData]# ls
myid version-2
[root@hadoop102 zkData]# echo 2 > myid
[root@hadoop102 zkData]# cat myid
2
[root@hadoop102 zkData]#
接下来,启动zookeeper
hadoop100机器的zookeeper
[root@hadoop100 zookeeper3.4.5]# bin/zkServer.sh start
JMX enabled by default
Using config: /usr/local/java/zookeeper/zookeeper3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@hadoop100 zookeeper3.4.5]# jps
8050 QuorumPeerMain
8068 Jps
[root@hadoop100 zookeeper3.4.5]#
hadoop101机器的zookeeper
[root@hadoop101 zkData]# cd ..
[root@hadoop101 zookeeper3.4.5]# bin/zkServer.sh start
JMX enabled by default
Using config: /usr/local/java/zookeeper/zookeeper3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@hadoop101 zookeeper3.4.5]# jps
7568 QuorumPeerMain
7586 Jps
[root@hadoop101 zookeeper3.4.5]#
hadoop102机器的zookeeper
[root@hadoop102 zkData]# cd ..
[root@hadoop102 zookeeper3.4.5]# bin/zkServer.sh start
JMX enabled by default
Using config: /usr/local/java/zookeeper/zookeeper3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@hadoop102 zookeeper3.4.5]# jps
7641 QuorumPeerMain
7659 Jps
[root@hadoop102 zookeeper3.4.5]#
搭建HA(高可用)集群
1、首先,创建ha文件
[root@hadoop100 opt]# mkdir ha
2、切换普通用户:MissZhou
[root@hadoop100 opt]# su MissZhou
[MissZhou@hadoop100 opt]$
3、给普通用户设权限,发现无法授权,只能使用root授权
[MissZhou@hadoop100 opt]$ sudo chown MissZhou:MissZhou ha
[sudo] password for MissZhou:
Sorry, user MissZhou is not allowed to execute '/bin/chown MissZhou:MissZhou ha' as root on hadoop100.
[MissZhou@hadoop100 opt]$ su root
Password:
[root@hadoop100 opt]# chown MissZhou:MissZhou ha
[root@hadoop100 opt]#
如图所示:
将/opt/hadoop/module/hadoop-2.7.2.tar.gz的压缩包拷贝到ha目录下,并对其解压
[root@hadoop100 module]# cp hadoop-2.7.2.tar.gz /opt/ha/
[root@hadoop100 module]# cd /opt/ha/
[root@hadoop100 ha]# ll
total 193028
-rw-r--r-- 1 root root 197657687 Dec 1 12:06 hadoop-2.7.2.tar.gz
[root@hadoop100 ha]# tar -zxvf hadoop-2.7.2.tar.gz
[root@hadoop100 ha]# ls
hadoop-2.7.2 hadoop-2.7.2.tar.gz
接着,进入配置文件,配置core-site.xml
[root@hadoop100 hadoop-2.7.2]# cd etc/hadoop/
[root@hadoop100 hadoop]# ls
capacity-scheduler.xml hadoop-env.sh httpfs-env.sh kms-env.sh mapred-env.sh ssl-server.xml.example
configuration.xsl hadoop-metrics2.properties httpfs-log4j.properties kms-log4j.properties mapred-queues.xml.template yarn-env.cmd
container-executor.cfg hadoop-metrics.properties httpfs-signature.secret kms-site.xml mapred-site.xml.template yarn-env.sh
core-site.xml hadoop-policy.xml httpfs-site.xml log4j.properties slaves yarn-site.xml
hadoop-env.cmd hdfs-site.xml kms-acls.xml mapred-env.cmd ssl-client.xml.example
[root@hadoop100 hadoop]#
[root@hadoop100 hadoop]# vim core-site.xml
<configuration>
<!--把两个NameNode的地址组装成一个集群mycluster -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster </value>
</property>
<!--指定hadoop运行时产生文件的存储目录-->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/ha/hadoop-2.7.2/data/tmp</value>
</property>
</configuration>
保存退出!
接着,进入配置文件,配置hdfs-site.xml
[root@hadoop100 hadoop]# vim hdfs-site.xml
<configuration>
<!--完全分布式集群名称-->
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<!--集群中NameNode节点都有哪些 -->
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>
<!--nn1的RPC通信地址-->
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>hadoop100:9000</value>
</property>
<!--nn2的RPC通信地址-->
<property>
</property>
</property>
<!--配置隔离机制,即同一时刻只能有一台服务器对外响应-->
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<!--使用隔离机制时需要ssh无密钥登陆-->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<!--声明journalnode服务器存储目录-->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/opt/ha/hadoop-2.7.2/data/jn</value>
</property>
<!--关闭权限检查 -->
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<!--访问代理类:client,mycluster,active配置日志失败自动切换实现方式-->
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
保存退出!
接下来分发别的机器
[root@hadoop100 hadoop]# scp -r /opt/ha/* root@hadoop101:/opt/
[root@hadoop100 hadoop]# scp -r /opt/ha/* root@hadoop102:/opt/
接下来,分别启动hadoop100、hadoop101、Hadoop102机器
[root@hadoop100 hadoop-2.7.2]# sbin/start-all.sh start journalnalnode
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
localhost: starting namenode, logging to /opt/ha/hadoop-2.7.2/logs/hadoop-root-namenode-hadoop100.out
localhost: starting datanode, logging to /opt/ha/hadoop-2.7.2/logs/hadoop-root-datanode-hadoop100.out
Starting journal nodes [hadoop100 hadoop101 hadoop102]
hadoop101: starting journalnode, logging to /opt/ha/hadoop-2.7.2/logs/hadoop-root-journalnode-hadoop101.out
hadoop100: starting journalnode, logging to /opt/ha/hadoop-2.7.2/logs/hadoop-root-journalnode-hadoop100.out
hadoop102: starting journalnode, logging to /opt/ha/hadoop-2.7.2/logs/hadoop-root-journalnode-hadoop102.out
starting yarn daemons
starting resourcemanager, logging to /opt/ha/hadoop-2.7.2/logs/yarn-root-resourcemanager-hadoop100.out
localhost: starting nodemanager, logging to /opt/ha/hadoop-2.7.2/logs/yarn-root-nodemanager-hadoop100.out
[root@hadoop100 hadoop-2.7.2]#
接着,格式化
[root@hadoop100 hadoop-2.7.2]# bin/hdfs namenode -format
启动namenode节点