ZooKeeper——ZooKeeper搭建及问题解决

ZooKeeper——ZooKeeper搭建及问题解决

---

ZooKeeper简介

Apache ZooKeeper是一项开发和维护开源服务器的工作，它能够实现高度可靠的分布式协调。Zookeeper官方介绍如下：

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

Zookeeper是一个用于维护配置信息、命名、提供分布式同步以及提供组服务的集中服务，所有这些类型的服务都以某种形式被分布式应用程序使用。每次它们被实现时，都有大量的工作需要去修复不可避免的bug和竞争条件。由于实现这类服务的困难，应用程序最初通常会略过它们，这使得它们在出现变化时变得脆弱，难以管理。即使做得正确，这些服务的不同实现在部署应用程序时也会导致管理复杂性。

ZooKeeper主要角色

• 领导者（leader）：负责投票的发起和决议，更新系统状态
• 跟随着（follower）：接收客户请求并向客户端返回结果，选主过程中参与投票
• 观察者（observer）：observer可以接受客户端连接，将写请求发送给leader节点，但observer不参与投票，只同步leader状态。observer作用是扩展系统，提高读取速度
• 客户端（client）：请求发起方

准备工作

ZooKeeper集群选择三台服务器进行搭建

• 192.168.108.128 node1
• 192.168.108.129 node2
• 192.168.108.130 node3

下载解压

使用3.5.5版本ZooKeeper。下载安装包apache-zookeeper-3.5.5-bin.tar.gz，解压tar -zxvf apache-zookeeper-3.5.5-bin.tar.gz
进入文件夹apache-zookeeper-3.5.5-bin，解压后的ZooKeeper文件夹目录结构

drwxr-xr-x. 2 2002 2002   232 4月   9 19:13 bin
drwxr-xr-x. 2 2002 2002   120 8月  23 15:16 conf
drwxr-xr-x. 5 2002 2002  4096 5月   3 20:07 docs
drwxr-xr-x. 2 root root  4096 8月  23 11:42 lib
-rw-r--r--. 1 2002 2002 11358 2月  15 2019 LICENSE.txt
drwxr-xr-x. 2 root root    45 8月  23 14:39 logs
-rw-r--r--. 1 2002 2002   432 4月   9 19:13 NOTICE.txt
-rw-r--r--. 1 2002 2002  1560 5月   3 19:41 README.md
-rw-r--r--. 1 2002 2002  1347 4月   2 21:05 README_packaging.txt

修改配置

修改配置文件，进入conf目录拷贝zoo_sample.cfg文件

cp zoo_sample.cfg zoo.cfg

修改zoo.cfg配置文件

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/var/lib/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

zoo.cfg常见配置说明

配置	作用	默认值
tickTime	心跳检测时间	2000
initLimit	leader与follower之间初始化连接时间上限	10
syncLimit	leader与follower之间的不同步时间上限	5
dataDir	存放服务器的信息，注：官方提示不要放在tmp下	/tmp/zookeeper
clientPort	客户端连接端口	2181
maxClientCnxns	客户端最大连接数	60

修改集群配置文件
192.168.108.128:

# 添加集群相关信息
server.1=0.0.0.0:2888:3888
server.2=192.168.108.129:2888:3888
server.3=192.169.108.130:2888:3888

192.168.108.129:

# 添加集群相关信息
server.1=192.168.108.128:2888:3888
server.2=0.0.0.0:2888:3888
server.3=192.168.108.130:2888:3888

192.168.108.130:

# 添加集群相关信息
server.1=192.168.108.128:2888:3888
server.2=192.168.108.129:2888:3888
server.3=0.0.0.0:2888:3888

添加的集群信息遵循server.X=hostname:peerPort:leaderPort的格式，具体格式说明

参数	意义	说明
X	服务器ID	服务器的唯一标识ID，必须是整数
hostname	服务器名或ip
peerPort	节点间通信端口
leaderPort	leader选举端口

创建myid文件

修改三台服务器的配置文件中dataDir属性值

# 修改dataDir
dataDir=/var/lib/zookeeper

然后手动创建zookeeper文件夹和myid文件

mkdir /var/lib/zookeeper
echo "1" > /var/lib/zookeeper/myid

这里在myid文件中可以追加任意内容

启动zk

进入bin目录，分别启动三台服务器的zk服务

zkServer.sh start

ZooKeeper JMX enabled by default
Using config: /component/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

可以看到，启动时会默认去conf目录下查找zoo.cfg配置文件，也可以通过命令参数config使用指定配置文件启动

查看ZooKeeper启动状态

输入jps查看java进程，进程中多了一个QuorumPeerMain进程

[root@node3 bin]# jps
37364 QuorumPeerMain
37514 Jps

通过ZooKeeper命令查看状态，zkServer.sh status

#128服务器为leader
[root@node1 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /component/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader

#129服务器为follower
[root@node2 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /component/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

#130服务器为follower
[root@node3 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /component/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

ZooKeeper集群集群搭建成功

---

搭建过程遇到的问题

配置错误，myid file is missing

org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /component/apache-zookeeper-3.5.5-bin/bin/../conf/zoo.cfg
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:154)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:113)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.lang.IllegalArgumentException: myid file is missing
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.checkValidity(QuorumPeerConfig.java:734)
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.setupQuorumPeerConfig(QuorumPeerConfig.java:605)
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:420)
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:150)
	... 2 more

解决方案：创建myid文件并随便写入

echo "3" > /var/lib/zookeeper/myid

无法连接集群

2019-08-23 14:44:01,690 [myid:1] - WARN  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumCnxManager@677] - Cannot open channel to 3 at election address /192.168.102.130:3888
java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:648)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:705)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:733)
	at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910)
	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247)
2019-08-23 14:44:01,690 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 60000

解决方案：首先关闭集群服务器防火墙，在zoo.cfg配置文件配置集群信息时，本机使用0.0.0.0代替IP。