工具之《zookeeper部署集群模式》

一、zookeeper集群模式
系统为CentOS7
采用一主两从
master 192.168.100.1
slave1 192.168.100.2
slave2 192.168.100.3

二、安装
1、建立目录&下载安装

mkdir -p /appserver/software
cd /appserver/software
wget https://downloads.apache.org/zookeeper/zookeeper-3.5.7/apache-zookeeper-3.5.7-bin.tar.gz
tar -zxvf apache-zookeeper-3.5.7-bin.tar.gz -C ../
chown -R root:root /appserver/apache-zookeeper-3.5.7-bin

注：zookeeper从3.5开始将可执行jar文件和源码包分开了，源码包只有代码
如果下错包，启动会报错：Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain

三、配置集群
1、复制配置文件&建立data目录

cd /appserver/apache-zookeeper-3.5.7-bin/conf
cp zoo_sample.cfg zoo.cfg
mkdir /appserver/apache-zookeeper-3.5.7-bin/data

2、编辑zoo.cfg
三台配置文件一致，leader由zookeeper选举得出

vi zoo.cfg

修改：
1）存储内存中数据库快照的位置

dataDir=/appserver/apache-zookeeper-3.5.7-bin/data

2）指定清理频率，单位是小时，默认是0，表示不开启自己清理功能

autopurge.purgeInterval=6

3）和上面的参数搭配使用，指定需要保留的文件数目，默认是保留3个

autopurge.snapRetainCount=5

4）服务列表

server.0=192.168.100.1:2888:3888
server.1=192.168.100.2:2888:3888
server.2=192.168.100.3:2888:3888

5）修改8080端口
zookeeper3.5.x使用jetty做web容器，会占用8080端口，如已使用可以换成其他的
该服务是Zookeeper AdminServer，主要用于获取服务器状态监控

admin.serverPort=8081

3、配置myid
在dataDir指定的目录下，创建myid文件，填入对应server.x中的数字。用来唯一标识这个服务。

四、启动命令
1、启动命令：
zkServer.sh start
2、停止命令：
zkServer.sh stop
3、重启命令：
zkServer.sh restart
4、查看集群节点状态：
zkServer.sh status
5、访问Zookeeper AdminServer
http://IP:8081/commands

五、启动日志
slaver1日志：

2020-02-21 09:14:11,428 [myid:1] - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@938] - minSessionTimeout set to 4000
2020-02-21 09:14:11,428 [myid:1] - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@947] - maxSessionTimeout set to 40000
2020-02-21 09:14:11,429 [myid:1] - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@166] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /appserver/apache-zookeeper-3.5.7-bin/data/version-2 snapdir /appserver/apache-zookeeper-3.5.7-bin/data/version-2
2020-02-21 09:14:11,431 [myid:1] - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Leader@464] - LEADING - LEADER ELECTION TOOK - 28 MS
2020-02-21 09:14:11,445 [myid:1] - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FileTxnSnapLog@404] - Snapshotting: 0x100000000 to /appserver/apache-zookeeper-3.5.7-bin/data/version-2/snapshot.100000000
2020-02-21 09:14:12,813 [myid:1] - INFO  [LearnerHandler-/192.168.100.1:49222:LearnerHandler@406] - Follower sid: 0 : info : 192.168.100.1:2888:3888:participant
2020-02-21 09:14:13,108 [myid:1] - INFO  [LearnerHandler-/192.168.100.1:49222:ZKDatabase@295] - On disk txn sync enabled with snapshotSizeFactor 0.33
2020-02-21 09:14:13,109 [myid:1] - INFO  [LearnerHandler-/192.168.100.1:49222:LearnerHandler@708] - Synchronizing with Follower sid: 0 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x100000000 peerLastZxid=0x0
2020-02-21 09:14:13,112 [myid:1] - INFO  [LearnerHandler-/192.168.100.1:49222:LearnerHandler@463] - Sending snapshot last zxid of peer is 0x0, zxid of leader is 0x200000000, send zxid of db as 0x100000000, 1 concurrent snapshots, snapshot was exempt from throttle
2020-02-21 09:14:13,123 [myid:1] - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Leader@1296] - Have quorum of supporters, sids: [ [0, 1],[0, 1] ]; starting up and setting last processed zxid: 0x200000000
2020-02-21 09:14:13,134 [myid:1] - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):CommitProcessor@256] - Configuring CommitProcessor with 1 worker threads.
2020-02-21 09:14:13,707 [myid:1] - INFO  [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ContainerManager@64] - Using checkIntervalMs=60000 maxPerMinute=10000
2020-02-21 09:15:07,995 [myid:1] - INFO  [/192.168.100.2:3888:QuorumCnxManager$Listener@924] - Received connection request 192.168.100.3:38234
2020-02-21 09:15:08,005 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), LEADING (my state)0 (n.config version)
2020-02-21 09:15:08,012 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 1 (n.leader), 0x100000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), LEADING (my state)0 (n.config version)
2020-02-21 09:15:10,489 [myid:1] - INFO  [LearnerHandler-/192.168.100.3:40110:LearnerHandler@406] - Follower sid: 2 : info : 192.168.100.3:2888:3888:participant
2020-02-21 09:15:15,772 [myid:1] - INFO  [LearnerHandler-/192.168.100.3:40110:ZKDatabase@295] - On disk txn sync enabled with snapshotSizeFactor 0.33
2020-02-21 09:15:15,773 [myid:1] - INFO  [LearnerHandler-/192.168.100.3:40110:LearnerHandler@708] - Synchronizing with Follower sid: 2 maxCommittedLog=0x0 minCommittedLog=0x0 lastProcessedZxid=0x200000000 peerLastZxid=0x0
2020-02-21 09:15:15,776 [myid:1] - INFO  [LearnerHandler-/192.168.100.3:40110:LearnerHandler@463] - Sending snapshot last zxid of peer is 0x0, zxid of leader is 0x200000000, send zxid of db as 0x200000000, 1 concurrent snapshots, snapshot was exempt from throttle

slaver2日志：

2020-02-21 09:15:09,388 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@938] - minSessionTimeout set to 4000
2020-02-21 09:15:09,388 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@947] - maxSessionTimeout set to 40000
2020-02-21 09:15:09,389 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@166] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /appserver/apache-zookeeper-3.5.7-bin/data/version-2 snapdir /appserver/apache-zookeeper-3.5.7-bin/data/version-2
2020-02-21 09:15:09,389 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@69] - FOLLOWING - LEADER ELECTION TOOK - 1373 MS
2020-02-21 09:15:15,778 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Learner@395] - Getting a snapshot from leader 0x200000000
2020-02-21 09:15:15,800 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Learner@546] - Learner received NEWLEADER message
2020-02-21 09:15:17,110 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FileTxnSnapLog@404] - Snapshotting: 0x200000000 to /appserver/apache-zookeeper-3.5.7-bin/data/version-2/snapshot.200000000
2020-02-21 09:15:20,901 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Learner@529] - Learner received UPTODATE message
2020-02-21 09:15:20,905 [myid:2] - INFO  [QuorumPeer[myid=2](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):CommitProcessor@256] - Configuring CommitProcessor with 1 worker threads.

master日志：

2020-02-21 09:14:11,379 [myid:0] - INFO  [QuorumPeer[myid=0](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@938] - minSessionTimeout set to 4000
2020-02-21 09:14:11,379 [myid:0] - INFO  [QuorumPeer[myid=0](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@947] - maxSessionTimeout set to 40000
2020-02-21 09:14:11,379 [myid:0] - INFO  [QuorumPeer[myid=0](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@166] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /appserver/apache-zookeeper-3.5.7-bin/data/version-2 snapdir /appserver/apache-zookeeper-3.5.7-bin/data/version-2
2020-02-21 09:14:11,380 [myid:0] - INFO  [QuorumPeer[myid=0](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@69] - FOLLOWING - LEADER ELECTION TOOK - 30 MS
2020-02-21 09:14:11,392 [myid:0] - WARN  [QuorumPeer[myid=0](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Learner@282] - Unexpected exception, tries=0, remaining init limit=20000, connecting to /192.168.100.2:2888
java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:233)
        at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:262)
        at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:77)
        at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1253)
2020-02-21 09:14:13,113 [myid:0] - INFO  [QuorumPeer[myid=0](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Learner@395] - Getting a snapshot from leader 0x100000000
2020-02-21 09:14:13,118 [myid:0] - INFO  [QuorumPeer[myid=0](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Learner@546] - Learner received NEWLEADER message
2020-02-21 09:14:13,121 [myid:0] - INFO  [QuorumPeer[myid=0](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FileTxnSnapLog@404] - Snapshotting: 0x100000000 to /appserver/apache-zookeeper-3.5.7-bin/data/version-2/snapshot.100000000
2020-02-21 09:14:13,711 [myid:0] - INFO  [QuorumPeer[myid=0](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Learner@529] - Learner received UPTODATE message
2020-02-21 09:14:13,720 [myid:0] - INFO  [QuorumPeer[myid=0](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):CommitProcessor@256] - Configuring CommitProcessor with 1 worker threads.
2020-02-21 09:15:07,982 [myid:0] - INFO  [/192.168.100.1:3888:QuorumCnxManager$Listener@924] - Received connection request 192.168.100.3:40460
2020-02-21 09:15:07,994 [myid:0] - INFO  [WorkerReceiver[myid=0]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)
2020-02-21 09:15:08,010 [myid:0] - INFO  [WorkerReceiver[myid=0]:FastLeaderElection@679] - Notification: 2 (message format version), 1 (n.leader), 0x100000000 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)

 七、安装管理控制台zkui
项目地址https://github.com/DeemOpen/zkui
1、将代码拉下来，编译成jar包
2、创建目录，上传jar包

mkdir -p /appserver/zkui

把zkui-2.0-SNAPSHOT-jar-with-dependencies.jar和代码目录下的config.cfg、zkui.sh上传到服务（任意一台即可，例如192.168.100.1）
3、修改config.cfg

zkServer=192.168.100.1:2181,192.168.100.2:2181,192.168.100.3:2181

4、修改zkui.sh

#!/usr/bin/env bash

PIDFILE=~/zkui.pid
ZKUIBINDIR=$(cd `dirname $0`; pwd)
ZKUICLASSNAME="zkui-2.0-SNAPSHOT-jar-with-dependencies.jar"
ZKUI_DAEMON_OUT=$ZKUIBINDIR/zkui.out

start(){
    echo  "Starting zkui ... $ZKUIBINDIR"
    if [ -f "$PIDFILE" ]; then
        if kill -0 `cat "$PIDFILE"` > /dev/null 2>&1; then
            echo zkui already running as process `cat "$PIDFILE"`. 
            exit 0
        fi
    fi
    nohup java -jar "$ZKUIBINDIR/$ZKUICLASSNAME" > "$ZKUI_DAEMON_OUT" 2>&1 < /dev/null &
    if [ $? -eq 0 ];
    then
        echo $!>$PIDFILE
        if [ $? -eq 0 ];
        then
            sleep 1
            echo STARTED
        else
            echo FAILED TO WRITE PID
            exit 1
        fi
    else
        echo SERVER DID NOT START
        exit 1
    fi
}

stop(){
    echo -n "Stopping zkui ... "
    if [ ! -f "$PIDFILE" ]
    then
        echo "no zkui to stop (could not find file $PIDFILE)"
    else
        kill -9 $(cat "$PIDFILE")
        rm "$PIDFILE"
        echo STOPPED
    fi
    exit 0
}

case "$1" in
start)
    start
    ;;
stop)
    stop
    ;;
restart)
    shift
    "$0" stop
    sleep 3
    "$0" start
    ;;
*)
    echo "Usage: $0 {start|stop|restart}" >&2
esac

5、命令
启动：zkui.sh start
停止：zkui.sh stop
重启：zkui.sh restart
6、访问zkui界面
http://IP:9090/login
默认使用admin/manager登录 

六、参考资料
https://www.cnblogs.com/ysocean/p/9860529.html
https://www.cnblogs.com/yuyijq/p/3438829.html
https://blog.csdn.net/gloriaied/article/details/103188940
https://blog.csdn.net/topc2000/article/details/100704084
https://blog.csdn.net/smartdt/article/details/79885270