Zookeeper分布式协调
一、Zookeeper是什么?
官方文档上这么解释zookeeper,它是一个分布式服务框架,是Apache Hadoop 的一个子项目,它主要是用来解决分布式应用中经常遇到的一些数据管理问题,如:统一命名服务、状态同步服务、集群管理、分布式应用配置项的管理等。
Zookeeper遵循是的CP原则,即保证了一致性,失去了可用性,体现在当Leader宕机后,zk 集群会马上进行新的 Leader 的选举,但是选举的这个过程是处于瘫痪状态的。所以其不满足可用性。
Eureka遵循的是AP原则,即保证了高可用,失去了一执行。每台服务器之间都有心跳检测机制,而且每台服务器都能进行读写,通过心跳机制完成数据共享同步,所以当一台机器宕机之后,其他的机器可以正常工作,但是可能此时宕机的机器还没有进行数据共享同步,所以其不满足一致性。
在ZAB中的三类角色
- Leader:ZK集群的老大,唯一的一个可以进行写数据的机器。
- Follower:ZK集群的具有一定职位的干活人。只能进行数据的读取,当老大(leader)机器挂了之后可以参与选举投票的机器。
- Observe:最小的干活小弟,只能进行数据读取,就算老大(leader)机器挂了,跟他一毛关系没有,不能参与选举投票的机器。
1、开启zookeeper服务及使用
Zookeeper下载地址
cd /home/installed/
tar -zxvf apache-zookeeper-3.7.0-bin.tar.gz
mv apache-zookeeper-3.7.0-bin zk1
cd /home/installed/zk1/conf
mv zoo_sample.cfg zoo.cfg
#复制另外两个节点
cp -r zk1 zk3
cp -r zk1 zk3
#修改配置文件,zk2和zk3分别是2182和2183且AdminServer不能重复
cd zk1/conf/
#数据存储路径
dataDir=/tmp/zookeeper/2181
#客户端端口
clientPort=2181
#修改AdminServer端口
admin.serverPort=8881
#dataDir分别创建对应的1、2、3
mkdir /tmp/zookeeper/2181
cd /tmp/zookeeper/2181
echo 1 >myid
#配置集群 server.服务器id=服务器IP地址:服务器直接通信端口:服务器之间选举投票端口
cd /home/installed/zk1/conf/
vim
server.1=127.0.0.1:2881:3881
server.2=127.0.0.1:2882:3882
server.3=127.0.0.1:2883:3883
#启动命令cd zk1/bin/
./zkServer.sh start
#查看节点状态
./zkServer.sh status
#停止节点
./zkServer.sh stop
#查看100行日志
cd /home/installed/zk1/logs
tail -100f zookeeper-root-server-VM-4-16-centos.out
启动成功
查看谁是主,谁是从,当主停止了,从会自动选举,主重启后会变成从
Zookeeper使用
1、使用 ls 命令来查看当前 ZooKeeper 中所包含的内容:
在bin目录下
./zkCli.sh -timeout 5000 -r -server ip:port
2、创建一个新的 znode ,使用 create /zkPro myData:
3、再次使用 ls 命令来查看现在 zookeeper 中所包含的内容:
4、下面我们运行 get 命令来确认第二步中所创建的 znode 是否包含我们所创建的字符串:
5、下面我们通过 set 命令来对 zk 所关联的字符串进行设置:
6、下面我们将刚才创建的 znode 删除,再用ls查询发现无了:
二、使用zookeeper
依赖
<!--zookeeper依赖-->
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.5.7</version>
</dependency>
1、连接zookeeper工具类
public class zkClient {
//连接地址
private String connectString = "111.111.111.111:2181";
//连接超时2000毫秒
private int sessionTimeout = 2000;
//连接
private ZooKeeper zkClient;
//计数器用于阻塞
private CountDownLatch latch = new CountDownLatch(1);
/**
* 连接客户端
*
* @throws IOException
*/
@Before
public void init() throws IOException, InterruptedException, KeeperException {
zkClient = new ZooKeeper(connectString, sessionTimeout, new Watcher() {
@Override
public void process(WatchedEvent watchedEvent) {
if (watchedEvent.getState() == Event.KeeperState.SyncConnected) {
//确认已经连接完毕后再进行操作
latch.countDown();
System.out.println("已经获得了连接");
}
}
});
//连接完成之前先等待,没有调用countDown则继续阻塞
latch.await();
//获取连接状态
ZooKeeper.States states = zkClient.getState();
System.out.println(states);
}
/**
* 创建子节点,
* 第一个个参数为那个目录,第二个参数为节点,第三个参数为权限,第四个参数为什么样的节点
*
* @throws InterruptedException
* @throws KeeperException
*/
@Test
public void create() throws InterruptedException, KeeperException {
String nodeCreated = zkClient.create("/yuange", "yuan1".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
}
/**
* 获取所有子节点
*
* @throws KeeperException
* @throws InterruptedException
*/
@Test
public void getChildren() throws KeeperException, InterruptedException {
//第二个参数代表是否要监听,为true的话代码在连接器的Watcher里
List<String> children = zkClient.getChildren("/", true);
for (String child : children) {
System.out.println(child);
}
// 延时
Thread.sleep(Long.MAX_VALUE);
}
/**
* 判断子节点是否存在
*
* @throws KeeperException
* @throws InterruptedException
*/
@Test
public void exist() throws KeeperException, InterruptedException {
Stat stat = zkClient.exists("/yuange", false);
System.out.println(stat == null ? "not exist " : "exist");
}
}
2、参数介绍
Zookeeper ZooDefs.Ids:
- OPEN_ACL_UNSAFE : 完全开放的ACL,任何连接的客户端都可以操作该属性znode
- CREATOR_ALL_ACL : 只有创建者才有ACL权限
- READ_ACL_UNSAFE:只能读取ACL
CreateMode:
- PERSISTENT:持久化目录节点,存储的数据不会丢失。
- PERSISTENT_SEQUENTIAL:顺序自动编号的持久化目录节点,存储的数据不会丢失,并且根据当前已近存在的节点数自动加 1,然后返回给客户端已经成功创建的目录节点名。
- EPHEMERAL:临时目录节点,一旦创建这个节点的客户端与服务器端口也就是session 超时,这种节点会被自动删除。
- EPHEMERAL_SEQUENTIAL: 临时自动编号节点,一旦创建这个节点的客户端与服务器端口也就是session 超时,这种节点会被自动删除,并且根据当前已近存在的节点数自动加 1,然后返回给客户端已经成功创建的目录节点名。
创建不同的节点:
- -s: 顺序节点,没有相对路径一说, 所有路径都是绝对路径。
[zk: localhost:2181(CONNECTED) 22] create /seq
Created /seq
[zk: localhost:2181(CONNECTED) 24] create -s /seq/Allen- "hello"
Created /seq/Allen-0000000001
[zk: localhost:2181(CONNECTED) 25] create -s /seq/Allen- "hello"
Created /seq/Allen-0000000002
[zk: localhost:2181(CONNECTED) 26] create -s /seq/Allen- "hello"
Created /seq/Allen-0000000003
[zk: localhost:2181(CONNECTED) 27] create -s /seq/Allen- "hello"
Created /seq/Allen-0000000004
[zk: localhost:2181(CONNECTED) 28] create -s /seq/Allen- "hello"
Created /seq/Allen-0000000005
[zk: localhost:2181(CONNECTED) 30] ls -R /seq
/seq
/seq/Allen-0000000001
/seq/Allen-0000000002
/seq/Allen-0000000003
/seq/Allen-0000000004
/seq/Allen-0000000005
[zk: localhost:2181(CONNECTED) 31]
- e: 临时节点,临时节点与持久节点的区别在于上面节点元数据信息的ephemeraOwner的值是不一样的。当客户端关掉,临时节点就没有了。临时节点下面是不能有子节点的。
[zk: localhost:2181(CONNECTED) 31] create -e /ephemeral "hello ephemeral"
Created /ephemeral
[zk: localhost:2181(CONNECTED) 32] get -s /ephemeral
hello ephemeral
cZxid = 0x15
ctime = Mon Apr 05 03:57:30 PDT 2021
mZxid = 0x15
mtime = Mon Apr 05 03:57:30 PDT 2021
pZxid = 0x15
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x1001f373c3a0001
dataLength = 15
numChildren = 0
- -c: 容器节点,唯一的区别是,当删除掉container节点下的所有子节点后, container节点本身也会被清除掉,默认被清除的时间是60s。
[zk: localhost:2181(CONNECTED) 3] create -c /container
Created /container
[zk: localhost:2181(CONNECTED) 4] create /container/sub1
Created /container/sub1
[zk: localhost:2181(CONNECTED) 5] create /container/sub2
Created /container/sub2
[zk: localhost:2181(CONNECTED) 6] create /container/sub3
Created /container/sub3
[zk: localhost:2181(CONNECTED) 7] ls -R /container
/container
/container/sub1
/container/sub2
/container/sub3
[zk: localhost:2181(CONNECTED) 8] delete /container/sub1
[zk: localhost:2181(CONNECTED) 9] delete /container/sub2
[zk: localhost:2181(CONNECTED) 10] delete /container/sub3
[zk: localhost:2181(CONNECTED) 11] ls /
[container, seq, seq0000000002, test1, zookeeper]
[zk: localhost:2181(CONNECTED) 12] ls /
[container, seq, seq0000000002, test1, zookeeper]
[zk: localhost:2181(CONNECTED) 16] ls /
[seq, seq0000000002, test1, zookeeper]
- -t: 可以给节点添加过期时间,默认禁用,需要通过系统参数启用,ttl节点的特性是可以创建一个打失效时间的节点,失效时间过来之后节点会被自动删除。
[zk: localhost:2181(CONNECTED) 0] create -t 5000 /ttl-node ttttt
Created /ttl-node
[zk: localhost:2181(CONNECTED) 1] ls /
[seq, seq0000000002, test1, ttl-node, zookeeper]
[zk: localhost:2181(CONNECTED) 2] ls /
[seq, seq0000000002, test1, ttl-node, zookeeper]
[zk: localhost:2181(CONNECTED) 9] ls /
[seq, seq0000000002, test1, ttl-node, zookeeper]
[zk: localhost:2181(CONNECTED) 10] ls /
[seq, seq0000000002, test1, zookeeper]
[zk: localhost:2181(CONNECTED) 11]
3、监听服务上下线提示
服务端
/**
* 服务端
*/
public class DistributeServer {
private String connectString = "123.123.123.123:2181";
private int sessionTimeout = 2000;
private ZooKeeper zk;
//计数器用于阻塞
private CountDownLatch latch = new CountDownLatch(1);
public static void main(String[] args) throws IOException, KeeperException, InterruptedException {
DistributeServer server = new DistributeServer();
// 1 获取zk连接
server.getConnect();
// 2 注册服务器到zk集群
server.regist(args[0]);
// 3 启动业务逻辑(睡觉)
server.business();
}
private void business() throws InterruptedException {
Thread.sleep(Long.MAX_VALUE);
}
private void regist(String hostname) throws KeeperException, InterruptedException {
String create = zk.create("/servers/" + hostname, hostname.getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);
System.out.println(hostname + " is online");
}
private void getConnect() throws IOException, InterruptedException {
zk = new ZooKeeper(connectString, sessionTimeout, new Watcher() {
@Override
public void process(WatchedEvent watchedEvent) {
if (watchedEvent.getState() == Event.KeeperState.SyncConnected) {
//确认已经连接完毕后再进行操作
latch.countDown();
System.out.println("已经获得了连接");
}
}
});
//连接完成之前先等待,没有调用countDown则继续阻塞
latch.await();
//获取连接状态
ZooKeeper.States states = zk.getState();
System.out.println(states);
}
}
客户端
/**
* 客户端
*/
public class DistributeClient {
private String connectString = "111.111.111.111:2181";
private int sessionTimeout = 2000;
private ZooKeeper zk;
//计数器用于阻塞
private CountDownLatch latch = new CountDownLatch(1);
public static void main(String[] args) throws IOException, KeeperException, InterruptedException {
DistributeClient client = new DistributeClient();
// 1 获取zk连接
client.getConnect();
// 2 监听/servers下面子节点的增加和删除
// client.getServerList();
// 3 业务逻辑(睡觉)
client.business();
}
private void business() throws InterruptedException {
Thread.sleep(Long.MAX_VALUE);
}
private void getServerList() throws KeeperException, InterruptedException {
List<String> children = zk.getChildren("/servers", true);
ArrayList<String> servers = new ArrayList<>();
for (String child : children) {
byte[] data = zk.getData("/servers/" + child, false, null);
servers.add(new String(data));
}
// 打印
System.out.println(servers);
}
private void getConnect() throws IOException, InterruptedException {
zk = new ZooKeeper(connectString, sessionTimeout, new Watcher() {
@Override
public void process(WatchedEvent watchedEvent) {
if (watchedEvent.getState() == Event.KeeperState.SyncConnected) {
//确认已经连接完毕后再进行操作
latch.countDown();
System.out.println("已经获得了连接");
}
try {
getServerList();
} catch (KeeperException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
//连接完成之前先等待,没有调用countDown则继续阻塞
latch.await();
//获取连接状态
ZooKeeper.States states = zk.getState();
System.out.println(states);
}
}
服务器上手动上线
测试上线
当服务端关闭后,节点自动下线
4、分布式锁
/**
* 客户端
*/
public class DistributedLock {
private final String connectString = "1.117.92.19:2181";
private final int sessionTimeout = 2000;
private final ZooKeeper zk;
private CountDownLatch connectLatch = new CountDownLatch(1);
private CountDownLatch waitLatch = new CountDownLatch(1);
private String waitPath;
private String currentMode;
public DistributedLock() throws IOException, InterruptedException, KeeperException {
// 获取连接
zk = new ZooKeeper(connectString, sessionTimeout, new Watcher() {
@Override
public void process(WatchedEvent watchedEvent) {
// connectLatch 如果连接上zk 可以释放
if (watchedEvent.getState() == Event.KeeperState.SyncConnected) {
connectLatch.countDown();
}
// waitLatch 需要释放,当删除节点时,并且获取路径和前一个路径是否相等,都符合则释放锁
if (watchedEvent.getType() == Event.EventType.NodeDeleted && watchedEvent.getPath().equals(waitPath)) {
waitLatch.countDown();
}
}
});
// 等待zk正常连接后,往下走程序,监听1
connectLatch.await();
// 判断根节点/locks是否存在
Stat stat = zk.exists("/locks", false);
if (stat == null) {
// 创建一下根节点
zk.create("/locks", "locks".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
}
}
// 对zk加锁
public void zklock() {
// 创建对应的临时带序号节点
try {
currentMode = zk.create("/locks/" + "seq-", null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);
// wait一小会, 让结果更清晰一些
Thread.sleep(10);
// 判断创建的节点是否是最小的序号节点,如果是获取到锁;如果不是,监听他序号前一个节点
List<String> children = zk.getChildren("/locks", false);
// 如果children 只有一个值,那就直接获取锁; 如果有多个节点,需要判断,谁最小
if (children.size() == 1) {
return;
} else {
Collections.sort(children);
// 获取节点名称 seq-00000000
String thisNode = currentMode.substring("/locks/".length());
// 通过seq-00000000获取该节点在children集合的位置
int index = children.indexOf(thisNode);
// 判断
if (index == -1) {
System.out.println("数据异常");
} else if (index == 0) {
// 就一个节点,可以获取锁了
return;
} else {
// 需要监听 他前一个节点变化
waitPath = "/locks/" + children.get(index - 1);
zk.getData(waitPath, true, new Stat());
// 等待监听2
waitLatch.await();
return;
}
}
} catch (KeeperException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
// 解锁
public void unZkLock() {
// 删除节点
try {
zk.delete(this.currentMode, -1);
} catch (InterruptedException e) {
e.printStackTrace();
} catch (KeeperException e) {
e.printStackTrace();
}
}
}
/**
* 分布式锁测试
*/
public class DistributedLockTest {
public static void main(String[] args) throws InterruptedException, IOException, KeeperException {
final DistributedLock lock1 = new DistributedLock();
final DistributedLock lock2 = new DistributedLock();
new Thread(new Runnable() {
@Override
public void run() {
try {
lock1.zklock();
System.out.println("线程1 启动,获取到锁");
Thread.sleep(5 * 1000);
lock1.unZkLock();
System.out.println("线程1 释放锁");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}).start();
new Thread(new Runnable() {
@Override
public void run() {
try {
lock2.zklock();
System.out.println("线程2 启动,获取到锁");
Thread.sleep(5 * 1000);
lock2.unZkLock();
System.out.println("线程2 释放锁");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}).start();
}
}
三、CuratorLock框架实现分布式锁
依赖
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-framework</artifactId>
<version>4.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
<version>4.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-client</artifactId>
<version>4.3.0</version>
</dependency>
测试代码
public class CuratorLockTest {
public static void main(String[] args) {
// 创建分布式锁1
InterProcessMutex lock1 = new InterProcessMutex(getCuratorFramework(), "/locks");
// 创建分布式锁2
InterProcessMutex lock2 = new InterProcessMutex(getCuratorFramework(), "/locks");
new Thread(new Runnable() {
@Override
public void run() {
try {
lock1.acquire();
System.out.println("线程1 获取到锁");
lock1.acquire();
System.out.println("线程1 再次获取到锁");
Thread.sleep(5 * 1000);
lock1.release();
System.out.println("线程1 释放锁");
lock1.release();
System.out.println("线程1 再次释放锁");
} catch (Exception e) {
e.printStackTrace();
}
}
}).start();
new Thread(new Runnable() {
@Override
public void run() {
try {
lock2.acquire();
System.out.println("线程2 获取到锁");
lock2.acquire();
System.out.println("线程2 再次获取到锁");
Thread.sleep(5 * 1000);
lock2.release();
System.out.println("线程2 释放锁");
lock2.release();
System.out.println("线程2 再次释放锁");
} catch (Exception e) {
e.printStackTrace();
}
}
}).start();
}
private static CuratorFramework getCuratorFramework() {
//参数一多少毫秒,参数二重试多少次
ExponentialBackoffRetry policy = new ExponentialBackoffRetry(10000, 3);
CuratorFramework client = CuratorFrameworkFactory.builder().connectString("123.123.123.123:2181")
.connectionTimeoutMs(10000)//连接时间输出
.sessionTimeoutMs(2000)//会话时间
.retryPolicy(policy).build();//连接失败后,多久后再连接
// 启动客户端
client.start();
System.out.println("zookeeper 启动成功");
return client;
}
}
效果
四、实践
项目会用到此注解@PostConstruct注解知识
五、其他
1、如何关闭 org.apache.zookeeper.clientcnxn 的(控制台大量输出)debug 日志
resources下创建logback.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!--定义日志文件的存储地址 勿在 LogBack 的配置中使用相对路径 -->
<property name="LOG_HOME" value="/home/webhome/mno-framework/logs/web" />
<!-- 控制台输出 -->
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<!--格式化输出:%d表示日期,%thread表示线程名,%-5level:级别从左显示5个字符宽度%msg:日志消息,%n是换行符 -->
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
</encoder>
</appender>
<!-- 按照每天生成日志文件 -->
<appender name="FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!--日志文件输出的文件名 -->
<FileNamePattern>${LOG_HOME}/mallweb.log.%d{yyyy-MM-dd}.log</FileNamePattern>
<!--日志文件保留天数 -->
<MaxHistory>30</MaxHistory>
</rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<!--格式化输出:%d表示日期,%thread表示线程名,%-5level:级别从左显示5个字符宽度%msg:日志消息,%n是换行符 -->
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} -
%msg%n</pattern>
</encoder>
<!--日志文件最大的大小 -->
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>10MB</MaxFileSize>
</triggeringPolicy>
</appender>
<logger name="com.xwtech.mnoframework" level="debug" />
<logger name="org.springframework" level="INFO" />
<logger name="com.alibaba.druid" level="INFO" />
<!--myibatis log configure -->
<logger name="org.apache.ibatis" level="debug" />
<logger name="java.sql.Connection" level="INFO" />
<logger name="java.sql.Statement" level="INFO" />
<logger name="java.sql.PreparedStatement" level="INFO" />
<logger name="log4j.logger.net" level="ERROR"/>
<logger name="log4j.logger.net.spy.memcached.transcoders.SerializingTranscoder" level="ERROR"/>
<!-- 日志输出级别 -->
<root level="INFO">
<appender-ref ref="STDOUT" />
<appender-ref ref="FILE" />
</root>
</configuration>