zookeeper使用分享

zookeeper是什么？

ZooKeeper是一个针对分布式应用的分布式、开源的协调服务。通过它可以实现更高级别的服务，用于数据同步、配置维护、服务分组和服务命名等。zookeeper设计考虑了易用性和简单性，它使用了一种类似于文件系统目录树结构的数据模型。协调服务容易出现竞争条件和死锁等错误。ZooKeeper背后的动机是为了减轻分布式应用从头开始实现协调服务的责任。

zookeeper设计目标：

简单/可靠/顺序性的/高性能（读）

zookeeper数据模型：

节点分为： 持久节点/持久化顺序节点/临时节点/临时顺序节点， 节点包含节点属性/节点acl/节点quata/节点数据/节点ttl等

疑惑？

• 集群环境客户端使用是否会读取到过期数据？
• zookeeper是否支持节点ttl？
• zookeeper写入同一个节点是否会出现并发问题？
• zookeeper临时节点是客户端进程关闭后会立即消失吗？
• 支持事务操作, 是否绝对可靠？

curator.inTransaction().create().forPath("/t2").and().create().forPath("/t4").and().create().forPath("/t3")
                    .and().commit();
实际zk服务器代码接手到请求是经过preRequestProcessor-->SyncRequestProcessor-->FinalRequestProcessor, 在preRequestProcessor中会对事务中所有的操作进行预校验,理论上是可靠的（服务器突然crash不可靠结果不能预知，也许还没进行snapshot操作）

一、基本原理

1.1 写入流程

1.在Client向Follwer发出一个写的请求

2.Follwer把请求发送给Leader

3.Leader接收到以后开始发起投票并通知Follwer进行投票

4.Follwer把投票结果发送给Leader

5.Leader将结果汇总后如果需要写入，则开始写入同时把写入操作通知给Follwer，然后commit;

6.Follwer把请求结果返回给Client

1.2 事件监听机制(watcher)

• 一次事件只通知一次, 需要重新注册watcher。

一个节点多次注册watcher只会收到一次事件(是收到一次事件)。

curator.getData().usingWatcher(new Watcher() {
               @Override
               public void process(WatchedEvent watchedEvent) {
                   if (watchedEvent.getType() == EventType.NodeDataChanged) {
                       System.out.println(watchedEvent.hashCode() + ": accept data changed event1!!!");
                   }
               }
           }).forPath("/t1");
 
curator.getData().usingWatcher(new Watcher() {
               @Override
               public void process(WatchedEvent watchedEvent) {
                   if (watchedEvent.getType() == EventType.NodeDataChanged) {
                       System.out.println(watchedEvent.hashCode() + ": accept data changed event2!!!");
                   }
               }
           }).forPath("/t1");

事件冒泡通知( 比如：删除/t1/t11, /t1会收到删除子节点事件，但是/节点不会收到消息)。

curator.getData().usingWatcher(new Watcher() {
                @Override
                public void process(WatchedEvent watchedEvent) {
                    if (watchedEvent.getType() == EventType.NodeDeleted) {
                        System.out.println(watchedEvent.hashCode() + ": /t1/t11 node deleted!!!");
                    }
                }
            }).forPath("/t1/t11");
 
            curator.getChildren().usingWatcher(new Watcher() {
                @Override
                public void process(WatchedEvent watchedEvent) {
                    if (watchedEvent.getType() == EventType.NodeChildrenChanged) {
                        System.out.println(watchedEvent.hashCode() + ": /t1 children changed !!!");
                    }
                }
            }).forPath("/t1");

• 客户端会保障它在看到相应的znode的新数据之前接收到watch事件 【未验证，看代码没有这个逻辑】。
• 事件可能会丢失的。【特别注意， 在使用watcher的时候要注意】

1.3 超时处理

(1) session超时

session超时后才删除临时节点。

(2) 连接超时

//注册zk连接监听器
zkUtils.getCuratorFramework().getConnectionStateListenable().addListener(new SessionConnectionListener());
 
class SessionConnectionListener implements ConnectionStateListener {
       @Override
       public void stateChanged(CuratorFramework client, ConnectionState newState) {
           if (newState == ConnectionState.LOST) {
               log.warn("connection lost, start to repiare connection!!!");
               while (true) {
                   try {
                       if (client.getZookeeperClient().blockUntilConnectedOrTimedOut()) {
                           log.warn("connection reconnected!!!");
                           register();
                           break;
                       }
                   } catch (InterruptedException e) {
                       break;
                   } catch (Exception e) {
                   }
               }
           }
       }
   }

二、分布式id中的应用

private SnowFlake getSnowFlake(BusinessType resourceType) {
    if (machineId == MACHINE_ID_REBOOT_V) {
        throw new RuntimeException("zk id base path is deleted, please fix the issue and reboot the service!!!");
    }
    if (machineId == MACHINE_ID_RETRY_V) {
        log.info("try to re-register machineId");
        synchronized (this) {
            machineId = getMachineId();
            if (machineId == MACHINE_ID_RETRY_V) {
                throw new RuntimeException("machineId register fail");
            }
        }
    }
    String cacheKey = String.format("%d-%d-%d", this.dataCenterId, machineId, resourceType.getValue());
    if (snowFlakeMap.containsKey(cacheKey)) {
        return snowFlakeMap.get(cacheKey);
    } else {
        synchronized (this) {
            SnowFlake snowFlake = new SnowFlake(this.dataCenterId, machineId, (long) resourceType.getValue());
            snowFlakeMap.put(cacheKey, snowFlake);
            return snowFlake;
        }
    }
}
 
private long getMachineId() {
    try {
        zkUtils.getCuratorFramework().getData().usingWatcher(new BasePathWatcher()).forPath(zkIdBasePath);
    } catch (Exception e) {
        log.error(e.getMessage(), e);
    }
 
    String ipAddr = NetUtil.getIp();
    if (ipAddr == null) {
        log.error("get host ip fail!");
        return MACHINE_ID_RETRY_V;
    }
    for (String invalidIp : invalidIps) {
        if (invalidIp.equals(ipAddr)) {
            log.error("hostname and ip address is not be configured correctly!!!!");
            return MACHINE_ID_RETRY_V;
        }
    }
 
    String currentNodeZkPath = zkIdBasePath + "/" + ipAddr;
    //step1: 尝试读取改节点之前注册的机器id
    long machineId = getZkMachineId(ipAddr);
    if (machineId != MACHINE_ID_RETRY_V) {
        log.info("machineId existed :{}", machineId);
        return machineId;
    }
    //step2:之前未注册过，重新注册一次机器id
    try {
        String path = zkUtils.getCuratorFramework().create().creatingParentsIfNeeded()
                .withMode(CreateMode.PERSISTENT_SEQUENTIAL).forPath(currentNodeZkPath);
        zkUtils.getCuratorFramework().getData().usingWatcher(new IdNodePathWatcher()).forPath(path);
        machineId = Long.valueOf(path.substring(currentNodeZkPath.length()));
        log.info("registered new machineId :{}", machineId);
        return machineId;
    } catch (Exception e) {
        log.error(e.getMessage(), e);
    }
    return MACHINE_ID_RETRY_V;
}
 
private class IdNodePathWatcher implements Watcher {
    @Override
    public void process(WatchedEvent watchedEvent) {
        boolean shouldResetMachineId = false;
        switch (watchedEvent.getState()) {
            case Expired:
            case AuthFailed:
            case Disconnected:
                shouldResetMachineId = true;
            case SyncConnected:
                if (watchedEvent.getType() == EventType.NodeDeleted) {
                    shouldResetMachineId = true;
                }
            default:
        }
 
        if (shouldResetMachineId && machineId != MACHINE_ID_REBOOT_V) {
            machineId = MACHINE_ID_RETRY_V;
        }
        try {
            zkUtils.getCuratorFramework().getData().usingWatcher(new IdNodePathWatcher())
                    .forPath(watchedEvent.getPath());
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        }
    }
}
 
private class BasePathWatcher implements Watcher {
    @Override
    public void process(WatchedEvent watchedEvent) {
        if (watchedEvent.getType() == EventType.NodeDeleted) {
            machineId = MACHINE_ID_REBOOT_V;
        }
        try {
            zkUtils.getCuratorFramework().getData().usingWatcher(new BasePathWatcher())
                    .forPath(watchedEvent.getPath());
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        }
    }
}
 
public long getZkMachineId(String nodePath) {
    try {
        List<String> childrenPaths = zkUtils.getCuratorFramework().getChildren().forPath(zkIdBasePath);
        if (CollectionUtils.isEmpty(childrenPaths)) {
            return MACHINE_ID_RETRY_V;
        }
        for (String currNodePath : childrenPaths) {
            if (!currNodePath.startsWith(nodePath)) {
                continue;
            }
            return Long.valueOf(currNodePath.substring(nodePath.length()));
        }
    } catch (Exception e) {
        log.error(e.getMessage(), e);
    }
    return MACHINE_ID_RETRY_V;
}

三、分布式调度中的应用(非重入公平的排他锁)

@Component
@Slf4j
public class OwnerCoordinator {
    @Autowired
    private ZkUtils zkUtils;
    private byte ownerStatus = 0;//0:pending,1:hold,2:waiting
    private final String zkOwnerPath = "/cleaner/match-task-owner";
 
    public boolean eligible() {
        if (ownerStatus == 0) {
            try {
                zkUtils.getCuratorFramework().create().creatingParentsIfNeeded().withMode(CreateMode.EPHEMERAL)
                        .forPath(zkOwnerPath, NetUtil.getHostname().getBytes());
                ownerStatus = 1;
                return true;
            } catch (Throwable throwable) {
                ownerStatus = 2;
                try {
                    zkUtils.getCuratorFramework().getData().usingWatcher(new NodeReleaseWatcher()).forPath(zkOwnerPath);
                } catch (Exception e) {
                    log.error(e.getMessage(), e);
                }
                return false;
            }
        } else if (ownerStatus == 1) {
            return true;
        } else if (ownerStatus == 2) {
            return false;
        }
        return false;
    }
 
    class NodeReleaseWatcher implements Watcher {
 
        @Override
        public void process(WatchedEvent watchedEvent) {
            if (watchedEvent.getType() == EventType.NodeDeleted
                    || watchedEvent.getState() == KeeperState.Disconnected) {
                log.info("has node offline!!!!, change this node to pending schedule");
                ownerStatus = 0;
            }
        }
    }
}

四、sharding策略中的应用

一共分为6个sharding，2个机器，每个机器承担一部分：

Node1：

Node2：

class RootNodeWatcher implements Watcher {
 
        @Override
        public void process(WatchedEvent watchedEvent) {
 
            if (watchedEvent.getType() == EventType.NodeChildrenChanged) {
                //重新分配节点route,所有节点停止工作(已经在队列的数据继续处理)
                List<String> nodePaths = null;
                try {
                    nodePaths = zkUtils.getCuratorFramework().getChildren().forPath(zkNodesRoot);
                    Map<String, List<Byte>> routesMap = new HashMap<>();
                    if (!CollectionUtils.isEmpty(nodePaths)) {
                        for (String nodePath : nodePaths) {
                            routesMap.put(nodePath, new ArrayList<>());
                        }
                    }
                    int pathIdx = 1;
                    byte routeIdx = 0;
                    int avgRoutesCount = shardingNum / routesMap.size();
                    int appendRoutesCount = shardingNum % routesMap.size();
 
                    for (String nodePath : routesMap.keySet()) {
                        List<Byte> routes = routesMap.get(nodePath);
                        for (int i = 0; i < avgRoutesCount; i++) {
                            routes.add(routeIdx++);
                        }
                        if (pathIdx++ <= appendRoutesCount) {
                            routes.add(routeIdx++);
                        }
                    }
                    for (String nodePath : routesMap.keySet()) {
                        String routeData = StringUtils.join(routesMap.get(nodePath), ",");
                        zkUtils.getCuratorFramework().setData()
                                .forPath(zkNodesRoot + "/" + nodePath, routeData.getBytes());
                    }
                } catch (Exception e) {
                    log.error(e.getMessage(), e);
                }
            }
            try {
                zkUtils.getCuratorFramework().getChildren().usingWatcher(new RootNodeWatcher()).forPath(zkNodesRoot);
            } catch (Exception e) {
                log.error(e.getMessage(), e);
            }
        }
    }
 
class NodeWatcher implements Watcher {
        @Override
        public void process(WatchedEvent watchedEvent) {
            if (watchedEvent.getType() == EventType.NodeDataChanged) {
                //重新分配节点route,所有节点停止工作(已经在队列的数据继续处理)
                try {
                    byte[] data = zkUtils.getCuratorFramework().getData().usingWatcher(new NodeWatcher())
                            .forPath(currNodePath);
                    if (data == null || data.length == 0) {
                        log.info("update {} route info to empty!", currNodePath);
                        routes.clear();
                        return;
                    }
                    String routeInfo = new String(data);
                    String[] routesStr = routeInfo.split(",");
                    routes.clear();
                    for (String route : routesStr) {
                        routes.add(Byte.valueOf(route));
                    }
                    log.info("update {} route info to :{}", currNodePath, routeInfo);
                } catch (Exception e) {
                    log.error(e.getMessage(), e);
                }
            } else {
                try {
                    zkUtils.getCuratorFramework().getData().usingWatcher(new NodeWatcher()).forPath(currNodePath);
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        }
    }