理解zookeeper的一致性及缺点

---

zookeeper使用ZAB协议达到了极高的一致性。所以在互联网业务中它经常被选作注册中心、配置中心、注册分布式锁等。

zookeeper保证

根据zookeeper官方文档，zookeeper提供了如下保证：

• Sequential Consistency - Updates from a client will be applied in the order that they were sent.
• Atomicity - Updates either succeed or fail. No partial results.
• Single System Image - A client will see the same view of the service regardless of the server that it connects to. i.e., a client will never see an older view of the system even if the client fails over to a different server with the same session. 如果client首先看到了新数据，再尝试重连到存有旧数据的follower，该follower会拒绝该连接（client的zxid高于follower）
• Reliability - Once an update has been applied, it will persist from that time forward until a client overwrites the update.
• Timeliness - The clients view of the system is guaranteed to be up-to-date within a certain time bound.

根据我的实践，认为zookeeper只是一个最终一致性的分布式系统，并且历史上zookeeper还经常爆出违反分布式共识的bug，比如expired ephemeral node reappears after ZK leader change这个，session expired之后，临时节点仍然存在

理解zookeeper的顺序一致性

ZooKeeper Programmer’s Guide提到：

Sometimes developers mistakenly assume one other guarantee that ZooKeeper does not in fact make. This is:
Simultaneously Conistent Cross-Client Views
ZooKeeper does not guarantee that at every instance in time, two different clients will have identical views of ZooKeeper data. Due to factors like network delays, one client may perform an update before another client gets notified of the change. Consider the scenario of two clients, A and B. If client A sets the value of a znode /a from 0 to 1, then tells client B to read /a, client B may read the old value of 0, depending on which server it is connected to. If it is important that Client A and Client B read the same value, Client B should should call the sync() method from the ZooKeeper API method before it performs its read.
So, ZooKeeper by itself doesn’t guarantee that changes occur synchronously across all servers, but ZooKeeper primitives can be used to construct higher level functions that provide useful client synchronization.

就是说zookeeper并不保证每次从其一个server读到的值是最新的，它只保证这个server中的值是顺序更新的，如果其他server节点想要读取最新的值，必须在get之前调用sync()(zoo_async)

zookeeper的缺点

1. Zab协议自身的限制导致了zookeeper的很多瓶颈，比如，单leader瓶颈，切主时服务不可用、系统存储的内容有限，可扩展性不足等等。
   • 另外zookeeper集群的一致性模型也并没有想象中完美，不提一些违背一致性的bug如ZOOKEEPER-2919，其本身的机制：更新操作都要forward给leader，读操作follower节点可以独立进行，就决定了zookeeper的一致性保证只能做到“Updates from a client will be applied in the order that they were sent”
2. 身为一个分布式系统，本身就免不了有许多bug。有很多论文调查、研究分布式系统历史上出现的各种bug，我列举了几篇放在参考链接中
3. zookeeper本身限制也导致了客户端的访问方式、处理事件的方式等等处处掣肘，客户端不管其上层承载的业务模型是怎样的，都要按照zookeeper的filesystem/trigger API去操作。

著名的zookeeper客户端库Curator专门总结了使用Zookeeper的Tech notes，我选择一些重要的翻译如下：

1. 所有的watcher事件都应该在同一个线程里执行，然后再这个线程里对访问的资源加锁（这个操作应该由zk库在zk线程里自己完成）
2. 认真对待session生命周期，如果expired就需要重连，如果session已经expired了，所有与这个session相关的操作也应该失败。session和临时节点是绑定的，session expired了临时节点也就没了
3. zookeeper不适合做消息队列，因为
   • zookeeper有1M的消息大小限制
   • zookeeper的children太多会极大的影响性能
   • znode太大也会影响性能
   • znode太大会导致重启zkserver耗时10-15分钟
   • zookeeper仅使用内存作为存储，所以不能存储太多东西。
4. 最好单线程操作zk客户端，不要并发，临界、竞态问题太多
5. Curator session 生命周期管理：
   • CONNECTED：第一次建立连接成功时收到该事件
   • READONLY：标明当前连接是read-only状态
   • SUSPENDED：连接目前断开了(收到KeeperState.Disconnected事件，也就是说curator目前没有连接到任何的zk server)，leader选举、分布式锁等操作遇到SUSPENED事件应该暂停上层的业务直到重连成功。Curator官方建议把SUSPENDED事件当作完全的连接断开来处理。意思就是收到SUSPENDED事件的时候就应该当作自己注册的所有临时节点已经掉了。
   • LOST：如下几种情况会进出LOST事件
     • curator收到zkserver发来的EXPIRED事件。
     • curator自己关掉当前zookeeper session
     • 当curator断定当前session被zkserver认为已经expired时设置该事件。在Curator 3.x，Curator会有自己的定时器，如果收到SUSPENDED事件一直没有没有收到重连成功的事件，超时一定时间（2/3 * session_timeout）。curator会认为当前session已经在server侧超时，并进入LOST事件。
   • RECONNECTED：重连成功

对于何时进入LOST状态，curator的建议：

When Curator receives a KeeperState.Disconnected message it changes its state to SUSPENDED (see TN12, errors, etc.). As always, our recommendation is to treat SUSPENDED as a complete connection loss. Exit all locks, leaders, etc. That said, since 3.x, Curator tries to simulate session expiration by starting an internal timer when KeeperState.Disconnected is received. If the timer expires before the connection is repaired, Curator changes its state to LOST and injects a session end into the managed ZooKeeper client connection. The duration of the timer is set to the value of the “negotiated session timeout” by calling ZooKeeper#getSessionTimeout().
The astute reader will realize that setting the timer to the full value of the session timeout may not be the correct value. This is due to the fact that the server closes the connection when 2/3 of a session have already elapsed. Thus, the server may close a session well before Curator’s timer elapses. This is further complicated by the fact that the client has no way of knowing why the connection was closed. There are at least three possible reasons for a client connection to close:

• The server has not received a heartbeat within 2/3 of a session
• The server crashed
• Some kind of general TCP error which causes a connection to fail

In situtation 1, the correct value for Curator’s timer is 1/3 of a session - i.e. Curator should switch to LOST if the connection is not repaired within 1/3 of a session as 2/3 of the session has already lapsed from the server’s point of view. In situations 2 and 3 however, Curator’s timer should be the full value of the session (possibly plus some “slop” value). In truth, there is no way to completely emulate in the client the session timing as managed by the ZooKeeper server. So, again, our recommendation is to treat SUSPENDED as complete connection loss.

curator默认使用100%的session timeout时间作为SUSPENDED到LOST的转换时间，但是用户可以根据需求配置为33%的session timeout以满足上文所说的情况的场景

可见，使用好zookeeper不是一件容易的事，笔者使用zookeeper的过程中也曾遇到以下问题：

1. zk session 处理
   • 忽略了connecting事件，client与server心跳超时之后没有将选主服务及时下线掉，导致双主脑裂。
   • 多个线程处理zk的连接状态，导致产生了多套zk线程连接zkserver。
   • zk超时时间不合理，导致重连频率太高，打爆zkserver。
   • 所有的zkserver全部重置（zk server全部状态被重置），这种情况下客户端不会受到expired事件，我之前实现的客户端也不会重新去建立zk session。导致之前的zkclient建立的session全部不可用，陷入无限重连而连不上的窘境。
2. 多线程竞态
   • zk客户端自己的线程do_completion会调用watcher的回调函数，和业务线程产生竞争，导致core dump。
3. 客户端同步api
   • 同步API没有超时时间，如果zkserver状态不对，发送给zkserver的rpc得不到回应，会导致调用同步zk API的线程阻塞卡死。
   • 供业务使用的api设计不当，导致初始化时调用的同步版本api造成死锁。

参考链接

1. What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems
2. TaxDC: A Taxonomy of Non-Deterministic Concurrency Bugs in Datacenter Distributed Systems
3. An Analysis of Network-Partitioning Failures in Cloud Systems
4. 可能是全网把 ZooKeeper 概念讲的最清楚的一篇文章
5. Zookeeper ZNodes – Characteristics & Example
6. ZooKeeper Recipes and Solutions
7. How to do distributed locking长文，里面画的图不错
8. ZAB协议简介braft介绍zab
9. Lease与最长宕机时间分析 zk提供选主服务，导致的不可服务的时间
10. Zab: High-performance broadcast for primary-backup systems
11. zookeeper Consistency Guarantees
12. 深入浅析zookeeper的一致性模型及其实现讲解了为什么zookeeper的一致性和其他一致性协议有区别
13. How ZooKeeper guarantees “Single System Image”?
14. ZooKeeper: Wait-free coordination for Internet-scale systems yahoo的论文
15. ZooKeeper FAQ 官方文档告诉你应该如何处理CONNECTION_LOSS，SESSION_EXPIRED等等，真的对zk有所了解的人都会问的问题。。。