分布式一致性协议Raft原理与实例

最新推荐文章于 2024-04-09 09:40:18 发布

hellochenlu

最新推荐文章于 2024-04-09 09:40:18 发布

阅读量931

点赞数 1

分类专栏：分布式系统

分布式系统专栏收录该内容

43 篇文章 0 订阅

订阅专栏

分布式一致性协议Raft原理与实例

1.Raft协议

1.1 Raft简介

Raft是由Stanford提出的一种更易理解的一致性算法，意在取代目前广为使用的Paxos算法。目前，在各种主流语言中都有了一些开源实现，比如本文中将使用的基于JGroups的Raft协议实现。关于Raft的原理，强烈推荐动画版Raft讲解。

1.2 Raft原理

在Raft中，每个结点会处于下面三种状态中的一种：

follower：所有结点都以follower的状态开始。如果没收到leader消息则会变成candidate状态
candidate：会向其他结点“拉选票”，如果得到大部分的票则成为leader。这个过程就叫做Leader选举(Leader Election)
leader：所有对系统的修改都会先经过leader。每个修改都会写一条日志(log entry)。leader收到修改请求后的过程如下，这个过程叫做日志复制(Log Replication)：
1. 复制日志到所有follower结点(replicate entry)
2. 大部分结点响应时才提交日志
3. 通知所有follower结点日志已提交
4. 所有follower也提交日志
5. 现在整个系统处于一致的状态

1.2.1 Leader Election

当follower在选举超时时间(election timeout)内未收到leader的心跳消息(append entries)，则变成candidate状态。为了避免选举冲突，这个超时时间是一个150~300ms之间的随机数。

成为candidate的结点发起新的选举期(election term)去“拉选票”：

重置自己的计时器
投自己一票
发送 Request Vote消息

如果接收结点在新term内没有投过票那它就会投给此candidate，并重置它自己的选举超时时间。candidate拉到大部分选票就会成为leader，并定时发送心跳——Append Entries消息，去重置各个follower的计时器。当前Term会继续直到某个follower接收不到心跳并成为candidate。

如果不巧两个结点同时成为candidate都去“拉票”怎么办？这时会发生Splite Vote情况。两个结点可能都拉到了同样多的选票，难分胜负，选举失败，本term没有leader。之后又有计时器超时的follower会变成candidate，将term加一并开始新一轮的投票。

1.2.2 Log Replication

当发生改变时，leader会复制日志给follower结点，这也是通过Append Entries心跳消息完成的。前面已经列举了Log Replication的过程，这里就不重复了。

Raft能够正确地处理网络分区（“脑裂”）问题。假设A~E五个结点，B是leader。如果发生“脑裂”，A、B成为一个子分区，C、D、E成为一个子分区。此时C、D、E会发生选举，选出C作为新term的leader。这样我们在两个子分区内就有了不同term的两个leader。这时如果有客户端写A时，因为B无法复制日志到大部分follower所以日志处于uncommitted未提交状态。而同时另一个客户端对C的写操作却能够正确完成，因为C是新的leader，它只知道D和E。

当网络通信恢复，B能够发送心跳给C、D、E了，却发现“改朝换代”了，因为C的term值更大，所以B自动降格为follower。然后A和B都回滚未提交的日志，并从新leader那里复制最新的日志。但这样是不是就会丢失更新？

2.JGroups-raft介绍

2.1 JGroups中的Raft

JGroups是Java里比较流行的网络通信框架，近期顺应潮流，它也推出了Raft基于JGroups的实现。简单试用了一下，还比较容易上手，底层Raft的内部机制都被API屏蔽掉了。下面就通过一个分布式计数器的实例来学习一下Raft协议在JGroups中的实际用法。

Maven依赖如下：

<code class="language-xml hljs  has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span>
        <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>org.jgroups<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>
        <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>jgroups-raft<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>
        <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>0.2<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>
    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

其实JGroups-raft的Jar包中已经自带了一个Counter的Demo，但仔细看了一下，有的地方写的有些麻烦，不太容易把握住Raft这根主线。所以这里就参照官方的例子，进行了简写，突出Raft协议的基本使用方法。JGroups-raft目前资料不多，InfoQ上的这篇文章很不错，还有官方文档。

2.2 核心API

使用JGroups-raft时，我们一般会实现两个接口：RAFT.RoleChange和StateMachine：

实现RAFT.RoleChange接口的方法能通知我们当前哪个结点是leader
实现StateMachine执行要实现一致性的操作

典型单点服务实现方式就是：

<code class="language-java hljs  has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">JChannel ch = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>;
RaftHandle handle = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> RaftHandle(ch, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>);
handle.addRoleListener(role -> {
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span>(role == Role.Leader)
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// start singleton services</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span>
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// stop singleton services</span>
});</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li></ul>

2.3 默认配置

jgroups-raft.jar中已经带了一个raft.xml配置文件，作为实例程序我们可以直接使用它。

简要解释一下最核心的几个配置项，参照GitHub上的文档：

UDP：IP多播配置
raft.NO_DUPES：是否检测新加入结点的ID与老结点有重复
raft.ELECTION：选举超时时间的随机化范围
raft.RAFT：所有Raft集群的成员必须在这里声明，也可以在运行时通过addServer/removeServer动态修改
raft.REDIRECT：是否转发请求给leader
raft.CLIENT：在哪个IP和端口上接收客户端请求

<code class="language-xml hljs  has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"><!--
  Default stack using IP multicasting. It is similar to the "udp"
  stack in stacks.xml, but doesn't use streaming state transfer and flushing
  author: Bela Ban
--></span>

<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">config</span> <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">xmlns</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"urn:org:jgroups"</span>
        <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">xmlns:xsi</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"http://www.w3.org/2001/XMLSchema-instance"</span>
        <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">xsi:schemaLocation</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd"</span>></span>
    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">UDP
</span>         <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">mcast_addr</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"228.5.5.5"</span>
         <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">mcast_port</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"${jgroups.udp.mcast_port:45588}"</span>
         <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">...</span> /></span>
    ...
    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">raft.NO_DUPES</span>/></span>
    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">raft.ELECTION</span> <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">election_min_interval</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"100"</span> <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">election_max_interval</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"500"</span>/></span>
    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">raft.RAFT</span> <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">members</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"A,B,C"</span> <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">raft_id</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"${raft_id:undefined}"</span>/></span>
    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">raft.REDIRECT</span>/></span>
    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">raft.CLIENT</span> <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">bind_addr</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"0.0.0.0"</span> /></span>
<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">config</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li></ul>

3.JGroups-raft实例

实例很简单，只有JGroupsRaftTest和CounterService两个类组成。JGroupsRaftTest是测试启动类，而CounterService就是利用Raft协议实现的分布式计数服务类。

3.1 JGroupsRaftTest

JGroupsRaftTest的职责主要有三个：

创建Raft协议的JChannel
创建CounterService
循环读取用户输入

目前简单实现了几种操作包括：初始化计数器、加一、减一、读取计数器、查看Raft日志、做Raft快照（用于压缩日志文件）等。其中对计数器的操作，因为要与其他Raft成员进行分布式通信，所以当前集群必须要多于一个结点时才能进行操作。如果要支持单结点时的操作，需要做特殊处理。

<code class="language-java hljs  has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.JChannel;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.protocols.raft.RAFT;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.util.Util;

<span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
 * Test jgroups raft algorithm implementation.
 */</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">JGroupsRaftTest</span> {</span>

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> String CLUSTER_NAME = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"ctr-cluster"</span>;
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> String COUNTER_NAME = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"counter"</span>;
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> String RAFT_XML = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"raft.xml"</span>;

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">main</span>(String[] args) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> Exception {
        JChannel ch = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> JChannel(RAFT_XML).name(args[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>]);
        CounterService counter = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> CounterService(ch);

        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
            doConnect(ch, CLUSTER_NAME);
            doLoop(ch, counter);
        } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">finally</span> {
            Util.close(ch);
        }
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">doConnect</span>(JChannel ch, String clusterName) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> Exception {
        ch.connect(clusterName);
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">doLoop</span>(JChannel ch, CounterService counter) {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">boolean</span> looping = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">true</span>;
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">while</span> (looping) {
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> key = Util.keyPress(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\n[0] Create [1] Increment [2] Decrement [3] Dump log [4] Snapshot [x] Exit\n"</span> +
                    <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"first-applied="</span> + ((RAFT) ch.getProtocolStack().findProtocol(RAFT.class)).log().firstApplied() +
                    <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">", last-applied="</span> + counter.lastApplied() +
                    <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">", commit-index="</span> + counter.commitIndex() +
                    <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">", log size="</span> + Util.printBytes(counter.logSize()) + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">": "</span>);

            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> ((key == <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'0'</span> || key == <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'1'</span> || key == <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'2'</span>) && !counter.isLeaderExist()) {
                System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Cannot perform cause there is no leader by now"</span>);
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">continue</span>;
            }

            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> val;
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">switch</span> (key) {
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'0'</span>:
                    counter.getOrCreateCounter(COUNTER_NAME, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>L);
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">break</span>;
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'1'</span>:
                    val = counter.incrementAndGet(COUNTER_NAME);
                    System.out.printf(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"%s: %s\n"</span>, COUNTER_NAME, val);
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">break</span>;
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'2'</span>:
                    val = counter.decrementAndGet(COUNTER_NAME);
                    System.out.printf(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"%s: %s\n"</span>, COUNTER_NAME, val);
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">break</span>;
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'3'</span>:
                    counter.dumpLog();
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">break</span>;
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'4'</span>:
                    counter.snapshot();
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">break</span>;
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'x'</span>:
                    looping = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">false</span>;
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">break</span>;
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'\n'</span>:
                    System.out.println(COUNTER_NAME + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">": "</span> + counter.get(COUNTER_NAME) + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\n"</span>);
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">break</span>;
            }
        }
    }

}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li></ul>

3.2 CounterService

CounterService是我们的核心类，利用Raft实现了分布式的计数器操作，它的API主要由四部分组成：

Raft Local API：操作本地Raft的状态，像日志大小、做快照等
Raft API：实现Raft的监听器和状态机的方法
- roleChanged：本地Raft的角色发生变化
- apply：分布式通信消息
- readContentFrom/writeContentTo：读写快照
Counter API：计数器的分布式API
Counter Native API：计数器的本地API。直接使用的话相当于脏读

<code class="language-java hljs  has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.Channel;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.protocols.raft.RAFT;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.protocols.raft.Role;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.protocols.raft.StateMachine;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.raft.RaftHandle;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.util.AsciiString;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.util.Bits;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.util.ByteArrayDataInputStream;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.util.ByteArrayDataOutputStream;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.jgroups.util.Util;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.DataInput;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.DataOutput;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.io.IOException;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.text.SimpleDateFormat;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.Date;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.HashMap;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.Map;

<span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
 * Distribute counter service based on Raft consensus algorithm.
 */</span>
class CounterService implements StateMachine, RAFT.RoleChange {

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> RaftHandle raft;

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> Map<String, Long> counters;

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">enum</span> Command {
        CREATE, INCREMENT_AND_GET, DECREMENT_AND_GET, GET, SET
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-title" style="box-sizing: border-box;">CounterService</span>(Channel ch) {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.raft = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> RaftHandle(ch, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>);
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.counters = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> HashMap<>();

        raft.raftId(ch.getName())
            .addRoleListener(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>);
    }

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// ===========================================</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//              Raft Status API</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// ===========================================</span>

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> <span class="hljs-title" style="box-sizing: border-box;">lastApplied</span>() {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> raft.lastApplied();
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> <span class="hljs-title" style="box-sizing: border-box;">commitIndex</span>() {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> raft.commitIndex();
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> <span class="hljs-title" style="box-sizing: border-box;">logSize</span>() {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> raft.logSize();
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">dumpLog</span>() {
        System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\nindex (term): command\n---------------------"</span>);
        raft.logEntries((entry, index) -> {
            StringBuilder log = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> StringBuilder()
                    .append(index)
                    .append(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" ("</span>).append(entry.term()).append(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"): "</span>);

            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (entry.command() == <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span> ) {
                System.out.println(log.append(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<marker record>"</span>));
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span>;
            } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (entry.internal()) {
                System.out.println(log.append(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"<internal command>"</span>));
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span>;
            }

            ByteArrayDataInputStream in = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> ByteArrayDataInputStream(
                    entry.command(), entry.offset(), entry.length()
            );
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
                Command cmd = Command.values()[in.readByte()];
                String name = Bits.readAsciiString(in).toString();
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">switch</span> (cmd) {
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> CREATE:
                        log.append(cmd)
                            .append(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"("</span>).append(name).append(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">", "</span>)
                            .append(Bits.readLong(in))
                            .append(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">")"</span>);
                        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">break</span>;
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> GET:
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> INCREMENT_AND_GET:
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> DECREMENT_AND_GET:
                        log.append(cmd)
                            .append(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"("</span>).append(name).append(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">")"</span>);
                        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">break</span>;
                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">default</span>:
                        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> IllegalArgumentException(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Command "</span> + cmd + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"is unknown"</span>);
                }
                System.out.println(log);
            }
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> (IOException e) {
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> IllegalStateException(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Error when dump log"</span>, e);
            }
        });
        System.out.println();
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">snapshot</span>() {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
            raft.snapshot();
        } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> (Exception e) {
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> IllegalStateException(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Error when snapshot"</span>, e);
        }
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">boolean</span> <span class="hljs-title" style="box-sizing: border-box;">isLeaderExist</span>() {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> raft.leader() != <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>;
    }

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// ===========================================</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//              Raft API</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// ===========================================</span>

    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">roleChanged</span>(Role role) {
        System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"roleChanged to: "</span> + role);
    }

    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">byte</span>[] <span class="hljs-title" style="box-sizing: border-box;">apply</span>(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">byte</span>[] data, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> offset, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> length) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> Exception {
        ByteArrayDataInputStream in = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> ByteArrayDataInputStream(data, offset, length);
        Command cmd = Command.values()[in.readByte()];
        String name = Bits.readAsciiString(in).toString();
        System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"["</span> + <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SimpleDateFormat(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"HH:mm:ss.SSS"</span>).format(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Date())
                + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"] Apply: cmd=["</span> + cmd + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"]"</span>);

        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> v1, retVal;
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">switch</span> (cmd) {
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> CREATE:
                v1 = Bits.readLong(in);
                retVal = create0(name, v1);
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> Util.objectToByteBuffer(retVal);
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> GET:
                retVal = get0(name);
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> Util.objectToByteBuffer(retVal);
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> INCREMENT_AND_GET:
                retVal = add0(name, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>L);
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> Util.objectToByteBuffer(retVal);
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> DECREMENT_AND_GET:
                retVal = add0(name, -<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>L);
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> Util.objectToByteBuffer(retVal);
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">default</span>:
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> IllegalArgumentException(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Command "</span> + cmd + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"is unknown"</span>);
        }
    }

    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">readContentFrom</span>(DataInput in) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> Exception {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> size = in.readInt();
        System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"ReadContentFrom: size=["</span> + size + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"]"</span>);
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> i = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>; i < size; i++) {
            AsciiString name = Bits.readAsciiString(in);
            Long value = Bits.readLong(in);
            counters.put(name.toString(), value);
        }
    }

    <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">writeContentTo</span>(DataOutput out) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> Exception {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">synchronized</span> (counters) {
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> size = counters.size();
            System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"WriteContentFrom: size=["</span> + size + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"]"</span>);
            out.writeInt(size);
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (Map.Entry<String, Long> entry : counters.entrySet()) {
                AsciiString name = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> AsciiString(entry.getKey());
                Long value = entry.getValue();
                Bits.writeAsciiString(name, out);
                Bits.writeLong(value, out);
            }
        }
    }

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// ===========================================</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//              Counter API</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// ===========================================</span>

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">getOrCreateCounter</span>(String name, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> initVal) {
        Object retVal = invoke(Command.CREATE, name, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">false</span>, initVal);
        counters.put(name, (Long) retVal);
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> <span class="hljs-title" style="box-sizing: border-box;">incrementAndGet</span>(String name) {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span>) invoke(Command.INCREMENT_AND_GET, name, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">false</span>);
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> <span class="hljs-title" style="box-sizing: border-box;">decrementAndGet</span>(String name) {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span>) invoke(Command.DECREMENT_AND_GET, name, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">false</span>);
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> <span class="hljs-title" style="box-sizing: border-box;">get</span>(String name) {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span>) invoke(Command.GET, name, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">false</span>);
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> Object <span class="hljs-title" style="box-sizing: border-box;">invoke</span>(Command cmd, String name, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">boolean</span> ignoreRetVal, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span>... values) {
        ByteArrayDataOutputStream out = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> ByteArrayDataOutputStream(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">256</span>);
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
            out.writeByte(cmd.ordinal());
            Bits.writeAsciiString(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> AsciiString(name), out);
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> val : values) {
                Bits.writeLong(val, out);
            }

            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">byte</span>[] rsp = raft.set(out.buffer(), <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>, out.position());
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> ignoreRetVal ? <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span> : Util.objectFromByteBuffer(rsp);
        }
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> (IOException ex) {
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> RuntimeException(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Serialization failure (cmd="</span>
                    + cmd + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">", name="</span> + name + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">")"</span>, ex);
        }
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> (Exception ex) {
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throw</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> RuntimeException(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Raft set failure (cmd="</span>
                    + cmd + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">", name="</span> + name + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">")"</span>, ex);
        }
    }

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// ===========================================</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//              Counter Native API</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// ===========================================</span>

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">synchronized</span> Long <span class="hljs-title" style="box-sizing: border-box;">create0</span>(String name, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> initVal) {
        counters.putIfAbsent(name, initVal);
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> counters.get(name);
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">synchronized</span> Long <span class="hljs-title" style="box-sizing: border-box;">get0</span>(String name) {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> counters.getOrDefault(name, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L);
    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">synchronized</span> Long <span class="hljs-title" style="box-sizing: border-box;">add0</span>(String name, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> delta) {
        Long oldVal = counters.getOrDefault(name, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L);
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> counters.put(name, oldVal + delta);
    }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li><li style="box-sizing: border-box; padding: 0px 5px;">90</li><li style="box-sizing: border-box; padding: 0px 5px;">91</li><li style="box-sizing: border-box; padding: 0px 5px;">92</li><li style="box-sizing: border-box; padding: 0px 5px;">93</li><li style="box-sizing: border-box; padding: 0px 5px;">94</li><li style="box-sizing: border-box; padding: 0px 5px;">95</li><li style="box-sizing: border-box; padding: 0px 5px;">96</li><li style="box-sizing: border-box; padding: 0px 5px;">97</li><li style="box-sizing: border-box; padding: 0px 5px;">98</li><li style="box-sizing: border-box; padding: 0px 5px;">99</li><li style="box-sizing: border-box; padding: 0px 5px;">100</li><li style="box-sizing: border-box; padding: 0px 5px;">101</li><li style="box-sizing: border-box; padding: 0px 5px;">102</li><li style="box-sizing: border-box; padding: 0px 5px;">103</li><li style="box-sizing: border-box; padding: 0px 5px;">104</li><li style="box-sizing: border-box; padding: 0px 5px;">105</li><li style="box-sizing: border-box; padding: 0px 5px;">106</li><li style="box-sizing: border-box; padding: 0px 5px;">107</li><li style="box-sizing: border-box; padding: 0px 5px;">108</li><li style="box-sizing: border-box; padding: 0px 5px;">109</li><li style="box-sizing: border-box; padding: 0px 5px;">110</li><li style="box-sizing: border-box; padding: 0px 5px;">111</li><li style="box-sizing: border-box; padding: 0px 5px;">112</li><li style="box-sizing: border-box; padding: 0px 5px;">113</li><li style="box-sizing: border-box; padding: 0px 5px;">114</li><li style="box-sizing: border-box; padding: 0px 5px;">115</li><li style="box-sizing: border-box; padding: 0px 5px;">116</li><li style="box-sizing: border-box; padding: 0px 5px;">117</li><li style="box-sizing: border-box; padding: 0px 5px;">118</li><li style="box-sizing: border-box; padding: 0px 5px;">119</li><li style="box-sizing: border-box; padding: 0px 5px;">120</li><li style="box-sizing: border-box; padding: 0px 5px;">121</li><li style="box-sizing: border-box; padding: 0px 5px;">122</li><li style="box-sizing: border-box; padding: 0px 5px;">123</li><li style="box-sizing: border-box; padding: 0px 5px;">124</li><li style="box-sizing: border-box; padding: 0px 5px;">125</li><li style="box-sizing: border-box; padding: 0px 5px;">126</li><li style="box-sizing: border-box; padding: 0px 5px;">127</li><li style="box-sizing: border-box; padding: 0px 5px;">128</li><li style="box-sizing: border-box; padding: 0px 5px;">129</li><li style="box-sizing: border-box; padding: 0px 5px;">130</li><li style="box-sizing: border-box; padding: 0px 5px;">131</li><li style="box-sizing: border-box; padding: 0px 5px;">132</li><li style="box-sizing: border-box; padding: 0px 5px;">133</li><li style="box-sizing: border-box; padding: 0px 5px;">134</li><li style="box-sizing: border-box; padding: 0px 5px;">135</li><li style="box-sizing: border-box; padding: 0px 5px;">136</li><li style="box-sizing: border-box; padding: 0px 5px;">137</li><li style="box-sizing: border-box; padding: 0px 5px;">138</li><li style="box-sizing: border-box; padding: 0px 5px;">139</li><li style="box-sizing: border-box; padding: 0px 5px;">140</li><li style="box-sizing: border-box; padding: 0px 5px;">141</li><li style="box-sizing: border-box; padding: 0px 5px;">142</li><li style="box-sizing: border-box; padding: 0px 5px;">143</li><li style="box-sizing: border-box; padding: 0px 5px;">144</li><li style="box-sizing: border-box; padding: 0px 5px;">145</li><li style="box-sizing: border-box; padding: 0px 5px;">146</li><li style="box-sizing: border-box; padding: 0px 5px;">147</li><li style="box-sizing: border-box; padding: 0px 5px;">148</li><li style="box-sizing: border-box; padding: 0px 5px;">149</li><li style="box-sizing: border-box; padding: 0px 5px;">150</li><li style="box-sizing: border-box; padding: 0px 5px;">151</li><li style="box-sizing: border-box; padding: 0px 5px;">152</li><li style="box-sizing: border-box; padding: 0px 5px;">153</li><li style="box-sizing: border-box; padding: 0px 5px;">154</li><li style="box-sizing: border-box; padding: 0px 5px;">155</li><li style="box-sizing: border-box; padding: 0px 5px;">156</li><li style="box-sizing: border-box; padding: 0px 5px;">157</li><li style="box-sizing: border-box; padding: 0px 5px;">158</li><li style="box-sizing: border-box; padding: 0px 5px;">159</li><li style="box-sizing: border-box; padding: 0px 5px;">160</li><li style="box-sizing: border-box; padding: 0px 5px;">161</li><li style="box-sizing: border-box; padding: 0px 5px;">162</li><li style="box-sizing: border-box; padding: 0px 5px;">163</li><li style="box-sizing: border-box; padding: 0px 5px;">164</li><li style="box-sizing: border-box; padding: 0px 5px;">165</li><li style="box-sizing: border-box; padding: 0px 5px;">166</li><li style="box-sizing: border-box; padding: 0px 5px;">167</li><li style="box-sizing: border-box; padding: 0px 5px;">168</li><li style="box-sizing: border-box; padding: 0px 5px;">169</li><li style="box-sizing: border-box; padding: 0px 5px;">170</li><li style="box-sizing: border-box; padding: 0px 5px;">171</li><li style="box-sizing: border-box; padding: 0px 5px;">172</li><li style="box-sizing: border-box; padding: 0px 5px;">173</li><li style="box-sizing: border-box; padding: 0px 5px;">174</li><li style="box-sizing: border-box; padding: 0px 5px;">175</li><li style="box-sizing: border-box; padding: 0px 5px;">176</li><li style="box-sizing: border-box; padding: 0px 5px;">177</li><li style="box-sizing: border-box; padding: 0px 5px;">178</li><li style="box-sizing: border-box; padding: 0px 5px;">179</li><li style="box-sizing: border-box; padding: 0px 5px;">180</li><li style="box-sizing: border-box; padding: 0px 5px;">181</li><li style="box-sizing: border-box; padding: 0px 5px;">182</li><li style="box-sizing: border-box; padding: 0px 5px;">183</li><li style="box-sizing: border-box; padding: 0px 5px;">184</li><li style="box-sizing: border-box; padding: 0px 5px;">185</li><li style="box-sizing: border-box; padding: 0px 5px;">186</li><li style="box-sizing: border-box; padding: 0px 5px;">187</li><li style="box-sizing: border-box; padding: 0px 5px;">188</li><li style="box-sizing: border-box; padding: 0px 5px;">189</li><li style="box-sizing: border-box; padding: 0px 5px;">190</li><li style="box-sizing: border-box; padding: 0px 5px;">191</li><li style="box-sizing: border-box; padding: 0px 5px;">192</li><li style="box-sizing: border-box; padding: 0px 5px;">193</li><li style="box-sizing: border-box; padding: 0px 5px;">194</li><li style="box-sizing: border-box; padding: 0px 5px;">195</li><li style="box-sizing: border-box; padding: 0px 5px;">196</li><li style="box-sizing: border-box; padding: 0px 5px;">197</li><li style="box-sizing: border-box; padding: 0px 5px;">198</li><li style="box-sizing: border-box; padding: 0px 5px;">199</li><li style="box-sizing: border-box; padding: 0px 5px;">200</li><li style="box-sizing: border-box; padding: 0px 5px;">201</li><li style="box-sizing: border-box; padding: 0px 5px;">202</li><li style="box-sizing: border-box; padding: 0px 5px;">203</li><li style="box-sizing: border-box; padding: 0px 5px;">204</li><li style="box-sizing: border-box; padding: 0px 5px;">205</li><li style="box-sizing: border-box; padding: 0px 5px;">206</li><li style="box-sizing: border-box; padding: 0px 5px;">207</li><li style="box-sizing: border-box; padding: 0px 5px;">208</li><li style="box-sizing: border-box; padding: 0px 5px;">209</li><li style="box-sizing: border-box; padding: 0px 5px;">210</li><li style="box-sizing: border-box; padding: 0px 5px;">211</li><li style="box-sizing: border-box; padding: 0px 5px;">212</li><li style="box-sizing: border-box; padding: 0px 5px;">213</li><li style="box-sizing: border-box; padding: 0px 5px;">214</li><li style="box-sizing: border-box; padding: 0px 5px;">215</li><li style="box-sizing: border-box; padding: 0px 5px;">216</li><li style="box-sizing: border-box; padding: 0px 5px;">217</li><li style="box-sizing: border-box; padding: 0px 5px;">218</li><li style="box-sizing: border-box; padding: 0px 5px;">219</li><li style="box-sizing: border-box; padding: 0px 5px;">220</li><li style="box-sizing: border-box; padding: 0px 5px;">221</li><li style="box-sizing: border-box; padding: 0px 5px;">222</li><li style="box-sizing: border-box; padding: 0px 5px;">223</li><li style="box-sizing: border-box; padding: 0px 5px;">224</li><li style="box-sizing: border-box; padding: 0px 5px;">225</li><li style="box-sizing: border-box; padding: 0px 5px;">226</li><li style="box-sizing: border-box; padding: 0px 5px;">227</li><li style="box-sizing: border-box; padding: 0px 5px;">228</li><li style="box-sizing: border-box; padding: 0px 5px;">229</li><li style="box-sizing: border-box; padding: 0px 5px;">230</li><li style="box-sizing: border-box; padding: 0px 5px;">231</li><li style="box-sizing: border-box; padding: 0px 5px;">232</li><li style="box-sizing: border-box; padding: 0px 5px;">233</li><li style="box-sizing: border-box; padding: 0px 5px;">234</li><li style="box-sizing: border-box; padding: 0px 5px;">235</li><li style="box-sizing: border-box; padding: 0px 5px;">236</li><li style="box-sizing: border-box; padding: 0px 5px;">237</li><li style="box-sizing: border-box; padding: 0px 5px;">238</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li><li style="box-sizing: border-box; padding: 0px 5px;">90</li><li style="box-sizing: border-box; padding: 0px 5px;">91</li><li style="box-sizing: border-box; padding: 0px 5px;">92</li><li style="box-sizing: border-box; padding: 0px 5px;">93</li><li style="box-sizing: border-box; padding: 0px 5px;">94</li><li style="box-sizing: border-box; padding: 0px 5px;">95</li><li style="box-sizing: border-box; padding: 0px 5px;">96</li><li style="box-sizing: border-box; padding: 0px 5px;">97</li><li style="box-sizing: border-box; padding: 0px 5px;">98</li><li style="box-sizing: border-box; padding: 0px 5px;">99</li><li style="box-sizing: border-box; padding: 0px 5px;">100</li><li style="box-sizing: border-box; padding: 0px 5px;">101</li><li style="box-sizing: border-box; padding: 0px 5px;">102</li><li style="box-sizing: border-box; padding: 0px 5px;">103</li><li style="box-sizing: border-box; padding: 0px 5px;">104</li><li style="box-sizing: border-box; padding: 0px 5px;">105</li><li style="box-sizing: border-box; padding: 0px 5px;">106</li><li style="box-sizing: border-box; padding: 0px 5px;">107</li><li style="box-sizing: border-box; padding: 0px 5px;">108</li><li style="box-sizing: border-box; padding: 0px 5px;">109</li><li style="box-sizing: border-box; padding: 0px 5px;">110</li><li style="box-sizing: border-box; padding: 0px 5px;">111</li><li style="box-sizing: border-box; padding: 0px 5px;">112</li><li style="box-sizing: border-box; padding: 0px 5px;">113</li><li style="box-sizing: border-box; padding: 0px 5px;">114</li><li style="box-sizing: border-box; padding: 0px 5px;">115</li><li style="box-sizing: border-box; padding: 0px 5px;">116</li><li style="box-sizing: border-box; padding: 0px 5px;">117</li><li style="box-sizing: border-box; padding: 0px 5px;">118</li><li style="box-sizing: border-box; padding: 0px 5px;">119</li><li style="box-sizing: border-box; padding: 0px 5px;">120</li><li style="box-sizing: border-box; padding: 0px 5px;">121</li><li style="box-sizing: border-box; padding: 0px 5px;">122</li><li style="box-sizing: border-box; padding: 0px 5px;">123</li><li style="box-sizing: border-box; padding: 0px 5px;">124</li><li style="box-sizing: border-box; padding: 0px 5px;">125</li><li style="box-sizing: border-box; padding: 0px 5px;">126</li><li style="box-sizing: border-box; padding: 0px 5px;">127</li><li style="box-sizing: border-box; padding: 0px 5px;">128</li><li style="box-sizing: border-box; padding: 0px 5px;">129</li><li style="box-sizing: border-box; padding: 0px 5px;">130</li><li style="box-sizing: border-box; padding: 0px 5px;">131</li><li style="box-sizing: border-box; padding: 0px 5px;">132</li><li style="box-sizing: border-box; padding: 0px 5px;">133</li><li style="box-sizing: border-box; padding: 0px 5px;">134</li><li style="box-sizing: border-box; padding: 0px 5px;">135</li><li style="box-sizing: border-box; padding: 0px 5px;">136</li><li style="box-sizing: border-box; padding: 0px 5px;">137</li><li style="box-sizing: border-box; padding: 0px 5px;">138</li><li style="box-sizing: border-box; padding: 0px 5px;">139</li><li style="box-sizing: border-box; padding: 0px 5px;">140</li><li style="box-sizing: border-box; padding: 0px 5px;">141</li><li style="box-sizing: border-box; padding: 0px 5px;">142</li><li style="box-sizing: border-box; padding: 0px 5px;">143</li><li style="box-sizing: border-box; padding: 0px 5px;">144</li><li style="box-sizing: border-box; padding: 0px 5px;">145</li><li style="box-sizing: border-box; padding: 0px 5px;">146</li><li style="box-sizing: border-box; padding: 0px 5px;">147</li><li style="box-sizing: border-box; padding: 0px 5px;">148</li><li style="box-sizing: border-box; padding: 0px 5px;">149</li><li style="box-sizing: border-box; padding: 0px 5px;">150</li><li style="box-sizing: border-box; padding: 0px 5px;">151</li><li style="box-sizing: border-box; padding: 0px 5px;">152</li><li style="box-sizing: border-box; padding: 0px 5px;">153</li><li style="box-sizing: border-box; padding: 0px 5px;">154</li><li style="box-sizing: border-box; padding: 0px 5px;">155</li><li style="box-sizing: border-box; padding: 0px 5px;">156</li><li style="box-sizing: border-box; padding: 0px 5px;">157</li><li style="box-sizing: border-box; padding: 0px 5px;">158</li><li style="box-sizing: border-box; padding: 0px 5px;">159</li><li style="box-sizing: border-box; padding: 0px 5px;">160</li><li style="box-sizing: border-box; padding: 0px 5px;">161</li><li style="box-sizing: border-box; padding: 0px 5px;">162</li><li style="box-sizing: border-box; padding: 0px 5px;">163</li><li style="box-sizing: border-box; padding: 0px 5px;">164</li><li style="box-sizing: border-box; padding: 0px 5px;">165</li><li style="box-sizing: border-box; padding: 0px 5px;">166</li><li style="box-sizing: border-box; padding: 0px 5px;">167</li><li style="box-sizing: border-box; padding: 0px 5px;">168</li><li style="box-sizing: border-box; padding: 0px 5px;">169</li><li style="box-sizing: border-box; padding: 0px 5px;">170</li><li style="box-sizing: border-box; padding: 0px 5px;">171</li><li style="box-sizing: border-box; padding: 0px 5px;">172</li><li style="box-sizing: border-box; padding: 0px 5px;">173</li><li style="box-sizing: border-box; padding: 0px 5px;">174</li><li style="box-sizing: border-box; padding: 0px 5px;">175</li><li style="box-sizing: border-box; padding: 0px 5px;">176</li><li style="box-sizing: border-box; padding: 0px 5px;">177</li><li style="box-sizing: border-box; padding: 0px 5px;">178</li><li style="box-sizing: border-box; padding: 0px 5px;">179</li><li style="box-sizing: border-box; padding: 0px 5px;">180</li><li style="box-sizing: border-box; padding: 0px 5px;">181</li><li style="box-sizing: border-box; padding: 0px 5px;">182</li><li style="box-sizing: border-box; padding: 0px 5px;">183</li><li style="box-sizing: border-box; padding: 0px 5px;">184</li><li style="box-sizing: border-box; padding: 0px 5px;">185</li><li style="box-sizing: border-box; padding: 0px 5px;">186</li><li style="box-sizing: border-box; padding: 0px 5px;">187</li><li style="box-sizing: border-box; padding: 0px 5px;">188</li><li style="box-sizing: border-box; padding: 0px 5px;">189</li><li style="box-sizing: border-box; padding: 0px 5px;">190</li><li style="box-sizing: border-box; padding: 0px 5px;">191</li><li style="box-sizing: border-box; padding: 0px 5px;">192</li><li style="box-sizing: border-box; padding: 0px 5px;">193</li><li style="box-sizing: border-box; padding: 0px 5px;">194</li><li style="box-sizing: border-box; padding: 0px 5px;">195</li><li style="box-sizing: border-box; padding: 0px 5px;">196</li><li style="box-sizing: border-box; padding: 0px 5px;">197</li><li style="box-sizing: border-box; padding: 0px 5px;">198</li><li style="box-sizing: border-box; padding: 0px 5px;">199</li><li style="box-sizing: border-box; padding: 0px 5px;">200</li><li style="box-sizing: border-box; padding: 0px 5px;">201</li><li style="box-sizing: border-box; padding: 0px 5px;">202</li><li style="box-sizing: border-box; padding: 0px 5px;">203</li><li style="box-sizing: border-box; padding: 0px 5px;">204</li><li style="box-sizing: border-box; padding: 0px 5px;">205</li><li style="box-sizing: border-box; padding: 0px 5px;">206</li><li style="box-sizing: border-box; padding: 0px 5px;">207</li><li style="box-sizing: border-box; padding: 0px 5px;">208</li><li style="box-sizing: border-box; padding: 0px 5px;">209</li><li style="box-sizing: border-box; padding: 0px 5px;">210</li><li style="box-sizing: border-box; padding: 0px 5px;">211</li><li style="box-sizing: border-box; padding: 0px 5px;">212</li><li style="box-sizing: border-box; padding: 0px 5px;">213</li><li style="box-sizing: border-box; padding: 0px 5px;">214</li><li style="box-sizing: border-box; padding: 0px 5px;">215</li><li style="box-sizing: border-box; padding: 0px 5px;">216</li><li style="box-sizing: border-box; padding: 0px 5px;">217</li><li style="box-sizing: border-box; padding: 0px 5px;">218</li><li style="box-sizing: border-box; padding: 0px 5px;">219</li><li style="box-sizing: border-box; padding: 0px 5px;">220</li><li style="box-sizing: border-box; padding: 0px 5px;">221</li><li style="box-sizing: border-box; padding: 0px 5px;">222</li><li style="box-sizing: border-box; padding: 0px 5px;">223</li><li style="box-sizing: border-box; padding: 0px 5px;">224</li><li style="box-sizing: border-box; padding: 0px 5px;">225</li><li style="box-sizing: border-box; padding: 0px 5px;">226</li><li style="box-sizing: border-box; padding: 0px 5px;">227</li><li style="box-sizing: border-box; padding: 0px 5px;">228</li><li style="box-sizing: border-box; padding: 0px 5px;">229</li><li style="box-sizing: border-box; padding: 0px 5px;">230</li><li style="box-sizing: border-box; padding: 0px 5px;">231</li><li style="box-sizing: border-box; padding: 0px 5px;">232</li><li style="box-sizing: border-box; padding: 0px 5px;">233</li><li style="box-sizing: border-box; padding: 0px 5px;">234</li><li style="box-sizing: border-box; padding: 0px 5px;">235</li><li style="box-sizing: border-box; padding: 0px 5px;">236</li><li style="box-sizing: border-box; padding: 0px 5px;">237</li><li style="box-sizing: border-box; padding: 0px 5px;">238</li></ul>

3.3 运行测试

我们分别以A、B、C为参数，启动三个JGroupsRaftTest服务。这样会自动在C:\Users\cdai\AppData\Local\Temp下生成A.log、B.log、C.log三个日志文件夹。

<code class="hljs livecodeserver has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">cdai@vm /cygdrive/c/Users/cdai/AppData/Local/Temp
$ tree A.<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">log</span>/ B.<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">log</span>/ C.<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">log</span>/
A.<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">log</span>/
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- 000005.sst</span>
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- 000006.log</span>
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- CURRENT</span>
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- LOCK</span>
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- LOG</span>
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- LOG.old</span>
`<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- MANIFEST-000004</span>
B.<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">log</span>/
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- 000003.log</span>
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- CURRENT</span>
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- LOCK</span>
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- LOG</span>
`<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- MANIFEST-000002</span>
C.<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">log</span>/
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- 000003.log</span>
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- CURRENT</span>
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- LOCK</span>
|<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- LOG</span>
`<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-- MANIFEST-000002</span>

<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">directories</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">17</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">files</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul>

3.3.1 分布式一致性

首先A创建计数器，B“加一”，C“减一”。可以看到尽管我们是分别在A、B、C上执行这三个操作，但三个结点都先后（leader提交日志后通知follower）通过apply()方法收到消息，并在本地的计数器Map上同步执行操作，保证了数据的一致性。最后停掉A服务，可以看到B通过roleChanged()得到消息，提升为新的Leader，并与C一同继续提供服务。

A的控制台输出：

<code class="hljs asciidoc has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-code" style="box-sizing: border-box;">-------------------------------------------------------------------
GMS: address=A, cluster=ctr-cluster, physical address=2001:0:9d38:6abd:cbb:1f78:3f57:50f6:50100
-------------------------------------------------------------------</span>

[0] Create [1] Increment [2] Decrement [3] Dump log [4] Snapshot [x] Exit
first-applied=0, last-applied=0, commit-index=0, log size=0b: 
roleChanged to: Candidate
roleChanged to: Leader
0
<span class="hljs-attribute" style="box-sizing: border-box;">[14:16:00.744] Apply: cmd=[CREATE]</span>

[0] Create [1] Increment [2] Decrement [3] Dump log [4] Snapshot [x] Exit
first-applied=0, last-applied=1, commit-index=1, log size=1b: 
<span class="hljs-attribute" style="box-sizing: border-box;">[14:16:07.002] Apply: cmd=[INCREMENT_AND_GET]</span>
<span class="hljs-attribute" style="box-sizing: border-box;">[14:16:14.264] Apply: cmd=[DECREMENT_AND_GET]</span>
3

<span class="hljs-header" style="box-sizing: border-box;">index (term): command
---------------------</span>
1 (29): CREATE(counter, 1)
2 (29): INCREMENT<span class="hljs-emphasis" style="box-sizing: border-box;">_AND_</span>GET(counter)
3 (29): DECREMENT<span class="hljs-emphasis" style="box-sizing: border-box;">_AND_</span>GET(counter)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li></ul>

B的控制台输出：

<code class="hljs sql has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-------------------------------------------------------------------</span>
GMS: address=B, cluster=ctr-cluster, physical address=2001:0:9d38:6abd:cbb:1f78:3f57:50f6:50101
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-------------------------------------------------------------------</span>

[0] <span class="hljs-operator" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">Create</span> [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>] Increment [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>] Decrement [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>] Dump log [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>] Snapshot [x] Exit
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">first</span>-applied=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">last</span>-applied=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">commit</span>-index=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>, log <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">size</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>b: 
[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">16</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01.300</span>] Apply: cmd=[<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">CREATE</span>]
<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>
counter: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>

[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>] <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">Create</span> [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>] Increment [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>] Decrement [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>] Dump log [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>] Snapshot [x] Exit
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">first</span>-applied=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">last</span>-applied=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">commit</span>-index=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, log <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">size</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>b: 
[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">16</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">07.299</span>] Apply: cmd=[INCREMENT_AND_GET]
[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">16</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14.304</span>] Apply: cmd=[DECREMENT_AND_GET]
roleChanged <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span>: Candidate
roleChanged <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span>: Leader</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li></ul>

C的控制台输出：

<code class="hljs sql has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-------------------------------------------------------------------</span>
GMS: address=C, cluster=ctr-cluster, physical address=2001:0:9d38:6abd:cbb:1f78:3f57:50f6:55800
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">-------------------------------------------------------------------</span>

[0] <span class="hljs-operator" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">Create</span> [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>] Increment [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>] Decrement [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>] Dump log [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>] Snapshot [x] Exit
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">first</span>-applied=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">last</span>-applied=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">commit</span>-index=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>, log <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">size</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>b: 
[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">16</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01.300</span>] Apply: cmd=[<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">CREATE</span>]
[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">16</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">07.299</span>] Apply: cmd=[INCREMENT_AND_GET]
<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
counter: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>

[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>] <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">Create</span> [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>] Increment [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>] Decrement [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>] Dump log [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>] Snapshot [x] Exit
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">first</span>-applied=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">last</span>-applied=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">commit</span>-index=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>, log <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">size</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>b: 
[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">16</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14.304</span>] Apply: cmd=[DECREMENT_AND_GET]</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul>

3.3.2 服务恢复

在只有B和C的集群中，我们执行了一次“加一”。当我们重新启动A服务时，它会自动执行这条日志，保持与B和C的一致。从日志的index能够看出，69是一个Term，也就是A为Leader时的“任期”，而70也就是B为Leader时。

A的控制台输出：

<code class="hljs asciidoc has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-code" style="box-sizing: border-box;">-------------------------------------------------------------------
GMS: address=A, cluster=ctr-cluster, physical address=2001:0:9d38:6abd:cbb:1f78:3f57:50f6:53237
-------------------------------------------------------------------</span>

[0] Create [1] Increment [2] Decrement [3] Dump log [4] Snapshot [x] Exit
first-applied=0, last-applied=3, commit-index=3, log size=3b: 
<span class="hljs-attribute" style="box-sizing: border-box;">[14:18:45.275] Apply: cmd=[INCREMENT_AND_GET]</span>
<span class="hljs-attribute" style="box-sizing: border-box;">[14:18:45.277] Apply: cmd=[GET]</span>
3

<span class="hljs-header" style="box-sizing: border-box;">index (term): command
---------------------</span>
1 (69): CREATE(counter, 1)
2 (69): INCREMENT<span class="hljs-emphasis" style="box-sizing: border-box;">_AND_</span>GET(counter)
3 (69): DECREMENT<span class="hljs-emphasis" style="box-sizing: border-box;">_AND_</span>GET(counter)
4 (70): INCREMENT<span class="hljs-emphasis" style="box-sizing: border-box;">_AND_</span>GET(counter)
5 (70): GET(counter)</code>

hellochenlu

关注

1
点赞
踩
7

收藏

觉得还不错? 一键收藏
0
评论
分布式一致性协议Raft原理与实例

分布式一致性协议Raft原理与实例1.Raft协议1.1 Raft简介Raft是由Stanford提出的一种更易理解的一致性算法，意在取代目前广为使用的Paxos算法。目前，在各种主流语言中都有了一些开源实现，比如本文中将使用的基于JGroups的Raft协议实现。关于Raft的原理，强烈推荐动画版Raft讲解。1.2 Raft原理在Raft中，每个结点
复制链接

扫一扫