来源:http://m635674608.iteye.com/blog/2283621
1.Raft协议
1.1 Raft简介
Raft是由Stanford提出的一种更易理解的一致性算法,意在取代目前广为使用的Paxos算法。目前,在各种主流语言中都有了一些开源实现,比如本文中将使用的基于JGroups的Raft协议实现。关于Raft的原理,强烈推荐动画版Raft讲解。
1.2 Raft原理
在Raft中,每个结点会处于下面三种状态中的一种:
- follower:所有结点都以follower的状态开始。如果没收到leader消息则会变成candidate状态
- candidate:会向其他结点“拉选票”,如果得到大部分的票则成为leader。这个过程就叫做Leader选举(Leader Election)
- leader:所有对系统的修改都会先经过leader。每个修改都会写一条日志(log entry)。leader收到修改请求后的过程如下,这个过程叫做日志复制(Log Replication):
- 复制日志到所有follower结点(replicate entry)
- 大部分结点响应时才提交日志
- 通知所有follower结点日志已提交
- 所有follower也提交日志
- 现在整个系统处于一致的状态
1.2.1 Leader Election
当follower在选举超时时间(election timeout)内未收到leader的心跳消息(append entries),则变成candidate状态。为了避免选举冲突,这个超时时间是一个150~300ms之间的随机数。
成为candidate的结点发起新的选举期(election term)去“拉选票”:
- 重置自己的计时器
- 投自己一票
- 发送 Request Vote消息
如果接收结点在新term内没有投过票那它就会投给此candidate,并重置它自己的选举超时时间。candidate拉到大部分选票就会成为leader,并定时发送心跳——Append Entries消息,去重置各个follower的计时器。当前Term会继续直到某个follower接收不到心跳并成为candidate。
如果不巧两个结点同时成为candidate都去“拉票”怎么办?这时会发生Splite Vote情况。两个结点可能都拉到了同样多的选票,难分胜负,选举失败,本term没有leader。之后又有计时器超时的follower会变成candidate,将term加一并开始新一轮的投票。
1.2.2 Log Replication
当发生改变时,leader会复制日志给follower结点,这也是通过Append Entries心跳消息完成的。前面已经列举了Log Replication的过程,这里就不重复了。
Raft能够正确地处理网络分区(“脑裂”)问题。假设A~E五个结点,B是leader。如果发生“脑裂”,A、B成为一个子分区,C、D、E成 为一个子分区。此时C、D、E会发生选举,选出C作为新term的leader。这样我们在两个子分区内就有了不同term的两个leader。这时如果 有客户端写A时,因为B无法复制日志到大部分follower所以日志处于uncommitted未提交状态。而同时另一个客户端对C的写操作却能够正确 完成,因为C是新的leader,它只知道D和E。
当网络通信恢复,B能够发送心跳给C、D、E了,却发现“改朝换代”了,因为C的term值更大,所以B自动降格为follower。然后A和B都回滚未提交的日志,并从新leader那里复制最新的日志。但这样是不是就会丢失更新?
2.JGroups-raft介绍
2.1 JGroups中的Raft
JGroups是Java里比较流行的网络通信框架,近期顺应潮流,它也推出了Raft基于JGroups的实现。简单试用了一下,还比较容易上 手,底层Raft的内部机制都被API屏蔽掉了。下面就通过一个分布式计数器的实例来学习一下Raft协议在JGroups中的实际用法。
Maven依赖如下:
- <code class="language-xml hljs has-numbering"> <span class="hljs-tag"><<span class="hljs-title">dependency</span>></span>
- <span class="hljs-tag"><<span class="hljs-title">groupId</span>></span>org.jgroups<span class="hljs-tag"></<span class="hljs-title">groupId</span>></span>
- <span class="hljs-tag"><<span class="hljs-title">artifactId</span>></span>jgroups-raft<span class="hljs-tag"></<span class="hljs-title">artifactId</span>></span>
- <span class="hljs-tag"><<span class="hljs-title">version</span>></span>0.2<span class="hljs-tag"></<span class="hljs-title">version</span>></span>
- <span class="hljs-tag"></<span class="hljs-title">dependency</span>></span></code>
- 1
- 2
- 3
- 4
- 5
其实JGroups-raft的Jar包中已经自带了一个Counter的Demo,但仔细看了一下,有的地方写的有些麻烦,不太容易把握住Raft这根主线。所以这里就参照官方的例子,进行了简写,突出Raft协议的基本使用方法。JGroups-raft目前资料不多,InfoQ上的这篇文章很不错,还有官方文档。
2.2 核心API
使用JGroups-raft时,我们一般会实现两个接口:RAFT.RoleChange和StateMachine:
- 实现RAFT.RoleChange接口的方法能通知我们当前哪个结点是leader
- 实现StateMachine执行要实现一致性的操作
典型单点服务实现方式就是:
- <code class="language-java hljs has-numbering">JChannel ch = <span class="hljs-keyword">null</span>;
- RaftHandle handle = <span class="hljs-keyword">new</span> RaftHandle(ch, <span class="hljs-keyword">this</span>);
- handle.addRoleListener(role -> {
- <span class="hljs-keyword">if</span>(role == Role.Leader)
- <span class="hljs-comment">// start singleton services</span>
- <span class="hljs-keyword">else</span>
- <span class="hljs-comment">// stop singleton services</span>
- });</code>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
2.3 默认配置
jgroups-raft.jar中已经带了一个raft.xml配置文件,作为实例程序我们可以直接使用它。
简要解释一下最核心的几个配置项,参照GitHub上的文档:
- UDP:IP多播配置
- raft.NO_DUPES:是否检测新加入结点的ID与老结点有重复
- raft.ELECTION:选举超时时间的随机化范围
- raft.RAFT:所有Raft集群的成员必须在这里声明,也可以在运行时通过addServer/removeServer动态修改
- raft.REDIRECT:是否转发请求给leader
- raft.CLIENT:在哪个IP和端口上接收客户端请求
- <code class="language-xml hljs has-numbering"><span class="hljs-comment"><!--
- Default stack using IP multicasting. It is similar to the "udp"
- stack in stacks.xml, but doesn't use streaming state transfer and flushing
- author: Bela Ban
- --></span>
- <span class="hljs-tag"><<span class="hljs-title">config</span> <span class="hljs-attribute">xmlns</span>=<span class="hljs-value">"urn:org:jgroups"</span>
- <span class="hljs-attribute">xmlns:xsi</span>=<span class="hljs-value">"http://www.w3.org/2001/XMLSchema-instance"</span>
- <span class="hljs-attribute">xsi:schemaLocation</span>=<span class="hljs-value">"urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd"</span>></span>
- <span class="hljs-tag"><<span class="hljs-title">UDP
- </span> <span class="hljs-attribute">mcast_addr</span>=<span class="hljs-value">"228.5.5.5"</span>
- <span class="hljs-attribute">mcast_port</span>=<span class="hljs-value">"${jgroups.udp.mcast_port:45588}"</span>
- <span class="hljs-attribute">...</span> /></span>
- ...
- <span class="hljs-tag"><<span class="hljs-title">raft.NO_DUPES</span>/></span>
- <span class="hljs-tag"><<span class="hljs-title">raft.ELECTION</span> <span class="hljs-attribute">election_min_interval</span>=<span class="hljs-value">"100"</span> <span class="hljs-attribute">election_max_interval</span>=<span class="hljs-value">"500"</span>/></span>
- <span class="hljs-tag"><<span class="hljs-title">raft.RAFT</span> <span class="hljs-attribute">members</span>=<span class="hljs-value">"A,B,C"</span> <span class="hljs-attribute">raft_id</span>=<span class="hljs-value">"${raft_id:undefined}"</span>/></span>
- <span class="hljs-tag"><<span class="hljs-title">raft.REDIRECT</span>/></span>
- <span class="hljs-tag"><<span class="hljs-title">raft.CLIENT</span> <span class="hljs-attribute">bind_addr</span>=<span class="hljs-value">"0.0.0.0"</span> /></span>
- <span class="hljs-tag"></<span class="hljs-title">config</span>></span></code>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
3.JGroups-raft实例
实例很简单,只有JGroupsRaftTest和CounterService两个类组成。JGroupsRaftTest是测试启动类,而CounterService就是利用Raft协议实现的分布式计数服务类。
3.1 JGroupsRaftTest
JGroupsRaftTest的职责主要有三个:
- 创建Raft协议的JChannel
- 创建CounterService
- 循环读取用户输入
目前简单实现了几种操作包括:初始化计数器、加一、减一、读取计数器、查看Raft日志、做Raft快照(用于压缩日志文件)等。其中对计数器的操作,因为要与其他Raft成员进行分布式通信,所以当前集群必须要多于一个结点时才能进行操作。如果要支持单结点时的操作,需要做特殊处理。
- <code class="language-java hljs has-numbering"><span class="hljs-keyword">import</span> org.jgroups.JChannel;
- <span class="hljs-keyword">import</span> org.jgroups.protocols.raft.RAFT;
- <span class="hljs-keyword">import</span> org.jgroups.util.Util;
- <span class="hljs-javadoc">/**
- * Test jgroups raft algorithm implementation.
- */</span>
- <span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">JGroupsRaftTest</span> {</span>
- <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">final</span> String CLUSTER_NAME = <span class="hljs-string">"ctr-cluster"</span>;
- <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">final</span> String COUNTER_NAME = <span class="hljs-string">"counter"</span>;
- <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">final</span> String RAFT_XML = <span class="hljs-string">"raft.xml"</span>;
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> <span class="hljs-title">main</span>(String[] args) <span class="hljs-keyword">throws</span> Exception {
- JChannel ch = <span class="hljs-keyword">new</span> JChannel(RAFT_XML).name(args[<span class="hljs-number">0</span>]);
- CounterService counter = <span class="hljs-keyword">new</span> CounterService(ch);
- <span class="hljs-keyword">try</span> {
- doConnect(ch, CLUSTER_NAME);
- doLoop(ch, counter);
- } <span class="hljs-keyword">finally</span> {
- Util.close(ch);
- }
- }
- <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> <span class="hljs-title">doConnect</span>(JChannel ch, String clusterName) <span class="hljs-keyword">throws</span> Exception {
- ch.connect(clusterName);
- }
- <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> <span class="hljs-title">doLoop</span>(JChannel ch, CounterService counter) {
- <span class="hljs-keyword">boolean</span> looping = <span class="hljs-keyword">true</span>;
- <span class="hljs-keyword">while</span> (looping) {
- <span class="hljs-keyword">int</span> key = Util.keyPress(<span class="hljs-string">"\n[0] Create [1] Increment [2] Decrement [3] Dump log [4] Snapshot [x] Exit\n"</span> +
- <span class="hljs-string">"first-applied="</span> + ((RAFT) ch.getProtocolStack().findProtocol(RAFT.class)).log().firstApplied() +
- <span class="hljs-string">", last-applied="</span> + counter.lastApplied() +
- <span class="hljs-string">", commit-index="</span> + counter.commitIndex() +
- <span class="hljs-string">", log size="</span> + Util.printBytes(counter.logSize()) + <span class="hljs-string">": "</span>);
- <span class="hljs-keyword">if</span> ((key == <span class="hljs-string">'0'</span> || key == <span class="hljs-string">'1'</span> || key == <span class="hljs-string">'2'</span>) && !counter.isLeaderExist()) {
- System.out.println(<span class="hljs-string">"Cannot perform cause there is no leader by now"</span>);
- <span class="hljs-keyword">continue</span>;
- }
- <span class="hljs-keyword">long</span> val;
- <span class="hljs-keyword">switch</span> (key) {
- <span class="hljs-keyword">case</span> <span class="hljs-string">'0'</span>:
- counter.getOrCreateCounter(COUNTER_NAME, <span class="hljs-number">1</span>L);
- <span class="hljs-keyword">break</span>;
- <span class="hljs-keyword">case</span> <span class="hljs-string">'1'</span>:
- val = counter.incrementAndGet(COUNTER_NAME);
- System.out.printf(<span class="hljs-string">"%s: %s\n"</span>, COUNTER_NAME, val);
- <span class="hljs-keyword">break</span>;
- <span class="hljs-keyword">case</span> <span class="hljs-string">'2'</span>:
- val = counter.decrementAndGet(COUNTER_NAME);
- System.out.printf(<span class="hljs-string">"%s: %s\n"</span>, COUNTER_NAME, val);
- <span class="hljs-keyword">break</span>;
- <span class="hljs-keyword">case</span> <span class="hljs-string">'3'</span>:
- counter.dumpLog();
- <span class="hljs-keyword">break</span>;
- <span class="hljs-keyword">case</span> <span class="hljs-string">'4'</span>:
- counter.snapshot();
- <span class="hljs-keyword">break</span>;
- <span class="hljs-keyword">case</span> <span class="hljs-string">'x'</span>:
- looping = <span class="hljs-keyword">false</span>;
- <span class="hljs-keyword">break</span>;
- <span class="hljs-keyword">case</span> <span class="hljs-string">'\n'</span>:
- System.out.println(COUNTER_NAME + <span class="hljs-string">": "</span> + counter.get(COUNTER_NAME) + <span class="hljs-string">"\n"</span>);
- <span class="hljs-keyword">break</span>;
- }
- }
- }
- }</code>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
3.2 CounterService
CounterService是我们的核心类,利用Raft实现了分布式的计数器操作,它的API主要由四部分组成:
- Raft Local API:操作本地Raft的状态,像日志大小、做快照等
- Raft API:实现Raft的监听器和状态机的方法
- roleChanged:本地Raft的角色发生变化
- apply:分布式通信消息
- readContentFrom/writeContentTo:读写快照
- Counter API:计数器的分布式API
- Counter Native API:计数器的本地API。直接使用的话相当于脏读
- <code class="language-java hljs has-numbering"><span class="hljs-keyword">import</span> org.jgroups.Channel;
- <span class="hljs-keyword">import</span> org.jgroups.protocols.raft.RAFT;
- <span class="hljs-keyword">import</span> org.jgroups.protocols.raft.Role;
- <span class="hljs-keyword">import</span> org.jgroups.protocols.raft.StateMachine;
- <span class="hljs-keyword">import</span> org.jgroups.raft.RaftHandle;
- <span class="hljs-keyword">import</span> org.jgroups.util.AsciiString;
- <span class="hljs-keyword">import</span> org.jgroups.util.Bits;
- <span class="hljs-keyword">import</span> org.jgroups.util.ByteArrayDataInputStream;
- <span class="hljs-keyword">import</span> org.jgroups.util.ByteArrayDataOutputStream;
- <span class="hljs-keyword">import</span> org.jgroups.util.Util;
- <span class="hljs-keyword">import</span> java.io.DataInput;
- <span class="hljs-keyword">import</span> java.io.DataOutput;
- <span class="hljs-keyword">import</span> java.io.IOException;
- <span class="hljs-keyword">import</span> java.text.SimpleDateFormat;
- <span class="hljs-keyword">import</span> java.util.Date;
- <span class="hljs-keyword">import</span> java.util.HashMap;
- <span class="hljs-keyword">import</span> java.util.Map;
- <span class="hljs-javadoc">/**
- * Distribute counter service based on Raft consensus algorithm.
- */</span>
- class CounterService implements StateMachine, RAFT.RoleChange {
- <span class="hljs-keyword">private</span> RaftHandle raft;
- <span class="hljs-keyword">private</span> <span class="hljs-keyword">final</span> Map<String, Long> counters;
- <span class="hljs-keyword">private</span> <span class="hljs-keyword">enum</span> Command {
- CREATE, INCREMENT_AND_GET, DECREMENT_AND_GET, GET, SET
- }
- <span class="hljs-keyword">public</span> <span class="hljs-title">CounterService</span>(Channel ch) {
- <span class="hljs-keyword">this</span>.raft = <span class="hljs-keyword">new</span> RaftHandle(ch, <span class="hljs-keyword">this</span>);
- <span class="hljs-keyword">this</span>.counters = <span class="hljs-keyword">new</span> HashMap<>();
- raft.raftId(ch.getName())
- .addRoleListener(<span class="hljs-keyword">this</span>);
- }
- <span class="hljs-comment">// ===========================================</span>
- <span class="hljs-comment">// Raft Status API</span>
- <span class="hljs-comment">// ===========================================</span>
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">int</span> <span class="hljs-title">lastApplied</span>() {
- <span class="hljs-keyword">return</span> raft.lastApplied();
- }
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">int</span> <span class="hljs-title">commitIndex</span>() {
- <span class="hljs-keyword">return</span> raft.commitIndex();
- }
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">int</span> <span class="hljs-title">logSize</span>() {
- <span class="hljs-keyword">return</span> raft.logSize();
- }
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">dumpLog</span>() {
- System.out.println(<span class="hljs-string">"\nindex (term): command\n---------------------"</span>);
- raft.logEntries((entry, index) -> {
- StringBuilder log = <span class="hljs-keyword">new</span> StringBuilder()
- .append(index)
- .append(<span class="hljs-string">" ("</span>).append(entry.term()).append(<span class="hljs-string">"): "</span>);
- <span class="hljs-keyword">if</span> (entry.command() == <span class="hljs-keyword">null</span> ) {
- System.out.println(log.append(<span class="hljs-string">"<marker record>"</span>));
- <span class="hljs-keyword">return</span>;
- } <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (entry.internal()) {
- System.out.println(log.append(<span class="hljs-string">"<internal command>"</span>));
- <span class="hljs-keyword">return</span>;
- }
- ByteArrayDataInputStream in = <span class="hljs-keyword">new</span> ByteArrayDataInputStream(
- entry.command(), entry.offset(), entry.length()
- );
- <span class="hljs-keyword">try</span> {
- Command cmd = Command.values()[in.readByte()];
- String name = Bits.readAsciiString(in).toString();
- <span class="hljs-keyword">switch</span> (cmd) {
- <span class="hljs-keyword">case</span> CREATE:
- log.append(cmd)
- .append(<span class="hljs-string">"("</span>).append(name).append(<span class="hljs-string">", "</span>)
- .append(Bits.readLong(in))
- .append(<span class="hljs-string">")"</span>);
- <span class="hljs-keyword">break</span>;
- <span class="hljs-keyword">case</span> GET:
- <span class="hljs-keyword">case</span> INCREMENT_AND_GET:
- <span class="hljs-keyword">case</span> DECREMENT_AND_GET:
- log.append(cmd)
- .append(<span class="hljs-string">"("</span>).append(name).append(<span class="hljs-string">")"</span>);
- <span class="hljs-keyword">break</span>;
- <span class="hljs-keyword">default</span>:
- <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> IllegalArgumentException(<span class="hljs-string">"Command "</span> + cmd + <span class="hljs-string">"is unknown"</span>);
- }
- System.out.println(log);
- }
- <span class="hljs-keyword">catch</span> (IOException e) {
- <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> IllegalStateException(<span class="hljs-string">"Error when dump log"</span>, e);
- }
- });
- System.out.println();
- }
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">snapshot</span>() {
- <span class="hljs-keyword">try</span> {
- raft.snapshot();
- } <span class="hljs-keyword">catch</span> (Exception e) {
- <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> IllegalStateException(<span class="hljs-string">"Error when snapshot"</span>, e);
- }
- }
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">boolean</span> <span class="hljs-title">isLeaderExist</span>() {
- <span class="hljs-keyword">return</span> raft.leader() != <span class="hljs-keyword">null</span>;
- }
- <span class="hljs-comment">// ===========================================</span>
- <span class="hljs-comment">// Raft API</span>
- <span class="hljs-comment">// ===========================================</span>
- <span class="hljs-annotation">@Override</span>
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">roleChanged</span>(Role role) {
- System.out.println(<span class="hljs-string">"roleChanged to: "</span> + role);
- }
- <span class="hljs-annotation">@Override</span>
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">byte</span>[] <span class="hljs-title">apply</span>(<span class="hljs-keyword">byte</span>[] data, <span class="hljs-keyword">int</span> offset, <span class="hljs-keyword">int</span> length) <span class="hljs-keyword">throws</span> Exception {
- ByteArrayDataInputStream in = <span class="hljs-keyword">new</span> ByteArrayDataInputStream(data, offset, length);
- Command cmd = Command.values()[in.readByte()];
- String name = Bits.readAsciiString(in).toString();
- System.out.println(<span class="hljs-string">"["</span> + <span class="hljs-keyword">new</span> SimpleDateFormat(<span class="hljs-string">"HH:mm:ss.SSS"</span>).format(<span class="hljs-keyword">new</span> Date())
- + <span class="hljs-string">"] Apply: cmd=["</span> + cmd + <span class="hljs-string">"]"</span>);
- <span class="hljs-keyword">long</span> v1, retVal;
- <span class="hljs-keyword">switch</span> (cmd) {
- <span class="hljs-keyword">case</span> CREATE:
- v1 = Bits.readLong(in);
- retVal = create0(name, v1);
- <span class="hljs-keyword">return</span> Util.objectToByteBuffer(retVal);
- <span class="hljs-keyword">case</span> GET:
- retVal = get0(name);
- <span class="hljs-keyword">return</span> Util.objectToByteBuffer(retVal);
- <span class="hljs-keyword">case</span> INCREMENT_AND_GET:
- retVal = add0(name, <span class="hljs-number">1</span>L);
- <span class="hljs-keyword">return</span> Util.objectToByteBuffer(retVal);
- <span class="hljs-keyword">case</span> DECREMENT_AND_GET:
- retVal = add0(name, -<span class="hljs-number">1</span>L);
- <span class="hljs-keyword">return</span> Util.objectToByteBuffer(retVal);
- <span class="hljs-keyword">default</span>:
- <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> IllegalArgumentException(<span class="hljs-string">"Command "</span> + cmd + <span class="hljs-string">"is unknown"</span>);
- }
- }
- <span class="hljs-annotation">@Override</span>
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">readContentFrom</span>(DataInput in) <span class="hljs-keyword">throws</span> Exception {
- <span class="hljs-keyword">int</span> size = in.readInt();
- System.out.println(<span class="hljs-string">"ReadContentFrom: size=["</span> + size + <span class="hljs-string">"]"</span>);
- <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i < size; i++) {
- AsciiString name = Bits.readAsciiString(in);
- Long value = Bits.readLong(in);
- counters.put(name.toString(), value);
- }
- }
- <span class="hljs-annotation">@Override</span>
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">writeContentTo</span>(DataOutput out) <span class="hljs-keyword">throws</span> Exception {
- <span class="hljs-keyword">synchronized</span> (counters) {
- <span class="hljs-keyword">int</span> size = counters.size();
- System.out.println(<span class="hljs-string">"WriteContentFrom: size=["</span> + size + <span class="hljs-string">"]"</span>);
- out.writeInt(size);
- <span class="hljs-keyword">for</span> (Map.Entry<String, Long> entry : counters.entrySet()) {
- AsciiString name = <span class="hljs-keyword">new</span> AsciiString(entry.getKey());
- Long value = entry.getValue();
- Bits.writeAsciiString(name, out);
- Bits.writeLong(value, out);
- }
- }
- }
- <span class="hljs-comment">// ===========================================</span>
- <span class="hljs-comment">// Counter API</span>
- <span class="hljs-comment">// ===========================================</span>
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">getOrCreateCounter</span>(String name, <span class="hljs-keyword">long</span> initVal) {
- Object retVal = invoke(Command.CREATE, name, <span class="hljs-keyword">false</span>, initVal);
- counters.put(name, (Long) retVal);
- }
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">long</span> <span class="hljs-title">incrementAndGet</span>(String name) {
- <span class="hljs-keyword">return</span> (<span class="hljs-keyword">long</span>) invoke(Command.INCREMENT_AND_GET, name, <span class="hljs-keyword">false</span>);
- }
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">long</span> <span class="hljs-title">decrementAndGet</span>(String name) {
- <span class="hljs-keyword">return</span> (<span class="hljs-keyword">long</span>) invoke(Command.DECREMENT_AND_GET, name, <span class="hljs-keyword">false</span>);
- }
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">long</span> <span class="hljs-title">get</span>(String name) {
- <span class="hljs-keyword">return</span> (<span class="hljs-keyword">long</span>) invoke(Command.GET, name, <span class="hljs-keyword">false</span>);
- }
- <span class="hljs-keyword">private</span> Object <span class="hljs-title">invoke</span>(Command cmd, String name, <span class="hljs-keyword">boolean</span> ignoreRetVal, <span class="hljs-keyword">long</span>... values) {
- ByteArrayDataOutputStream out = <span class="hljs-keyword">new</span> ByteArrayDataOutputStream(<span class="hljs-number">256</span>);
- <span class="hljs-keyword">try</span> {
- out.writeByte(cmd.ordinal());
- Bits.writeAsciiString(<span class="hljs-keyword">new</span> AsciiString(name), out);
- <span class="hljs-keyword">for</span> (<span class="hljs-keyword">long</span> val : values) {
- Bits.writeLong(val, out);
- }
- <span class="hljs-keyword">byte</span>[] rsp = raft.set(out.buffer(), <span class="hljs-number">0</span>, out.position());
- <span class="hljs-keyword">return</span> ignoreRetVal ? <span class="hljs-keyword">null</span> : Util.objectFromByteBuffer(rsp);
- }
- <span class="hljs-keyword">catch</span> (IOException ex) {
- <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> RuntimeException(<span class="hljs-string">"Serialization failure (cmd="</span>
- + cmd + <span class="hljs-string">", name="</span> + name + <span class="hljs-string">")"</span>, ex);
- }
- <span class="hljs-keyword">catch</span> (Exception ex) {
- <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> RuntimeException(<span class="hljs-string">"Raft set failure (cmd="</span>
- + cmd + <span class="hljs-string">", name="</span> + name + <span class="hljs-string">")"</span>, ex);
- }
- }
- <span class="hljs-comment">// ===========================================</span>
- <span class="hljs-comment">// Counter Native API</span>
- <span class="hljs-comment">// ===========================================</span>
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">synchronized</span> Long <span class="hljs-title">create0</span>(String name, <span class="hljs-keyword">long</span> initVal) {
- counters.putIfAbsent(name, initVal);
- <span class="hljs-keyword">return</span> counters.get(name);
- }
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">synchronized</span> Long <span class="hljs-title">get0</span>(String name) {
- <span class="hljs-keyword">return</span> counters.getOrDefault(name, <span class="hljs-number">0</span>L);
- }
- <span class="hljs-keyword">public</span> <span class="hljs-keyword">synchronized</span> Long <span class="hljs-title">add0</span>(String name, <span class="hljs-keyword">long</span> delta) {
- Long oldVal = counters.getOrDefault(name, <span class="hljs-number">0</span>L);
- <span class="hljs-keyword">return</span> counters.put(name, oldVal + delta);
- }
- }</code>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
- 116
- 117
- 118
- 119
- 120
- 121
- 122
- 123
- 124
- 125
- 126
- 127
- 128
- 129
- 130
- 131
- 132
- 133
- 134
- 135
- 136
- 137
- 138
- 139
- 140
- 141
- 142
- 143
- 144
- 145
- 146
- 147
- 148
- 149
- 150
- 151
- 152
- 153
- 154
- 155
- 156
- 157
- 158
- 159
- 160
- 161
- 162
- 163
- 164
- 165
- 166
- 167
- 168
- 169
- 170
- 171
- 172
- 173
- 174
- 175
- 176
- 177
- 178
- 179
- 180
- 181
- 182
- 183
- 184
- 185
- 186
- 187
- 188
- 189
- 190
- 191
- 192
- 193
- 194
- 195
- 196
- 197
- 198
- 199
- 200
- 201
- 202
- 203
- 204
- 205
- 206
- 207
- 208
- 209
- 210
- 211
- 212
- 213
- 214
- 215
- 216
- 217
- 218
- 219
- 220
- 221
- 222
- 223
- 224
- 225
- 226
- 227
- 228
- 229
- 230
- 231
- 232
- 233
- 234
- 235
- 236
- 237
- 238
3.3 运行测试
我们分别以A、B、C为参数,启动三个JGroupsRaftTest服务。这样会自动在C:\Users\cdai\AppData\Local\Temp下生成A.log、B.log、C.log三个日志文件夹。
- <code class="hljs livecodeserver has-numbering">cdai@vm /cygdrive/c/Users/cdai/AppData/Local/Temp
- $ tree A.<span class="hljs-built_in">log</span>/ B.<span class="hljs-built_in">log</span>/ C.<span class="hljs-built_in">log</span>/
- A.<span class="hljs-built_in">log</span>/
- |<span class="hljs-comment">-- 000005.sst</span>
- |<span class="hljs-comment">-- 000006.log</span>
- |<span class="hljs-comment">-- CURRENT</span>
- |<span class="hljs-comment">-- LOCK</span>
- |<span class="hljs-comment">-- LOG</span>
- |<span class="hljs-comment">-- LOG.old</span>
- `<span class="hljs-comment">-- MANIFEST-000004</span>
- B.<span class="hljs-built_in">log</span>/
- |<span class="hljs-comment">-- 000003.log</span>
- |<span class="hljs-comment">-- CURRENT</span>
- |<span class="hljs-comment">-- LOCK</span>
- |<span class="hljs-comment">-- LOG</span>
- `<span class="hljs-comment">-- MANIFEST-000002</span>
- C.<span class="hljs-built_in">log</span>/
- |<span class="hljs-comment">-- 000003.log</span>
- |<span class="hljs-comment">-- CURRENT</span>
- |<span class="hljs-comment">-- LOCK</span>
- |<span class="hljs-comment">-- LOG</span>
- `<span class="hljs-comment">-- MANIFEST-000002</span>
- <span class="hljs-number">0</span> <span class="hljs-built_in">directories</span>, <span class="hljs-number">17</span> <span class="hljs-built_in">files</span></code>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
3.3.1 分布式一致性
首先A创建计数器,B“加一”,C“减一”。可以看到尽管我们是分别在A、B、C上执行这三个操作,但三个结点都先后(leader提交日志后通知follower)通过apply()方法收到消息,并在本地的计数器Map上同步执行操作,保证了数据的一致性。最后停掉A服务,可以看到B通过roleChanged()得到消息,提升为新的Leader,并与C一同继续提供服务。
A的控制台输出:
- <code class="hljs asciidoc has-numbering"><span class="hljs-code">-------------------------------------------------------------------
- GMS: address=A, cluster=ctr-cluster, physical address=2001:0:9d38:6abd:cbb:1f78:3f57:50f6:50100
- -------------------------------------------------------------------</span>
- [0] Create [1] Increment [2] Decrement [3] Dump log [4] Snapshot [x] Exit
- first-applied=0, last-applied=0, commit-index=0, log size=0b:
- roleChanged to: Candidate
- roleChanged to: Leader
- 0
- <span class="hljs-attribute">[14:16:00.744] Apply: cmd=[CREATE]</span>
- [0] Create [1] Increment [2] Decrement [3] Dump log [4] Snapshot [x] Exit
- first-applied=0, last-applied=1, commit-index=1, log size=1b:
- <span class="hljs-attribute">[14:16:07.002] Apply: cmd=[INCREMENT_AND_GET]</span>
- <span class="hljs-attribute">[14:16:14.264] Apply: cmd=[DECREMENT_AND_GET]</span>
- 3
- <span class="hljs-header">index (term): command
- ---------------------</span>
- 1 (29): CREATE(counter, 1)
- 2 (29): INCREMENT<span class="hljs-emphasis">_AND_</span>GET(counter)
- 3 (29): DECREMENT<span class="hljs-emphasis">_AND_</span>GET(counter)</code>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
B的控制台输出:
- <code class="hljs sql has-numbering"><span class="hljs-comment">-------------------------------------------------------------------</span>
- GMS: address=B, cluster=ctr-cluster, physical address=2001:0:9d38:6abd:cbb:1f78:3f57:50f6:50101
- <span class="hljs-comment">-------------------------------------------------------------------</span>
- [0] <span class="hljs-operator"><span class="hljs-keyword">Create</span> [<span class="hljs-number">1</span>] Increment [<span class="hljs-number">2</span>] Decrement [<span class="hljs-number">3</span>] Dump log [<span class="hljs-number">4</span>] Snapshot [x] Exit
- <span class="hljs-keyword">first</span>-applied=<span class="hljs-number">0</span>, <span class="hljs-keyword">last</span>-applied=<span class="hljs-number">0</span>, <span class="hljs-keyword">commit</span>-index=<span class="hljs-number">0</span>, log <span class="hljs-keyword">size</span>=<span class="hljs-number">0</span>b:
- [<span class="hljs-number">14</span>:<span class="hljs-number">16</span>:<span class="hljs-number">01.300</span>] Apply: cmd=[<span class="hljs-keyword">CREATE</span>]
- <span class="hljs-number">1</span>
- counter: <span class="hljs-number">2</span>
- [<span class="hljs-number">0</span>] <span class="hljs-keyword">Create</span> [<span class="hljs-number">1</span>] Increment [<span class="hljs-number">2</span>] Decrement [<span class="hljs-number">3</span>] Dump log [<span class="hljs-number">4</span>] Snapshot [x] Exit
- <span class="hljs-keyword">first</span>-applied=<span class="hljs-number">0</span>, <span class="hljs-keyword">last</span>-applied=<span class="hljs-number">2</span>, <span class="hljs-keyword">commit</span>-index=<span class="hljs-number">1</span>, log <span class="hljs-keyword">size</span>=<span class="hljs-number">2</span>b:
- [<span class="hljs-number">14</span>:<span class="hljs-number">16</span>:<span class="hljs-number">07.299</span>] Apply: cmd=[INCREMENT_AND_GET]
- [<span class="hljs-number">14</span>:<span class="hljs-number">16</span>:<span class="hljs-number">14.304</span>] Apply: cmd=[DECREMENT_AND_GET]
- roleChanged <span class="hljs-keyword">to</span>: Candidate
- roleChanged <span class="hljs-keyword">to</span>: Leader</span></code>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
C的控制台输出:
- <code class="hljs sql has-numbering"><span class="hljs-comment">-------------------------------------------------------------------</span>
- GMS: address=C, cluster=ctr-cluster, physical address=2001:0:9d38:6abd:cbb:1f78:3f57:50f6:55800
- <span class="hljs-comment">-------------------------------------------------------------------</span>
- [0] <span class="hljs-operator"><span class="hljs-keyword">Create</span> [<span class="hljs-number">1</span>] Increment [<span class="hljs-number">2</span>] Decrement [<span class="hljs-number">3</span>] Dump log [<span class="hljs-number">4</span>] Snapshot [x] Exit
- <span class="hljs-keyword">first</span>-applied=<span class="hljs-number">0</span>, <span class="hljs-keyword">last</span>-applied=<span class="hljs-number">0</span>, <span class="hljs-keyword">commit</span>-index=<span class="hljs-number">0</span>, log <span class="hljs-keyword">size</span>=<span class="hljs-number">0</span>b:
- [<span class="hljs-number">14</span>:<span class="hljs-number">16</span>:<span class="hljs-number">01.300</span>] Apply: cmd=[<span class="hljs-keyword">CREATE</span>]
- [<span class="hljs-number">14</span>:<span class="hljs-number">16</span>:<span class="hljs-number">07.299</span>] Apply: cmd=[INCREMENT_AND_GET]
- <span class="hljs-number">2</span>
- counter: <span class="hljs-number">3</span>
- [<span class="hljs-number">0</span>] <span class="hljs-keyword">Create</span> [<span class="hljs-number">1</span>] Increment [<span class="hljs-number">2</span>] Decrement [<span class="hljs-number">3</span>] Dump log [<span class="hljs-number">4</span>] Snapshot [x] Exit
- <span class="hljs-keyword">first</span>-applied=<span class="hljs-number">0</span>, <span class="hljs-keyword">last</span>-applied=<span class="hljs-number">3</span>, <span class="hljs-keyword">commit</span>-index=<span class="hljs-number">2</span>, log <span class="hljs-keyword">size</span>=<span class="hljs-number">3</span>b:
- [<span class="hljs-number">14</span>:<span class="hljs-number">16</span>:<span class="hljs-number">14.304</span>] Apply: cmd=[DECREMENT_AND_GET]</span></code>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
3.3.2 服务恢复
在只有B和C的集群中,我们执行了一次“加一”。当我们重新启动A服务时,它会自动执行这条日志,保持与B和C的一致。从日志的index能够看出,69是一个Term,也就是A为Leader时的“任期”,而70也就是B为Leader时。
A的控制台输出:
- <code class="hljs asciidoc has-numbering"><span class="hljs-code">-------------------------------------------------------------------
- GMS: address=A, cluster=ctr-cluster, physical address=2001:0:9d38:6abd:cbb:1f78:3f57:50f6:53237
- -------------------------------------------------------------------</span>
- [0] Create [1] Increment [2] Decrement [3] Dump log [4] Snapshot [x] Exit
- first-applied=0, last-applied=3, commit-index=3, log size=3b:
- <span class="hljs-attribute">[14:18:45.275] Apply: cmd=[INCREMENT_AND_GET]</span>
- <span class="hljs-attribute">[14:18:45.277] Apply: cmd=[GET]</span>
- 3
- <span class="hljs-header">index (term): command
- ---------------------</span>
- 1 (69): CREATE(counter, 1)
- 2 (69): INCREMENT<span class="hljs-emphasis">_AND_</span>GET(counter)
- 3 (69): DECREMENT<span class="hljs-emphasis">_AND_</span>GET(counter)
- 4 (70): INCREMENT<span class="hljs-emphasis">_AND_</span>GET(counter)
- 5 (70): GET(counter)</code>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17