版权声明:本文为博主原创文章,未经博主允许不得转载。
Apache Curator入门实战
Curator是Netflix公司开源的一个Zookeeper客户端,与Zookeeper提供的原生客户端相比,Curator的抽象层次更高,简化了Zookeeper客户端的开发量。
1.Zookeeper安装部署
Zookeeper的部署很简单,如果已经有Java运行环境的话,下载tarball解压后即可运行。
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@vm</span> <span class="hljs-constant" style="box-sizing: border-box;">Temp</span>]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>wget <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">http:</span>/<span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/mirror.bit.edu.cn/apache</span><span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/zookeeper/zookeeper</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>/zookeeper-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>.tar.gz [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@vm</span> <span class="hljs-constant" style="box-sizing: border-box;">Temp</span>]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>tar zxvf zookeeper-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>.tar.gz [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@vm</span> <span class="hljs-constant" style="box-sizing: border-box;">Temp</span>]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>cd zookeeper-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span> [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@vm</span> zookeeper-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>cp conf/zoo_sample.cfg conf/zoo.cfg [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@vm</span> zookeeper-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>export <span class="hljs-constant" style="box-sizing: border-box;">ZOOKEEPER_HOME</span>=<span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/usr/local</span><span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/src/zookeeper</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span> [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@vm</span> zookeeper-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>export <span class="hljs-constant" style="box-sizing: border-box;">PATH</span>=<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ZOOKEEPER_HOME</span>/<span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">bin:</span><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$PATH</span> [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@vm</span> zookeeper-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>bin/zkServer.sh start [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@vm</span> zookeeper-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>bin/zkCli.sh -server <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">127.0</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li></ul>
2.客户端常用操作
用zkCli.sh连接上Zookeeper服务后,用help能列出所有命令:
<code class="hljs r has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[root@BC-VM-edce4ac67d304079868c0bb265337bd4 zookeeper-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.6</span>]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># bin/zkCli.sh -127.0.0.1:2181</span> Connecting to localhost:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">06</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">55</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">387</span> [myid:] - INFO [main:Environment@<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span>] - Client environment:zookeeper.version=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.6</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1569965</span>, built on <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">02</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2014</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span> GMT <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">...</span> [zk: localhost:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span>(CONNECTED) <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>] help ZooKeeper -server host:port cmd args connect host:port get path [watch] ls path [watch] set path data [version] rmr path delquota [-n|-b] path quit printwatches on|off create [-s] [-e] path data acl stat path [watch] close ls2 path [watch] history listquota path setAcl path acl getAcl path sync path redo cmdno addauth scheme auth delete path [version] setquota -n|-b val path</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li></ul>
下面就试验一下常用的命令:
- create:创建路径结点。
- ls:查看路径下的所有结点。
- get:获得结点上的值。
- set:修改结点上的值。
- delete:删除结点。
<code class="hljs makefile has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[zk: localhost:2181(CONNECTED) 6] create /zktest mydata Created /zktest [zk: localhost:2181(CONNECTED) 12] ls / [zktest, zookeeper] [zk: localhost:2181(CONNECTED) 7] ls /zktest [] [zk: localhost:2181(CONNECTED) 13] get /zktest mydata <span class="hljs-constant" style="box-sizing: border-box;">cZxid</span> = 0x1c <span class="hljs-constant" style="box-sizing: border-box;">ctime</span> = Thu Jun 11 10:58:06 CST 2015 <span class="hljs-constant" style="box-sizing: border-box;">mZxid</span> = 0x1c <span class="hljs-constant" style="box-sizing: border-box;">mtime</span> = Thu Jun 11 10:58:06 CST 2015 <span class="hljs-constant" style="box-sizing: border-box;">pZxid</span> = 0x1c <span class="hljs-constant" style="box-sizing: border-box;">cversion</span> = 0 <span class="hljs-constant" style="box-sizing: border-box;">dataVersion</span> = 0 <span class="hljs-constant" style="box-sizing: border-box;">aclVersion</span> = 0 <span class="hljs-constant" style="box-sizing: border-box;">ephemeralOwner</span> = 0x0 <span class="hljs-constant" style="box-sizing: border-box;">dataLength</span> = 6 <span class="hljs-constant" style="box-sizing: border-box;">numChildren</span> = 0 [zk: localhost:2181(CONNECTED) 14] set /zktest junk <span class="hljs-constant" style="box-sizing: border-box;">cZxid</span> = 0x1c <span class="hljs-constant" style="box-sizing: border-box;">ctime</span> = Thu Jun 11 10:58:06 CST 2015 <span class="hljs-constant" style="box-sizing: border-box;">mZxid</span> = 0x1f <span class="hljs-constant" style="box-sizing: border-box;">mtime</span> = Thu Jun 11 10:59:08 CST 2015 <span class="hljs-constant" style="box-sizing: border-box;">pZxid</span> = 0x1c <span class="hljs-constant" style="box-sizing: border-box;">cversion</span> = 0 <span class="hljs-constant" style="box-sizing: border-box;">dataVersion</span> = 1 <span class="hljs-constant" style="box-sizing: border-box;">aclVersion</span> = 0 <span class="hljs-constant" style="box-sizing: border-box;">ephemeralOwner</span> = 0x0 <span class="hljs-constant" style="box-sizing: border-box;">dataLength</span> = 4 <span class="hljs-constant" style="box-sizing: border-box;">numChildren</span> = 0 [zk: localhost:2181(CONNECTED) 15] delete /zktest [zk: localhost:2181(CONNECTED) 16] ls / [zookeeper]</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li></ul>
3.用Curator管理Zookeeper
Curator的Maven依赖如下,一般直接使用curator-recipes就行了,如果需要自己封装一些底层些的功能的话,例如增加连接管理重试机制等,则可以引入curator-framework包。
<code class="language-xml hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>org.apache.curator<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>curator-recipes<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>2.7.0<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>
3.1 Client操作
利用Curator提供的客户端API,可以完全实现上面原生客户端的功能。值得注意的是,Curator采用流式风格API。
<code class="language-java hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.cdai.codebase.bigdata.hadoop.zookeeper.curator; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.CuratorFramework; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.CuratorFrameworkFactory; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.retry.RetryNTimes; <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/** * Curator framework's client test. * Output: * $ create /zktest hello * $ ls / * [zktest, zookeeper] * $ get /zktest * hello * $ set /zktest world * $ get /zktest * world * $ delete /zktest * $ ls / * [zookeeper] */</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">CuratorClientTest</span> {</span> <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/** Zookeeper info */</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> String ZK_ADDRESS = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"192.168.1.100:2181"</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> String ZK_PATH = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/zktest"</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">main</span>(String[] args) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> Exception { <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 1.Connect to zk</span> CuratorFramework client = CuratorFrameworkFactory.newClient( ZK_ADDRESS, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> RetryNTimes(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5000</span>) ); client.start(); System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"zk client start successfully!"</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 2.Client API test</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 2.1 Create node</span> String data1 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hello"</span>; print(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"create"</span>, ZK_PATH, data1); client.create(). creatingParentsIfNeeded(). forPath(ZK_PATH, data1.getBytes()); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 2.2 Get node and data</span> print(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"ls"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/"</span>); print(client.getChildren().forPath(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/"</span>)); print(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"get"</span>, ZK_PATH); print(client.getData().forPath(ZK_PATH)); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 2.3 Modify data</span> String data2 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"world"</span>; print(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"set"</span>, ZK_PATH, data2); client.setData().forPath(ZK_PATH, data2.getBytes()); print(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"get"</span>, ZK_PATH); print(client.getData().forPath(ZK_PATH)); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 2.4 Remove node</span> print(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"delete"</span>, ZK_PATH); client.delete().forPath(ZK_PATH); print(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"ls"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/"</span>); print(client.getChildren().forPath(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/"</span>)); } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">print</span>(String... cmds) { StringBuilder text = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> StringBuilder(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"$ "</span>); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (String cmd : cmds) { text.append(cmd).append(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" "</span>); } System.out.println(text.toString()); } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">print</span>(Object result) { System.out.println( result <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">instanceof</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">byte</span>[] ? <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> String((<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">byte</span>[]) result) : result); } }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li></ul>
3.2 监听器
Curator提供了三种Watcher(Cache)来监听结点的变化:
- Path Cache:监视一个路径下1)孩子结点的创建、2)删除,3)以及结点数据的更新。产生的事件会传递给注册的PathChildrenCacheListener。
- Node Cache:监视一个结点的创建、更新、删除,并将结点的数据缓存在本地。
- Tree Cache:Path Cache和Node Cache的“合体”,监视路径下的创建、更新、删除事件,并缓存路径下所有孩子结点的数据。
下面就测试一下最简单的Path Watcher:
<code class="language-java hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.cdai.codebase.bigdata.hadoop.zookeeper.curator; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.CuratorFramework; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.CuratorFrameworkFactory; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.recipes.cache.ChildData; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.recipes.cache.PathChildrenCache; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.recipes.cache.PathChildrenCache.StartMode; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.retry.RetryNTimes; <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/** * Curator framework watch test. */</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">CuratorWatcherTest</span> {</span> <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/** Zookeeper info */</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> String ZK_ADDRESS = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"192.168.1.100:2181"</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> String ZK_PATH = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/zktest"</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">main</span>(String[] args) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> Exception { <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 1.Connect to zk</span> CuratorFramework client = CuratorFrameworkFactory.newClient( ZK_ADDRESS, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> RetryNTimes(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5000</span>) ); client.start(); System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"zk client start successfully!"</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 2.Register watcher</span> PathChildrenCache watcher = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> PathChildrenCache( client, ZK_PATH, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">true</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// if cache data</span> ); watcher.getListenable().addListener((client1, event) -> { ChildData data = event.getData(); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (data == <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>) { System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"No data in event["</span> + event + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"]"</span>); } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> { System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Receive event: "</span> + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"type=["</span> + event.getType() + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"]"</span> + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">", path=["</span> + data.getPath() + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"]"</span> + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">", data=["</span> + <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> String(data.getData()) + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"]"</span> + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">", stat=["</span> + data.getStat() + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"]"</span>); } }); watcher.start(StartMode.BUILD_INITIAL_CACHE); System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Register zk watcher successfully!"</span>); Thread.sleep(Integer.MAX_VALUE); } }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li></ul>
下面是在zkCli.sh中操作时Java程序的输出:
<code class="hljs http has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-attribute" style="box-sizing: border-box;">Java</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">zk client start successfully!</span> <span class="hljs-attribute" style="box-sizing: border-box;">Java</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">Register zk watcher successfully!</span> <span class="http" style="box-sizing: border-box;"><span class="hljs-attribute" style="box-sizing: border-box;">zkCli</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">[zk: localhost:2181(CONNECTED) 11] create /zktest/hello mydata</span> <span class="hljs-attribute" style="box-sizing: border-box;">Java</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">Receive event: type=[CHILD_ADDED], path=[/zktest/hello], data=[mydata], stat=[121,121,1434001221097,1434001221097,0,0,0,0,6,0,121]</span> <span class="http" style="box-sizing: border-box;"><span class="hljs-attribute" style="box-sizing: border-box;">zkCli</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">[zk: localhost:2181(CONNECTED) 12] set /zktest/hello otherdata</span> <span class="hljs-attribute" style="box-sizing: border-box;">Java</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">Receive event: type=[CHILD_UPDATED], path=[/zktest/hello], data=[otherdata], stat=[121,122,1434001221097,1434001228467,1,0,0,0,9,0,121]</span> <span class="http" style="box-sizing: border-box;"><span class="hljs-attribute" style="box-sizing: border-box;">zkCli</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">[zk: localhost:2181(CONNECTED) 13] delete /zktest/hello</span> <span class="hljs-attribute" style="box-sizing: border-box;">Java</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">Receive event: type=[CHILD_REMOVED], path=[/zktest/hello], data=[otherdata], stat=[121,122,1434001221097,1434001228467,1,0,0,0,9,0,121]</span></span></span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li></ul>
4.Curator“菜谱”
既然Maven包叫做curator-recipes,那说明Curator有它独特的“菜谱”:
- 锁:包括共享锁、共享可重入锁、读写锁等。
- 选举:Leader选举算法。
- Barrier:阻止分布式计算直至某个条件被满足的“栅栏”,可以看做JDK Concurrent包中Barrier的分布式实现。
- 缓存:前面提到过的三种Cache及监听机制。
- 持久化结点:连接或Session终止后仍然在Zookeeper中存在的结点。
- 队列:分布式队列、分布式优先级队列等。
4.1 分布式锁
分布式编程时,比如最容易碰到的情况就是应用程序在线上多机部署,于是当多个应用同时访问某一资源时,就需要某种机制去协调它们。例如,现在一台应用正在rebuild缓存内容,要临时锁住某个区域暂时不让访问;又比如调度程序每次只想一个任务被一台应用执行等等。
下面的程序会启动两个线程t1和t2去争夺锁,拿到锁的线程会占用5秒。运行多次可以观察到,有时是t1先拿到锁而t2等待,有时又会反过来。Curator会用我们提供的lock路径的结点作为全局锁,这个结点的数据类似这种格式:[_c_64e0811f-9475-44ca-aa36-c1db65ae5350-lock-0000000005],每次获得锁时会生成这种串,释放锁时清空数据。
<code class="language-java hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.cdai.codebase.bigdata.hadoop.zookeeper.curator; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.CuratorFramework; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.CuratorFrameworkFactory; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.recipes.locks.InterProcessMutex; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.retry.RetryNTimes; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.concurrent.TimeUnit; <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/** * Curator framework's distributed lock test. */</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">CuratorDistrLockTest</span> {</span> <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/** Zookeeper info */</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> String ZK_ADDRESS = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"192.168.1.100:2181"</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> String ZK_LOCK_PATH = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/zktest"</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">main</span>(String[] args) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> InterruptedException { <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 1.Connect to zk</span> CuratorFramework client = CuratorFrameworkFactory.newClient( ZK_ADDRESS, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> RetryNTimes(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5000</span>) ); client.start(); System.out.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"zk client start successfully!"</span>); Thread t1 = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Thread(() -> { doWithLock(client); }, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"t1"</span>); Thread t2 = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Thread(() -> { doWithLock(client); }, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"t2"</span>); t1.start(); t2.start(); } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">doWithLock</span>(CuratorFramework client) { InterProcessMutex lock = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> InterProcessMutex(client, ZK_LOCK_PATH); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> { <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (lock.acquire(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span> * <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>, TimeUnit.SECONDS)) { System.out.println(Thread.currentThread().getName() + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" hold lock"</span>); Thread.sleep(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5000</span>L); System.out.println(Thread.currentThread().getName() + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" release lock"</span>); } } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> (Exception e) { e.printStackTrace(); } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">finally</span> { <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> { lock.release(); } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> (Exception e) { e.printStackTrace(); } } } }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li></ul>
4.2 Leader选举
当集群里的某个服务down机时,我们可能要从slave结点里选出一个作为新的master,这时就需要一套能在分布式环境中自动协调的Leader选举方法。Curator提供了LeaderSelector监听器实现Leader选举功能。同一时刻,只有一个Listener会进入takeLeadership()方法,说明它是当前的Leader。注意:当Listener从takeLeadership()退出时就说明它放弃了“Leader身份”,这时Curator会利用Zookeeper再从剩余的Listener中选出一个新的Leader。autoRequeue()方法使放弃Leadership的Listener有机会重新获得Leadership,如果不设置的话放弃了的Listener是不会再变成Leader的。
<code class="language-java hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> com.cdai.codebase.bigdata.hadoop.zookeeper.curator; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.CuratorFramework; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.CuratorFrameworkFactory; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.recipes.leader.LeaderSelector; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.recipes.leader.LeaderSelectorListener; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.framework.state.ConnectionState; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.retry.RetryNTimes; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.curator.utils.EnsurePath; <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/** * Curator framework's leader election test. * Output: * LeaderSelector-2 take leadership! * LeaderSelector-2 relinquish leadership! * LeaderSelector-1 take leadership! * LeaderSelector-1 relinquish leadership! * LeaderSelector-0 take leadership! * LeaderSelector-0 relinquish leadership! * ... */</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">CuratorLeaderTest</span> {</span> <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/** Zookeeper info */</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> String ZK_ADDRESS = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"192.168.1.100:2181"</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> String ZK_PATH = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/zktest"</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">main</span>(String[] args) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> InterruptedException { LeaderSelectorListener listener = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> LeaderSelectorListener() { <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">takeLeadership</span>(CuratorFramework client) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> Exception { System.out.println(Thread.currentThread().getName() + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" take leadership!"</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// takeLeadership() method should only return when leadership is being relinquished.</span> Thread.sleep(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5000</span>L); System.out.println(Thread.currentThread().getName() + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" relinquish leadership!"</span>); } <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">stateChanged</span>(CuratorFramework client, ConnectionState state) { } }; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Thread(() -> { registerListener(listener); }).start(); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Thread(() -> { registerListener(listener); }).start(); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Thread(() -> { registerListener(listener); }).start(); Thread.sleep(Integer.MAX_VALUE); } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">registerListener</span>(LeaderSelectorListener listener) { <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 1.Connect to zk</span> CuratorFramework client = CuratorFrameworkFactory.newClient( ZK_ADDRESS, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> RetryNTimes(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5000</span>) ); client.start(); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 2.Ensure path</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> { <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> EnsurePath(ZK_PATH).ensure(client.getZookeeperClient()); } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> (Exception e) { e.printStackTrace(); } <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 3.Register listener</span> LeaderSelector selector = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> LeaderSelector(client, ZK_PATH, listener); selector.autoRequeue(); selector.start(); } }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li></ul>