warm创建docker集群:服务发现-etcd

• 不详细介绍swarm、docker的原理、架构
• 通过Docker + Etcd + Swarm 搭建docker集群

术语

本文中用到的术语书名

Docker节点(docker node)：docker的运行环境，通常为物理机或者虚拟机

测试环境

• 2个docker主机的配置如下所示

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># cat /etc/redhat-release</span>
<span class="hljs-constant" style="box-sizing: border-box;">CentOS</span> <span class="hljs-constant" style="box-sizing: border-box;">Linux</span> release <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7.1</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1503</span> (<span class="hljs-constant" style="box-sizing: border-box;">Core</span>) 
[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># docker --version</span>
<span class="hljs-constant" style="box-sizing: border-box;">Docker</span> version <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.8</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>-el7.centos, build a01dc02/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.8</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

配置docker

主要是配置docker的api端口，具体参考docker的安装&配置

准备Docker镜像

<code class="hljs  has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">docker pull swarm
docker pull etcd</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

启动服务发现：etcd

在docker集群的master节点上执行下面命令(其实可以在任何一个docker环境中执行下面的命令，只要注意后面使用etcd时的ip即可，将其假设为[etcd_ip])

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">docker run <span class="hljs-attribute" style="box-sizing: border-box;">-ti</span> <span class="hljs-attribute" style="box-sizing: border-box;">-d</span> <span class="hljs-attribute" style="box-sizing: border-box;">-p</span> <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">[</span>etcd_port<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">]</span><span class="hljs-markup" style="box-sizing: border-box;">:4001 --name etcd docker.io/microbox/etcd -name discovery</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

其中[map_port]需要替换成自己环境中可用的端口，比如4001。

该步骤是启动了etcd，并在4001端口监听，以便各个docker节点进行上报。

启动swarm manger

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">docker run <span class="hljs-attribute" style="box-sizing: border-box;">-ti</span> <span class="hljs-attribute" style="box-sizing: border-box;">-d</span> <span class="hljs-attribute" style="box-sizing: border-box;">-p</span> <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">[</span>swarm_manager_port<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">]</span><span class="hljs-markup" style="box-sizing: border-box;">:2375 --name swarm-manager docker.io/swarm manage etcd://</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">[</span>etcd_ip<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">]</span><span class="hljs-markup" style="box-sizing: border-box;">:</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">[</span>etcd_port<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">]</span><span class="hljs-markup" style="box-sizing: border-box;"></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

• 将[swarm_manager_port]替换成自己环境中可用的端口，比如2376
• 将[etcd_ip]换成etcd所在环境的ip，例如docker集群master节点的ip（etcd运行的位置决定了该ip的值）
• 将[etcd_port]换成启动etcd时映射到host机后的端口，比如4001

docker节点加入集群

在每个要加入集群docker节点上执行下面的命令。

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">docker run <span class="hljs-attribute" style="box-sizing: border-box;">-ti</span> <span class="hljs-attribute" style="box-sizing: border-box;">-d</span> <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">--</span>name swarm<span class="hljs-attribute" style="box-sizing: border-box;">-agent</span> docker<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>io/swarm <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">join</span> <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">--</span>addr <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">[</span>docker_node_ip<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">]</span><span class="hljs-markup" style="box-sizing: border-box;">:</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">[</span>docker_port<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">]</span><span class="hljs-markup" style="box-sizing: border-box;"> etcd://</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">[</span>etcd_ip<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">]</span><span class="hljs-markup" style="box-sizing: border-box;">:</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">[</span>etcd_port<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">]</span><span class="hljs-markup" style="box-sizing: border-box;">
</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

• 将[docker_node_ip]替换成当前docker节点的ip
• 将docker_port替换成当前docker监听的端口，比如2375。端口的配置在本文上面已经介绍，或者参考具体参考docker的安装&配置
• 将[etcd_ip]换成etcd所在环境的ip，例如docker集群master节点的ip（etcd运行的位置决定了该ip的值）
• 将[etcd_port]换成启动etcd时映射到host机后的端口，比如4001

验证

<code class="hljs css has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">docker</span> <span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">-H</span> <span class="hljs-attr_selector" style="color: rgb(0, 136, 0); box-sizing: border-box;">[swarm_ip]</span>:<span class="hljs-attr_selector" style="color: rgb(0, 136, 0); box-sizing: border-box;">[swarm_manager_port]</span> <span class="hljs-tag" style="color: rgb(0, 0, 0); box-sizing: border-box;">info</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

• 将swarm_ip换成swarm所在的节点的ip。
• 将swarm_manager_port换成swarm的端口，例如2376.

• 可以看到如下内容：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">
<span class="hljs-label" style="box-sizing: border-box;">Containers:</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>
<span class="hljs-label" style="box-sizing: border-box;">Images:</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">26</span>
<span class="hljs-label" style="box-sizing: border-box;">Role:</span> primary
<span class="hljs-label" style="box-sizing: border-box;">Strategy:</span> spread
<span class="hljs-label" style="box-sizing: border-box;">Filters:</span> health, port, dependency, affinity, constraint
<span class="hljs-label" style="box-sizing: border-box;">Nodes:</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>
 ftd<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.master</span>: [docker_node_ip_1]:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2375</span>
  └ Status: Healthy
  └ Containers: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>
  └ Reserved CPUs: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> / <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>
  └ Reserved Memory: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> B / <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">32.79</span> GiB
  └ Labels: executiondriver=native-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.2</span>, kernelversion=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.10</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">229.</span>el7<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.x</span>86_64, operatingsystem=CentOS Linux <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span> (Core), storagedriver=devicemapper
 localhost<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.localdomain</span>: [docker_node_ip_2]:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2375</span>
  └ Status: Healthy
  └ Containers: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>
  └ Reserved CPUs: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> / <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>
  └ Reserved Memory: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> B / <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.888</span> GiB
  └ Labels: executiondriver=native-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.2</span>, kernelversion=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.10</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">229.</span>el7<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.x</span>86_64, operatingsystem=CentOS Linux <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span> (Core), storagedriver=devicemapper
<span class="hljs-label" style="box-sizing: border-box;">CPUs:</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">12</span>
Total Memory: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">36.67</span> GiB
<span class="hljs-label" style="box-sizing: border-box;">Name:</span> xxxxxxxxxx</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li></ul>

集群中创建容器

创建两个容器：执行2次docker run busybox。 
通过docker -H [swarm_ip]:[swarm_port] ps -a可以看到swarm中创建的docker容器. 
通过docker ps -a可以看到2个容器分布在不同的docker节点。

至此，Docker + Etcd + Swarm已经实验完成