docker-machine使用discovery模式搭建etcd集群

创建三个docker machine

docker-machine create --driver virtualbox box0
docker-machine create --driver virtualbox box1
docker-machine create --driver virtualbox box2

生成discovery

curl -s http://discovery.etcd.io/new?size=$3
得到
http://discovery.etcd.io/e77afb997af5a84983baa98fd42cf12f

分别在box0、box1、box2

启动etcd0

docker-machine ip box0
192.168.99.101

docker run --rm -p 4001:4001 -p 2380:2380 -p 2379:2379 \
 --name etcd0 quay.io/coreos/etcd \
 -name etcd0 \
 -advertise-client-urls http://192.168.99.101:2379,http://192.168.99.101:4001 \
 -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \
 -initial-advertise-peer-urls http://192.168.99.101:2380 \
 -listen-peer-urls http://0.0.0.0:2380 \
 -discovery http://discovery.etcd.io/e77afb997af5a84983baa98fd42cf12f

输出

2016-06-04 12:49:27.231469 I | etcdmain: etcd Version: 2.3.6
2016-06-04 12:49:27.231591 I | etcdmain: Git SHA: 128344c
2016-06-04 12:49:27.231604 I | etcdmain: Go Version: go1.6.2
2016-06-04 12:49:27.231612 I | etcdmain: Go OS/Arch: linux/amd64
2016-06-04 12:49:27.231620 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2016-06-04 12:49:27.231629 W | etcdmain: no data-dir provided, using default data-dir ./etcd0.etcd
2016-06-04 12:49:27.231699 I | etcdmain: listening for peers on http://0.0.0.0:2380
2016-06-04 12:49:27.231728 I | etcdmain: listening for client requests on http://0.0.0.0:2379
2016-06-04 12:49:27.231757 I | etcdmain: listening for client requests on http://0.0.0.0:4001
2016-06-04 12:49:28.832408 N | discovery: found self 4eb2dcba58da982f in the cluster
2016-06-04 12:49:28.832545 N | discovery: found 1 peer(s), waiting for 2 more
2016-06-04 12:50:28.951308 E | discovery: error #0: client: etcd member http://discovery.etcd.io returns server error [Gateway Timeout]
2016-06-04 12:50:28.951331 I | discovery: waiting for other nodes: error connecting to http://discovery.etcd.io, retrying in 2s
2016-06-04 12:50:31.330137 N | discovery: found self 4eb2dcba58da982f in the cluster
2016-06-04 12:50:31.330269 N | discovery: found 1 peer(s), waiting for 2 more
2016-06-04 12:51:19.825750 N | discovery: found peer aa5569b385caf33b in the cluster
2016-06-04 12:51:19.825772 N | discovery: found 2 peer(s), waiting for 1 more
2016-06-04 12:51:22.896913 N | discovery: found peer f84fe4a4e816e778 in the cluster
2016-06-04 12:51:22.897058 N | discovery: found 3 needed peer(s)
2016-06-04 12:51:22.897666 I | etcdserver: name = etcd0
2016-06-04 12:51:22.897721 I | etcdserver: data dir = etcd0.etcd
2016-06-04 12:51:22.897914 I | etcdserver: member dir = etcd0.etcd/member
2016-06-04 12:51:22.897949 I | etcdserver: heartbeat = 100ms
2016-06-04 12:51:22.898162 I | etcdserver: election = 1000ms
2016-06-04 12:51:22.898230 I | etcdserver: snapshot count = 10000
2016-06-04 12:51:22.898676 I | etcdserver: discovery URL= http://discovery.etcd.io/e77afb997af5a84983baa98fd42cf12f
2016-06-04 12:51:22.898733 I | etcdserver: advertise client URLs = http://192.168.99.101:2379,http://192.168.99.101:4001
2016-06-04 12:51:22.899048 I | etcdserver: initial advertise peer URLs = http://192.168.99.101:2380
2016-06-04 12:51:22.899107 I | etcdserver: initial cluster = etcd0=http://192.168.99.101:2380
2016-06-04 12:51:22.900519 I | etcdserver: starting member 4eb2dcba58da982f in cluster f5c81c838c280635
2016-06-04 12:51:22.900606 I | raft: 4eb2dcba58da982f became follower at term 0
2016-06-04 12:51:22.900851 I | raft: newRaft 4eb2dcba58da982f [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-06-04 12:51:22.900887 I | raft: 4eb2dcba58da982f became follower at term 1
2016-06-04 12:51:22.916188 I | etcdserver: starting server... [version: 2.3.6, cluster version: to_be_decided]
2016-06-04 12:51:22.923197 N | etcdserver: added local member 4eb2dcba58da982f [http://192.168.99.101:2380] to cluster f5c81c838c280635
2016-06-04 12:51:22.923368 N | etcdserver: added member aa5569b385caf33b [http://192.168.99.103:2380] to cluster f5c81c838c280635
2016-06-04 12:51:22.924024 N | etcdserver: added member f84fe4a4e816e778 [http://192.168.99.102:2380] to cluster f5c81c838c280635
2016-06-04 12:51:22.928413 I | rafthttp: the connection with aa5569b385caf33b became active
2016-06-04 12:51:23.402216 I | raft: 4eb2dcba58da982f is starting a new election at term 1
2016-06-04 12:51:23.402366 I | raft: 4eb2dcba58da982f became candidate at term 2
2016-06-04 12:51:23.402638 I | raft: 4eb2dcba58da982f received vote from 4eb2dcba58da982f at term 2
2016-06-04 12:51:23.402775 I | raft: 4eb2dcba58da982f [logterm: 1, index: 3] sent vote request to aa5569b385caf33b at term 2
2016-06-04 12:51:23.403004 I | raft: 4eb2dcba58da982f [logterm: 1, index: 3] sent vote request to f84fe4a4e816e778 at term 2
2016-06-04 12:51:23.405960 I | raft: 4eb2dcba58da982f received vote from aa5569b385caf33b at term 2
2016-06-04 12:51:23.406086 I | raft: 4eb2dcba58da982f [quorum:2] has received 2 votes and 0 vote rejections
2016-06-04 12:51:23.406481 I | raft: 4eb2dcba58da982f became leader at term 2
2016-06-04 12:51:23.406538 I | raft: raft.node: 4eb2dcba58da982f elected leader 4eb2dcba58da982f at term 2
2016-06-04 12:51:23.411623 I | etcdserver: published {Name:etcd0 ClientURLs:[http://192.168.99.101:2379 http://192.168.99.101:4001]} to cluster f5c81c838c280635
2016-06-04 12:51:23.508414 I | rafthttp: the connection with f84fe4a4e816e778 became active
2016-06-04 12:51:23.521271 I | etcdserver: setting up the initial cluster version to 2.3
2016-06-04 12:51:23.524790 N | etcdserver: set the initial cluster version to 2.3

启动etcd1

docker-machine ip box1
192.168.99.102

docker run --rm -p 4001:4001 -p 2380:2380 -p 2379:2379 \
  --name etcd1 quay.io/coreos/etcd \
  -name etcd1 \
  -advertise-client-urls http://192.168.99.102:2379,http://192.168.99.102:4001 \
  -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \
  -initial-advertise-peer-urls http://192.168.99.102:2380 \
  -listen-peer-urls http://0.0.0.0:2380 \
  -discovery http://discovery.etcd.io/e77afb997af5a84983baa98fd42cf12f

输出

2016-06-04 12:51:22.104124 I | etcdmain: etcd Version: 2.3.6
2016-06-04 12:51:22.104227 I | etcdmain: Git SHA: 128344c
2016-06-04 12:51:22.104616 I | etcdmain: Go Version: go1.6.2
2016-06-04 12:51:22.104651 I | etcdmain: Go OS/Arch: linux/amd64
2016-06-04 12:51:22.104894 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2016-06-04 12:51:22.104943 W | etcdmain: no data-dir provided, using default data-dir ./etcd1.etcd
2016-06-04 12:51:22.105248 I | etcdmain: listening for peers on http://0.0.0.0:2380
2016-06-04 12:51:22.105301 I | etcdmain: listening for client requests on http://0.0.0.0:2379
2016-06-04 12:51:22.105592 I | etcdmain: listening for client requests on http://0.0.0.0:4001
2016-06-04 12:51:23.486319 N | discovery: found peer 4eb2dcba58da982f in the cluster
2016-06-04 12:51:23.486453 N | discovery: found peer aa5569b385caf33b in the cluster
2016-06-04 12:51:23.486788 N | discovery: found self f84fe4a4e816e778 in the cluster
2016-06-04 12:51:23.486828 N | discovery: found 3 needed peer(s)
2016-06-04 12:51:23.487083 I | etcdserver: name = etcd1
2016-06-04 12:51:23.487123 I | etcdserver: data dir = etcd1.etcd
2016-06-04 12:51:23.487333 I | etcdserver: member dir = etcd1.etcd/member
2016-06-04 12:51:23.487369 I | etcdserver: heartbeat = 100ms
2016-06-04 12:51:23.487624 I | etcdserver: election = 1000ms
2016-06-04 12:51:23.487660 I | etcdserver: snapshot count = 10000
2016-06-04 12:51:23.487865 I | etcdserver: discovery URL= http://discovery.etcd.io/e77afb997af5a84983baa98fd42cf12f
2016-06-04 12:51:23.487930 I | etcdserver: advertise client URLs = http://192.168.99.102:2379,http://192.168.99.102:4001
2016-06-04 12:51:23.488121 I | etcdserver: initial advertise peer URLs = http://192.168.99.102:2380
2016-06-04 12:51:23.488160 I | etcdserver: initial cluster = etcd1=http://192.168.99.102:2380
2016-06-04 12:51:23.489236 I | etcdserver: starting member f84fe4a4e816e778 in cluster f5c81c838c280635
2016-06-04 12:51:23.489310 I | raft: f84fe4a4e816e778 became follower at term 0
2016-06-04 12:51:23.489614 I | raft: newRaft f84fe4a4e816e778 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-06-04 12:51:23.489652 I | raft: f84fe4a4e816e778 became follower at term 1
2016-06-04 12:51:23.507153 I | etcdserver: starting server... [version: 2.3.6, cluster version: to_be_decided]
2016-06-04 12:51:23.510253 I | raft: f84fe4a4e816e778 [term: 1] received a MsgHeartbeat message with higher term from 4eb2dcba58da982f [term: 2]
2016-06-04 12:51:23.510365 I | raft: f84fe4a4e816e778 became follower at term 2
2016-06-04 12:51:23.510764 I | raft: raft.node: f84fe4a4e816e778 elected leader 4eb2dcba58da982f at term 2
2016-06-04 12:51:23.511757 N | etcdserver: added member 4eb2dcba58da982f [http://192.168.99.101:2380] to cluster f5c81c838c280635
2016-06-04 12:51:23.511867 N | etcdserver: added member aa5569b385caf33b [http://192.168.99.103:2380] to cluster f5c81c838c280635
2016-06-04 12:51:23.512205 N | etcdserver: added local member f84fe4a4e816e778 [http://192.168.99.102:2380] to cluster f5c81c838c280635
2016-06-04 12:51:23.513200 I | rafthttp: the connection with 4eb2dcba58da982f became active
2016-06-04 12:51:23.520368 I | rafthttp: the connection with aa5569b385caf33b became active
2016-06-04 12:51:23.521514 I | raft: f84fe4a4e816e778 [logterm: 1, index: 3, vote: 0] voted for 4eb2dcba58da982f [logterm: 1, index: 3] at term 2
2016-06-04 12:51:23.530335 N | etcdserver: set the initial cluster version to 2.3
2016-06-04 12:51:23.530774 I | etcdserver: published {Name:etcd1 ClientURLs:[http://192.168.99.102:2379 http://192.168.99.102:4001]} to cluster f5c81c838c280635

启动etcd2

docker-machine ip box2
192.168.99.103

  docker run --rm -p 4001:4001 -p 2380:2380 -p 2379:2379 \
   --name etcd2 quay.io/coreos/etcd \
   -name etcd2 \
   -advertise-client-urls http://192.168.99.103:2379,http://192.168.99.103:4001 \
   -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \
   -initial-advertise-peer-urls http://192.168.99.103:2380 \
   -listen-peer-urls http://0.0.0.0:2380 \
   -discovery http://discovery.etcd.io/e77afb997af5a84983baa98fd42cf12f

输出

2016-06-04 12:51:18.952183 I | etcdmain: etcd Version: 2.3.6
2016-06-04 12:51:18.952268 I | etcdmain: Git SHA: 128344c
2016-06-04 12:51:18.952279 I | etcdmain: Go Version: go1.6.2
2016-06-04 12:51:18.952286 I | etcdmain: Go OS/Arch: linux/amd64
2016-06-04 12:51:18.952294 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2016-06-04 12:51:18.952302 W | etcdmain: no data-dir provided, using default data-dir ./etcd2.etcd
2016-06-04 12:51:18.952358 I | etcdmain: listening for peers on http://0.0.0.0:2380
2016-06-04 12:51:18.952383 I | etcdmain: listening for client requests on http://0.0.0.0:2379
2016-06-04 12:51:18.952409 I | etcdmain: listening for client requests on http://0.0.0.0:4001
2016-06-04 12:51:20.438255 N | discovery: found peer 4eb2dcba58da982f in the cluster
2016-06-04 12:51:20.438410 N | discovery: found self aa5569b385caf33b in the cluster
2016-06-04 12:51:20.438647 N | discovery: found 2 peer(s), waiting for 1 more
2016-06-04 12:51:22.900419 N | discovery: found peer f84fe4a4e816e778 in the cluster
2016-06-04 12:51:22.900633 N | discovery: found 3 needed peer(s)
2016-06-04 12:51:22.901192 I | etcdserver: name = etcd2
2016-06-04 12:51:22.901237 I | etcdserver: data dir = etcd2.etcd
2016-06-04 12:51:22.901462 I | etcdserver: member dir = etcd2.etcd/member
2016-06-04 12:51:22.901493 I | etcdserver: heartbeat = 100ms
2016-06-04 12:51:22.901694 I | etcdserver: election = 1000ms
2016-06-04 12:51:22.901726 I | etcdserver: snapshot count = 10000
2016-06-04 12:51:22.901955 I | etcdserver: discovery URL= http://discovery.etcd.io/e77afb997af5a84983baa98fd42cf12f
2016-06-04 12:51:22.901993 I | etcdserver: advertise client URLs = http://192.168.99.103:2379,http://192.168.99.103:4001
2016-06-04 12:51:22.902258 I | etcdserver: initial advertise peer URLs = http://192.168.99.103:2380
2016-06-04 12:51:22.902296 I | etcdserver: initial cluster = etcd2=http://192.168.99.103:2380
2016-06-04 12:51:22.903566 I | etcdserver: starting member aa5569b385caf33b in cluster f5c81c838c280635
2016-06-04 12:51:22.903652 I | raft: aa5569b385caf33b became follower at term 0
2016-06-04 12:51:22.904045 I | raft: newRaft aa5569b385caf33b [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-06-04 12:51:22.904102 I | raft: aa5569b385caf33b became follower at term 1
2016-06-04 12:51:22.923049 I | etcdserver: starting server... [version: 2.3.6, cluster version: to_be_decided]
2016-06-04 12:51:22.926160 I | rafthttp: the connection with 4eb2dcba58da982f became active
2016-06-04 12:51:22.928142 N | etcdserver: added member 4eb2dcba58da982f [http://192.168.99.101:2380] to cluster f5c81c838c280635
2016-06-04 12:51:22.928340 N | etcdserver: added local member aa5569b385caf33b [http://192.168.99.103:2380] to cluster f5c81c838c280635
2016-06-04 12:51:22.928736 N | etcdserver: added member f84fe4a4e816e778 [http://192.168.99.102:2380] to cluster f5c81c838c280635
2016-06-04 12:51:23.402085 I | raft: aa5569b385caf33b [term: 1] received a MsgVote message with higher term from 4eb2dcba58da982f [term: 2]
2016-06-04 12:51:23.402268 I | raft: aa5569b385caf33b became follower at term 2
2016-06-04 12:51:23.402676 I | raft: aa5569b385caf33b [logterm: 1, index: 3, vote: 0] voted for 4eb2dcba58da982f [logterm: 1, index: 3] at term 2
2016-06-04 12:51:23.405786 I | raft: raft.node: aa5569b385caf33b elected leader 4eb2dcba58da982f at term 2
2016-06-04 12:51:23.409927 I | etcdserver: published {Name:etcd2 ClientURLs:[http://192.168.99.103:2379 http://192.168.99.103:4001]} to cluster f5c81c838c280635
2016-06-04 12:51:23.508791 I | rafthttp: the connection with f84fe4a4e816e778 became active
2016-06-04 12:51:23.523118 N | etcdserver: set the initial cluster version to 2.3

查看是否成功

访问
http://discovery.etcd.io/e77afb997af5a84983baa98fd42cf12f

返回

{
    "action": "get", 
    "node": {
        "key": "/_etcd/registry/e77afb997af5a84983baa98fd42cf12f", 
        "dir": true, 
        "nodes": [
            {
                "key": "/_etcd/registry/e77afb997af5a84983baa98fd42cf12f/4eb2dcba58da982f", 
                "value": "etcd0=http://192.168.99.101:2380", 
                "modifiedIndex": 1113260929, 
                "createdIndex": 1113260929
            }, 
            {
                "key": "/_etcd/registry/e77afb997af5a84983baa98fd42cf12f/aa5569b385caf33b", 
                "value": "etcd2=http://192.168.99.103:2380", 
                "modifiedIndex": 1113261715, 
                "createdIndex": 1113261715
            }, 
            {
                "key": "/_etcd/registry/e77afb997af5a84983baa98fd42cf12f/f84fe4a4e816e778", 
                "value": "etcd1=http://192.168.99.102:2380", 
                "modifiedIndex": 1113261726, 
                "createdIndex": 1113261726
            }
        ], 
        "modifiedIndex": 1113260647, 
        "createdIndex": 1113260647
    }
}

坑

如果discovery的地址使用https，则需要自己去搞证书，比较麻烦，改为http的就ok。另外如果启动集群失败了，discovery要重新生成，因为比如之前注册了etcd0，失败了再次启动，会报etcd0已经被注册。

• 请教搭建自己的etcd Discovery后：getsockopt: connection refused[已解决]

• Clustering etcd using Discovery

参考

• docker-compose跑单个etcd实例

• docker-etcd集群-单个machine实例

• Running etcd under Docker(static模式)

• Playing with ETCD cluster in Docker on Local