linux工匠之mesos marathon consul的集群试验

我们今天做mesos、marathon和consul的集成试验，通过consul的自动发现，配合named服务，在docker容器中能够自动识别服务的域名，以便在mesos+marathon中部署docker实例能够通过域名互通。也就是consul在docker容器中的自动发现。

如果是线上，consul server最少是3台做集群，consul client是每个mesos-slave上跑一个，mesos-master也最少3台集群，marathon和zookeeper、bind都应该是集群的模式。集群配置在前两篇博文有说明，consul集群的服务注册发现和模板使用  mesos marathon和chronos构建分布式系统，请予查看。

1 2 3 4 5 6 7 8

# cat /etc/centos-release CentOS Linux release 7.2.1511 (Core) # named -v BIND 9.9.4-RedHat-9.9.4-29.el7_2.4 (Extended Support Version) # consul -v Consul v0.7.0 # docker -v Docker version 1.12.2, build bb80604

部署结构说明，总共3台虚拟机：

1 2 3

master    mesos-master  marathon  consul-server  bind  10.211.55.10 slave      mesos-slave  docker    consul-client        10.211.55.16 vm01        zookeeper              consul-client        10.211.55.4

consul的配置如下：

1 2 3 4 5 6 7 8 9 10

[root@master ~]# consul agent -server -bootstrap-expect=1 -data-dir=/tmp/consul -node=agent-one -bind=10.211.55.10 -config-dir=/etc/consul.d -client 0.0.0.0 -dc=dc1 -ui & [root@slave ~]# consul agent -advertise 10.211.55.16 -data-dir=/tmp/consul -config-dir=/etc/consul.d -node=agent-slave -bind=10.211.55.16 -join 10.211.55.10 -dc=dc1 & [root@vm01 ~]# consul agent -advertise 10.211.55.4 -data-dir=/tmp/1/consul -config-dir=/etc/consul.d -node=zookeeper -bind=10.211.55.4 -join 10.211.55.10 -dc=dc1 &   [root@master ~]# consul members     2016/12/07 03:57:02 [INFO] agent.rpc: Accepted client: 127.0.0.1:55339 Node         Address            Status  Type    Build  Protocol  DC agent-one    10.211.55.10:8301  alive   server  0.7.0  2         dc1 agent-slave  10.211.55.16:8301  alive   client  0.7.0  2         dc1 zookeeper    10.211.55.4:8301   alive   client  0.7.0  2         dc1

bind服务配置：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

[root@master~]# yum install bind-utils -y   #dig命令 [root@master~]# yum install bind [root@master~]# cat /etc/named.conf options{         listen-onport53{127.0.0.1;};         listen-on-v6port53{::1;};         directory      "/var/named";         dump-file      "/var/named/data/cache_dump.db";         statistics-file"/var/named/data/named_stats.txt";         memstatistics-file"/var/named/data/named_mem_stats.txt";         allow-query    {localhost;};           recursionyes;           #dnssec-enable yes;         dnssec-enableno;         #dnssec-validation yes;         dnssec-validationno;           /* Path to ISC DLV key */         bindkeys-file"/etc/named.iscdlv.key";           managed-keys-directory"/var/named/dynamic";           pid-file"/run/named/named.pid";         session-keyfile"/run/named/session.key"; };   logging{         channeldefault_debug{                 file"data/named.run";                 severitydynamic;         }; };   zone"."IN{         typehint;         file"named.ca"; };   include"/etc/named.rfc1912.zones"; include"/etc/named.root.key"; include"/etc/named/consul.conf";   [root@master~]# cat /etc/named/consul.conf zone"consul"IN{   typeforward;   forwardonly;   forwarders{127.0.0.1port8600;}; };   # systemctl restart named

我们现在注册consul服务，在master主机注册marathon服务

1 2	[root@master ~]# cat /etc/consul.d/marathon.json {"service": {"name": "marathon", "tags": ["marathon"], "port": 8080, "check": {"script": "curl 127.0.0.1:8080 >/dev/null 2>&1", "interval": "10s"}}}

在vm01主机注册zookeeper服务

1 2	[root@vm01~]# cat /etc/consul.d/zookeeper.json {"service":{"name":"zookeeper","tags":["zookeeper"],"port":2181}}

 

ok，来master主机测试dig的结果

1 2 3 4 5 6 7 8 9 10

[root@master ~]# dig @localhost -p 8600 consul.service.consul +short 10.211.55.10 [root@master ~]# dig @127.0.0.1 -p 8600 consul.service.consul SRV +short 1 1 8300 agent-one.node.dc1.consul. [root@master ~]# dig @localhost -p 8600 zookeeper.service.consul +short 10.211.55.4 [root@master ~]# dig @localhost -p 53 zookeeper.service.consul +short 10.211.55.4 [root@master ~]# dig @localhost zookeeper.service.consul +short 10.211.55.4

因为在docker中用到bind服务，所以修改docker的配置，添加dns选项：

1 2 3 4

[root@slave~]# egrep -v "^$|^#" /etc/sysconfig/docker OPTIONS='--log-driver=journald --dns 10.211.55.10 --dns-search service.consul' DOCKER_CERT_PATH=/etc/docker # systemctl restart docker

终于上面都完成了，开始最后的试验：在mesos-master中新建一个application，并且在mesos-slave主机查看docker的运行情况

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

[root@master ~]# cat mesos-consul.json {   "args": [     "--zk=zk://zookeeper.service.consul:2181/mesos",     "--log-level=debug",     "--consul",     "--refresh=5s"   ],   "container": {     "type": "DOCKER",     "docker": {       "network": "BRIDGE",       "image": "ciscocloud/mesos-consul"     }   },   "id": "mesos-consul",   "instances": 1,   "cpus": 0.1,   "mem": 256 } [root@master ~]# curl -X POST -H "Content-Type: application/json" http://10.211.55.10:8080/v2/apps -d @mesos-consul.json {"id":"/mesos-consul","cmd":null,"args":["--zk=zk://zookeeper.service.consul:2181/mesos","--log-level=debug","--consul","--refresh=5s"],"user":null,"env":{},"instances":1,"cpus":0.1,"mem":256,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"ciscocloud/mesos-consul","network":"BRIDGE","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":1,"maximumOverCapacity":1},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-12-07T08:23:23.720Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[0],"portDefinitions":[{"port":0,"protocol":"tcp","labels":{}}],"requirePorts":false,"tasksStaged":0,"tasksRunning":0,"tasksHealthy":0,"tasksUnhealthy":0,"deployments":[{"id":"f9c87ccd-1d1d-4718-9004-1ab1b49249b0"}],"tasks":[]}   因为docker没有ciscocloud/mesos-consul这个image，所以等待他下载完成后自动运行 [root@master ~]# curl http://0.0.0.0:8080/v2/apps/mesos-consul | python -m json.tool

 

1 2 3 4 5

[root@slave~]# docker images|grep consul docker.io/ciscocloud/mesos-consul                    latest              9733c42aa65d        4monthsago        15.01MB [root@slave~]# docker ps CONTAINERID        IMAGE                    COMMAND                  CREATED              STATUS              PORTS              NAMES 0e4bfbc51998        ciscocloud/mesos-consul  "/bin/mesos-consul --"  Aboutaminuteago  UpAboutaminute                      mesos-ecfc0bd0-dc7f-48c0-a2ba-21de888e9230-S0.10f970f3-08f3-46cc-a07f-727a61e88ae3

 

这时候已经完成了试验，不过似乎没发现什么，我们再来看consul的service，发现多了一个mesos

 

最终的结果是需要在docker中能够和consul提供的域名通信，所以启动一个docker镜像，在里面ping consul.service.consul 此域名查看

1 2 3 4 5 6 7 8 9 10 11 12 13 14

[root@slave~]# docker run -it docker.io/nginx /bin/bash root@a5959acf6785:/# ping consul.service.consul -c 1 PINGconsul.service.consul(10.211.55.10):56databytes 64bytesfrom10.211.55.10:icmp_seq=0ttl=63time=0.262ms ---consul.service.consulpingstatistics--- 1packetstransmitted,1packetsreceived,0%packetloss round-tripmin/avg/max/stddev=0.262/0.262/0.262/0.000ms root@a5959acf6785:/# root@a5959acf6785:/# ping zookeeper.service.consul -c 1 PINGzookeeper.service.consul(10.211.55.4):56databytes 64bytesfrom10.211.55.4:icmp_seq=0ttl=63time=0.366ms ---zookeeper.service.consulpingstatistics--- 1packetstransmitted,1packetsreceived,0%packetloss round-tripmin/avg/max/stddev=0.366/0.366/0.366/0.000ms

你会发现，在docker中已经可以访问consul服务注册的域名，这个才是重要的结果。我们不是常常用一个域名反代多个主机提供的服务来做集群的吗，consul正合适！

 

• 转自 https://bbotte.github.io/service_config/try-on-mesos-marathon-consul-cluster
• 文章信息: bbotte 2016年12月08日 于 linux工匠 关注运维自动化|Python开发|linux高可用集群|数据库维护|性能提优|系统架构 发表