搭建mcollective高可用,使puppet架构更加安全、稳定

一、配置Rabbitmq

安装(略),可参考http://kisspuppet.com/2013/11/10/mcollective-middleware/http://rsyslog.org/2013/11/10/mcollective-middleware/

1. 开启插件rabbitmq_stomp

1
2
3
4
[root@linuxmaster1poc ~]# rabbitmq-plugins enable rabbitmq_stomp
The following plugins have been enabled:
   rabbitmq_stomp
Plugin configuration has changed. Restart RabbitMQ  for  changes to take effect.

2. 添加tcp监听端口和范围

1
2
3
4
[root@linuxmaster1poc ~]# vim /etc/rabbitmq/rabbitmq.config
[
   {rabbitmq_stomp, [{tcp_listeners, [ 61613 ]}]}
].

备注:可参考http://www.rabbitmq.com/stomp.html

3. 创建账户并设置权限

如果你以前配置过,建议将配置清空

1
2
3
4
5
6
7
8
9
[root@linuxmaster1poc ~]# rabbitmqctl stop_app
Stopping node rabbit@linuxmaster1poc ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl reset
Resetting node rabbit@linuxmaster1poc ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl start_app
Starting node rabbit@linuxmaster1poc ...
...done.

删除默认用户guest,添加三个用户(webadmin-http访问用,admin--管理员,mcrabbitmq--mcollective链接用)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@linuxmaster1poc ~]# rabbitmqctl list_users
Listing users ...
guest   [administrator]
...done.
[root@linuxmaster1poc ~]# rabbitmqctl delete_user guest
Deleting user  "guest"  ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl add_user mc_rabbitmq  123 .com
Creating user  "mc_rabbitmq"  ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl add_user admin password= 123 .com
Creating user  "admin"  ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl add_user web_admin  123 .com
Creating user  "web_admin"  ...
...done.

设置用户的角色

1
2
3
4
5
6
[root@linuxmaster1poc ~]# rabbitmqctl set_user_tags admin administrator
Setting tags  for  user  "admin"  to [administrator] ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl set_user_tags web_admin monitoring
Setting tags  for  user  "web_admin"  to [monitoring] ...
...done.


创建虚拟主机组

1
2
3
[root@linuxmaster1poc ~]# rabbitmqctl add_vhost /mcollective
Creating vhost  "/mcollective"  ...
...done.

设置用户访问虚拟主机组的权限

1
2
3
4
5
6
7
8
9
[root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p  "/mcollective"  mc_rabbitmq   ".*"  ".*"  ".*"
Setting permissions  for  user  "mc_rabbitmq"  in  vhost  "/mcollective"  ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p  "/mcollective"  admin   ".*"  ".*"  ".*"
Setting permissions  for  user  "admin"  in  vhost  "/mcollective"  ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p  "/mcollective"  web_admin   ".*"  ".*"  ".*"
Setting permissions  for  user  "web_admin"  in  vhost  "/mcollective"  ...
...done.

重启rabbitmq-server服务

1
2
3
[root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server restart
Restarting rabbitmq-server: SUCCESS
rabbitmq-server.

查看用户以及角色是否创建成功

1
2
3
4
5
6
[root@linuxmaster1poc ~]# rabbitmqctl list_users
Listing users ...
admin   [administrator]
mc_rabbitmq []
web_admin   [monitoring]
...done.

查看虚拟主机组“/mcollective”中所有用户的权限

1
2
3
4
5
6
7
[root@linuxmaster1poc ~]# rabbitmqctl list_permissions -p  "/mcollective"
Listing permissions  in  vhost  "/mcollective"  ...
admin   .*  .*  .*
mc_rabbitmq .*  .*  .*
web_admin   .*  .*  .*
...done.
[root@linuxmaster1poc ~]#

4、登录http://192.168.100.120:15672/设置虚拟主机“/mcollective”exchanges

默认配置

1
2
3
4
5
6
7
8
9
10
[root@linuxmaster1poc ~]# rabbitmqctl list_exchanges -p  "/mcollective"
Listing exchanges ...
     direct
amq.direct  direct
amq.fanout  fanout
amq.headers headers
amq.match   headers
amq.rabbitmq. trace   topic
amq.topic   topic
...done.

mcollective_rabbitmq_high_availability-1

设置后更新配置

mcollective_rabbitmq_high_availability.j


1
2
3
4
5
6
7
8
9
10
11
12
[root@linuxmaster1poc ~]# rabbitmqctl list_exchanges -p  "/mcollective"
Listing exchanges ...
     direct
amq.direct  direct
amq.fanout  fanout
amq.headers headers
amq.match   headers
amq.rabbitmq. trace   topic
amq.topic   topic
mcollective_broadcast   topic
mcollective_directed    direct
...done.

备注:可参考官网设置https://www.rabbitmq.com/man/rabbitmqctl.1.man.html

二、配置MCollective:

1. 配置mcollective client

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[root@linuxmaster1poc testing]# cat /etc/mcollective/client.cfg
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
logger_type = console
#loglevel = debug
loglevel = warn
# Plugins
securityprovider = psk
plugin.psk = a36cd839414370e10fd281b8a38a4f48
direct_addressing =  1
connector = rabbitmq
plugin.rabbitmq.vhost = /mcollective  #虚拟主机
plugin.rabbitmq.pool.size =  2  #设置地址池里有两个mq
plugin.rabbitmq.initial_reconnect_delay =  0.01
plugin.rabbitmq.max_reconnect_delay =  30.0   #重连时间
plugin.rabbitmq.use_exponential_back_off =  true
plugin.rabbitmq.back_off_multiplier =  2
plugin.rabbitmq.max_reconnect_attempts =  0
plugin.rabbitmq.randomize =  false
plugin.rabbitmq.timeout = - 1
plugin.rabbitmq.pool. 1 .host =  192.168 . 100.120
plugin.rabbitmq.pool. 1 .port =  61613
plugin.rabbitmq.pool. 1 .user = mc_rabbitmq
plugin.rabbitmq.pool. 1 .password =  123 .com
plugin.rabbitmq.pool. 1 .ssl =  false
plugin.rabbitmq.pool. 2 .host =  192.168 . 100.121
plugin.rabbitmq.pool. 2 .port =  61613
plugin.rabbitmq.pool. 2 .user = mc_rabbitmq
plugin.rabbitmq.pool. 2 .password =  123 .com
plugin.rabbitmq.pool. 2 .ssl =  false
# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml

2. 配置mcollective server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
[root@linux57poc tmp]# cat /etc/mcollective/server.cfg
# --Global--
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
logfile = / var /log/puppet/mcollective.log
loglevel = info
daemonize =  1
# --rabbitmq Plugins--
securityprovider = psk
plugin.psk = a36cd839414370e10fd281b8a38a4f48
direct_addressing =  1
connector = rabbitmq
plugin.rabbitmq.vhost = /mcollective
plugin.rabbitmq.pool.size =  2
plugin.rabbitmq.initial_reconnect_delay =  0.01
plugin.rabbitmq.max_reconnect_delay =  30.0
plugin.rabbitmq.use_exponential_back_off =  true
plugin.rabbitmq.back_off_multiplier =  2
plugin.rabbitmq.max_reconnect_attempts =  0
plugin.rabbitmq.randomize =  false
plugin.rabbitmq.timeout = - 1
plugin.rabbitmq.pool. 1 .host =  192.168 . 100.120
plugin.rabbitmq.pool. 1 .port =  61613
plugin.rabbitmq.pool. 1 .user = mc_rabbitmq
plugin.rabbitmq.pool. 1 .password =  123 .com
plugin.rabbitmq.pool. 1 .ssl =  false
plugin.rabbitmq.pool. 2 .host =  192.168 . 100.121
plugin.rabbitmq.pool. 2 .port =  61613
plugin.rabbitmq.pool. 2 .user = mc_rabbitmq
plugin.rabbitmq.pool. 2 .password =  123 .com
plugin.rabbitmq.pool. 2 .ssl =  false
# --Puppet provider specific options--
plugin.service.provider = puppet
plugin.service.puppet.hasstatus =  true
plugin.service.puppet.hasrestart =  true
plugin.puppet.command = puppet agent
plugin.puppet.splay =  true
plugin.puppet.splaylimit =  30
plugin.puppet.config = /etc/puppet/puppet.conf
# --Facts--
factsource = yaml
##factsource = facter
plugin.yaml = /etc/mcollective/facts.yaml

三、高可用测试

特别注意:节点mcollectiveserver.cfgpool是有优先级的,默认数字小的生效,这点需要注意,也就是说当所有节点都连接在MQ2上的时候,启动MQ1mco命令是无法使用的,因为它在运行的时候连接的是MQ1,而所有节点都连接在MQ2上。

1. 停止MQ1,查看切换状态

1.1 先看当前的节点连接状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@linuxmaster1poc ~]# mco ping   #查看连接的节点
linux57poc                               time= 69.46  ms
linux58poc                               time= 70.05  ms
linux64poc                               time= 70.59  ms
---- ping statistics ----
3  replies max:  70.59  min:  69.46  avg:  70.03
[root@linuxmaster1poc ~]# mco shell  "lsof -i:61613"  #查看所有节点监听的端口情况,可以看到目前都连接在linuxmaster1poc上。
Do you really want to send  this  command unfiltered? (y/n): y
Discovering hosts using the mc method  for  2  second(s) ....  3
Host: linux64poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ruby     36625  root    6u  IPv4   27771       0t0  TCP linux64poc: 40493 ->linuxmaster1poc: 61613  (ESTABLISHED)
Host: linux58poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ruby     11060  root    6u  IPv4   34046       0t0  TCP linux58poc: 36295 ->linuxmaster1poc: 61613  (ESTABLISHED)
Host: linux57poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE  DEVICE SIZE NODE NAME
ruby     18076  root    6u  IPv4  1351365        TCP linux57poc: 24698 ->linuxmaster1poc: 61613  (ESTABLISHED)
[root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server stop
Stopping rabbitmq-server: rabbitmq-server.

1.2 再次运行mco查看切换状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@linuxmaster1poc ~]# mco ping
linux58poc                               time= 73.54  ms
linux64poc                               time= 74.61  ms
linux57poc                               time= 75.39  ms
---- ping statistics ----
3  replies max:  75.39  min:  73.54  avg:  74.51
[root@linuxmaster1poc ~]# mco shell  "lsof -i:61613"
Do you really want to send  this  command unfiltered? (y/n): y
Discovering hosts using the mc method  for  2  second(s) ....  3
Host: linux58poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ruby     11060  root    6u  IPv4   34046       0t0  TCP linux58poc: 36295 ->linuxmaster1poc: 61613  (CLOSE_WAIT)
ruby     11060  root    9u  IPv4   34137       0t0  TCP linux58poc: 47200 ->linuxmaster2poc: 61613  (ESTABLISHED)
Host: linux64poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ruby     36625  root    6u  IPv4   27771       0t0  TCP linux64poc: 40493 ->linuxmaster1poc: 61613  (CLOSE_WAIT)
ruby     36625  root    8u  IPv4   27877       0t0  TCP linux64poc: 37472 ->linuxmaster2poc: 61613  (ESTABLISHED)
Host: linux57poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE  DEVICE SIZE NODE NAME
ruby     18076  root    9u  IPv4  1351484        TCP linux57poc: 9309 ->linuxmaster2poc: 61613  (ESTABLISHED)

通过日志查看

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[root@linuxmaster1poc ~]# mco shell  "lsof -i:61613"
Do you really want to send  this  command unfiltered? (y/n): y
Discovering hosts using the mc method  for  2  second(s) ....  3
Host: linux58poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ruby     11428  root    6u  IPv4   34283       0t0  TCP linux58poc: 36300 ->linuxmaster1poc: 61613  (CLOSE_WAIT)
ruby     11428  root    8u  IPv4   34338       0t0  TCP linux58poc: 47205 ->linuxmaster2poc: 61613  (ESTABLISHED)
Host: linux57poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE  DEVICE SIZE NODE NAME
ruby     18447  root    6u  IPv4  1351559        TCP linux57poc: 59343 ->linuxmaster1poc: 61613  (CLOSE_WAIT)
ruby     18447  root    8u  IPv4  1351622        TCP linux57poc: 29757 ->linuxmaster2poc: 61613  (ESTABLISHED)
Host: linux64poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ruby     37054  root    4u  IPv4   28036       0t0  TCP linux64poc: 37476 ->linuxmaster2poc: 61613  (ESTABLISHED)
ruby     37054  root    6u  IPv4   27990       0t0  TCP linux64poc: 40497 ->linuxmaster1poc: 61613  (CLOSE_WAIT)

总结:可以看到之前的连接已经变成CLOSE_WAIT,新的连接被建立

2. 停止MQ2,启动MQ1 查看切换状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@linuxmaster2poc rabbitmq]# /etc/init.d/rabbitmq-server stop
Stopping rabbitmq-server: rabbitmq-server.
[root@linux57poc service]# lsof -i: 61613
COMMAND   PID USER   FD   TYPE  DEVICE SIZE NODE NAME
ruby     18447  root    6u  IPv4  1351559        TCP linux57poc: 59343 ->linuxmaster1poc: 61613  (CLOSE_WAIT)
ruby     18447  root    8u  IPv4  1351622        TCP linux57poc: 29757 ->linuxmaster2poc: 61613  (CLOSE_WAIT)
[root@linux58poc ~]# lsof -i: 61613
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ruby     11428  root    6u  IPv4   34283       0t0  TCP linux58poc: 36300 ->linuxmaster1poc: 61613  (CLOSE_WAIT)
ruby     11428  root    8u  IPv4   34338       0t0  TCP linux58poc: 47205 ->linuxmaster2poc: 61613  (CLOSE_WAIT)
[root@linux64poc ~]# lsof -i: 61613
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ruby     37054  root    4u  IPv4   28036       0t0  TCP linux64poc: 37476 ->linuxmaster2poc: 61613  (CLOSE_WAIT)
ruby     37054  root    6u  IPv4   27990       0t0  TCP linux64poc: 40497 ->linuxmaster1poc: 61613  (CLOSE_WAIT)
[root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server start
Starting rabbitmq-server: SUCCESS
rabbitmq-server.

根据 plugin.rabbitmq.maxreconnectdelay =30.0,需要过最多30秒,mcollective服务端会重新建立连接请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[root@linuxmaster1poc ~]# tailf  / var /log/rabbitmq/rabbit\@linuxmaster1poc.log
=INFO REPORT====  24 -Dec- 2013 :: 11 : 00 : 45  ===
accepting STOMP connection < 0.332 . 0 > ( 192.168 . 100.126 : 36316  ->  192.168 . 100.120 : 61613 )
=INFO REPORT====  24 -Dec- 2013 :: 11 : 00 : 45  ===
accepting STOMP connection < 0.348 . 0 > ( 192.168 . 100.125 : 18945  ->  192.168 . 100.120 : 61613 )
=INFO REPORT====  24 -Dec- 2013 :: 11 : 00 : 45  ===
accepting STOMP connection < 0.382 . 0 > ( 192.168 . 100.127 : 40513  ->  192.168 . 100.120 : 61613 )
[root@linuxmaster1poc ~]# mco ping
linux58poc                               time= 70.60  ms
linux57poc                               time= 71.32  ms
linux64poc                               time= 111.56  ms
---- ping statistics ----
3  replies max:  111.56  min:  70.60  avg:  84.49
[root@linuxmaster1poc ~]# mco shell  "lsof -i:61613"
Do you really want to send  this  command unfiltered? (y/n): y
Discovering hosts using the mc method  for  2  second(s) ....  3
Host: linux58poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ruby     11428  root    6u  IPv4   34283       0t0  TCP linux58poc: 36300 ->linuxmaster1poc: 61613  (CLOSE_WAIT)
ruby     11428  root    8u  IPv4   34338       0t0  TCP linux58poc: 47205 ->linuxmaster2poc: 61613  (CLOSE_WAIT)
ruby     11428  root   10u  IPv4   34444       0t0  TCP linux58poc: 36316 ->linuxmaster1poc: 61613  (ESTABLISHED)
Host: linux57poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE  DEVICE SIZE NODE NAME
ruby     18447  root   10u  IPv4  1351723        TCP linux57poc: 18945 ->linuxmaster1poc: 61613  (ESTABLISHED)
Host: linux64poc
Statuscode:  0
Output:
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ruby     37054  root    4u  IPv4   28036       0t0  TCP linux64poc: 37476 ->linuxmaster2poc: 61613  (CLOSE_WAIT)
ruby     37054  root    6u  IPv4   27990       0t0  TCP linux64poc: 40497 ->linuxmaster1poc: 61613  (CLOSE_WAIT)
ruby     37054  root    9u  IPv4   28206       0t0  TCP linux64poc: 40513 ->linuxmaster1poc: 61613  (ESTABLISHED)

本文转自凌激冰51CTO博客,原文链接:http://blog.51cto.com/dreamfire/1344492,如需转载请自行联系原作者


  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值