一、(单台服务器)docker-compose搭建RabbitMQ集群
这里我们提到的是用一台Centos7服务器上用docker搭建的RabbitMQ集群,只能用作测试和练手用。
1.环境准备
服务器IP | 容器分配网段 | 容器hostname | 容器的IP |
---|---|---|---|
172.17.202.82 | 172.24.0.0/16 | rabbit1 | 172.24.0.5 |
rabbit2 | 172.24.0.6 | ||
rabbit3 | 172.24.0.7 |
2.Centos7关闭防火墙
启动一个服务:systemctl start firewalld.service
关闭一个服务:systemctl stop firewalld.service
重启一个服务:systemctl restart firewalld.service
显示一个服务的状态:systemctl status firewalld.service
在开机时启用一个服务:systemctl enable firewalld.service
在开机时禁用一个服务:systemctl disable firewalld.service
选择合适的命令,关闭防火墙
3.安装Docker,Docker-compose
略
4.单机版docker-compose文件
说明:docker-compose.yml文件中创建了network桥接网卡,和三个RabbitMQ的容器
version: '3.6'
networks:
rabbit:
driver: bridge
ipam:
config:
- subnet: 172.24.0.0/16
services:
rabbit1:
image: rabbitmq:management-alpine
hostname: rabbit1
ports:
- "25672:25672"
- "5672:5672"
- "15672:15672"
- "4369:4369"
networks:
rabbit:
ipv4_address: 172.24.0.5
environment:
- RABBITMQ_NODENAME=rabbit1
- RABBITMQ_ERLANG_COOKIE=CURIOAPPLICATION
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=admin
volumes:
- "/var/lib/rabbitmq/.erlang.cookie:/root/.erlang.cookie"
rabbit2:
image: rabbitmq:management-alpine
hostname: rabbit2
environment:
- RABBITMQ_NODENAME=rabbit2
- RABBITMQ_ERLANG_COOKIE=CURIOAPPLICATION
ports:
- "5673:5672"
- "15673:15672"
networks:
rabbit:
ipv4_address: 172.24.0.6
volumes:
- "/var/lib/rabbitmq/.erlang.cookie:/root/.erlang.cookie"
rabbit3:
image: rabbitmq:management-alpine
hostname: rabbit3
environment:
- RABBITMQ_NODENAME=rabbit3
- RABBITMQ_ERLANG_COOKIE=CURIOAPPLICATION
ports:
- "5674:5672"
- "15674:15672"
networks:
rabbit:
ipv4_address: 172.24.0.7
volumes:
- "/var/lib/rabbitmq/.erlang.cookie:/root/.erlang.cookie"
5.创建并启动RabbitMQ容器
执行命令:
[root@localhost home]# docker-compose –up -d
6.组成集群
说明: rabbit1@rabbit1 第一个RabbitMQ的节点名,第二个是容器主机名
如果想用ram存储就用第一个shell文件, 用硬盘存储就用第二个shell文件
ram集群服务器shell文件
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit1@rabbit1 --ram
rabbitmqctl start_app
disk集群服务器shell文件
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit1@rabbit1
rabbitmqctl start_app
Rabbit2加入rabbit1节点:
[root@linuxpei sqa]# docker exec -it 675f /bin/bash
bash-4.4# hostname
rabbit2
bash-4.4#
bash-4.4# rabbitmqctl stop_app
Stopping rabbit application on node rabbit2@rabbit2 ...
bash-4.4#
bash-4.4# rabbitmqctl reset
Resetting node rabbit2@rabbit2 ...
bash-4.4#
bash-4.4# rabbitmqctl join_cluster rabbit1@rabbit1 --ram
Clustering node rabbit2@rabbit2 with rabbit1@rabbit1
bash-4.4#
bash-4.4# rabbitmqctl start_app
Starting node rabbit2@rabbit2 ...
completed with 3 plugins.
bash-4.4# exit
exit
[root@linuxpei sqa]#
Rabbit3加入rabbit1节点:
[root@linuxpei sqa]# docker exec -it f634 /bin/bash
bash-4.4# hostname
rabbit3
bash-4.4#
bash-4.4# rabbitmqctl stop_app
Stopping rabbit application on node rabbit3@rabbit3 ...
bash-4.4#
bash-4.4# rabbitmqctl reset
Resetting node rabbit3@rabbit3 ...
bash-4.4#
bash-4.4# rabbitmqctl join_cluster rabbit1@rabbit1
Clustering node rabbit3@rabbit3 with rabbit1@rabbit1
bash-4.4#
bash-4.4# rabbitmqctl start_app
Starting node rabbit3@rabbit3 ...
completed with 3 plugins.
bash-4.4#
bash-4.4# exit
exit
[root@linuxpei sqa]#
7.查看结果
在rabbit1容器中查看集群状态,此时可以看到集群中有三个节点在运行
[root@linuxpei sqa]# docker exec -it 7793d /bin/bash
bash-4.4#
bash-4.4# hostname
rabbit1
bash-4.4# rabbitmqctl cluster_status
Cluster status of node rabbit1@rabbit1 ...
[{nodes,[{disc,[rabbit1@rabbit1,rabbit3@rabbit3]},{ram,[rabbit2@rabbit2]}]},
{running_nodes,[rabbit3@rabbit3,rabbit2@rabbit2,rabbit1@rabbit1]},
{cluster_name,<<"rabbit1@rabbit1">>},
{partitions,[]},
{alarms,[{rabbit3@rabbit3,[]},{rabbit2@rabbit2,[]},{rabbit1@rabbit1,[]}]}]
bash-4.4#
bash-4.4#
登陆网页:http://172.17.202.82:15672 登陆用户名密码:admin/admin 查看结果:
8.RabbitMQ集群设置镜像队列策略
在任意一个节点(如:rabbit1)上执行:
[root@linuxpei sqa]# docker exec -it 7793d /bin/bash
bash-4.4#
bash-4.4# hostname
rabbit1
bash-4.4#
bash-4.4# rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'
Setting policy "ha-all" for pattern "^" to "{"ha-mode":"all"}" with priority "0" for vhost "/" ...
bash-4.4#
bash-4.4#
将所有队列设置为镜像队列,即队列会被复制到各个节点,各个节点状态保持一直。
我们可以在rabbit2上查看策略
[root@linuxpei sqa]# docker exec -it 675 /bin/bash
bash-4.4#
bash-4.4# hostname
rabbit2
bash-4.4# rabbitmqctl list_policies
Listing policies for vhost "/" ...
vhost name pattern apply-to definition priority
/ ha-all ^ all {"ha-mode":"all"} 0
bash-4.4#
二、(多台服务器)docker-compose搭建RabbitMQ集群
上面我们使用Docker搭建出了一个RabbitMQ单机集群,这种模式只能用来测试玩玩,无法再生产环境中使用。在生产环境中,需要把3个节点的集群分布到各个主机上面去。这个时候docker-compose就需要做调整,环境部署如下:
1.环境准备
服务器IP | 容器分配网段 | 容器hostname | 容器的IP |
---|---|---|---|
172.17.202.82 | 172.24.0.0/16 | rabbit1 | 172.24.0.5 |
172.17.202.144 | 172.21.0.0/16 | rabbit4 | 172.21.0.4 |
172.17.202.155 | 172.22.0.0/16 | rabbit5 | 172.22.0.2 |
2.Centos7关闭防火墙
启动一个服务:systemctl start firewalld.service
关闭一个服务:systemctl stop firewalld.service
重启一个服务:systemctl restart firewalld.service
显示一个服务的状态:systemctl status firewalld.service
在开机时启用一个服务:systemctl enable firewalld.service
在开机时禁用一个服务:systemctl disable firewalld.service
3.安装Docker,Docker-compose
略
4.多机版docker-compose文件
在服务器:172.17.202.82上创建rabbit1(IP:172.24.0.5)容器
version: '3.6'
networks:
rabbit:
driver: bridge
ipam:
config:
- subnet: 172.24.0.0/16
services:
rabbit1:
image: rabbitmq:management-alpine
hostname: rabbit1
ports:
- "25672:25672"
- "5672:5672"
- "15672:15672"
- "4369:4369"
networks:
rabbit:
ipv4_address: 172.24.0.5
environment:
- RABBITMQ_NODENAME=rabbit1
- RABBITMQ_ERLANG_COOKIE=CURIOAPPLICATION
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=admin
volumes:
- "/var/lib/rabbitmq/.erlang.cookie:/root/.erlang.cookie"
在服务器:172.17.202.144上创建rabbit4(IP:172.21.0.4)容器
version: '3.6'
networks:
rabbit:
driver: bridge
ipam:
config:
- subnet: 172.21.0.0/16
services:
rabbit4:
image: rabbitmq:management-alpine
hostname: rabbit4
environment:
- RABBITMQ_NODENAME=rabbit4
- RABBITMQ_ERLANG_COOKIE=CURIOAPPLICATION
ports:
- "5672:5672"
- "15672:15672"
- "25672:25672"
- "4369:4369"
networks:
rabbit:
ipv4_address: 172.21.0.4
extra_hosts:
- "rabbit1:172.24.0.5"
volumes:
- "/var/lib/rabbitmq/.erlang.cookie:/root/.erlang.cookie"
在服务器:172.17.202.155上创建rabbit5(IP:172.22.0.2)容器
version: '3.6'
networks:
rabbit:
driver: bridge
ipam:
config:
- subnet: 172.22.0.0/16
services:
rabbit5:
image: rabbitmq:management-alpine
hostname: rabbit5
environment:
- RABBITMQ_NODENAME=rabbit5
- RABBITMQ_ERLANG_COOKIE=CURIOAPPLICATION
ports:
- "5672:5672"
- "15672:15672"
- "25672:25672"
- "4369:4369"
networks:
rabbit:
ipv4_address: 172.22.0.2
extra_hosts:
- "rabbit1:172.24.0.5"
volumes:
- "/var/lib/rabbitmq/.erlang.cookie:/root/.erlang.cookie"
5.创建并启动RabbitMQ容器
执行命令:
[root@localhost home]# docker-compose –up -d
6.两个主机的容器互访测试
要将两个主机上容器组成一个集群,必须要确保这两个容器能够互访。
在(172.17.202.144)服务器的容器内做以下测试:
144的容器ip 操作 目标服务器
172.21.0.4 ping 172.17.202.144 (ping本容器的主机)
172.21.0.4 ping 172.17.202.82 (ping82主机)
172.21.0.4 ping 172.21.0.4 (ping本容器)
172.21.0.4 ping 172.24.0.5 (ping82主机的容器->不通)
可以发现,ping 82主机的容器 ping不通。 跨主机容器互访失败
[root@whsqadt02 ~]# docker exec -it c3f8 /bin/bash
bash-4.4# hostname
rabbit4
bash-4.4#
bash-4.4# ping 172.17.202.144
PING 172.17.202.144 (172.17.202.144): 56 data bytes
64 bytes from 172.17.202.144: seq=0 ttl=64 time=0.257 ms
64 bytes from 172.17.202.144: seq=1 ttl=64 time=0.124 ms
bash-4.4#
bash-4.4# ping 172.17.202.82
PING 172.17.202.82 (172.17.202.82): 56 data bytes
64 bytes from 172.17.202.82: seq=0 ttl=63 time=0.922 ms
64 bytes from 172.17.202.82: seq=1 ttl=63 time=0.868 ms
bash-4.4#
bash-4.4# ping 172.21.0.4
PING 172.21.0.4 (172.21.0.4): 56 data bytes
64 bytes from 172.21.0.4: seq=0 ttl=64 time=0.145 ms
64 bytes from 172.21.0.4: seq=1 ttl=64 time=0.115 ms
bash-4.4#
bash-4.4# ping 172.24.0.5
PING 172.21.0.6 (172.24.0.5): 56 data bytes
没有响应
7.配置路由规则
方法一、添加路由方式
在144主机上执行以下命令行,添加路由规则:
route add -net 172.24.0.0/16 gw 172.17.202.82
或
ip route add 172.24.0.0/16 via 172.17.202.82
可以看到,以上命令是加的82主机和82容器的相关配置
反之,对82服务器添加路由规则:
route add -net 172.21.0.0/16 gw 172.17.202.144
或
ip route add 172.21.0.0/16 via 172.17.202.144
可以看到,以上命令是加的144主机和144容器的相关配置
删除路由方式:
route del -net 172.24.0.2 netmask 255.255.255.254 gw 172.17.202.82
注:172.24.0.0/16填写16还是24 是根据容器的网段网关决定,如下:
[root@linuxpei ~]# docker exec -it 779 /bin/bash
bash-4.4#
bash-4.4# hostname
rabbit1
bash-4.4#
bash-4.4# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
72: eth0@if73: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:18:00:05 brd ff:ff:ff:ff:ff:ff
inet 172.24.0.5/16 brd 172.24.255.255 scope global eth0
valid_lft forever preferred_lft forever
bash-4.4#
可以执行以下命令查看添加的路由规则:
[root@whsqadt02 ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
172.17.202.0 0.0.0.0 255.255.254.0 U 100 0 0 ens33
172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-
172.24.0.0 linuxpei 255.255.0.0 UG 0 0 0 ens33
192.168.192.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
[root@whsqadt02 ~]#
再登陆到144主机的容器中,对82主机的容器进行ping操作:
[root@whsqadt02 ~]# docker exec -it c3f8 /bin/bash
bash-4.4#
bash-4.4# ping 172.24.0.5
PING 172.24.0.5 (172.24.0.5): 56 data bytes
64 bytes from 172.24.0.5: seq=0 ttl=62 time=1.124 ms
64 bytes from 172.24.0.5: seq=1 ttl=62 time=1.128 ms
bash-4.4#
反之,再登陆82主机的容器做同样ping操作,查看互访结果。
同理:在对82和155服务器做以上操作
注:以上方式添加的路由规则,在重启网络后,路由规则会消失
方法二、添加永久路由
1、在144主机上编辑 /etc/sysconfig/static-routes : (没有static-routes的话就手动建立一个这样的文件)
例如:
any net 172.24.0.0/16 gw 172.17.202.82
any net 172.21.0.0/16 gw 172.17.202.144
或
any net 172.24.0.0 netmask 255.255.0.0 gw 172.17.202.82
any net 172.21.0.0 netmask 255.255.0.0 gw 172.17.202.144
2、在144主机上编辑/etc/sysconfig/network: (写入以下内容)
# Add non interface-specific static-routes.
if [ -f /etc/sysconfig/static-routes ]; then
grep "^any" /etc/sysconfig/static-routes | while read ignore args ; do
/sbin/route add -$args
done
fi
做好以上操作后,重启144服务器,然后登陆容器内,进行144->82,144->155容器间互信测试。
同理:在对82和155服务器做以上操作
8.组成集群
说明: rabbit1@rabbit1 第一个RabbitMQ的节点名,第二个是容器主机名
如果想用ram存储就用第一个shell文件, 用硬盘存储就用第二个shell文件
ram集群服务器shell文件
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit1@rabbit1 --ram
rabbitmqctl start_app
disk集群服务器shell文件
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit1@rabbit1
rabbitmqctl start_app
Rabbit4加入rabbit1节点:
[root@whsqadt02 sqa]# docker exec -it 675f /bin/bash
bash-4.4# hostname
rabbit4
bash-4.4#
bash-4.4# rabbitmqctl stop_app
Stopping rabbit application on node rabbit2@rabbit2 ...
bash-4.4#
bash-4.4# rabbitmqctl reset
Resetting node rabbit4@rabbit4 ...
bash-4.4#
bash-4.4# rabbitmqctl join_cluster rabbit1@rabbit1 --ram
Clustering node rabbit4@rabbit4 with rabbit1@rabbit1
bash-4.4#
bash-4.4# rabbitmqctl start_app
Starting node rabbit4@rabbit4 ...
completed with 3 plugins.
bash-4.4# exit
exit
[root@whsqadt02 sqa]#
rabbit5加入rabbit1节点:
[root@whsqadt03 sqa]# docker exec -it f634 /bin/bash
bash-4.4# hostname
rabbit5
bash-4.4#
bash-4.4# rabbitmqctl stop_app
Stopping rabbit application on node rabbit5@rabbit5 ...
bash-4.4#
bash-4.4# rabbitmqctl reset
Resetting node rabbit5@rabbit5 ...
bash-4.4#
bash-4.4# rabbitmqctl join_cluster rabbit1@rabbit1
Clustering node rabbit5@rabbit5 with rabbit1@rabbit1
bash-4.4#
bash-4.4# rabbitmqctl start_app
Starting node rabbit5@rabbit5 ...
completed with 3 plugins.
bash-4.4#
bash-4.4# exit
exit
[root@whsqadt03 sqa]#
9.查看结果
在rabbit1容器中查看集群状态,此时可以看到集群中有三个节点在运行
[root@linuxpei sqa]# docker exec -it 7793d /bin/bash
bash-4.4#
bash-4.4# hostname
rabbit1
bash-4.4# rabbitmqctl cluster_status
Cluster status of node rabbit1@rabbit1 ...
[{nodes,[{disc,[rabbit1@rabbit1,rabbit5@rabbit5]},{ram,[rabbit4@rabbit4]}]},
{running_nodes,[rabbit4@rabbit4,rabbit5@rabbit5,rabbit1@rabbit1]},
{cluster_name,<<"rabbit1@rabbit1">>},
{partitions,[]},
{alarms,[{rabbit4@rabbit4,[]},{rabbit5@rabbit5,[]},{rabbit1@rabbit1,[]}]}]
bash-4.4#
bash-4.4#
登陆网页:http://172.17.202.82:15672 登陆用户名密码:admin/admin 查看结果:
10.RabbitMQ集群设置镜像队列策略
在任意一个节点(如:rabbit1)上执行:
[root@linuxpei sqa]# docker exec -it 7793d /bin/bash
bash-4.4#
bash-4.4# hostname
rabbit1
bash-4.4#
bash-4.4# rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'
Setting policy "ha-all" for pattern "^" to "{"ha-mode":"all"}" with priority "0" for vhost "/" ...
bash-4.4#
bash-4.4#
将所有队列设置为镜像队列,即队列会被复制到各个节点,各个节点状态保持一直。
我们可以在rabbit4上查看策略
[root@whsqadt03 sqa]# docker exec -it 675 /bin/bash
bash-4.4#
bash-4.4# hostname
rabbit4
bash-4.4# rabbitmqctl list_policies
Listing policies for vhost "/" ...
vhost name pattern apply-to definition priority
/ ha-all ^ all {"ha-mode":"all"} 0
bash-4.4#
@ 参考其他的博客文档,并结合自己的整理,完成部署教程,共勉。