【rabbitmq】Rabbitmq+haproxy+keepalived构建双主热备高可用镜像集群

最新推荐文章于 2024-02-28 02:30:00 发布

X-Huger

最新推荐文章于 2024-02-28 02:30:00 发布

阅读量593

点赞数

原文链接：https://blog.csdn.net/GPF19950411/article/details/82794558

版权

服务器准备

rabbitMq主备需要两台；haproxy和keepalived共用或者分开，需要两台或者4台服务器，VIP一个，用于LVS分发

软件准备

需要准备软件：rabbitMq3.6.15：rabbitmq-server-generic-unix-3.6.15.tar.xz；haproxy1.5.9：haproxy-1.5.9.tar.gz；keepalived1.2.7: keepalived-1.2.7-3.el6.x86_64.rpm.

rabbitmq可以到官方网站下载该版本：https://github.com/rabbitmq/rabbitmq-server/releases/tag/rabbitmq_v3_6_15
haproxy可以到官网下载也可以到地址下下载：https://download.csdn.net/download/gpf19950411/10679951
keepalived+lvs可以到该地址下载：https://download.csdn.net/download/gpf19950411/10679954

机器列表：

10.242.154.177（rabbitMq）
10.242.154.176（rabbitMq）
10.244.124.33（Haproxy+keepalived）
10.244.124.34（Haproxy+keepalived）
10.244.124.36（VIP）

安装rabbitMq

RabbitMQ的集群是依赖erlang集群，而erlang集群是通过这个cookie进行通信认证的
首先我们在两台服务器上分别安装erlang

安装erlang
下载`otp_src_20.3.tar.gz`
解压tar -xivf otp_src_20.3.tar.gz
cd otp_src_20.3
./configure --prefix=/opt/erlang
注意进度，如果出现报错如No curses library functions found ncurses；
通过yum安装：
yum -y install ncurses-devel
然后继续安装 
make FREFIX=/opt/erlang
make install FREFIX=/opt/erlang

然后配置环境变量：
ERLANG_HOME=/opt/erlang
export PATH=$PATH:$ERLANG_HOME/bin
export ERLANG_HOME

source /etc/profile

验证：
[root@archpredb221 opt]# erl
Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.3  (abort with ^G)
1>
这样就表示安装成功

接下来是安装rabbitmq：
下载rabbitmq：`rabbitmq-server-generic-unix-3.6.15.tar.xz`
解压：tar -xivf rabbitmq-server-generic-unix-3.6.15.tar.xz -C /opt
mv rabbitmq_server-3.6.15 rabbitmq
配置环境变量：
export PATH=$PATH:/opt/rabbitmq/sbin

source /etc/profile

在/opt/rabbitmq/etc/rabbitmq目录下新建文件：
touch rabbitmq-env.conf
配置相关参数：
RABBITMQ_MNESIA_BASE=/data/rabbitMQ/mnesia
RABBITMQ_LOG_BASE=/data/rabbitMQ/log
RABBITMQ_PLUGINS_DIR=/data/rabbitMQ/plugins

启动rabbitmq：
rabbitmq-server -detached 
这是把rabbitmq当做守护进程进行启动的；

rabbitmqctl status查看状态；
[root@archpredb221 rabbitmq]# rabbitmqctl status
Status of node rabbit@archpredb221
[{pid,5066},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.6.15"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.15"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.15"},
      {rabbit,"RabbitMQ","3.6.15"},
      {cowboy,"Small, fast, modular HTTP server.","1.0.4"},
      {ranch,"Socket acceptor pool for TCP protocols.","1.3.2"},
      {ssl,"Erlang/OTP SSL application","8.2.4"},
      {public_key,"Public key infrastructure","1.5.2"},
      {asn1,"The Erlang ASN1 compiler version 5.0.5","5.0.5"},
      {amqp_client,"RabbitMQ AMQP Client","3.6.15"},
      {cowlib,"Support library for manipulating Web protocols.","1.0.2"},
      {crypto,"CRYPTO","4.2.1"},
      {os_mon,"CPO  CXC 138 46","2.4.4"},
      {rabbit_common,
          "Modules shared by rabbitmq-server and rabbitmq-erlang-client",
          "3.6.15"},
      {recon,"Diagnostic tools for production use","2.3.2"},
      {xmerl,"XML parser","1.3.16"},
      {compiler,"ERTS  CXC 138 10","7.1.5"},
      {inets,"INETS  CXC 138 49","6.5"},
      {mnesia,"MNESIA  CXC 138 12","4.15.3"},
      {syntax_tools,"Syntax tools","2.1.4"},
      {sasl,"SASL  CXC 138 11","3.1.1"},
      {stdlib,"ERTS  CXC 138 10","3.4.4"},
      {kernel,"ERTS  CXC 138 10","5.4.3"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:64] [hipe] [kernel-poll:true]\n"},
 {memory,
     [{connection_readers,341320},
      {connection_writers,7400},
      {connection_channels,17240},
      {connection_other,252712},
      {queue_procs,106296},
      {queue_slave_procs,41085704},
      {plugins,2312216},
      {other_proc,18334000},
      {metrics,217688},
      {mgmt_db,873976},
      {mnesia,139248},
      {other_ets,1954200},
      {binary,16497232},
      {msg_index,48024},
      {code,25094576},
      {atom,1041593},
      {other_system,11009303},
      {allocated_unused,37990536},
      {reserved_unallocated,49991680},
      {total,207314944}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"},{http,15672,"::"}]},
 {vm_memory_calculation_strategy,rss},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,1606834585},
 {disk_free_limit,50000000},
 {disk_free,98908790784},
 {file_descriptors,
     [{total_limit,51100},
      {total_used,8},
      {sockets_limit,45988},
      {sockets_used,5}]},
 {processes,[{limit,1048576},{used,399}]},
 {run_queue,0},
 {uptime,782812},
 {kernel,{net_ticktime,60}}]

查看集群状态：目前是一个节点：
[root@archpredb221 rabbitmq]# rabbitmqctl cluster_status
Cluster status of node rabbit@archpredb221
[{nodes,[{disc,[rabbit@archpredb221]}]},
 {running_nodes,[rabbit@archpredb222]},
 {cluster_name,<<"dbrtmap_pre">>},
 {partitions,[]},
 {alarms,[{rabbit@archpredb222,[]}]}]

默认情况下：RabbitMq会有一个默认用户：guest,但是是限定为localhost访问的；
新建用户：
[root@archpredb221 rabbitmq]# rabbitmqctl add_user root root
Creating user "root"
授予权限：
[root@archpredb221 rabbitmq]# rabbitmqctl set_permissions -p / root ".* " ".* " ".*"
Setting permissions for user "root" in vhost "/"
设置root用户角色：
[root@archpredb221 rabbitmq]# rabbitmqctl set_user_tags root administrator 
Setting tags for user "root" to [administrator]

集群配置：
首先需要将所有节点：
10.242.154.176
10.242.154.177
配置好互信
rabbitmq集群需要通过主机名互相访问 所以必须在三台机器 /etc/hosts配置以上内容 确保主机名正确 
三台机器使用rabbitmq进行通信 使用的安全策略是 erlang.cookie 就是一个随机字符串 三个必须保持一致 否则 无法添加节点到集群
该cookie文件 可能位于  /var/lib/rabbitmq/.erlang.cookie 或者 ~/.erlang.cookie 启动任何一台机器，刚才已经启动了一个节点
把启动节点的.erlang.cookie拷贝到其他节点相同目录下；
上面是单节点的配置，配置好后，配置第二个节点并且启动；目前现在这两个节点是独立；
接下来开始组建集群：
假设已其中一个节点为基准：
在另外一个节点执行：
 rabbitmqctl stop_app
 rabbitmqctl reset
 rabbitmqctl join_cluster rabbit@archpredb222
 rabbitmqctl start_app
接下来查看集群状态：
[root@archpredb221 ~]#  rabbitmqctl cluster_status
Cluster status of node rabbit@archpredb221
[{nodes,[{disc,[rabbit@archpredb221,rabbit@archpredb222]}]},
 {running_nodes,[rabbit@archpredb222,rabbit@archpredb221]},
 {cluster_name,<<"dbrtmap_pre">>},
 {partitions,[]},
 {alarms,[{rabbit@archpredb222,[]},{rabbit@archpredb221,[]}]}]

剔除节点：
方法一：
在节点上执行：
rabbitmqctl stop
rabbitmqctl forget_cluster_node rabbit@archpredb222 --offline 

注意上面在使用 rabbitmqctl forget_cluster_node 命令的时候用到了
"--offline" 参数，如果不添加这个参数，就需要保证 archpredb222 节点中的 RabbitMQ 服务处于
运行状态，而在这种情况下， archpredb222 无法先行启动， 则 "--offline" 参数的添加让其可以在非运行状态下将 nodel 剥离出当前集群。
方法二：
在节点上执行：
rabbitmqctl stop_app
rabbitmqctl reset
这样也能剔除该节点

这种高可用配置只是将结构复制到另外一个集群上，但是并不会保证数据不丢失；rabbitmq可以配置镜像集群来保证数据两边一致，但是这样也会降低队列的性能，一般生产上都是使用镜像集群的配置方式

图像话界面配置：
在这里插入图片描述

这样集群基本配置完成！

安装Haproxy

10.244.124.33
10.244.124.34
上安装haproxy

user add haproxy
首先下载haproxy;`haproxy-1.5.9.tar.gz`
tar -xivf haproxy-1.5.9.tar.gz -C /opt
编译安装：
make TARGET=linux26 CPU=x86_64 PREFIX=/opt/Haproxy USE_OPENSSL=1 ADDLIB=-lz
make install  PREFIX=/opt/Haproxy
其中TARGET=Linux26 是通过uname -a 来查看Linux内核版本的
cp -r /opt/haproxy-1.5.9/examples/errorfiles/ /opt/Haproxy
常见目录：
mkdir /usr/share/haproxy
在/opt/Haproxy中建立文件夹：mkdir -p /opt/Haproxy/conf
创建配置文件：touch haproxy.cfg
配置项如下：

###########全局配置#########
global
    log     127.0.0.1       local0 #local0是设备，对应于 /etc/rsyslog.conf中的配置，默认回收info的日志级
    chroot /usr/share/haproxy
    pidfile  /usr/share/haproxy/haproxy.pid
    maxconn  100000
    user   haproxy
    group   haproxy
    daemon
    nbproc 1
    node dbrtmap_rabbitmq002
    description dbrtmap_rabbitmq002
###########默认配置#########
defaults
    log     127.0.0.1       local0
    mode    http
    option  httplog
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    option  httpclose
    option  redispatch
    timeout connect 10000
    maxconn     100000
    retries     3
    errorfile 400 /opt/Haproxy/errorfiles/400.http
    errorfile 403 /opt/Haproxy/errorfiles/403.http
    errorfile 408 /opt/Haproxy/errorfiles/408.http
    errorfile 500 /opt/Haproxy/errorfiles/500.http
    errorfile 502 /opt/Haproxy/errorfiles/502.http
    errorfile 503 /opt/Haproxy/errorfiles/503.http
    errorfile 504 /opt/Haproxy/errorfiles/504.http
####################################################################
listen admin_stats
        bind 0.0.0.0:1080           #监听端口
        stats refresh 30s           #统计页面自动刷新时间
        stats uri /haproxy?stats    #统计页面url
        stats realm Haproxy Manager #统计页面密码框上提示文本
        stats auth admin:admin      #统计页面用户名和密码设置
        #stats hide-version         #隐藏统计页面上HAProxy的版本信息

#####################我把RabbitMQ的管理界面###############################
listen rabbitmq_admin
    bind 0.0.0.0:8004
    server archpredb221 10.242.154.176:15672
    server archpredb222 10.242.154.177:15672
####################################################################
listen rabbitmq_cluster
    bind 0.0.0.0:5672
    option tcplog
    mode tcp
    timeout client  3h
    timeout server  3h
    option          clitcpka
    balance roundrobin      #负载均衡算法（#banlance roundrobin 轮询，balance source 保存session值，支持static-rr，leastconn，first，uri等参数ï    #balance url_param userid
    server   archpredb221 10.242.154.176:5672 check inter 5s rise 2 fall 3 weight 10  #check inter 2000 是检测心跳频率，rise #2是2次正确认为服务器可用，fall 3是3次失败认为服务器不可用
    server   archpredb222 10.242.154.177:5672 check inter 5s rise 2 fall 3 weight 10

配置haproxy日志

cd /etc/rsyslog.d/
touch haproxy.conf
配置：
$ModLoad imudp
$UDPServerRun 514
local0.*     /var/log/haproxy.log
#如果不加下面的的配置则除了在/var/log/haproxy.log中写入日志外，也会写入message文件
&~

------
查看rsyslog配置
vim /etc/rsyslog.conf
查看是否有这行：
$IncludeConfig /etc/rsyslog.d/*.conf

[root@dbrtmapprdapp23 rsyslog.d]# /etc/init.d/rsyslog restart
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]

重启服务
然后启动 
hproxy -f /opt/Haproxy/conf/haproxy.cng &
查看/var/log/下有没有haproxy.log
有则配置成功
两台机器配置一致即可

安装keepalived

下载软件：
ipvsadm-1.26-2.el6.x86_64.rpm
keepalived-1.2.7-3.el6.x86_64.rpm
安装：
rpm -ivh ipvsadm-1.26-2.el6.x86_64.rpm
rpm -ivh keepalived-1.2.7-3.el6.x86_64.rpm
安装完毕之后配置LVS：
iptables -t mangle -I PREROUTING -d 192.168.212.189 -p tcp -m tcp --dport 5672 -m mac ! --mac-source   06:75:76:00:12:5D  -j MARK --set-mark 0x2 
#  192.168.212.252是读VIP  00:50:56:92:44:2F 为对方的mac地址号  3为代号(本机的keepalived配置的就是自己的这个代号)
-t 表
-d 目的地址（destination）
-p 协议（protocol）
-j 执行目标（jump to target）
两台机器配置好后配置keepalived
vim /etc/keepalived/keepalived.conf

------
keepalived主节点配置：
global_defs {
   notification_email {
   }
   smtp_connect_timeout 30
   router_id 10.244.124.33
}

vrrp_instance VI_1 {
    state master
    interface eth0
    virtual_router_id 165
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
       10.244.124.36
    }
 }

virtual_server fwmark 1 {
        delay_loop 10
        lb_algo rr
        lb_kind DR
#    persistence_timeout 2
        protocol TCP
        real_server 10.244.124.33 5672 {
                weight 3
                TCP_CHECK {
                        connect_timeout 10
                        nb_get_retry 3
                    delay_before_retry 3
                        connect_port 5672
                }
        }

        real_server 10.244.124.34 5672 {
                weight 3
                TCP_CHECK {
                        connect_timeout 10
                        nb_get_retry 3
                    delay_before_retry 3
                        connect_port 5672
                }
        }

}

keepalived备节点配置：
global_defs {
   notification_email {
   }
   smtp_connect_timeout 30
   router_id 10.244.124.33
}

vrrp_instance VI_1 {
    state backup
    interface eth0
    virtual_router_id 165
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
       10.244.124.36
    }
 }

virtual_server fwmark 2 {
        delay_loop 10
        lb_algo rr
        lb_kind DR
#    persistence_timeout 2
        protocol TCP
        real_server 10.244.124.33 5672 {
                weight 3
                TCP_CHECK {
                        connect_timeout 10
                        nb_get_retry 3
                    delay_before_retry 3
                        connect_port 5672
                }
        }

        real_server 10.244.124.34 5672 {
                weight 3
                TCP_CHECK {
                        connect_timeout 10
                        nb_get_retry 3
                    delay_before_retry 3
                        connect_port 5672
                }
        }

}

在两台机器上lo网卡上增加VIP：
ip addr add 10.244.124.36/32 dev lo
然后启动keepalived:
/etc/init.d/keepalived start
还有其他命令为：
Usage: /etc/init.d/keepalived {start|stop|status|restart|condrestart|try-restart|reload|force-reload}  #　顺序：先主后从，　停顺序：先从后主
然后查看分发：
[root@dbrtmapprdapp23 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  2 rr
  -> 10.244.124.34:5672          Route   3      0          0         
  -> 10.244.124.33:5672          Local   3      0          0

高可用rabbitmq集群基本配置完毕

X-Huger

关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
0
评论
【rabbitmq】Rabbitmq+haproxy+keepalived构建双主热备高可用镜像集群

服务器准备rabbitMq主备需要两台；haproxy和keepalived共用或者分开，需要两台或者4台服务器，VIP一个，用于LVS分发软件准备需要准备软件：rabbitMq3.6.15：rabbitmq-server-generic-unix-3.6.15.tar.xz；haproxy1.5.9：haproxy-1.5.9.tar.gz；keepalived1.2.7:keep...
复制链接

扫一扫