Ceph RGW高可用HA集群keepalived+Haproxy

我不是程序猿

已于 2022-06-13 14:30:31 修改

阅读量1k

点赞数 1

分类专栏： Ceph 文章标签：负载均衡

于 2022-04-20 11:27:10 首次发布

本文链接：https://blog.csdn.net/qq_25126841/article/details/124293218

版权

Ceph 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

在这里插入图片描述
部署了入口服务的主机有 N 个。每个主机都有一个 haproxy 守护进程和一个 keepalived 守护进程。一次仅在其中一台主机上自动配置虚拟 IP。

每个keepalived 守护进程每隔几秒检查同一主机上的haproxy 守护进程是否正在响应。Keepalived 还将检查主 keepalived 守护进程是否正常运行。如果“主”keepalived 守护进程或活动 haproxy 没有响应，则以备份模式运行的剩余的 keepalived 守护进程之一将被选为主进程，并且虚拟 IP 将移动到该节点。

主动 haproxy 就像一个负载均衡器，在所有可用的 RGW 守护进程、NFS守护进程之间分配所有请求
参考文档：

部署Keepalived

下载并安装

[root@ceph-node3 ~]# mkdir -p /data/keepalived
[root@ceph-node3 ~]# cd /data/keepalived
[root@ceph-node3 keepalived]#  wget https://www.keepalived.org/software/keepalived-2.2.4.tar.gz
[root@ceph-node3 keepalived]#  tar -zxvf keepalived-2.2.4.tar.gz
[root@ceph-node3 keepalived]#  cd keepalived-2.2.4
#指定安装路径
[root@ceph-node3 keepalived-2.2.4]# ./configure --prefix=/usr/local/keepalived-2.2.4 
#如果配置报错请安装openssl-devel
configure: error: 
  !!! OpenSSL is not properly installed on your system. !!!
  !!! Can not include OpenSSL headers files.            !!!
[root@ceph-node2 keepalived-2.2.4]# yum install openssl-devel
[root@ceph-node2 keepalived-2.2.4]# make && make install

配置keepalived

[root@ceph-node3 init.d]# cd /usr/local/keepalived-2.2.4
[root@ceph-node3 keepalived-2.2.4]# cp etc/keepalived/keepalived.conf /etc/keepalived
[root@ceph-node3 keepalived-2.2.4]# cp etc/sysconfig/keepalived /etc/sysconfig/
#编辑system service文件，实现服务自启动
[root@ceph-node3 keepalived-2.2.4]# vim /lib/systemd/system/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target 
Wants=network-online.target 
Documentation=man:keepalived(8)
Documentation=man:keepalived.conf(5)
Documentation=man:genhash(1)
Documentation=https://keepalived.org

[Service]
Type=forking
PIDFile=/run/keepalived.pid
KillMode=process
EnvironmentFile=-/usr/local/keepalived-2.2.4/etc/sysconfig/keepalived
ExecStart=/usr/local/keepalived-2.2.4/sbin/keepalived  $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target
[root@ceph-node3 keepalived-2.2.4]# systemctl daemon-reload
[root@ceph-node3 keepalived-2.2.4]# systemctl enable keepalived
#编辑keepalived.conf文件
[root@ceph-node3 keepalived-2.2.4]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
#全局定义配置
global_defs {
   #邮件通知配置  
   #notification_email {
   #  acassen@firewall.loc
   #  failover@firewall.loc
   #  sysadmin@firewall.loc
   #}
   #notification_email_from Alexandre.Cassen@firewall.loc
   #smtp_server 192.168.200.1
   #smtp_connect_timeout 30
   
   #负载均衡唯一标识
   router_id ceph
   vrrp_skip_check_adv_addr
   vrrp_script
   vrrp_garp_interval 0
   vrrp_gna_interval 0
   script_user root
   enable_script_security
}
vrrp_script chk_haproxy {                        #Haproxy服务启动
   script "/etc/keepalived/check_haproxy.sh"     #监控haproxy进程的脚本, 根据自己的实际路径放置
   interval 5
   weight -30
}
#VRRP定义模块
vrrp_instance VI_1 {
    state BACKUP
    interface enp1s0f0
    virtual_router_id 101
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {                #调用haproxy进程检测脚本
       chk_haproxy
    }
    virtual_ipaddress {
       10.18.101.12/24 brd 10.18.101.255 dev enp1s0f0
    }
    nopreempt              #非抢占模式:允许低优先级节点继续担任MASTER
    preempt_delay 2        #抢占延迟时间：发现低优先级MASTER后多少秒开始抢占
}
[root@ceph-node3 keepalived-2.2.4]# systemctl start keepalived
#检查VIP地址是否漂移
[root@ceph-node3 sysconfig]# ip addr|grep 10.18.101.12
    inet 10.18.101.12/24 brd 10.18.101.255 scope global secondary enp1s0f0

部署Haproxy

下载并安装

[root@ceph-node3 ~]# yum install -y haproxy
[root@ceph-node3 ~]# systemctl enable haproxy

配置haproxy

[root@ceph-node3 ~]# vim /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4096
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         1m
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 1m
    timeout check           5s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend   rgw *:8080
     use_backend rgw
frontend   nfs *:2050
     use_backend nfs
#frontend  main *:5000
#    acl url_static       path_beg       -i /static /images /javascript /stylesheets
#    acl url_static       path_end       -i .jpg .gif .png .css .js
#
#    use_backend static          if url_static
#    default_backend             app

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
#backend static
#    balance     roundrobin
#    server      static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend rgw
    balance roundrobin
    server  rgw-node1 10.18.101.9:80 check inter 2000 fall 5 maxconn 1000
    server  rgw-node2 10.18.101.6:80 check inter 2000 fall 5 maxconn 1000
    server  rgw-node3 10.18.101.5:80 check inter 2000 fall 5 maxconn 1000
backend nfs
    balance roundrobin
    server  nfs-node1 10.18.101.9:2049 check inter 2000 fall 5 maxconn 1000
    server  nfs-node2 10.18.101.6:2049 check inter 2000 fall 5 maxconn 1000
    server  nfs-node3 10.18.101.5:2049 check inter 2000 fall 5 maxconn 1000
#状态检测
listen status 0.0.0.0:8081        #监控页面的端口
    mode http
    stats enable
    stats refresh 10s                  #统计页面自动刷新时间
    stats uri /haproxy-admin           #监控页面的访问地址
    stats realm Haproxy\ Statistics    #统计页面密码框上提示文本
    stats auth admin:W@gs2021*             #统计页面用户名和密码设置
    stats hide-version                 #隐藏统计页面上HAProxy的版本信息
[root@ceph-node3 ~]# systemctl daemon-reload
[root@ceph-node3 ~]# systemctl start haproxy

配置检测脚本

[root@ceph-node3 ~]# vim /etc/keepalived/check_haproxy.sh
#!/bin/bash
if [ $(ps -C haproxy --no-header | wc -l) -eq 0 ]; then    ###判断haproxy是否已经启动
    systemctl start haproxy                                ###如果没有启动，则启动haproxy程序
fi
sleep 2                                                    ###睡眠两秒钟，等待haproxy完全启动
if [ $(ps -C haproxy --no-header | wc -l) -eq 0 ]; then    ###判断haproxy是否已经启动
    systemctl stop keepalived                              ###如果haproxy没有启动起来，则将keepalived停掉，则VIP自动漂移到另外一台haproxy机器,实现了对haproxy的高可用
fi

验证故障转移

#断开haproxy
[root@ceph-node3 ~] systemctl stop haproxy
#验证当前节点是否有VIP地址
[root@ceph-node3 ~] ip addr|grep 10.18.101.12
#启动haproxy
[root@ceph-node3 ~] systemctl start haproxy

keepalived官网文档：https://www.keepalived.org/doc
haproxy官方文档: https://www.haproxy.org/
keepalived+nginx实现高可用负载均衡:https://segmentfault.com/a/1190000022186901