FreeSwitch 使用keepalived进行主备切换高可用部署

FreeSWITCH的高可用部署方式有两种：主备切换和负载均衡，官方文档介绍的主备切换部署是采用Corosync & Pacemaker，负载均衡采用前置opensips。但对使用keepalived进行主备切换的高可用方式没有介绍，同时网上对该种部署方式也没有介绍。

本人对Corosync & Pacemaker不熟悉，目前在职的公司web应用大部分采用keepalived+haproxy，所以对keepalived稍微熟悉一点，因此尝试使用keepalived进行freeswitch进行主备切换的高可用部署。

使用keepalived进行freeswitch进行主备切换的高可用部署比较简单，本文介绍的部署方案有两个亮点（自以为）：1、主节点不抢占VIP，否则可能导致主节点恢复后VIP切换导致正在交互的SIP信令处理失败；2、FS可用性检测脚本和主备切换后话务接管脚本。

一、前提

1、freeswicth和keepalived均能通过服务启动；

2、两个节点freeswitch连接同一个mysql数据库；

二、环境

OS：Red Hat Enterprise Linux Server release 7.9

A节点IP：100.207.104.77

B节点IP：100.207.104.78

VIP：100.207.104.89

freeswitch域名：callcenter.xxxxx.com  ---域名也可以直接使用VIP ，---域名暂无启用。

三、配置

3.1、允许应用绑定非本机IP

在两个节点均执行以下命令：

echo 'net.ipv4.ip_nonlocal_bind=1' >> /etc/sysctl.conf

 cat > /etc/sysctl.conf << EOF
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
kernel.sysrq = 1
kernel.pid_max = 524288
fs.file-max = 6553600
kernel.sem = 500 256000 250 8192
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_retries2 = 5
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 262144
net.core.wmem_max = 262144
net.core.somaxconn = 1024
vm.swappiness = 0
net.ipv4.ip_nonlocal_bind=1
net.ipv4.neigh.default.gc_thresh1 = 10240
net.ipv4.neigh.default.gc_thresh2 = 20480
net.ipv4.neigh.default.gc_thresh3 = 40960
EOF

sysctl -p

3.2 配置freeswitch

修改“/usr/local/freeswitch/conf/vars.xml ”   ---具体路径视实际情况

将“local_ip_v4”的值修改为VIP：“100.207.104.89”

将“domain”的值修改为：“callcenter.xxxxx.com” ---域名暂无启用。

freeswitch 使用systemctl 启用服务方法

cat >/lib/systemd/system/freeswitch.service<<EOF
[Unit]
Description=FreeSWITCH
After=syslog.target network.target
After=postgresql.service postgresql-9.3.service postgresql-9.4.service mysqld.service httpd.service
[Service]
User=appdeploy
EnvironmentFile=-/usr/local/freeswitch/etc/sysconfig/freeswitch
WorkingDirectory=/usr/local/freeswitch
ExecStart=/usr/local/freeswitch/bin/freeswitch -nc -nf $FREESWITCH_PARAMS
ExecReload=/usr/bin/kill -HUP $MAINPID
#ExecStop=/usr/local/freeswitch/bin/freeswitch -stop $FREESWITCH_PARAMS
ExecStop=/bin/kill -9 $MAINPID


[Install]
WantedBy=multi-user.target

EOF

freeswitch启动、检查、停止、重启命令

systemctl start freeswitch

systemctl status freeswitch

systemctl stop freeswitch

systemctl restart freeswitch

开机自启动

systemctl enable freeswitch

关闭开机自启动

systemctl disable freeswitch

加载配置
systemctl daemon-reload

过滤查看启动项
systemctl list-unit-files --type=service |grep enabled

3.3 安装、配置keepalived

下载 keepalived-2.2.7.tar.gz
wget https://www.keepalived.org/software/keepalived-2.2.7.tar.gz  --no-check-certificate

安装keepalived-2.2.7.tar.gz

yum install curl gcc openssl-devel libnl3-devel net-snmp-devel

解压：tar zxvf keepalived-2.2.7.tar.gz

tar zxvf keepalived-2.2.7.tar.gz
cd keepalived-2.2.7
./configure --prefix=/app/keepalived
make
make install

systemctl 启动 keepalived服务

cat > /lib/systemd/system/keepalived.service <<EOF

[Unit]
Description=keepalive
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/app/keepalived/sbin/keepalived -f /app/keepalived/etc/keepalived/keepalived.conf -p /app/keepalived/run/keep.pid -r /app/keepalived/run/vrrp.pid
ExecStop=/bin/kill -HUP $MAINPID
Restart=always
User=appdeploy
Group=mwopr

[Install]
WantedBy=multi-user.target

EOF

keepalived 要想使用普通用户运行，必须使用setcap授权

setcap cap_net_admin,cap_net_raw=eip /app/keepalived/sbin/keepalived

setcap 'CAP_SETUID+eip CAP_SETGID+eip CAP_NET_RAW+eip CAP_NET_ADMIN+eip cap_net_bind_service+eip'  /app/keepalived/sbin/keepalived

CAP_NET_ADMIN:允许执行网络管理任务
CAP_NET_RAW:允许使用原始套接字

CAP_SETGID: 允许改变进程的组ID

CAP_SETUID: 允许改变进程的用户ID

CAP_NET_BIND_SERVICE: 允许绑定到小于1024的端口

A、B节点 创建日志和进程ID目录：

mkdir -p /app/keepalived/log

mkdir -p /app/keepalived/run

keepalived启动、检查、停止、重启命令

systemctl start keepalived

systemctl status keepalived

systemctl stop keepalived

systemctl restart keepalived

开机自启动

systemctl enable keepalived

关闭开机自启动

systemctl disable keepalived

加载配置
systemctl daemon-reload

过滤查看启动项
systemctl list-unit-files --type=service |grep enabled

配置keepalived

keepalived默认配置文件路径“/app/keepalived/etc/keepalived/keepalived.conf ”

A节点keepalived配置：100.207.104.77

cat > /app/keepalived/etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
    #每个keepalived取个不同名称
    router_id 40
}

vrrp_script check_fs {
    script "/app/keepalived/script/check_fs.sh"
    interval 1
    weight 2
}

vrrp_instance VI_1 {
    # MASTER为主实例,BACKUP 为从实例
    state MASTER
    # 网卡名称
    interface eth0
    # 主备这里要配置为同样的
    virtual_router_id 68
    # 优先级,主要高于备. 一般主配置为100  备配置为80
    priority 100
    advert_int 1
    # 主动抢占,主备都开启的话,服务器抢占过去后,要等这个服务器keepalived停掉才会漂移到另一台
    nopreempt
    authentication {
        # 主备必须配置成同样的
        auth_type PASS
        # 主备必须配置成同样的
        auth_pass 1111
    }

    track_script {
        check_fs
    }

    virtual_ipaddress {
      # vip,主备必须配置一样
      10.207.104.89
    }
    notify_master "/app/keepalived/script/fs_recover.sh"
}
EOF

添加普通用户启动权限

# 全局定义
global_defs {
  # 路由id
  router_id LVS_DEVEL
  # 设置运行脚本默认用户和组。如果没有指定，则默认用户为keepalived_script(需要该用户存在)，否则为root用户。默认groupname同username
  script_user appdeploy
  # 如果脚本路径的任一部分对于非root用户来说，都具有可写权限，则不会以root身份运行脚本
  enable_script_security
}

# 全局定义
global_defs {
  notification_email {
    acassen@firewall.loc
    failover@firewall.loc
    sysadmin@firewall.loc
  }
  notification_email_from Alexandre.Cassen@firewall.loc
  smtp_server 192.168.0.123
  smtp_connect_timeout 30
  # 路由id
  router_id LVS_DEVEL
  # 设置运行脚本默认用户和组。如果没有指定，则默认用户为keepalived_script(需要该用户存在)，否则为root用户。默认groupname同username
  script_user appdeploy
  # 如果脚本路径的任一部分对于非root用户来说，都具有可写权限，则不会以root身份运行脚本
  enable_script_security
}
# 检测Nginx是否还存活的脚本
vrrp_script check_nginx {
  # 脚本文件的位置
  script "/app/app/keepalived/etc/keepalived/check_nginx.sh"
  interval 2 # 检测脚本执行的间隔
  weight 2 # 设置当服务器的权重
}
# 虚拟IP的配置
vrrp_instance VI_1 {
  state BACKUP # 表示当前服务器是MASTER还是BACKUP
  interface eth0 # 绑定的网卡名称
  virtual_router_id 51 # 主、备机的virtual_router_id必须相同
  priority 80 # 主、备机取不同的优先级，主机值较大，备份机值较小
  advert_int 1 # 每隔多久发送一次心跳，默认是1秒
  # 权限校验
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  # 虚拟IP地址
  virtual_ipaddress {
    192.168.3.20
  }
  # 指定监控脚本
  track_script {
    # 这里的check_nginx对应上面的vrrp_script的名字
    check_nginx
  }
}

B节点keepalived配置：100.207.104.78

cat > /app/keepalived/etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
    #每个keepalived取个不同名称
    router_id 40
}

vrrp_script check_fs {
    script "/app/keepalived/script/check_fs.sh"
    interval 1
    weight 2
}

vrrp_instance VI_1 {
    # MASTER为主实例,BACKUP 为从实例
    state BACKUP
    # 网卡名称
    interface eth0
    # 主备这里要配置为同样的
    virtual_router_id 68
    # 优先级,主要高于备. 一般主配置为100  备配置为80
    priority 80
    advert_int 1
    # 主动抢占,主备都开启的话,服务器抢占过去后,要等这个服务器keepalived停掉才会漂移到另一台
    nopreempt
    authentication {
        # 主备必须配置成同样的
        auth_type PASS
        # 主备必须配置成同样的
        auth_pass 1111
    }

    track_script {
        check_fs
    }

    virtual_ipaddress {
      # vip,主备必须配置一样
      10.207.104.89
    }
    notify_master "/app/keepalived/script/fs_recover.sh"
}
EOF

3.4 检测脚本

A、B节点 检测脚本路径：

mkdir -p /app/keepalived/script/
检测可用性脚本

cat > /app/keepalived/script/check_fs.sh << EOF
#!/bin/sh
FS_CLI_PROG='/usr/local/bin/fs_cli'
FS_CLI_HOST='127.0.0.1'
FS_CLI_PORT='8021'
FS_CLI_PASS='ClueCon'
PROFILES='10.207.104.89'
VIP='10.207.104.89'

fs_cli() {
   $FS_CLI_PROG -H $FS_CLI_HOST -P $FS_CLI_PORT -p $FS_CLI_PASS -x "$1"
}

sofia_profile_started() {
   fs_cli "sofia xmlstatus" | grep "<name>$1</name>" | wc -l
}

save_log(){
   count=1
   str_tmp="`date +%Y-%m-%d_%H:%M:%S` "
   while [ $# -ge 1 ];do
        str_tmp="$str_tmp $1"
        count=count+1
        shift
   done
   echo $str_tmp >>/app/keepalived/log/check_fs_`date +%Y-%m-%d`.log
}

check_vrrp(){
   ip a|grep $VIP|wc -l
}

check_fs_service(){
   ps -ef |grep freeswitch.service|grep -v 'grep'|wc -l
}

# fs_cli "sofia recover"
for p in $PROFILES; do
if [ `sofia_profile_started "$p"` -eq 0 ]; then
# echo "$p DOWN"
      log_str="$p DOWN"
      save_log $log_str
   if [ `check_vrrp` -eq 1 ];then
      save_log "本机已经绑定VRRP，即将重启keepalived和FreeSWITCH。"
      #service keepalived restart
      systemctl restart keepalived
      save_log "vrrp切换完成！"
   if [ `check_fs_service` -eq 1 ];then
      save_log "freeswitch服务正在操作中。"
   else
      #service freeswitch restart
      systemctl restart keepalived
      save_log "freeswitch重启成功！"
   fi
else
   if [ `check_fs_service` -eq 1 ];then
      save_log "freeswitch服务正在操作中。 "
   else
      save_log "本机没有绑定VRRP，重启FreeSWITCH。"
      #service freeswitch restart
      systemctl restart keepalived
      save_log "freeswitch重启成功！"
   fi
fi
exit 1
fi
done
   save_log "freeswitch状态检测:OK!"
#echo "OK"
exit 0
EOF

切换为主节点后恢复通话脚本

cat > /app/keepalived/script/fs_recover.sh << EOF
#!/bin/sh
FS_CLI_PROG='/usr/local/bin/fs_cli'
FS_CLI_HOST='127.0.0.1'
FS_CLI_PORT='8021'
FS_CLI_PASS='ClueCon'
PROFILES='10.207.104.89'
VIP='10.207.104.89'

fs_cli() {
   $FS_CLI_PROG -H $FS_CLI_HOST -P $FS_CLI_PORT -p $FS_CLI_PASS -x "$1"
}

save_log(){
   counti=1
   str_tmp="`date +%Y-%m-%d_%H:%M:%S` "
   while [ $# -ge 1 ];do
      str_tmp="$str_tmp $1"
      count=count+1
      shift
   done
   echo $str_tmp >>/app/keepalived/log/check_fs_`date +%Y-%m-%d`.log
}
save_log "本节点切换为主用状态，开始接管切换前的通话。"
fs_cli "sofia recover"
fs_cli "raloadxml"
exit 0
EOF