zabbix监控脑裂

最新推荐文章于 2024-03-02 15:23:01 发布

Eternity_zzh

最新推荐文章于 2024-03-02 15:23:01 发布

阅读量158

点赞数

文章标签：服务器数据库 java

版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。

本文链接：https://blog.csdn.net/eternity_zzh/article/details/127240892

版权

zabbix监控脑裂

目录

zabbix监控脑裂

脑裂产生的原因

keepalived脑裂监控

keepalived安装

keepalived配置

让keepalived监控nginx负载均衡机

配置keepalived加入监控脚本的配置

添加监控项

添加触发器

选择邮箱媒介

脑裂

在高可用（HA）系统中，当联系2个节点的“心跳线”断开时，本来为一整体、动作协调的HA系统，就分裂成为2个独立的个体。由于相互失去了联系，都以为是对方出了故障。两个节点上的HA软件像“裂脑人”一样，争抢“共享资源”、争起“应用服务”，就会发生严重后果——或者共享资源被瓜分、两边“服务”都起不来了；或者两边“服务”都起来了，但同时读写“共享存储”，导致数据损坏（常见如数据库轮询着的联机日志出错）。
1、对付HA系统“裂脑”的对策，目前达成共识的的大概有以下几条：
（1）添加冗余的心跳线，例如：双心跳线（心跳线也HA），尽量减少“裂脑”发生几率；
（2）启用磁盘锁。正在服务一方锁住共享磁盘，“裂脑”发生时，让对方完全“抢不走”共享磁盘资源。但使用锁磁盘也会有一个不小的问题，如果占用共享盘的一方不主动“解锁”，另一方就永远得不到共享磁盘。现实中假如服务节点突然死机或崩溃，就不可能执行解锁命令。后备节点也就接管不了共享资源和应用服务。于是有人在HA中设计了“智能”锁。即：正在服务的一方只在发现心跳线全部断开（察觉不到对端）时才启用磁盘锁。平时就不上锁了
（3）设置仲裁机制。例如设置参考IP（如网关IP），当心跳线完全断开时，2个节点都各自ping一下参考IP，不通则表明断点就出在本端。不仅“心跳”、还兼对外“服务”的本端网络链路断了，即使启动（或继续）应用服务也没有用了，那就主动放弃竞争，让能够ping通参考IP的一端去起服务。更保险一些，ping不通参考IP的一方干脆就自我重启，以彻底释放有可能还占用着的那些共享资源

脑裂产生的原因

一般来说，脑裂的发生，有以下几种原因：
（1）高可用服务器对之间心跳线链路发生故障，导致无法正常通信
（2）因心跳线坏了（包括断了，老化）
（3）因网卡及相关驱动坏了，ip配置及冲突问题（网卡直连）
（4）因心跳线间连接的设备故障（网卡及交换机）
（5）因仲裁的机器出问题（采用仲裁的方案）
（6）高可用服务器上开启了 iptables防火墙阻挡了心跳消息传输
（7）高可用服务器上心跳网卡地址等信息配置不正确，导致发送心跳失败
（8）其他服务配置不当等原因，如心跳方式不同，心跳广插冲突、软件Bug等
对脑裂的监控应在备用服务器上进行，通过添加zabbix自定义监控进行。
监控什么信息呢？监控备上有无VIP地址
备机上出现VIP有两种情况：
（1）发生了脑裂
（2）正常的主备切换
监控只是监控发生脑裂的可能性，不能保证一定是发生了脑裂，因为正常的主备切换VIP也会到备份节点上。

keepalived脑裂监控

keepalived安装

//配置master

//配置网络源

[root@master ~]# dnf -y install epel-release

//安装keepalived

[root@master ~]# dnf -y install keepalived

//安装nginx

[root@master ~]# dnf -y install nginx

[root@master ~]# vim /etc/nginx/nginx.conf

server {

listen 8080 default_server; //更改端口号为8080

listen [::]:8080 default_server; //更改端口号为8080

[root@master ~]# cd /usr/share/nginx/html/

[root@master html]# mv index.html{,.bak}

[root@master html]# echo 'master' > index.html

[root@master html]# ls

404.html index.html nginx-logo.png

50x.html index.html.bak poweredby.png

[root@master html]# systemctl start nginx

[root@master html]# systemctl enable nginx

Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /usr/lib/systemd/system/nginx.service.

[root@master html]# ss -antl

State Recv-Q Send-Q Local Address:Port Peer Address:Port Process

LISTEN 0 128 0.0.0.0:10050 0.0.0.0:*

LISTEN 0 128 0.0.0.0:10051 0.0.0.0:*

LISTEN 0 128 0.0.0.0:9000 0.0.0.0:*

LISTEN 0 128 0.0.0.0:8080 0.0.0.0:*

LISTEN 0 128 0.0.0.0:22 0.0.0.0:*

LISTEN 0 80 *:3306 *:*

LISTEN 0 128 [::]:8080 [::]:*

LISTEN 0 128 *:80 *:*

LISTEN 0 128 [::]:22 [::]:*


//配置backup

//配置网络源

[root@backup ~]# dnf -y install epel-release

//安装keepalived

[root@backup ~]# dnf -y install keepalived

//安装nginx

[root@backup ~]# dnf -y install nginx

[root@backup ~]# vim /etc/nginx/nginx.conf

[root@master ~]# vim /etc/nginx/nginx.conf

server {

listen 8080 default_server; //更改端口号为8080

listen [::]:8080 default_server; //更改端口号为8080

[root@backup ~]# cd /usr/share/nginx/html/

[root@backup html]# mv index.html{,.bak}

[root@backup html]# echo 'backup' > index.html

[root@backup html]# ls

404.html index.html nginx-logo.png

50x.html index.html.bak poweredby.png

[root@backup html]# systemctl start nginx

[root@backup html]# ss -antl

State Recv-Q Send-Q Local Address:Port Peer Address:Port Process

LISTEN 0 128 0.0.0.0:8080 0.0.0.0:*

LISTEN 0 128 0.0.0.0:22 0.0.0.0:*

LISTEN 0 128 0.0.0.0:10050 0.0.0.0:*

LISTEN 0 128 [::]:8080 [::]:*

LISTEN 0 128 *:80 *:*

LISTEN 0 128 [::]:22 [::]:*

keepalived配置

//配置master

[root@master ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived


global_defs {

router_id lb01

}


vrrp_instance VI_1 {

state MASTER

interface ens33

virtual_router_id 51

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass lijiang

}

virtual_ipaddress {

192.168.205.250

}

}


virtual_server 192.168.205.250 80 {

delay_loop 6

lb_algo rr

lb_kind DR

persistence_timeout 50

protocol TCP


real_server 192.168.205.144 8080 {

weight 1

TCP_CHECK {

connect_port 80

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

}


real_server 192.168.205.147 8080 {

weight 1

TCP_CHECK {

connect_port 80

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

}

}

[root@master ~]# systemctl enable keepalived

Created symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /usr/lib/systemd/system/keepalived.service.

[root@master ~]# systemctl start keepalived

[root@master ~]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

link/ether 00:0c:29:63:98:e5 brd ff:ff:ff:ff:ff:ff

inet 192.168.205.144/24 brd 192.168.205.255 scope global dynamic noprefixroute ens33

valid_lft 1578sec preferred_lft 1578sec

inet 192.168.205.250/32 scope global ens33

valid_lft forever preferred_lft forever

inet6 fe80::9e26:e42:e48e:6000/64 scope link noprefixroute

valid_lft forever preferred_lft forever

inet6 fe80::7ea6:279d:2d84:19ea/64 scope link dadfailed tentative noprefixroute

valid_lft forever preferred_lft forever


//配置backup

[root@backup ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived


global_defs {

router_id lb02

}


vrrp_instance VI_1 {

state BACKUP

interface ens33

virtual_router_id 51

priority 90

advert_int 1

authentication {

auth_type PASS

auth_pass lijiang

}

virtual_ipaddress {

192.168.205.250

}

}


virtual_server 192.168.205.250 80 {

delay_loop 6

lb_algo rr

lb_kind DR

persistence_timeout 50

protocol TCP


real_server 192.168.205.144 8080 {

weight 1

TCP_CHECK {

connect_port 80

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

}


real_server 192.168.205.147 8080 {

weight 1

TCP_CHECK {

connect_port 80

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

}

}

[root@backup ~]# systemctl start keepalived

[root@backup ~]# systemctl enable keepalived

Created symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /usr/lib/systemd/system/keepalived.service.

[root@backup ~]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

link/ether 00:0c:29:6a:fe:0a brd ff:ff:ff:ff:ff:ff

inet 192.168.205.147/24 brd 192.168.205.255 scope global dynamic noprefixroute ens33

valid_lft 1324sec preferred_lft 1324sec

inet6 fe80::9e26:e42:e48e:6000/64 scope link dadfailed tentative noprefixroute

valid_lft forever preferred_lft forever

inet6 fe80::7ea6:279d:2d84:19ea/64 scope link noprefixroute

valid_lft forever preferred_lft forever

让keepalived监控nginx负载均衡机

//配置master

[root@master ~]# mkdir /scripts

[root@master ~]# cd /scripts/

[root@master scripts]# vim check_nginx.sh

#!/bin/bash

nginx_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bnginx\b'|wc -l)

if [ $nginx_status -lt 1 ];then

systemctl stop keepalived

fi

[root@master scripts]# chmod +x check_nginx.sh

[root@master scripts]# ll

total 4

-rwxr-xr-x. 1 root root 143 Oct 9 20:35 check_nginx.sh


[root@master scripts]# vim notify.sh

#!/bin/bash

VIP=$2

case "$1" in

master)

nginx_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bnginx\b'|wc -l)

if [ $nginx_status -lt 1 ];then

systemctl start nginx

fi

;;

backup)

nginx_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bnginx\b'|wc -l)

if [ $nginx_status -gt 0 ];then

systemctl stop nginx

fi

;;

*)

echo "Usage:$0 master|backup VIP"

;;

esac

[root@master scripts]# chmod +x notify.sh

[root@master scripts]# ll

total 8

-rwxr-xr-x. 1 root root 143 Oct 9 20:35 check_nginx.sh

-rwxr-xr-x. 1 root root 418 Oct 9 20:39 notify.sh


//配置backup

[root@backup ~]# cd /scripts/

[root@backup scripts]# vim notify.sh

#!/bin/bash

VIP=$2

case "$1" in

master)

nginx_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bnginx\b'|wc -l)

if [ $nginx_status -lt 1 ];then

systemctl start nginx

fi

;;

backup)

nginx_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bnginx\b'|wc -l)

if [ $nginx_status -gt 0 ];then

systemctl stop nginx fi

;; *)

echo "Usage:$0 master|backup VIP"

;;

esac

[root@backup scripts]# chmod +x notify.sh

[root@backup scripts]# ll

total 12

-rw-r--r--. 1 root root 0 Sep 6 18:49 check_httpd.sh

-rwxr-xr-x. 1 root root 119 Sep 6 18:54 check_proces.sh

-rwxr-xr-x. 1 root root 1854 Mar 22 2020 log.py

-rwxr-xr-x. 1 root root 422 Oct 9 20:46 notify.sh

配置keepalived加入监控脚本的配置

//配置master

[root@master ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived


global_defs {

router_id lb01

}


vrrp_script nginx_check {

script "/scripts/check_nginx.sh"

interval 1

weight -20

}


vrrp_instance VI_1 {

state MASTER

interface ens33

virtual_router_id 51

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass lijiang

}

virtual_ipaddress {

192.168.205.250

}

track_script {

nginx_check

}

notify_master "/scripts/notify.sh master"

notify_backup "/scripts/notify.sh backup"

}


virtual_server 192.168.205.250 8080 {

delay_loop 6

lb_algo rr

lb_kind DR

persistence_timeout 50

protocol TCP


real_server 192.168.205.144 8080 {

weight 1

TCP_CHECK {

connect_port 8080

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

}


real_server 192.168.205.147 8080 {

weight 1

TCP_CHECK {

connect_port 8080

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

}

}

[root@master ~]# systemctl restart keepalived


//配置master

[root@backup ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived


global_defs {

router_id lb02

}


vrrp_instance VI_1 {

state BACKUP

interface ens33

virtual_router_id 51

priority 90

advert_int 1

authentication {

auth_type PASS

auth_pass lijiang

}

virtual_ipaddress {

192.168.205.250

}

notify_master "/scripts/notify.sh master"

notify_backup "/scripts/notify.sh backup"

}


virtual_server 192.168.205.250 8080 {

delay_loop 6

lb_algo rr

lb_kind DR

persistence_timeout 50

protocol TCP


real_server 192.168.205.144 8080 {

weight 1

TCP_CHECK {

connect_port 8080

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

}


real_server 192.168.205.147 8080 {

weight 1

TCP_CHECK {

connect_port 8080

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

}

}

[root@backup ~]# systemctl restart keepalived


//脑裂进行监控脚本

[root@backup ~]# cd /scripts/

[root@backup scripts]# vim check_keepalived.sh

#!/bin/bash

if [ `ip a show ens33 |grep 192.168.205.250|wc -l` -ne 0 ]

then

echo "keepalived is error!"

else

echo "keepalived is OK !"

fi

[root@backup scripts]# chmod +x check_keepalived.sh


[root@backup scripts]# vim /usr/local/etc/zabbix_agentd.conf

UnsafeUserParameters=1

UserParameter=check_keepalived,/scripts/check_keepalived.sh

添加监控项

添加触发器

选邮箱媒介

添加动作

模拟脑裂

评论

被折叠的条评论为什么被折叠?

到【灌水乐园】发言

查看更多评论

添加红包

成就一亿技术人!

hope_wisdom

发出的红包

实付元

使用余额支付

点击重新获取

扫码支付

钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。