ubuntu lvs keepalived mysql_Lvs+Keepalived实现MySQL高可用

LVS+Keepalived+MySQL高可用配置

本文所有配置前提是已实现MySQL双主备份(MySQL双主)

安装前的准备:

VIP:192.168.0.201

Keepalived:

Keepalived-Master:192.168.0.241

Keepalived-Backup:192.168.0.277

RealServer:

MySQL1:192.168.0.225

MySQL2:192.168.0.226

kernel-devel各版本下载地址:http://rpmfind.net/linux/rpm2html/search.php?query=kernel-devel

keepalived下载地址:http://www.keepalived.org/download.html

kernel-devel.2.6.32-431.el6.x86_64.rpm (根据系统实际内核版本)

keepalived-1.2.7.tar.gz

ipvsadm-1.26.tar.gz

依赖包 :

yum install openssl-devle popt-devel popt-static libnl*

一、软件安装

1、安装kernel-devel:

rpm -ivh kernel-devel-2.6.32-431.29.2.el6.x86_64.rpm

2、安装lvs管理工具ipsadm:

tar zxvf ipvsadm-1.26.tar.gz

cd ipvsadm

make

make install

3、安装keepalived:

tar zxvf keepalived-1.2.7.tar.gz

cd keepalived

./configure --prefix=/usr/local/keepalived

make

make install

4、将keepalived添加到系统服务:

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/cp/usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/cp/usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/mkdir/etc/keepalived

cp/usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

二、配置

1、MySQL服务器中的配置 :

将realserver脚本放在/etc/init.d/目录下，执行即可:

#!/bin/sh

VIP=192.168.0.201sh/etc/rc.d/init.d/functions

case"$1" in

#禁用本地的ARP请求、绑定本地回环地址

start)/sbin/ifconfig lo down/sbin/ifconfig lo up

echo"1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo"2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo"1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo"2" >/proc/sys/net/ipv4/conf/all/arp_announce/sbin/sysctl -p >/dev/null 2>&1

/sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 up #在回环地址上绑定VIP，设定掩码，与Direct Server(自身)上的IP保持通信

/sbin/route add -host $VIP dev lo:0

echo"LVS-DR real server starts successfully.\n";;

stop)/sbin/ifconfig lo:0 down/sbin/route del $VIP >/dev/null 2>&1echo"1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo"2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo"1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo"2" >/proc/sys/net/ipv4/conf/all/arp_announce

echo"LVS-DR real server stopped.\n";;

status)

isLoOn=`/sbin/ifconfig lo:0 | grep "$VIP"`

isRoOn=`/bin/netstat -rn | grep "$VIP"`if [ "$isLoON" == "" -a "$isRoOn" == ""]; then

echo"LVS-DR real server has run yet."

elseecho"LVS-DR real server is running."fi

exit3;;*)

echo"Usage: $0 {start|stop|status}"exit1esac

exit 0

然后使用 ip addr 命令查看VIP是否绑定:

1: lo: mtu 16436qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet127.0.0.1/8scope host lo

inet192.168.0.201/32 brd 192.168.0.201 scope globallo:0

inet6 ::1/128scope host

valid_lft forever preferred_lft forever2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 52:54:00:59:f1:3f brd ff:ff:ff:ff:ff:ff

inet192.168.0.225/24 brd 192.168.0.255 scope globaleth0

inet6 fe80::5054:ff:fe59:f13f/64scope link

valid_lft forever preferred_lft forever

可以看到vip已经绑定在了lo上。两台msyql服务器执行同样的操作。

2、Keepalived的配置:

2.1、首先关闭系统防火墙iptables和selinux

2.2、开启路由转发及非本机ip绑定支持功能:

echo 1 >/proc/sys/net/ipv4/ip_forward

echo1 >/proc/sys/net/ipv4/ip_nonlocal_bind

2.3、修改Keepalived的配置文件/etc/keepalived/keepalived.conf:

! Configuration File forkeepalived

global_defs {

notification_email {

123@sina.com

}

notification_email_from 321@sina.com

smtp_server192.168.0.241smtp_connect_timeout30router_id LVS1

}

vrrp_instance VI_1 {

state BACKUP ＃角色,若都为BACKUP，则表示当其中一台挂掉时，以另一台为MASTER，(即不会主动抢VIP)

interface eth0 ＃主备之间健康检查的网卡

virtual_router_id51 ＃主备id相同priority100advert_int1authentication {

auth_type PASS ＃主备之间通信认证

auth_pass0711}

virtual_ipaddress {192.168.0.201}

}

virtual_server192.168.0.201 3306{

delay_loop6lb_algo rr

lb_kind DR

nat_mask255.255.255.0persistence_timeout50protocol TCP

real_server192.168.0.225 3306{

weight1TCP_CHECK {

connect_timeout3nb_get_retry3delay_before_retry3connect_prot3306}

}

real_server192.168.0.226 3306{

weight1TCP_CHECK {

connect_timeout3nb_get_retry3delay_before_retry3connect_prot3306}

}

}

keepalived的Master和Backup配置文件不同的是，

Master: state为MASTER，priority的值比BACKUP大。

BACKUP:state为BACKUP，priority的值比MASTER小。

2.4、启动Keepalived:

service keepalived start

然后使用ipvsadm工具查看集群列表:

主、备结果一样。

[root@localhost ~]#ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags->RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP192.168.0.201:3306 rr persistent 50

-> 192.168.0.225:3306 Route 10 0-> 192.168.0.226:3306 Route 1 0 0

查看ip地址情况:

MASTER:

[root@localhost ~]#ip addr

1: lo: mtu 16436qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet127.0.0.1/8scope host lo

inet6 ::1/128scope host

valid_lft forever preferred_lft forever2: p4p1: mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 64:00:6a:34:c3:e2 brd ff:ff:ff:ff:ff:ff

inet192.168.0.241/24 brd 192.168.0.255 scope globalp4p1

inet192.168.0.201/32 scope globalp4p1

inet6 fe80::6600:6aff:fe34:c3e2/64scope link

valid_lft forever preferred_lft forever

BACKUP：

[root@localhost ~]#ip addr

1: lo: mtu 16436qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet127.0.0.1/8scope host lo

inet6 ::1/128scope host

valid_lft forever preferred_lft forever2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 52:54:00:b3:34:e6 brd ff:ff:ff:ff:ff:ff

inet192.168.0.227/24 brd 192.168.0.255 scope globaleth0

inet6 fe80::5054:ff:feb3:34e6/64scope link

valid_lft forever preferred_lft forever

可以看到，MASTER上绑定了VIP，而BACKUP没有。

3、测试:

3.1keepalived主从测试:

把MASTER的keepalived服务停掉，service keepalived stop ，然后再查看ip:

[root@localhost ~]#service keepalived stop

Stopping keepalived: [ OK ]

[root@localhost~]#ip addr

1: lo: mtu 16436qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet127.0.0.1/8scope host lo

inet6 ::1/128scope host

valid_lft forever preferred_lft forever2: p4p1: mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 64:00:6a:34:c3:e2 brd ff:ff:ff:ff:ff:ff

inet192.168.0.241/24 brd 192.168.0.255 scope globalp4p1

inet6 fe80::6600:6aff:fe34:c3e2/64scope link

valid_lft forever preferred_lft forever

发现VIP没了。此时再查看BACKUP的ip:

[root@localhost ~]#ip addr

1: lo: mtu 16436qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet127.0.0.1/8scope host lo

inet6 ::1/128scope host

valid_lft forever preferred_lft forever2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 52:54:00:b3:34:e6 brd ff:ff:ff:ff:ff:ff

inet192.168.0.227/24 brd 192.168.0.255 scope globaleth0

inet192.168.0.201/32 scope globaleth0

inet6 fe80::5054:ff:feb3:34e6/64scope link

valid_lft forever preferred_lft forever

VIP成功漂移到了BACKUP上。

同时，通过keepalived的日志也可以看出，当master停掉后，backup主动成为master:

cat /var/log/messages

Jan 25 16:18:28 localhost Keepalived_healthcheckers[1199]: SMTP connection ERROR to [192.168.0.227]:25.

Jan25 16:42:41 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Transition to MASTER STATE

Jan25 16:42:42 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Entering MASTER STATEJan25 16:42:42 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) setting protocol VIPs.

Jan25 16:42:42 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.201Jan25 16:42:42 localhost Keepalived_healthcheckers[1199]: Netlink reflector reports IP 192.168.0.201added

Jan25 16:42:47 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.201

3.2MySQL高可用测试:

通过VIP连接mysql:

localhost:~ ahaii$ mysql -h192.168.0.201 -uroot -p

Enter password:

Welcome to the MySQL monitor. Commands end with ;or\g.

Your MySQL connection idis 643Server version:5.5.54-log Source distribution

Copyright (c)2000, 2015, Oracle and/orits affiliates. All rights reserved.

Oracleis a registered trademark of Oracle Corporation and/orits

affiliates. Other names may be trademarks of their respective

owners.

Type'help;' or '\h' for help. Type '\c'to clear the current input statement.

mysql>

连接成功。

同时在keepalived-master上查看lvs分配情况:

[root@localhost ~]#ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags->RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP192.168.0.201:3306 rr persistent 50

-> 192.168.0.225:3306 Route 10 0-> 192.168.0.226:3306 Route 1 10

[root@localhost~]#

显示现在连接的是226的那台mysql服务器。

此时，将226服务器上的mysql服务停掉，然后再通过vip连接mysql，并查看lvs分配情况:

依然能通过vip连接mysql:

localhost:~ ahaii$ mysql -h192.168.0.201 -uroot -p

Enter password:

Welcome to the MySQL monitor. Commands end with ;or\g.

Your MySQL connection idis 37014Server version:5.5.54-log Source distribution

Copyright (c)2000, 2015, Oracle and/orits affiliates. All rights reserved.

Oracleis a registered trademark of Oracle Corporation and/orits

affiliates. Other names may be trademarks of their respective

owners.

Type'help;' or '\h' for help. Type '\c'to clear the current input statement.

mysql>

此时，226已被keepalived集群剔除掉，并且客户端连接的是225服务器 ：

[root@localhost ~]#ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags->RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP192.168.0.201:3306 rr persistent 50

-> 192.168.0.225:3306 Route 1 1 0

当把226服务器的mysql服务重新启动时，keepalived又将其加入到集群中:

[root@localhost ~]#ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags->RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP192.168.0.201:3306 rr persistent 50

-> 192.168.0.225:3306 Route 1 10-> 192.168.0.226:3306 Route 1 0 0

keepalived剔除、添加226服务器的日志记录如下:

Jan 25 17:00:27 localhost Keepalived_healthcheckers[1429]: TCP connection to [192.168.0.226]:3306failed !!!

Jan25 17:00:27 localhost Keepalived_healthcheckers[1429]: Removing service [192.168.0.226]:3306 from VS [192.168.0.201]:3306Jan25 17:03:27 localhost Keepalived_healthcheckers[1429]: TCP connection to [192.168.0.226]:3306success.

Jan25 17:03:27 localhost Keepalived_healthcheckers[1429]: Adding service [192.168.0.226]:3306 to VS [192.168.0.201]:3306

疑惑:

1、客户点请求VIP时，由于调度器和realserver都有VIP，如何保证是调度器接收请求，而不是realserver？

为了保证是调度器接收客户端的请求，在realserver上都做了如下配置:

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo"2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo"1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo"2" >/proc/sys/net/ipv4/conf/all/arp_announce

用arp_ignore参数设定当请求的目标地址是本机IP时，对其ARP的查询作出不同级别的响应。

1:表示只回答目标IP地址是本地网卡的arp请求

2:表示只回答目标IP地址是本地网卡的arp请求，并且来访IP与本机IP在同一网段内。

用arp_announce参数设定本机向外宣告自身IP-MAC时的级别。

1:表示尽量避免响应ARP请求中MAC不是本网卡的。如一个主机有多块网卡，其中一块网卡接收到了ARP请求，发现所请求的MAC是本机另一块网卡的，这个时候接收到ARP请求的这块网卡就尽量避免响应。

2:表示总是使用最合适的网卡来响应。一个主机有多块网卡，其中一块网卡接收到了ARP请求，发现所请求的MAC是本机另一块网卡的，这个时候接收到ARP请求的这块网卡就一定不响应，只有发现请求的MAC是自己的才给与响应。

以上设置，就是为了接收客户点请求的是调度器的VIP，而不是realserver的VIP。

2、为什么负载均衡器和realserver上都有VIP呢？

当均衡器收到客户端请求后，根据配置的算法选取一台realserver，将客户端请求报文的目标MAC地址修改为选取的realserver的MAC地址后重新封装该数据帧并发送给选取的realserver。realserver收到该数据帧后检查IP报文发现该报文的目标IP(VIP)是自己，然后对该请求报文做出响应，将响应结果直接发送给客户端(整个过程中源地址不变)。

以上是整个安装配置和测试的过程。