###############keepalived+lvs#############
一.知识梳理
LVS+keepalived基于完成开源软件的构架实现负载均衡高可用
1.LVS
LVS是Linux Virtual Server的简写,即Linux虚拟服务器,是一个虚拟的服务集群系统。有三种负载均衡技术(VS/NAT、VS/TUN、VS/DR),八种调度算法。
2.keeplived
keepalived在这里主要做realserver健康检查以及loadbalance主机和backup主机之间failover的实现。
keepalived是一个基于VRRP协议来实现WEB服务高可用方案,可以用来避免单点故障,一个web服务器至少有两台服务器运行keepalived,一台主服务器(MASTER),一台备用服务器(BACKUP),但是对外只表现一个vip,主服务器会发送特定消息给备用服务器,当备用服务器收到消息时,备用服务器就会接管ipswich,继续提供服务,从而保证了高可用性。
3.kepalive工作原理
Layer3:keepalived使用layer3的方式工作时,keepalived会定期向服务器群中发送一个ICMP的数据包(即我们平时用的ping程序),如果发现某台服务器的IP地址没有激活,keepalived便会报告这台服务器是小,并将他从服务器群中剔除。Layer3的方式是以服务器的IP第孩子是否有效作为服务器工作正常与否的标准。
Layer4:主要以TCP端口的状态来决定服务器工作正常与否。如web sercer的服务端口一般是80.如果keepalived检测到80端口没有启动,则keepalived将这台服务器从服务群中删除。
Layer5:layer5就是工作载具体的应用层,比layer3,4要复杂一点,载网络上占用的宽带也要打一些。Keepalived将根据用户的设定检查服务器的运行是否正常。如果设定不相符,则keepalived将把服务器从群中踢除。
二.实现
1.lvs配置
server1
[root@server1 ha.d]# yum install -y ipvsadm
[root@server1 ha.d]# ipvsadm -A -t 172.25.38.100:80 -s rr
##添加策略 指定vip 172.25.66.100:80,-t指tcp,-s rr 指定轮询
[root@server1 ha.d]# ip addr add 172.25.66.100/24 dev eth0 ##添加虚拟ip
[root@server1 ha.d]# ipvsadm -L ##列出ipvsadm策略
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:http rr
[root@server1 ha.d]# ipvsadm -a -t 172.25.66.100:80 -r 172.25.66.2:80 -g
[root@server1 ha.d]# ipvsadm -a -t 172.25.66.100:80 -r 172.25.66.3:80 -g
##将虚拟ip映射到真实的ip,-r指定realserver -t指定tcp协议 -g指dr模式
[root@server1 ha.d]# ipvsadm -L ##查看策略已加入
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:http rr
-> server2:http Route 1 0 0
-> server3:http Route 1 0 0
[root@server1 ha.d]# /etc/init.d/ipvsadm save ##保存策略,不保存的话关机策略即失效
ipvsadm: Saving IPVS table to /etc/sysconfig/ipvsadm: [ OK ]
[root@server1 ha.d]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:80 rr
-> 172.25.66.2:80 Route 1 0 0
-> 172.25.66.3:80 Route 1 0 0
server2
[root@server2 ~]# ip addr add 172.25.66.100/32 dev eth0 ##在realserver上加一个和vip相同的ip地址
[root@server2 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:64:ed:04 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.2/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fe64:ed04/64 scope link
valid_lft forever preferred_lft forever
[root@server2 ~]# yum install -y arptables_jf
为防止客户端在访问vip时直接访问到realserver,在realserver设置一定的策略
[root@server2 ~]# arptables -A IN -d 172.25.66.100 -j DROP
##访问realserver的100ip时的请求直接丢弃
[root@server2 ~]# arptables -A OUT -s 172.25.66.100 -j mangle --mangle-ip-s 172.25.66.2
##realserver发出的信息全部转为他的真实ip
[root@server2 ~]# /etc/init.d/arptables_jf save ##保存策略
Saving current rules to /etc/sysconfig/arptables: [ OK ]
[root@server2 ~]# /etc/init.d/httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.25.6.2 for ServerName
[ OK ]
server3(两个realserver做同样的配置)
[root@server3 ~]# ip addr add 172.25.66.100/32 dev eth0
[root@server3 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:c7:a3:48 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.3/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fec7:a348/64 scope link
valid_lft forever preferred_lft forever
[root@server3 ~]# yum install -y arptables_jf
[root@server3 ~]# arptables -A IN -d 172.25.66.100 -j DROP
[root@server3 ~]# arptables -A OUT -s 172.25.66.100 -j mangle --mangle-ip-s 172.25.66.3
[root@server3 ~]# /etc/init.d/arptables_jf save
Saving current rules to /etc/sysconfig/arptables: [ OK ]
[root@server3 ~]# /etc/init.d/httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.25.6.3 for ServerName
[ OK ]
2.keeplived配置
[root@server1 new]# tar zxf keepalived-1.2.20.tar.gz
[root@server1 new]# cd keepalived-1.2.20
[root@server1 keepalived-1.2.20]# yum install -y openssl-devel ##解决依赖性
[root@server1 keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived ##源码编译安装
[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/keepalived/ /etc/
[root@server1 keepalived]# cd /etc/keepalived/
[root@server1 keepalived]# ls
keepalived.conf samples
[root@server1 keepalived]# ll /etc/init.d/keepalived
lrwxrwxrwx 1 root root 48 Jul 26 13:30 /etc/init.d/keepalived -> /usr/local/keepalived/etc/rc.d/init.d/keepalived
[root@server1 keepalived]# vim keepalived.conf ##编辑主配置文件
1 ! Configuration File for keepalived
2
3 global_defs {
4 notification_email {
5 root@localhost ##设置报警邮件地址,如果要开启邮件报警,要开启本机的sendmail服务
6 }
7 notification_email_from keepalived@server1##设置邮件的发送地址
8 smtp_server 127.0.0.1 ##设置smpt server邮件服务端的地址
9 smtp_connect_timeout 30 ##设置连接smptserver的超时时间
10 router_id LVS_DEVEL
11 vrrp_skip_check_adv_addr
12 vrrp_strict
13 }
14
15 vrrp_instance VI_1 {
16 state MASTER ##指定keepalived的角色,MASTER表示此主机为主服务器,若为BACKUP则表示备用服务器
17 interface eth0 ##指定HA检测网络的接口
18 virtual_router_id 51 ##虚拟路由标识,MASTER和BACKUP的必须相同
19 priority 100 ##定义优先级,数字越大优先级越高,因此MASTER的必须年比BACKUP的数字大,否则就会有冲突
20 advert_int 1 ##设定MASTER和BACKUP负载均衡之间同步检查的时间间隔,单位为秒
21 authentication { ##设定验证类型和密码
22 auth_type PASS
23 auth_pass 1111
24 }
25 virtual_ipaddress { ##指定虚拟ip
26 172.25.66.100
27 }
28 }
29 ##虚拟服务器定义部分
30 virtual_server 172.25.66.100 80 { ##设定虚拟服务器ip地址及端口,两者以空格隔开
31 delay_loop 6 ##设置运行情况检查时间,单位为秒
32 lb_algo rr ##设置负载均衡算法,这里设置为rr算法,即轮询算法
33 lb_kind DR ##设置lvs实现负载均衡机制,有NAT,TUN,DR三种模式
34 #persistence_timeout 50 ##会话保留时间,单位为秒
35 protocol TCP ##指定转发协议类型
36
37
38
39 real_server 172.25.66.2 80 { ##配置realserver,需要指定真实ip与其端口,用空格隔开
40 weight 1 ##配置服务节点的权重值
41 TCP_CHECK { ##realserver检查状态设置部分
42 connect_timeout 3 ##3s无响应时超时
43 nb_get_retry 3 ##重复次数
44 delay_before_retry 3 ##重复间隔
45 }
46 }
47 real_server 172.25.66.3 80 {
48 weight 1
49 TCP_CHECK {
50 connect_timeout 3
51 nb_get_retry 3
52 delay_before_retry 3
53 }
54 }
55 }
test2
[root@server1 local]# scp -r keepalived/ test2:/usr/local/
[root@test2 local]# ls
bin etc games include keepalived lib lib64 libexec sbin share src
[root@test2 local]# cd keepalived/
[root@test2 keepalived]# ls
bin etc sbin share
[root@test2 keepalived]# ln -s /usr/local/keepalived/sbin/keepalived /sbin/
[root@test2 keepalived]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[root@test2 keepalived]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@test2 ~]# mkdir /etc/keepalived
[root@server1 keepalived]# scp keepalived.conf 172.25.66.11:/etc/keepalived/
[root@test2 keepalived]# cd /etc/keepalived/
[root@test2 keepalived]# ls
keepalived.conf
[root@test2 keepalived]# vim keepalived.conf
1 ! Configuration File for keepalived
2
3 global_defs {
4 notification_email {
5 root@localhost
6 }
7 notification_email_from keepalived@test2
8 smtp_server 127.0.0.1
9 smtp_connect_timeout 30
10 router_id LVS_DEVEL
11 vrrp_skip_check_adv_addr
12 vrrp_strict
13 }
14
15 vrrp_instance VI_1 {
16 state BACKUP ##备用服务器
17 interface eth0
18 virtual_router_id 51
19 priority 50 ##优先级必须低于主服务器
20 advert_int 1
21 authentication {
22 auth_type PASS
23 auth_pass 1111
24 }
25 virtual_ipaddress {
26 172.25.66.100
27 }
28 }
29
30 virtual_server 172.25.66.100 80 {
31 delay_loop 6
32 lb_algo rr
33 lb_kind DR
34 #persistence_timeout 50
35 protocol TCP
36
37
38
39 real_server 172.25.66.2 80 {
40 weight 1
41 TCP_CHECK {
42 connect_timeout 3
43 nb_get_retry 3
44 delay_before_retry 3
45 }
46 }
47 real_server 172.25.66.3 80 {
48 weight 1
49 TCP_CHECK {
50 connect_timeout 3
51 nb_get_retry 3
52 delay_before_retry 3
53 }
54 }
55 }
[root@server1 keepalived]# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
[root@server1 keepalived]# tail -f /var/log/messages
Jul 26 13:46:25 server1 Keepalived_healthcheckers[24222]: Removing service [172.25.66.3]:80 from VS [172.25.66.100]:80
Jul 26 13:46:25 server1 Keepalived_healthcheckers[24222]: Remote SMTP server [127.0.0.1]:25 connected.
Jul 26 13:46:25 server1 Keepalived_healthcheckers[24222]: SMTP alert successfully sent.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: TCP connection to [172.25.66.2]:80 failed.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Check on service [172.25.66.2]:80 failed after 1 retry.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Removing service [172.25.66.2]:80 from VS [172.25.66.100]:80
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Lost quorum 1-0=1 > 0 for VS [172.25.66.100]:80
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Remote SMTP server [127.0.0.1]:25 connected.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: SMTP alert successfully sent.
Jul 26 13:46:28 server1 Keepalived_vrrp[24223]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.25.66.100
[root@server1 keepalived]# ip addr ##接管vip
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:c1:37:57 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.1/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fec1:3757/64 scope link
valid_lft forever preferred_lft forever
[root@server1 keepalived]# ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:http rr
-> server2:http Route 1 0 3
测试:
server2和server3实现lvs
server1和test2实现keepalived
1)当server1和test2的keepalived服务都正常时,server1的keepalived工作(默认master工作,这里server1被设置为master)
[kiosk@foundation6 Desktop]$ curl 172.25.66.100
<h1>server2-www.westos.org</h1>
[kiosk@foundation6 Desktop]$ curl 172.25.66.100
<h1>server3-www.westos.org</h1>
[kiosk@foundation6 Desktop]$ arp -an | grep 100
? (172.25.66.100) at 52:54:00:c1:37:57 [ether] on br0
[root@server1 local]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:c1:37:57 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.1/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fec1:3757/64 scope link
valid_lft forever preferred_lft forever
2)当master即server1挂掉时,test2的keepalived工作
[root@server1 local]# /etc/init.d/keepalived stop
Stopping keepalived: [ OK ]
[kiosk@foundation6 Desktop]$ curl 172.25.66.100
<h1>server3-www.westos.org</h1>
[kiosk@foundation6 Desktop]$ arp -an | grep 100
? (172.25.66.100) at 52:54:00:7a:98:49 [ether] on br0
[root@test2 keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:7a:98:49 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.11/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fe7a:9849/64 scope link
valid_lft forever preferred_lft forever
keepalived+ftp
[root@server1 ~]# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
[root@server1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:c1:37:57 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.1/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fec1:3757/64 scope link
valid_lft forever preferred_lft forever
[root@server1 ~]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:http rr
-> server2:http Route 1 0 1
-> server3:http Route 1 0 1
[root@server1 ~]# vim /etc/keepalived/keepalived.conf
57 virtual_server 172.25.66.100 21 {
58 delay_loop 6
59 lb_algo rr
60 lb_kind DR
61 persistence_timeout 50
62 protocol TCP
63
64
65
66 real_server 172.25.66.2 21 {
67 weight 1
68 TCP_CHECK {
69 connect_timeout 3
70 nb_get_retry 3
71 delay_before_retry 3
72 }
73 }
74 real_server 172.25.66.3 21 {
75 weight 1
76 TCP_CHECK {
77 connect_timeout 3
78 nb_get_retry 3
79 delay_before_retry 3
80 }
81 }
82 }
##这里我们主要强调persistence_timeout 50这个参数,persistence_timeout会话保持时间,单位是秒。这个选项对动态网站很有用处:当用户从远程用帐号进行登陆网站时,有了这个会话保持功能,就能把用户的请求转发给同一个应用服务器。在这里,我们来做一个假设,假定现在有一个lvs 环境,使用DR转发模式,真实服务器有3个,负载均衡器不启用会话保持功能。当用户第一次访问的时候,他的访问请求被负载均衡器转给某个真实服务器,这样他看到一个登陆页面,第一次访问完毕;接着他在登陆框填写用户名和密码,然后提交;这时候,问题就可能出现了---登陆不能成功。因为没有会话保持,负载均衡器可能会把第2次的请求转发到其他的服务器,因此在配置有些需要保持绘画的服务时,这里必须做相应的时常配置,否则会影响业务或数据处理,
[root@server1 ~]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:ftp rr persistent 50
-> server2:ftp Route 1 0 0
-> server3:ftp Route 1 0 0
TCP 172.25.66.100:http rr
-> server2:http Route 1 0 0
-> server3:http Route 1 0 0
[root@server2 ~]# yum install y vsftpd
[root@server2 ~]# /etc/init.d/vsftpd start
Starting vsftpd for vsftpd:
[root@server2 ~]# touch /var/ftp/serve2
[root@server3 ~]# yum install -y vsftpd
[root@server3 ~]# /etc/init.d/vsftpd start
Starting vsftpd for vsftpd: [ OK ]
[root@server3 ~]# touch /var/ftp/server3
停止任意一台真机服务,不会影响虚拟FTP服务,虚拟FTP服务会自动连上可以提供服务的FTP服务器,如果所有的FTP真机服务都宕机了,则虚拟FTP也将无法访问。当真机FTP服务恢复正常后,虚拟FTP将自动添加真机服务至虚拟队列中,无人值守。
一.知识梳理
LVS+keepalived基于完成开源软件的构架实现负载均衡高可用
1.LVS
LVS是Linux Virtual Server的简写,即Linux虚拟服务器,是一个虚拟的服务集群系统。有三种负载均衡技术(VS/NAT、VS/TUN、VS/DR),八种调度算法。
2.keeplived
keepalived在这里主要做realserver健康检查以及loadbalance主机和backup主机之间failover的实现。
keepalived是一个基于VRRP协议来实现WEB服务高可用方案,可以用来避免单点故障,一个web服务器至少有两台服务器运行keepalived,一台主服务器(MASTER),一台备用服务器(BACKUP),但是对外只表现一个vip,主服务器会发送特定消息给备用服务器,当备用服务器收到消息时,备用服务器就会接管ipswich,继续提供服务,从而保证了高可用性。
3.kepalive工作原理
Layer3:keepalived使用layer3的方式工作时,keepalived会定期向服务器群中发送一个ICMP的数据包(即我们平时用的ping程序),如果发现某台服务器的IP地址没有激活,keepalived便会报告这台服务器是小,并将他从服务器群中剔除。Layer3的方式是以服务器的IP第孩子是否有效作为服务器工作正常与否的标准。
Layer4:主要以TCP端口的状态来决定服务器工作正常与否。如web sercer的服务端口一般是80.如果keepalived检测到80端口没有启动,则keepalived将这台服务器从服务群中删除。
Layer5:layer5就是工作载具体的应用层,比layer3,4要复杂一点,载网络上占用的宽带也要打一些。Keepalived将根据用户的设定检查服务器的运行是否正常。如果设定不相符,则keepalived将把服务器从群中踢除。
二.实现
1.lvs配置
server1
[root@server1 ha.d]# yum install -y ipvsadm
[root@server1 ha.d]# ipvsadm -A -t 172.25.38.100:80 -s rr
##添加策略 指定vip 172.25.66.100:80,-t指tcp,-s rr 指定轮询
[root@server1 ha.d]# ip addr add 172.25.66.100/24 dev eth0 ##添加虚拟ip
[root@server1 ha.d]# ipvsadm -L ##列出ipvsadm策略
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:http rr
[root@server1 ha.d]# ipvsadm -a -t 172.25.66.100:80 -r 172.25.66.2:80 -g
[root@server1 ha.d]# ipvsadm -a -t 172.25.66.100:80 -r 172.25.66.3:80 -g
##将虚拟ip映射到真实的ip,-r指定realserver -t指定tcp协议 -g指dr模式
[root@server1 ha.d]# ipvsadm -L ##查看策略已加入
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:http rr
-> server2:http Route 1 0 0
-> server3:http Route 1 0 0
[root@server1 ha.d]# /etc/init.d/ipvsadm save ##保存策略,不保存的话关机策略即失效
ipvsadm: Saving IPVS table to /etc/sysconfig/ipvsadm: [ OK ]
[root@server1 ha.d]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:80 rr
-> 172.25.66.2:80 Route 1 0 0
-> 172.25.66.3:80 Route 1 0 0
server2
[root@server2 ~]# ip addr add 172.25.66.100/32 dev eth0 ##在realserver上加一个和vip相同的ip地址
[root@server2 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:64:ed:04 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.2/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fe64:ed04/64 scope link
valid_lft forever preferred_lft forever
[root@server2 ~]# yum install -y arptables_jf
为防止客户端在访问vip时直接访问到realserver,在realserver设置一定的策略
[root@server2 ~]# arptables -A IN -d 172.25.66.100 -j DROP
##访问realserver的100ip时的请求直接丢弃
[root@server2 ~]# arptables -A OUT -s 172.25.66.100 -j mangle --mangle-ip-s 172.25.66.2
##realserver发出的信息全部转为他的真实ip
[root@server2 ~]# /etc/init.d/arptables_jf save ##保存策略
Saving current rules to /etc/sysconfig/arptables: [ OK ]
[root@server2 ~]# /etc/init.d/httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.25.6.2 for ServerName
[ OK ]
server3(两个realserver做同样的配置)
[root@server3 ~]# ip addr add 172.25.66.100/32 dev eth0
[root@server3 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:c7:a3:48 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.3/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fec7:a348/64 scope link
valid_lft forever preferred_lft forever
[root@server3 ~]# yum install -y arptables_jf
[root@server3 ~]# arptables -A IN -d 172.25.66.100 -j DROP
[root@server3 ~]# arptables -A OUT -s 172.25.66.100 -j mangle --mangle-ip-s 172.25.66.3
[root@server3 ~]# /etc/init.d/arptables_jf save
Saving current rules to /etc/sysconfig/arptables: [ OK ]
[root@server3 ~]# /etc/init.d/httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.25.6.3 for ServerName
[ OK ]
2.keeplived配置
[root@server1 new]# tar zxf keepalived-1.2.20.tar.gz
[root@server1 new]# cd keepalived-1.2.20
[root@server1 keepalived-1.2.20]# yum install -y openssl-devel ##解决依赖性
[root@server1 keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived ##源码编译安装
[root@server1 keepalived-1.2.20]# make && make instal
l
[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/keepalived/ /etc/
[root@server1 keepalived]# cd /etc/keepalived/
[root@server1 keepalived]# ls
keepalived.conf samples
[root@server1 keepalived]# ll /etc/init.d/keepalived
lrwxrwxrwx 1 root root 48 Jul 26 13:30 /etc/init.d/keepalived -> /usr/local/keepalived/etc/rc.d/init.d/keepalived
[root@server1 keepalived]# vim keepalived.conf ##编辑主配置文件
1 ! Configuration File for keepalived
2
3 global_defs {
4 notification_email {
5 root@localhost ##设置报警邮件地址,如果要开启邮件报警,要开启本机的sendmail服务
6 }
7 notification_email_from keepalived@server1##设置邮件的发送地址
8 smtp_server 127.0.0.1 ##设置smpt server邮件服务端的地址
9 smtp_connect_timeout 30 ##设置连接smptserver的超时时间
10 router_id LVS_DEVEL
11 vrrp_skip_check_adv_addr
12 vrrp_strict
13 }
14
15 vrrp_instance VI_1 {
16 state MASTER ##指定keepalived的角色,MASTER表示此主机为主服务器,若为BACKUP则表示备用服务器
17 interface eth0 ##指定HA检测网络的接口
18 virtual_router_id 51 ##虚拟路由标识,MASTER和BACKUP的必须相同
19 priority 100 ##定义优先级,数字越大优先级越高,因此MASTER的必须年比BACKUP的数字大,否则就会有冲突
20 advert_int 1 ##设定MASTER和BACKUP负载均衡之间同步检查的时间间隔,单位为秒
21 authentication { ##设定验证类型和密码
22 auth_type PASS
23 auth_pass 1111
24 }
25 virtual_ipaddress { ##指定虚拟ip
26 172.25.66.100
27 }
28 }
29 ##虚拟服务器定义部分
30 virtual_server 172.25.66.100 80 { ##设定虚拟服务器ip地址及端口,两者以空格隔开
31 delay_loop 6 ##设置运行情况检查时间,单位为秒
32 lb_algo rr ##设置负载均衡算法,这里设置为rr算法,即轮询算法
33 lb_kind DR ##设置lvs实现负载均衡机制,有NAT,TUN,DR三种模式
34 #persistence_timeout 50 ##会话保留时间,单位为秒
35 protocol TCP ##指定转发协议类型
36
37
38
39 real_server 172.25.66.2 80 { ##配置realserver,需要指定真实ip与其端口,用空格隔开
40 weight 1 ##配置服务节点的权重值
41 TCP_CHECK { ##realserver检查状态设置部分
42 connect_timeout 3 ##3s无响应时超时
43 nb_get_retry 3 ##重复次数
44 delay_before_retry 3 ##重复间隔
45 }
46 }
47 real_server 172.25.66.3 80 {
48 weight 1
49 TCP_CHECK {
50 connect_timeout 3
51 nb_get_retry 3
52 delay_before_retry 3
53 }
54 }
55 }
test2
[root@server1 local]# scp -r keepalived/ test2:/usr/local/
[root@test2 local]# ls
bin etc games include keepalived lib lib64 libexec sbin share src
[root@test2 local]# cd keepalived/
[root@test2 keepalived]# ls
bin etc sbin share
[root@test2 keepalived]# ln -s /usr/local/keepalived/sbin/keepalived /sbin/
[root@test2 keepalived]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[root@test2 keepalived]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@test2 ~]# mkdir /etc/keepalived
[root@server1 keepalived]# scp keepalived.conf 172.25.66.11:/etc/keepalived/
[root@test2 keepalived]# cd /etc/keepalived/
[root@test2 keepalived]# ls
keepalived.conf
[root@test2 keepalived]# vim keepalived.conf
1 ! Configuration File for keepalived
2
3 global_defs {
4 notification_email {
5 root@localhost
6 }
7 notification_email_from keepalived@test2
8 smtp_server 127.0.0.1
9 smtp_connect_timeout 30
10 router_id LVS_DEVEL
11 vrrp_skip_check_adv_addr
12 vrrp_strict
13 }
14
15 vrrp_instance VI_1 {
16 state BACKUP ##备用服务器
17 interface eth0
18 virtual_router_id 51
19 priority 50 ##优先级必须低于主服务器
20 advert_int 1
21 authentication {
22 auth_type PASS
23 auth_pass 1111
24 }
25 virtual_ipaddress {
26 172.25.66.100
27 }
28 }
29
30 virtual_server 172.25.66.100 80 {
31 delay_loop 6
32 lb_algo rr
33 lb_kind DR
34 #persistence_timeout 50
35 protocol TCP
36
37
38
39 real_server 172.25.66.2 80 {
40 weight 1
41 TCP_CHECK {
42 connect_timeout 3
43 nb_get_retry 3
44 delay_before_retry 3
45 }
46 }
47 real_server 172.25.66.3 80 {
48 weight 1
49 TCP_CHECK {
50 connect_timeout 3
51 nb_get_retry 3
52 delay_before_retry 3
53 }
54 }
55 }
[root@server1 keepalived]# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
[root@server1 keepalived]# tail -f /var/log/messages
Jul 26 13:46:25 server1 Keepalived_healthcheckers[24222]: Removing service [172.25.66.3]:80 from VS [172.25.66.100]:80
Jul 26 13:46:25 server1 Keepalived_healthcheckers[24222]: Remote SMTP server [127.0.0.1]:25 connected.
Jul 26 13:46:25 server1 Keepalived_healthcheckers[24222]: SMTP alert successfully sent.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: TCP connection to [172.25.66.2]:80 failed.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Check on service [172.25.66.2]:80 failed after 1 retry.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Removing service [172.25.66.2]:80 from VS [172.25.66.100]:80
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Lost quorum 1-0=1 > 0 for VS [172.25.66.100]:80
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: Remote SMTP server [127.0.0.1]:25 connected.
Jul 26 13:46:28 server1 Keepalived_healthcheckers[24222]: SMTP alert successfully sent.
Jul 26 13:46:28 server1 Keepalived_vrrp[24223]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.25.66.100
[root@server1 keepalived]# ip addr ##接管vip
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:c1:37:57 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.1/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fec1:3757/64 scope link
valid_lft forever preferred_lft forever
[root@server1 keepalived]# ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:http rr
-> server2:http Route 1 0 3
-> server3:http Route 1 0 4
测试:
server2和server3实现lvs
server1和test2实现keepalived
1)当server1和test2的keepalived服务都正常时,server1的keepalived工作(默认master工作,这里server1被设置为master)
[kiosk@foundation6 Desktop]$ curl 172.25.66.100
<h1>server2-www.westos.org</h1>
[kiosk@foundation6 Desktop]$ curl 172.25.66.100
<h1>server3-www.westos.org</h1>
[kiosk@foundation6 Desktop]$ arp -an | grep 100
? (172.25.66.100) at 52:54:00:c1:37:57 [ether] on br0
[root@server1 local]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:c1:37:57 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.1/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fec1:3757/64 scope link
valid_lft forever preferred_lft forever
2)当master即server1挂掉时,test2的keepalived工作
[root@server1 local]# /etc/init.d/keepalived stop
Stopping keepalived: [ OK ]
[kiosk@foundation6 Desktop]$ curl 172.25.66.100
<h1>server3-www.westos.org</h1>
[kiosk@foundation6 Desktop]$ arp -an | grep 100
? (172.25.66.100) at 52:54:00:7a:98:49 [ether] on br0
[root@test2 keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:7a:98:49 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.11/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fe7a:9849/64 scope link
valid_lft forever preferred_lft forever
keepalived+ftp
[root@server1 ~]# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
[root@server1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:c1:37:57 brd ff:ff:ff:ff:ff:ff
inet 172.25.66.1/24 brd 172.25.66.255 scope global eth0
inet 172.25.66.100/32 scope global eth0
inet6 fe80::5054:ff:fec1:3757/64 scope link
valid_lft forever preferred_lft forever
[root@server1 ~]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:http rr
-> server2:http Route 1 0 1
-> server3:http Route 1 0 1
[root@server1 ~]# vim /etc/keepalived/keepalived.conf
57 virtual_server 172.25.66.100 21 {
58 delay_loop 6
59 lb_algo rr
60 lb_kind DR
61 persistence_timeout 50
62 protocol TCP
63
64
65
66 real_server 172.25.66.2 21 {
67 weight 1
68 TCP_CHECK {
69 connect_timeout 3
70 nb_get_retry 3
71 delay_before_retry 3
72 }
73 }
74 real_server 172.25.66.3 21 {
75 weight 1
76 TCP_CHECK {
77 connect_timeout 3
78 nb_get_retry 3
79 delay_before_retry 3
80 }
81 }
82 }
##这里我们主要强调persistence_timeout 50这个参数,persistence_timeout会话保持时间,单位是秒。这个选项对动态网站很有用处:当用户从远程用帐号进行登陆网站时,有了这个会话保持功能,就能把用户的请求转发给同一个应用服务器。在这里,我们来做一个假设,假定现在有一个lvs 环境,使用DR转发模式,真实服务器有3个,负载均衡器不启用会话保持功能。当用户第一次访问的时候,他的访问请求被负载均衡器转给某个真实服务器,这样他看到一个登陆页面,第一次访问完毕;接着他在登陆框填写用户名和密码,然后提交;这时候,问题就可能出现了---登陆不能成功。因为没有会话保持,负载均衡器可能会把第2次的请求转发到其他的服务器,因此在配置有些需要保持绘画的服务时,这里必须做相应的时常配置,否则会影响业务或数据处理,
[root@server1 ~]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.66.100:ftp rr persistent 50
-> server2:ftp Route 1 0 0
-> server3:ftp Route 1 0 0
TCP 172.25.66.100:http rr
-> server2:http Route 1 0 0
-> server3:http Route 1 0 0
[root@server2 ~]# yum install y vsftpd
[root@server2 ~]# /etc/init.d/vsftpd start
Starting vsftpd for vsftpd:
[root@server2 ~]# touch /var/ftp/serve2
[root@server3 ~]# yum install -y vsftpd
[root@server3 ~]# /etc/init.d/vsftpd start
Starting vsftpd for vsftpd: [ OK ]
[root@server3 ~]# touch /var/ftp/server3
停止任意一台真机服务,不会影响虚拟FTP服务,虚拟FTP服务会自动连上可以提供服务的FTP服务器,如果所有的FTP真机服务都宕机了,则虚拟FTP也将无法访问。当真机FTP服务恢复正常后,虚拟FTP将自动添加真机服务至虚拟队列中,无人值守。