LVS(Linux Virtual server)
1.1LVS简介
1.1.1LVS的起源和目的
LVS起源于1998年,由章文嵩博士发起,当时,随着互联网的快速发展,网络服务的需求日益增长,单个服务器往往难以承受大量的并发请求,容易出现性能瓶颈和可靠性问题。为了解决这一难题,章文嵩博士致力于开发一种能够有效整合多个服务器资源、实现负载均衡的技术,于是便有了 LVS,来提供一个稳定、高效、可扩展的负载均衡系统,以满足日益增长的网络服务需求。
1.1.2LVS集群体系结构
LVS集群分为三层结构
- 负载调度器(Load Blancer):是整个LVS集群对外的前端,负责把client的请求发送到服务器上执行
- 服务器集群(server Cluster):一组真正执行clinet请求的服务器,一般是web服务器;除了web,还有FTP、MAIL、DNS等
- 共享存储(shared stord):它为server Cluster提供了一个共享的存储区,很容易让服务器池拥有相同的内容,提供相同的服务
1.1.3LVS的相关术语
缩写 | 全写 | 概括 |
VS | Virtual Server | 负责调度 |
RS | Real Server | 后端真实的工作服务器。 |
VIP | Virtual IP | 向外部直接面向用户请求,作为用户请求的目标的IP地址。 |
DIP | Director Server IP | 主要用于和内部主机通讯的IP地址。 |
RIP | Real Server IP | 后端服务器的IP地址。 |
CIP | Client IP | 访问客户端的IP地址。 |
1.2LVS的工作模式
1.2.1NAT模式
工作原理:客户端向LVS的VIP发送请求,LVS负载均衡器利用NAT把请求数据包的源和目标IP进行转换后发给真实服务器。真实服务器处理完将响应发给负载均衡器,负载均衡器再次转换IP后把响应返回给客户端。
1.2.2DR模式
工作原理:客户端向LVS负载均衡器的VIP发送请求,负载均衡器依据算法选定真实服务器,接着修改请求的目的MAC地址为所选真实服务器的MAC地址,随后通过数据链路层把请求直接发给真实服务器。真实服务器收到请求后进行处理,完成后直接把响应回传给客户端,无需再经负载均衡器中转。
要配置DR我们需要了解两个内核参数(kernelparameter),第一个是arp_ignore定义接受到ARP请求时的相应级别;第二个是arp_announce定义将自己地址向外通告时的通告级别。
arp_ignore:定义接受到ARP请求时的相应级别
0: 只要本地配置的有相应地址,就给予响应。(默认)
1: 仅回应目标IP地址是本地的入网地址的arp请求。
arp_announce:定义将自己地址向外通告是的通告级别
0: 将本地任何接口上的任何地址向外通告
1: 试图仅向目标网络通告与其网络匹配的地址
2: 仅向与本地接口上地址匹配的网络进行通告
1.2.3TUN模式
工作原理:客户端向LVS的VIP发送请求,LVS负载均衡器根据算法选定真实服务器,将请求重新封装通过IP隧道发给真实服务器。真实服务器解封装处理请求后直接响应客户端,不经过负载均衡器,从而实现负载均衡和高效的数据传输。
1.3LVS部署
1.3.1ipvsadm
实现部署前要先了解ipvsadm,ipvsadm是ipvs的管理器,需要yum安装。
ipvsadm命令
ipvsadm -A|E -t|u|f service-address [-s scheduler] [-p [timeout]]
-A | 添加虚拟服务器规则,后面跟随虚拟服务器的 IP 地址和端口号 |
-E | 修改一个虚拟服务 |
-D | 删除一个虚拟服务 |
-C | 清除所有虚拟服务 |
-R | 从标准输入获取ipvsadm命令 |
-a | 为虚拟服务添加一个real server |
-e | 修改real server |
-L | 列出虚拟服务表中的所有虚拟服务 |
-t | 指定虚拟服务为tcp服务 |
-u | 使用udp服务 |
-s | 调度算法可以指定以下10种: lvs静态调度算法:rr(轮询),wrr(权重),sh(源地址哈希),dh(目的地址哈希) lvs动态调度算法:lc(最后连接),wlc(权重),lblc(本地最后连接),lblcr(带复制的本地最后连接),sed(最小期望延迟),nq(永不排队) |
-p | 设置持久连接,这个模式可以使来自客户的多个请求被送到同一个真实服务器 |
-f | 用firewall mark取代虚拟地址来指定要被负载均衡的数据包,可以通过这个命令实现把不同址、端口的虚拟地址整合成一个虚拟服务,可以让虚拟服务器同时截获处理去往多个不同地址的数据包 |
-r | 为虚拟服务指定数据可以转发到的真实服务器的地址。可以添加端口号。如果没有指定端口号,则等效于使用虚拟地址的端口号。 |
-g | 使用网关(即直接路由),此模式是默认模式。(DR) |
-i | 使用ipip隧道模式(TUN) |
-m | 使用NAT模式 |
-w | 设置权重。权重是0~65535的整数。如果将某个真实服务器的权重设置为0,那么它不会收到新的连接,但是已有连接还会继续维持 |
-Z | 将所有数据相关的记录清零。这些记录一般用于调度策略 |
-n | 不对ip地址进行dns查询 |
--stats | 状态信息 |
--rate | 传输速率 |
1.3.2NAT模式实验
测试环境:
Director | DIP:192.168.0.100 VIP:172.25.254.100 |
webserver1 | RIP:192.168.0.10 |
webserver2 | RIP:192.168.0.20 |
[root@lvs ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.25.254.2 0.0.0.0 UG 100 0 0 eth0
172.25.254.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 101 0 0 eth1
[root@web1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.100 0.0.0.0 UG 100 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
[root@web2 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.100 0.0.0.0 UG 100 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
对lvs进行nat配置:
[root@lvs ~]# ipvsadm -A -t 172.25.254.100:80 -s rr
[root@lvs ~]# ipvsadm -a -t 172.25.254.100:80 -r 192.168.0.10:80 -m
[root@lvs ~]# ipvsadm -a -t 172.25.254.100:80 -r 192.168.0.20:80 -m
进行测试:
[root@lvs ~]# for i in {1..10}
> do
> curl 172.25.254.100
> done
web1 - 192.168.0.10
web2 - 192.168.0.20
web1 - 192.168.0.10
web2 - 192.168.0.20
web1 - 192.168.0.10
web2 - 192.168.0.20
web1 - 192.168.0.10
web2 - 192.168.0.20
web1 - 192.168.0.10
web2 - 192.168.0.20
1.3.3DR模式实验
测试环境:
主机名 | ip | vip | 角色 |
client | 172.25.254.200 NAT ,GW 172.25.254.100 | null | 测试主机 |
router | NAT-eth0:172.25.254.100,仅主机 eth1:192.168.0.100 | null | 路由器 |
lvs | 192.168.0.50,GW 192.168.0.100 仅主机 | lo:192.168.0.200 | 调度器 |
RS1 | 192.168.0.10,GW 192.168.0.100 仅主机 | lo:192.168.0.200 | web服务器1 |
RS2 | 192.168.0.20, GW 192.168.0.100 仅主机 | lo:192.168.0.200 | web服务器2 |
[root@client ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.200 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::b8f0:aac1:e8a9:3b1b prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:a1:b3:de txqueuelen 1000 (Ethernet)
RX packets 416 bytes 69358 (67.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 218 bytes 20494 (20.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 11 bytes 1814 (1.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11 bytes 1814 (1.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0[root@client ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.25.254.100 0.0.0.0 UG 100 0 0 eth0
172.25.254.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
[root@router ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.100 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::176f:ed9:5470:1f48 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c5:6d:16 txqueuelen 1000 (Ethernet)
RX packets 492 bytes 69479 (67.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 323 bytes 33141 (32.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.100 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::675b:358b:f97f:5b32 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c5:6d:20 txqueuelen 1000 (Ethernet)
RX packets 246 bytes 44385 (43.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 107 bytes 12848 (12.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 96 bytes 12004 (11.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 96 bytes 12004 (11.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0[root@router ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.25.254.2 0.0.0.0 UG 102 0 0 eth0
172.25.254.0 0.0.0.0 255.255.255.0 U 102 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 103 0 0 eth1
[root@lvs ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.50 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::176f:ed9:5470:1f48 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:fd:53:de txqueuelen 1000 (Ethernet)
RX packets 2015 bytes 257026 (251.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1217 bytes 119440 (116.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 41 bytes 12887 (12.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 41 bytes 12887 (12.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0[root@lvs ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.100 0.0.0.0 UG 100 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
[root@web1 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.10 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::1649:5487:efd6:de68 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c9:ac:5e txqueuelen 1000 (Ethernet)
RX packets 1122 bytes 188941 (184.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 446 bytes 36374 (35.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 161 bytes 15670 (15.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 161 bytes 15670 (15.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0[root@web1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.100 0.0.0.0 UG 100 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
[root@web1 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.10 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::1649:5487:efd6:de68 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c9:ac:5e txqueuelen 1000 (Ethernet)
RX packets 1122 bytes 188941 (184.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 446 bytes 36374 (35.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 161 bytes 15670 (15.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 161 bytes 15670 (15.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0[root@web1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.100 0.0.0.0 UG 100 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
使rs主机中vip不对外响应:
[root@web1 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@web1 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@web1 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
[root@web1 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@web2 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@web2 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@web2 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
[root@web2 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
对lvs,rs1/2添加vip
[root@lvs ~]# ip a a 192.168.0.200/32 dev lo
[root@lvs ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 192.168.0.200/32 scope global lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:fd:53:de brd ff:ff:ff:ff:ff:ff
altname enp3s0
altname ens160
inet 192.168.0.50/24 brd 192.168.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::176f:ed9:5470:1f48/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@web1 ~]# ip a a 192.168.0.200/32 dev lo
[root@web1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 192.168.0.200/32 scope global lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:c9:ac:5e brd ff:ff:ff:ff:ff:ff
altname enp3s0
altname ens160
inet 192.168.0.10/24 brd 192.168.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::1649:5487:efd6:de68/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@web2 ~]# ip a a 192.168.0.200/32 dev lo
[root@web2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 192.168.0.200/32 scope global lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:af:ca:be brd ff:ff:ff:ff:ff:ff
altname enp3s0
altname ens160
inet 192.168.0.20/24 brd 192.168.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::b8f0:aac1:e8a9:3b1b/64 scope link noprefixroute
valid_lft forever preferred_lft forever
对lvs中添加配置:
[root@lvs ~]# ipvsadm -A -t 192.168.0.200:80 -s wrr
[root@lvs ~]# ipvsadm -a -t 192.168.0.200:80 -r 192.168.0.10:80 -g -w 1
[root@lvs ~]# ipvsadm -a -t 192.168.0.200:80 -r 192.168.0.20:80 -g -w 1
在客户机进行测试:
[root@client ~]# for i in {1..10}
> do
> curl 192.168.0.200
> done
web2 - 192.168.0.20
web1 - 192.168.0.10
web2 - 192.168.0.20
web1 - 192.168.0.10
web2 - 192.168.0.20
web1 - 192.168.0.10
web2 - 192.168.0.20
web1 - 192.168.0.10
web2 - 192.168.0.20
web1 - 192.168.0.10