1、简介
LVS是Linux Virtual Server的简写,意即Linux虚拟服务器,是一个虚拟的服务器集群系统。在1998年5月由章文嵩博士成立,是中国国内最早出现的自由软件项目之一。
LVS 工作原理简述:用户请求LVS VIP,LVS根据转发方式和算法,将请求转发给后端服务器,后端服务器接受到请求,返回给用户。对于用户来说,看不到WEB后端具体的应用。
2、LVS 负载均衡原理
实现LVS负载均衡转发方式有三种,分别为NAT、DR、TUN模式,LVS均衡算法包括:RR(round-robin)、LC(least_connection)、W(weight)RR、WLC模式等(RR为轮询模式,LC为最少连接模式)。
LVS NAT原理:用户请求LVS VIP到达director(LVS服务器:LB),director将请求的报文的目标IP地址改成后端的realserver IP地址,同时将报文的目标端口也改成后端选定的realserver相应端口,最后将报文发送到realserver,realserver将数据返给director,director再把数据发送给用户。(两次请求都经过director,所以访问大的话,director会成为瓶颈)。
注意事项:
1、LVS服务器至少2块物理网卡,一块连接公网(VIP),一块连接内网;
2、后端Realserver机器的默认网关设置为LVS的内网IP地址;
3、保证LVS内网网卡通常跟Realserver在同一网段;
4、LVS NAT模式后端Realserver机器数量不超过30台;
5、用户的请求进入和返回均会经过LVS,LVS会成为瓶颈。
LVS DR原理:用户请求LVS VIP到达director(LB均衡器),director将请求的报文的目标MAC地址改成后端的realserver MAC地址,目标IP为VIP(不变),源IP为用户IP地址(保持不变),然后Director将报文发送到realserver,realserver检测到目标为自己本地VIP,如果在同一个网段,然后将请求直接返给用户。如果用户跟realserver不在一个网段,则通过网关返回用户。
注意事项:
1、LVS服务器和后端Realserver必须在同网段;
2、LVS修改请求报文的目标MAC,目标IP(VIP)不变;
3、realServer端需将VIP配置在lo网卡以保证多个相同的VIP之间不冲突;
4、realServer端的网关指向路由器下一跳(默认网关),保证数据能够出去(访问外网);
5、所有的realServer端服务器要抑制VIP ARP广播,禁止VIP响应解析,而且保证真实网卡不能抑制ARP广播。
LVS TUN原理:用户请求LVS到达director,director通过IP-TUN加密技术将请求的报文的目标MAC地址改成后端的realserver MAC地址,目标IP为VIP(不变),源IP为用户IP地址(保持不变),然后Director将报文发送到realserver,realserver基于IP-TUN解密,然后检测到目标为自己本地VIP,如果在同一个网段,然后将请求直接返给用户。如果用户跟realserver不在一个网段,则通过网关返回用户。
3、LVS 负载均衡实现–NAT模式
LVS负载均衡技术实现是基于Linux内核模块IP_VS。IP_VS模块与iptables一样是直接工作在内核中,互联网主流的Linux发行版默认都已经集成了ipvs模块,因此只需安装管理工具ipvsadm。
LVS 端机器配置:
安装ipvsadm
yum install -y ipvsadm
机器配置双网卡,一个对内一个对外
[root@localhost ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.159 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 fe80::20ff:bdcd:8409:96e9 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:b7:36:91 txqueuelen 1000 (Ethernet)
RX packets 13589 bytes 18950235 (18.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5881 bytes 445749 (435.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens37: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.30.78 netmask 255.255.255.0 broadcast 192.168.30.255
inet6 fe80::f862:a92f:f830:61ef prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:b7:36:9b txqueuelen 1000 (Ethernet)
RX packets 16826 bytes 1910791 (1.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 123 bytes 10246 (10.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 1207 bytes 106216 (103.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1207 bytes 106216 (103.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# 添加虚拟服务器IP
[root@localhost ~]# ipvsadm -A -t 10.0.0.159:80 -s rr
# 添加realserver后端服务
[root@localhost ~]# ipvsadm -a -t 10.0.0.159:80 -r 192.168.30.79:80 -m -w 2
# 查看
[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.159:80 rr
-> 192.168.30.79:80 Masq 2 0 0
ipvsadm 参数解析
-A 增加一台虚拟服务器VIP地址;
-t 虚拟服务器提供的是tcp服务;
-s 使用的调度算法;
-a 在虚拟服务器中增加一台后端真实服务器;
-r 指定真实服务器地址;
-w 后端真实服务器的权重;
-m 设置当前转发方式为NAT模式;-g为直接路由模式;-i 模式为隧道模式
-D 删除虚拟服务器VIP地址
-d 删除虚拟服务器中后端真实服务器
开启 IP 转发
[root@localhost ~]# cat /proc/sys/net/ipv4/ip_forward
0
[root@localhost ~]# echo 1 > /proc/sys/net/ipv4/ip_forward
[root@localhost ~]# cat /proc/sys/net/ipv4/ip_forward
1
realServer端配置:
将192.168.30.79 机器网关设置为LVS内网网卡Ip地址,然后重启生效
[root@localhost ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
GATEWAY=192.168.30.78
[root@localhost ~]# /etc/init.d/network restart
启动nginx
[root@localhost ~]# systemctl start nginx
访问 LVS 外网IP,跳转到realServer
LVS 服务机器查看
[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.159:80 rr
-> 192.168.30.79:80 Masq 2 1 1
4、LVS 负载均衡实现–DR模式
realServer配置:
添加lo子网卡
[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp ifcfg-lo ifcfg-lo:1
[root@localhost network-scripts]# vim ifcfg-lo:1
DEVICE=lo:1
IPADDR=10.0.0.152
NETMASK=255.255.255.255
NETWORK=127.0.0.0
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback
[root@localhost network-scripts]# ifup lo:1
[root@localhost network-scripts]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.150 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 fe80::996:6962:aea5:f905 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:68:03:e2 txqueuelen 1000 (Ethernet)
RX packets 667 bytes 79050 (77.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 423 bytes 60923 (59.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 51642 bytes 78221180 (74.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 51642 bytes 78221180 (74.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo:1: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 10.0.0.152 netmask 255.255.255.255
loop txqueuelen 1000 (Local Loopback)
此时可以从外部ping通VIP,需进行arp抑制,阻止lo网卡响应组播消息。
进行arp抑制
# arp_ignore 参数 1 表示只响应目标IP是本地真实网卡上配置的IP
# arp_announce 参数2 表示忽略报文的源IP地址,使用主机上能够跟用户通信的真实网卡发生数据
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
LVS 端配置:
设置ip_vs模块
[root@localhost ~]# lsmod |grep -i ip_vs
[root@localhost ~]# modprobe ip_vs
[root@localhost ~]# lsmod |grep -i ip_vs
ip_vs 145497 0
nf_conntrack 133095 1 ip_vs
libcrc32c 12644 3 xfs,ip_vs,nf_conntrack
添加VIP和realServer
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@localhost ~]# ipvsadm -A -t 10.0.0.152:80 -s rr
[root@localhost ~]# ipvsadm -a -t 10.0.0.152:80 -r 10.0.0.150 -g -w 100
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.152:80 rr
-> 10.0.0.150:80 Route 100 0 0
添加子网卡,设置VIP
[root@localhost network-scripts]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp ifcfg-ens33 ifcfg-ens33:1
[root@localhost network-scripts]# vim ifcfg-ens33:1
TYPE=Ethernet
BOOTPROTO=static
DEVICE=ens33:1
ONBOOT=yes
IPADDR=10.0.0.152
NETMASK=255.255.255.0
[root@localhost network-scripts]# ifup ifcfg-ens33:1
[root@localhost network-scripts]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.148 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 fe80::fe63:b144:f5f4:17ed prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:cd:2b:78 txqueuelen 1000 (Ethernet)
RX packets 18476 bytes 17269630 (16.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 20558 bytes 1492968 (1.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.152 netmask 255.0.0.0 broadcast 10.255.255.255
ether 00:0c:29:cd:2b:78 txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 8741 bytes 769266 (751.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8741 bytes 769266 (751.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
浏览器访问:
5、LVS + keepalived
LVS和keepalived结合使用,可以省去 添加VIP和realServer 步骤。LVS+Keepalived负载均衡高可用集群架构适用于千万级并发网站,在互联网企业得到大力的应用。
[root@localhost network-scripts]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
***@163.com
}
notification_email_from ***@163.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
}
# VIP1
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 151
priority 100
advert_int 5
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.152
}
}
virtual_server 10.0.0.152 80{
delay_loop 6
lb_algo rr
lb_kind DR
persistence_timeout 60
protocol TCP
real_server 10.0.0.150 80 {
weight 100
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
[root@localhost network-scripts]# systemctl start keepalived
[root@localhost network-scripts]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.152:80 rr persistent 60
-> 10.0.0.150:80 Route 100 0 0
访问浏览器