LVS-集群
集群Cluster
Cluster: 集群是为了解决某个特定问题将堕胎计算机组合起来形成的单个系统
Cluster常见的三种类型:
-
LB:LoadBalancing(负载均衡)由多个主机组成,每个主机只承担一部分访问
-
HA:High Availiablity(高可用)SPOF(single Point Of failure)
-
MTBF:平均无故障时间(正常时间)
-
MTTR:平均恢复前时间(故障时间)
-
A=MTBF/(MTBF+MTTR),故障时间越短越趋近100%
-
-
HPC:High-performance computing(高性能计算)
lvs工作模式
LVS工作原理:根据tcp/ip请求报文的目标ip和目标协议及端口将其调度转发到某个rs,根据调度算法来挑选 rs(真实主机)。
cip(客户机ip),vip(vs外网的ip),dip(lvs访问rs的ip),rip(rs的ip)
lvs访问流程:cip>vip==dip>rip,dip转为rip就是通过lvs的集群来实现,集群的类型:
-
lvs-nat,修改请求报文的目标ip,将cip>vip通过lvs转换为dip再传到rip,不改变源只改目的
-
lvs-dr, 解决lvs压力,server不再rip>dip,直接传给主机,不改变端口
-
lvs-tun,报文到达lvs时在前面加一个请求报文,传递以后再回到lvs时将请求报文删掉
-
lvs-fullnat,将请求报文的源和目的都修改,将cip传到vip改为从dip传到rip
NAT模式 | TUN模式 | DR模式 | |
---|---|---|---|
RS操作系统 | 不限 | 支持隧道 | 禁用arp |
调度器和服务器网络 | 可跨网络 | 可跨网络 | 不可跨网络 |
调度服务器数量服务器数量 | 少 | 多 | 多 |
RS服务器网关 | 指向到调度器DIP | 指向到路由 | 指向到路由 |
LVS-ipvsadm常用参数
1.管理集群服务中的增删改
ipvsadm -A|E -t|u|f service-address [-s scheduler] [-p [timeout]]
-A #添加
-E #修改
-t #tcp服务
-u #udp服务
-s #指定调度算法,默认为WLC
-p #设置持久连接超时,持久连接可以理解为在同一个时间段同一个来源的请求调度到同一Realserver
-f #firewall mask 火墙标记,是一个数字
2.管理集群中RealServer的增删改查
ipvsadm -a|e -t|u|f service-address -r realserver-address [-g|i|m] [-w weight]
-a #添加realserver
-e #更改realserver
-t #tcp协议
-u #udp协议
-f #火墙 标签
-r #realserver地址
-g #直连路由模式
-i #ipip隧道模式
-m #nat模式
-w #设定权重
-Z #清空计数器
-C #清空lvs策略
-L #查看lvs策略
-n #不做解析
--rate :输出速率信息
LVS-NAT模式部署实验
准备:两台webserver,一台lvs,一台客户端,(均以rhel9为例)
lvs上拥有两块网卡,分别为仅主机模式和nat模式(为了区分网络,使其处于不同的vlan中)
webserver均为仅主机模式,便于实验均关闭防火墙和selinux
1.配置好三台机器的环境
LVS:
webserver1:
webserver2:
2.lvs中打开内核路由功能
3.配置http
[root@webserver1 ~] dnf install httpd -y
[root@webserver1 ~] echo webserver1 - 192.168.0.10 > /var/www/html/index.html
[root@webserver1 ~] systemctl enable httpd --now
[root@webserver2 ~] dnf install httpd -y
[root@webserver2 ~] echo webserver2 - 192.168.0.20 > /var/www/html/index.html
[root@webserver2 ~] systemctl enable httpd --now
[root@lvs ~] curl 192.168.0.10 #检测
webserver1 - 192.168.0.10
[root@lvs ~] curl 192.168.0.20 #检测
webserver2 - 192.168.0.20
4.配置lvs
[root@lvs ~] dnf install ipvsadm-1.31-6.el9.x86_64 -y #下载lvs程序包
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@lvs ~] ipvsadm -A -t 172.25.254.100 -s rr #因为没加端口显示报错
Zero port specified for non-persistent service
[root@lvs ~] ipvsadm -A -t 172.25.254.100:80 -s rr
# -A:增加虚拟server,-t:调度为tcp,-s:指定调度算法,rr:轮询调度算法(静态算法)
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.254.100:80 rr
[root@lvs ~] ipvsadm -a -t 172.25.254.100:80 -r 192.168.0.10:80 -m
# -a:增加realserver -r:realserver -m:nat模式
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.254.100:80 rr
-> 192.168.0.10:80 Masq 1 0 0
测试后,同样的做法增加一条20
[root@lvs ~] ipvsadm -a -t 172.25.254.100:80 -r 192.168.0.20:80 -m
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.254.100:80 rr
-> 192.168.0.10:80 Masq 1 0 0
-> 192.168.0.20:80 Masq 1 0 0
5.最终测试
[root@client ~] for i in {1..10}
> do
> curl 192.168.0.200
> done
webserver2 - 192.168.0.20
webserver1 - 192.168.0.10
webserver2 - 192.168.0.20
webserver2 - 192.168.0.20
webserver1 - 192.168.0.10
webserver2 - 192.168.0.20
webserver2 - 192.168.0.20
webserver1 - 192.168.0.10
webserver2 - 192.168.0.20
webserver2 - 192.168.0.20
LVS-DR模式的实现实验
准备:一台客户端,一个路由器,一个lvs,两台realserver,均关闭防火墙和selinux
内网ip负责通信,vip负责对外
lvs与两台server共同组成了内网,都要把网关指向route上的192.168.0.100,nat将网关指向route上的172.25.254.100
1.配置各个主机的环境
lvs:网关指到192.168.0.100
router:eth1仅作为通信接口,不需要网关;开启内核路由功能
client:网关指到172.25.254.100
2.解决vip响应问题
DR模型中各主机上均需要配置VIP,解决地址冲突的方式有三种:
(1)在前端网关做静态绑定
(2)在各RS使用arptables
(3)在各RS修改内核参数,来限制arp响应和通告的级别
限制响应级别
-
0:默认值,表示可使用本地任意接口上配置的任意地址进行响应
-
1:仅在请求的目标IP配置在本地主机的接收到请求报文的接口上时,才给予响应
限制通告级别
-
0:默认值,把本机所有接口的所有信息向每个接口的网络进行通告
-
1:尽量避免将接口信息向非直接连接网络进行通告
-
2:必须避免将接口信息向非本网络进行通告
[root@webserver1 ~] echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@webserver1 ~] echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@webserver1 ~] echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
[root@webserver1 ~] echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
#在rs中vip不对外响应
[root@webserver2 ~] echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@webserver2 ~] echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@webserver2 ~] echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
[root@webserver2 ~] echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
注意,此配置时临时的,重启以后不生效,永久设定则将配置写道sysctl.conf里
3.添加vip
[root@lvs ~] ip a a 192.168.0.200/32 dev lo
[root@lvs ~] nmcli connection reload
[root@lvs ~] nmcli connection up lo
[root@webserver1 ~] ip a a 192.168.0.200/32 dev lo
[root@webserver1 ~] nmcli connection reload
[root@webserver1 ~] nmcli connection up lo
[root@webserver2 ~] ip a a 192.168.0.200/32 dev lo
[root@webserver2 ~] nmcli connection reload
[root@webserver2 ~] nmcli connection up lo
4.配置lvs
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.254.100:80 rr
[root@lvs ~] ipvsadm -C
[root@lvs ~] ipvsadm -A -t 192.168.0.200:80 -s wrr
[root@lvs ~] ipvsadm -a -t 192.168.0.200:80 -r 192.168.0.10:80 -g -w
-w: missing argument
[root@lvs ~] ipvsadm -a -t 192.168.0.200:80 -r 192.168.0.10:80 -g -w 1
[root@lvs ~] ipvsadm -a -t 192.168.0.200:80 -r 192.168.0.10:80 -g -w 2
Destination already exists
[root@lvs ~] ipvsadm -a -t 192.168.0.200:80 -r 192.168.0.20:80 -g -w 2
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.200:80 wrr
-> 192.168.0.10:80 Route 1 0 0
-> 192.168.0.20:80 Route 2 0 0
5.测试
[root@client ~] for i in {1..10}
> do
> curl 192.168.0.200
> done
webserver2 - 192.168.0.20
webserver1 - 192.168.0.10
webserver2 - 192.168.0.20
webserver2 - 192.168.0.20
webserver1 - 192.168.0.10
webserver2 - 192.168.0.20
webserver2 - 192.168.0.20
webserver1 - 192.168.0.10
webserver2 - 192.168.0.20
webserver2 - 192.168.0.20
lvs火墙标记实验
1.配置环境
如同之前LVS-DR模式实验
[root@server1 ~] yum install mod_ssl -y #安装mod ssl模块 让rs支持https
[root@server1 ~] systemctl restart httpd
[root@server2 ~] yum install mod_ssl -y
[root@server2 ~] systemctl restart httpd
#测试
[root@lvs ~] curl 192.168.0.10
webserver1 - 192.168.0.10
[root@lvs ~] curl https://192.168.0.10
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
[root@lvs ~] curl -k https://192.168.0.10 #-k跳过证书
webserver1 - 192.168.0.10
2.配置lvs
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.200:80 rr
-> 192.168.0.10:80 Route 1 0 0
-> 192.168.0.20:80 Route 1 0 0
[root@lvs ~] ipvsadm -A -t 192.168.0.200:443 -s rr
[root@lvs ~] ipvsadm -a -t 192.168.0.200:443 -r 192.168.0.10:443 -g
[root@lvs ~] ipvsadm -a -t 192.168.0.200:443 -r 192.168.0.20:443 -g
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.200:80 rr
-> 192.168.0.10:80 Route 1 0 0
-> 192.168.0.20:80 Route 1 0 0
TCP 192.168.0.200:443 rr
-> 192.168.0.10:443 Route 1 0 0
-> 192.168.0.20:443 Route 1 0 0
#检测
[root@client ~] curl 192.168.0.200;curl -k https://192.168.0.200
webserver1 - 192.168.0.10
webserver1 - 192.168.0.10
[root@client ~] curl 192.168.0.200;curl -k https://192.168.0.200
webserver2 - 192.168.0.20
webserver2 - 192.168.0.20
[root@client ~] curl 192.168.0.200;curl -k https://192.168.0.200
webserver1 - 192.168.0.10
webserver1 - 192.168.0.10
通过检测结果可以看见,同时访问192.168.0.200上的不同端口得到的结果都为20或10,不符合预期,因为配置的调度80和443端口是分开的两个调度,为了解决需要使用防护墙标记来解决轮询问题
3.配置端口标记
[root@lvs ~] ipvsadm -C
[root@lvs ~] iptables -t mangle -nL
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
[root@lvs ~] iptables -t mangle -A PREROUTING -d 192.168.0.200 -p tcp -m multiport --dports 80,443 -j MARK --set-mark 666
# -A:添加,-d:标记目的地址,-p:协议,-m:做多端口标记,-j:动作
[root@lvs ~] iptables -t mangle -nL
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK tcp -- 0.0.0.0/0 192.168.0.200 multiport dports 80,443 MARK set 0x29a
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@lvs ~] ipvsadm -A -f 666 -s rr
[root@lvs ~] ipvsadm -a -f 666 -r 192.168.0.10 -g
[root@lvs ~] ipvsadm -a -f 666 -r 192.168.0.20 -g
4.检测
[root@client ~] curl 192.168.0.200;curl -k https://192.168.0.200
webserver2 - 192.168.0.20
webserver1 - 192.168.0.10
[root@client ~] curl 192.168.0.200;curl -k https://192.168.0.200
webserver2 - 192.168.0.20
webserver1 - 192.168.0.10
LVS-SESSION会话问题解决
lvs持久链接
在我们客户上网过程中有很多情况下需要和服务器进行交互,客户需要提交响应信息给服务器,如果单纯的进行调度会导致客户填写的表单丢失,虽然可以使用sh算法解决,但是sh算法可能会导致调度失衡,所以采用将所有的同源数据包记录在内存中,即将这个源的主机调度到rs上,如果短期内(默认360s)再次同源访问仍按照内存记录的信息调度到同一rs上。
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 666 rr
-> 192.168.0.10:0 Route 1 0 0
-> 192.168.0.20:0 Route 1 0 0
[root@lvs ~] ipvsadm -E -f 666 -s rr -p
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 666 rr persistent 360
-> 192.168.0.10:0 Route 1 0 0
-> 192.168.0.20:0 Route 1 0 0
[root@lvs ~] ipvsadm -E -f 666 -s rr -p 2 #设定具体时间
[root@lvs ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 666 rr persistent 2
-> 192.168.0.10:0 Route 1 0 0
-> 192.168.0.20:0 Route 1 0 0