负载均衡和高可用集群（LVS，Keeplived）
1.群集的含义和类型
- 含义
群集（或集群）来自于英文单词“Cluster”，表示一群、一串的意思，用再服务器领域则表示大量服务器的集合，以区分单个服务器。 - 类型
- 负载均衡群集：以提高应用的响应能力、尽可能处理更多的访问请求、减少延迟为目标，获得高并发、高负载的整体性能
- 高可用群集：提高应用系统的可靠性、尽可能的减少中断时间为目标，确保服务的连续性，达到高可用的容错模式。工作方式包括双工、主从两种模式
- 高性能运算群集：提高系统的CPU运算速度、扩展硬件资源和分析能力为目标获得大型、超级计算机的高性能运算能力，高性能依赖于‘分布式计算’‘并行计算’，通过专用硬件和软件将多个服务器的CPU、内存等资源整合，实现超大规模的计算能力
2.负载均衡的分层结构
在负载均衡群集中，包括三个层次的组件，前端至少有一个负载调度器负责响应并分发来自客户机的访问请求，后端由大量真实服务器构成服务器池，提供实际的应用服务，整个群集的伸缩性通过增加、删除服务节点来完成，而这些过程对客户机是透明的，为了保持服务的一致性，所有节点使用共享存储设备。
- 负载调度器
访问整个集群系统的唯一入口，对外使用所有服务器共有的VIP（虚拟IP），也称为群集IP地址，通常会配置主、备两台调度器实现热备份，主调度器失效后平滑替换到备调度器。 - 服务器池
群集所提供的所有应用服务由服务器池承担，其中每个节点具有独立的RIP（真实IP），只处理调度器分发过来的客户机请求，当某个节点暂时失效时，负载调度器的容错机制会将其隔离，等待错误排除以后再重新纳入服务器池。 - 共享存储
为服务器池中的所有节点提供稳定、一致的文件存储服务，确保整个群集的统一性，共享存储体哦概念股NFS共享服务的专用服务器。
3.负载均衡的工作模式
群集的负载调度技术，可以基于IP、端口、内容等进行分发，其中基于IP的负载调度时效率最高的，基于IP 的负载均衡模式中，常见的由地址转换、IP隧道和直接路由三种工作模式
- 地址转换
简称NAT模式类似于防火墙的私有网络结构，负载调度器作为所有客户机的访问入口，也是各节点回应客户机的访问出口，服务器系欸但使用私有IP地址，于负载调度器位于同一个物理网络，安全性要优于其他两种方式 - IP隧道
采用开放式大的网络结构，负载调度器仅作为客户机的访问入口，各节点通过各自的lnternet连接直接回应客户机，而不再经过负载调度器，服务器节点分散在互联网中的不同位置，具有独立的IP地址，通过转筒IP隧道相互通信 - 直接路由
采用半开放式的网络结构，于IP隧道的结构类似，但个节点并不是分散在各地，二十于调度器位于同一个物理网络，负载调度器于个节点服务器通过本地网络连接，不再需要建立专用的IP隧道
4.LVS的负载调度算法
- 轮询
将收到的访问请求按照顺序轮流分配给群集中的个节点，均等的对待每一台服务器，而不管服务器的实际连接数和几桶负载 - 加权轮询
根据正式服务器的处理能力轮流分配收到的访问请求，调度器可以自动查询个节点的负载情况，并动态调整其权重，可以保证能力强的服务器承担更多的访问。 - 最少连接
根据真实服务器已建立的连接数进行分配，将受到的访问请求优先分配给连接数最少的节点 - 加权最少连接
在服务器系欸但的性能差异较大时，可以为真实服务器自动调整权重，权重较高的节点承担更大的活动连接负载。
5.ipvsadm命令解析
ipvsadm命令选项解析：
-C：清除内核虚拟服务器表中的所有记录
-D 删除整个虚拟服务器
-A：增加一台新的虚拟服务器
-d 删除某一个节点
-t：说明虚拟服务器提供的是tcp的服务
-s rr：启用轮询算法
-a：在一个虚拟服务器中增加一台新的真实服务器
-r：指定RIP地址及TCP端口
-m：指定LVS的工作模式为NAT模式
-g 表示使用 DR模式
-i 表示使用 TUN模式
-w 设置权重 (权重为 0 时表示暂停节点)
-p 60 表示保持长连接60秒
-l 列表查看 LVS 虚拟服务器 (默认为查看所有)
-n 以数字形式显示地址、端口等信息。
ipvsadm：启用LVS功能
6.NAT模式LVS负载均衡集群部署
- 准备
LVS负载调度器：ens33：192.168.20.11 ens37：192.168.26.100
web节点服务器1：192.168.20.22
web节点服务器2：192.168.20.33
NFS服务器：192.168.20.44
客户端：192.168.26.20 - 部署共享存储NFS
[root@localhost ~]# yum install -y nfs-utils rpcbind //下载服务
[root@localhost ~]# systemctl start nfs.service //开启服务并开机自启
[root@localhost ~]# systemctl start rpcbind.service
[root@localhost ~]# systemctl start nfs.service
[root@localhost ~]# systemctl start rpcbind.service
[root@localhost ~]# mkdir /opt/tom /opt/jerry //创建共享目录
[root@localhost ~]# chmod 777 /opt/tom /opt/jerry //给全部权限
[root@localhost ~]# vim /etc/exports
/opt/tom 192.168.20.0/24(rw,sync)
/opt/jerry 192.168.20.0/24(rw,sync)
[root@localhost ~]# exportfs -rv //发布共享
exporting 192.168.20.0/24:/opt/jerry
exporting 192.168.20.0/24:/opt/tom
[root@localhost ~]# showmount -e //查看共享
Export list for localhost.localdomain:
/opt/jerry 192.168.20.0/24
/opt/tom 192.168.20.0/24
- 配置节点服务器20.33、20.22
- 两台都配置
[root@localhost ~]# yum install -y httpd //下载httpd服务
[root@localhost ~]# systemctl start httpd.service
[root@localhost ~]# systemctl start rpcbind.service
[root@localhost ~]# systemctl enable rpcbind.service
[root@localhost ~]# showmount -e 192.168.20.44
Export list for 192.168.20.44:
/opt/jerry 192.168.20.0/24
/opt/tom 192.168.20.0/24
- web1
[root@localhost ~]# mount.nfs 192.168.20.44:/opt/tom /var/www/html/ //挂载
[root@localhost ~]# echo 'this is tom' > /var/www/html/index.html //网页文件
[root@localhost ~]# vim /etc/fstab
192.168.20.44:/opt/tom /var/www/html nfs defaults,netdev 0 0
[root@localhost ~]# mount -a
[root@localhost ~]#vim /etc/sysconfig/network-scripts/ifcfg-ens33
GATEWAY=192.168.20.11
- web2
[root@localhost ~]# mount.nfs 192.168.20.44:/opt/jerry /var/www/html/
[root@localhost ~]# echo 'this is jerry' > /var/www/html/index.html
[root@localhost ~]# vim /etc/fstab
192.168.20.44:/opt/jerry /var/www/html nfs defaults,netdev 0 0
[root@localhost ~]# mount -a
[root@localhost ~]#vim /etc/sysconfig/network-scripts/ifcfg-ens33
GATEWAY=192.168.20.11
- LVS配置
添加网卡为vmnet1
[root@localhost ~]# ifconfig ens37 192.168.26.100 //添加ens37网卡
[root@localhost ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.20.11 netmask 255.255.255.0 broadcast 192.168.20.255
inet6 fe80::a127:8d6a:5ef8:bcc6 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:7e:9c:75 txqueuelen 1000 (Ethernet)
RX packets 656621 bytes 928813225 (885.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 309427 bytes 18999557 (18.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens37: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.26.100 netmask 255.255.255.0 broadcast 192.168.26.255
inet6 fe80::a3b8:9a92:ac49:2b5c prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:7e:9c:7f txqueuelen 1000 (Ethernet)
RX packets 3 bytes 746 (746.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 39 bytes 7929 (7.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
- 配置转发规则
[root@localhost ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward=1
[root@localhost ~]# sysctl -p
net.ipv4.ip_forward = 1
[root@localhost ~]# iptables -t nat -F
[root@localhost ~]# iptables -F
[root@localhost ~]# iptables -t nat -nL
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
[root@localhost ~]# systemctl start firewalld.service
[root@localhost ~]# iptables -t nat -A POSTROUTING -s 192.168.20.0/24 -o ens37 -j SNAT --to-source 192.168.26.100 //添加规则
[root@localhost ~]# iptables -t nat -nL
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
PREROUTING_direct all -- 0.0.0.0/0 0.0.0.0/0
PREROUTING_ZONES_SOURCE all -- 0.0.0.0/0 0.0.0.0/0
PREROUTING_ZONES all -- 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
OUTPUT_direct all -- 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
RETURN all -- 192.168.122.0/24 224.0.0.0/24
RETURN all -- 192.168.122.0/24 255.255.255.255
MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24
POSTROUTING_direct all -- 0.0.0.0/0 0.0.0.0/0
POSTROUTING_ZONES_SOURCE all -- 0.0.0.0/0 0.0.0.0/0
POSTROUTING_ZONES all -- 0.0.0.0/0 0.0.0.0/0
SNAT all -- 192.168.20.0/24 0.0.0.0/0 to:192.168.26.100
Chain OUTPUT_direct (1 references)
target prot opt source destination
Chain POSTROUTING_ZONES (1 references)
target prot opt source destination
POST_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
POST_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
POST_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
Chain POSTROUTING_ZONES_SOURCE (1 references)
target prot opt source destination
Chain POSTROUTING_direct (1 references)
target prot opt source destination
Chain POST_public (3 references)
target prot opt source destination
POST_public_log all -- 0.0.0.0/0 0.0.0.0/0
POST_public_deny all -- 0.0.0.0/0 0.0.0.0/0
POST_public_allow all -- 0.0.0.0/0 0.0.0.0/0
Chain POST_public_allow (1 references)
target prot opt source destination
Chain POST_public_deny (1 references)
target prot opt source destination
Chain POST_public_log (1 references)
target prot opt source destination
Chain PREROUTING_ZONES (1 references)
target prot opt source destination
PRE_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
PRE_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
PRE_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
Chain PREROUTING_ZONES_SOURCE (1 references)
target prot opt source destination
Chain PREROUTING_direct (1 references)
target prot opt source destination
Chain PRE_public (3 references)
target prot opt source destination
PRE_public_log all -- 0.0.0.0/0 0.0.0.0/0
PRE_public_deny all -- 0.0.0.0/0 0.0.0.0/0
PRE_public_allow all -- 0.0.0.0/0 0.0.0.0/0
Chain PRE_public_allow (1 references)
target prot opt source destination
Chain PRE_public_deny (1 references)
target prot opt source destination
Chain PRE_public_log (1 references)
target prot opt source destination
[root@localhost ~]# modprobe ip_vs //手动加载ip_vs模块
[root@localhost ~]# cat /proc/net/ip_vs //查看版本信息
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@localhost ~]# yum install -y ipvsadm //安装管理工具
[root@localhost ~]# ipvsadm-save > /etc/sysconfig/ipvsadm //必须要先保存分配策略
[root@localhost ~]# systemctl start ipvsadm.service //开启策略
[root@localhost ~]# ipvsadm -C //清空规则
[root@localhost ~]# ipvsadm -A -t 192.168.26.100:80 -s rr //写入规则
[root@localhost ~]# ipvsadm -a -t 192.168.26.100:80 -r 192.168.20.22:80 -m -w 1
[root@localhost ~]# ipvsadm -a -t 192.168.26.100:80 -r 192.168.20.33:80 -m -w 1
[root@localhost ~]# ipvsadm //开启策略
[root@localhost ~]# ipvsadm-save > /etc/sysconfig/ipvsadm //保存策略
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP localhost.localdomain:http rr
-> 192.168.20.22:http Masq 1 0 0
-> 192.168.20.33:http Masq 1 0 0
[root@localhost ~]# watch -n 1 ipvsadm //实时刷新查看
Every 1.0s: ipvsadm Mon Jul 26 18:31:39 2021
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP localhost.localdomain:http rr
-> 192.168.20.22:http Masq 1 0 0
-> 192.168.20.33:http Masq 1 0 0
- 正在Win 10上面访问并刷新测试负载均衡
7.LVS-DR模式
7.1LVS-DR数据包流量分析（局域网）
- 客户端向目标VIP发送请求，负载均衡器接收
- 负载均衡器根据负载均衡算法选择后端真实服务器，不修改也不封装IP报文，而是将数据帧的MAC地址改为后端正式服务器的MAC地址，然后再局域网上发送。
- 后端真实服务器收到这个帧，解封后发现目标IP和本机的匹配（提前绑定VIP），于是处理报文
- 随后重新封装报文，将响应报文通过lo接口传送给物理网卡然后向外发出，客户端将受到回复报文，客户端认为已经得到了服务，而不会知道是哪一台雾浮起处理的如果跨网段，则报文通过路由器经由internet返回给用户。
7.2LVS-DR中的ARP问题
- 在LVS-DR负载均衡集中，负载均衡于节点服务器都要配置相同的VIP地址。
- 在局域网中具有相同的IP地址，会造成各服务器ARP通信的紊乱，
当ARP广播发送到集群时，因为负载均衡器和节点服务器都是连接到相同网络上，他们都会收到ARP广播，只有前端的负载均衡器进行响应，其他节点服务器不应该响应ARP广播。 - 对节点服务器进行处理，使其不响应针对VIP的ARP请求
使用虚接口lo：0承载VIP地址设置内阁参数arp_ignore=1：系统只响应目的IP为本地IP的ARP请求 - RealServer返回报文经由路由器转发，重新封装报文时，需要先获取路由器的MAC地址
- 发送ARP请求时，Linux默认使用IP包的源IP地址作为ARP请求包中的源IP地址，而不适用发送接口的IP地址
- 路由器收到ARP请求后，将更新ARP表
- 原有的VIP对应Director的MAC地址会被更新为VIP对应的RealServer的MAC地址
- 路由器根据Arp表，将新来的请求报文转发给RealServer，导致Director的VIP失效
对节点服务器进行处理，设置内核参数arp_announce=2：系统不使用IP包的源地址来设置ARP请求的源地址，而选择发送接口的IP地址。
上文两个问题的设置方法
修改 /etc/sysctl.conf 文件
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
7.3LVS-DR处理问题后的流量分析
- 客户端发送请求到Director server（负载均衡器），请求的数据报文到达内核空间
- Director Server和Real server在同一个网络中，数据通过二层数据链路层来传输
- 内核空间判断数据包的目标IP就是本机VIP，此时IPVS（IP虚拟服务器）比对数据包请求的服务是否是集群服务，时集群服务就重新封装数据包，修改源MAC地址为Director Server的MAC地址，修改目标MAC地址为Real server的MAC地址，源IP地址于目标IP地址没有变动，然后将数据包发送给Real server
- 到达Real server的请求报文的MAC地址时自身的MAC地址，就接受此报文。数据包重新封装报文，将响应报文通过lo接口给物理网卡然后向外发出
- Real server直接将响应报文传送到客户端
7.4LVS-DR 特性
- Director server和real server必须在同一个物理网络中
- real server可以使用私有地址，也可以使用公网地址，如果使用公网地址，可以通过互联网对RIP进行直接访问
- Director server作为群集的访问入口，但不作为网关使用，所有的请求报文经由Director server，但回复响应报文不能经过Director server
- Real server的网关不允许指向director server IP
- Real server 的loi恶口配置VIP的IP地址。
8.keepalived
- 支持故障自动切换
- 支持节点健康状态检查
- 判断LVS负载调度器、节点服务器的可用性，当master主机出现故障及时切换到backup节点保证业务正常，当master故障主机恢复后将其重新加入群集并且业务重新切换回master节点
8.1kepplived实现原理
keeplived曹勇VRRP热备份协议实现Linux服务器的多机热备功能，VRRP（虚拟路由冗余协议）是针对路由器的一种备份结局方案，由多台路由器组成一个热备份组，通过共有的虚拟IP对峙对外提供服务，每个热备组内同时只有一台主路由提供服务，其他路由器处于冗余状态，若当前在线的路由器失效，则其他路由器会根据设置的优先级自动接替虚拟IP地址，继续提供服务。
9.LVS-DR
- 负载调度器
ens33：（192.168.20.11）
ens33:0 (192.168.20.12)
[root@localhost ~]# yum -y install ipvsadm //下载ipvsadm
[root@localhost ~]# modprobe ip_vs #加载ip_vs模块
[root@localhost ~]# cat /proc/net/ip_vs #查看ip_vs版本信息
[root@localhost ~]# cd /etc/sysconfig/network-scripts/ //配置ens33:0虚拟网卡
[root@localhost network-scripts]# cp -p ifcfg-ens33 ifcfg-ens33:0 //复制模板
[root@localhost network-scripts]# vim ifcfg-ens33:0 //按照模板写如下列字段
DEVICE=ens33:0
ONBOOT=yes
IPADDR=192.168.20.12
NETMASK=255.255.255.255
[root@localhost network-scripts]# ifup ens33:0 //开启ens33：0
[root@localhost network-scripts]# ifconfig //查看是否成功
ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.20.12 netmask 255.255.255.255 broadcast 192.168.20.12
ether 00:0c:29:7e:9c:75 txqueuelen 1000 (Ethernet)
[root@localhost network-scripts]# vim /etc/sysctl.conf //调整proc响应参数（lvs和个节点需要公用VIP地址，所以要关闭内核的重定向响应参数，不充当路由器）
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
[root@localhost network-scripts]# sysctl -p //从指定的文件加载系统参数
[root@localhost ~]# modprobe ip_vs //开启ipvsadm
[root@localhost ~]# cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@localhost ~]# ipvsadm-save > /etc/sysconfig/ipvsadm //配置负载分配策略
[root@localhost ~]# systemctl start ipvsadm
[root@localhost opt]# vim /opt/dr.sh //设置规则
#!/bin/bash
ipvsadm -C //清空规则
ipvsadm -A -t 192.168.20.12:80 -s rr //添加虚拟服务器并指定vip地址和tcp端口指定轮询模式
ipvsadm -a -t 192.168.20.12:80 -r 192.168.20.22:80 -g //指定真实服务器指定RIP和tcp端口使用DR模式
ipvsadm -a -t 192.168.20.12:80 -r 192.168.20.33:80 -g
ipvsadm //开启
[root@localhost ~]# ipvsadm //开启
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP localhost.localdomain:http rr
-> 192.168.20.22:http Route 1 1 0
-> 192.168.20.33:http Route 1 0 1
[root@localhost ~]# ipvsadm -Lnc //查看节点状态
IPVS connection entries
pro expire state source virtual destination
[root@localhost ~]# cd /opt
[root@localhost opt]# sh -x dr.sh //可执行脚本虽然之前开启过了
+ ipvsadm -C
+ ipvsadm -A -t 192.168.20.12:80 -s rr
+ ipvsadm -a -t 192.168.20.12:80 -r 192.168.20.22:80 -g
+ ipvsadm -a -t 192.168.20.12:80 -r 192.168.20.33:80 -g
+ ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP localhost.localdomain:http rr
-> 192.168.20.22:http Route 1 0 0
-> 192.168.20.33:http Route 1 0 0
[root@localhost opt]# ipvsadm -Lnc
IPVS connection entries
pro expire state source virtual destination
- web服务器1
ens33（192.168.20.22）
**lo:0 (VIP):（192.168.20.12) **
[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp -p ifcfg-lo ifcfg-lo:0 //模板，照着写
[root@localhost network-scripts]# vim ifcfg-lo:0 //配置虚拟接口
DEVICE=lo:0
ONBOOT=yes
IPADDR=192.168.20.12
NETMASK=255.255.255.255
[root@localhost network-scripts]# ifup lo:0 //开启
[root@localhost network-scripts]# ifconfig lo:0 //查看
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.20.12 netmask 255.255.255.255
loop txqueuelen 1000 (Local Loopback)
[root@localhost network-scripts]# route add -host 192.168.226.100 dev lo:0 //禁锢路由
[root@localhost network-scripts]# route -n //查看路由表
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.20.2 0.0.0.0 UG 100 0 0 ens33
192.168.20.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
192.168.20.12 0.0.0.0 255.255.255.255 UH 0 0 0 lo
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
[root@localhost network-scripts]# vim /etc/rc.local //插入下列字段管理执行路由禁锢
/sbin/route add -host 192.168.20.12 dev lo:0
[root@localhost network-scripts]# yum -y install httpd // 下载httpd
[root@localhost network-scripts]# vim /etc/sysctl.conf //调整内阁ARP响应参数和组织更新VIP的MAC地址，避免冲突
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
[root@localhost network-scripts]# sysctl -p
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
[root@localhost network-scripts]# systemctl start httpd
[root@localhost network-scripts]# vim /var/www/html/index.html //页面
<html>
<body>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8">
<h1>this is tom web</h1>
</body>
</html>
- web服务器2
ens33 192.168.20.33
**lo:0 (VIP)：192.168.20.12 **
[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp -p ifcfg-lo ifcfg-lo:0
[root@localhost network-scripts]# vim ifcfg-lo:0
[root@localhost network-scripts]# ifup lo:0
[root@localhost network-scripts]# ifconfig lo:0
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.20.12 netmask 255.255.255.255
loop txqueuelen 1000 (Local Loopback)
[root@localhost network-scripts]# route add -host 192.168.20.12 dev lo:0
[root@localhost network-scripts]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.20.2 0.0.0.0 UG 100 0 0 ens33
192.168.20.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
192.168.20.12 0.0.0.0 255.255.255.255 UH 0 0 0 lo
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
[root@localhost network-scripts]# vim /etc/rc.local
/sbin/route add -host 192.168.20.12 dev lo:0
[root@localhost network-scripts]# yum -y install httpd
[root@localhost network-scripts]# vim /etc/sysctl.conf
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
[root@localhost network-scripts]# sysctl -p
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
[root@localhost network-scripts]# systemctl start httpd
[root@localhost network-scripts]# vim /var/www/html/index.html
<html>
<body>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8">
<h1>this is jerry web</h1>
</body>
</html>
- 在LVS端
[root@localhost opt]# watch -n 1 ipvsadm -Lnc //实时查看
然后通过在win10访问观察是否轮询
[root@localhost opt]# ipvsadm -Lnc
IPVS connection entries
pro expire state source virtual destination
TCP 01:08 FIN_WAIT 192.168.20.20:50060 192.168.20.12:80 192.168.20.22:80
TCP 14:39 ESTABLISHED 192.168.20.42:56829 192.168.20.12:80 192.168.20.33:80
TCP 00:50 FIN_WAIT 192.168.20.42:52352 192.168.20.12:80 192.168.20.22:80
TCP 14:16 ESTABLISHED 192.168.20.20:50061 192.168.20.12:80 192.168.20.33:80
TCP 00:41 FIN_WAIT 192.168.20.42:50206 192.168.20.12:80 192.168.20.33:80
TCP 14:14 ESTABLISHED 192.168.20.20:50062 192.168.20.12:80 192.168.20.22:80
[root@localhost opt]#
[root@localhost opt]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.20.12:80 rr
-> 192.168.20.22:80 Route 1 1 2
-> 192.168.20.33:80 Route 1 2 1
10.LVS-DR-keepalived
那么这个是基于上面的LVS+轮询基础上加入keepaliced
两个LVS负载调度器，一个主、一个备
主ens33：（192.168.20.11）
ens33:0 (192.168.20.12)
备ens33：（192.168.20.44）
ens33：0（192.168.20.12）
web服务器不变
- 在主LVS上
[root@lvs1 opt]# yum install -y keepalived //下载keepalived
[root@lvs1 opt]# vim /etc/keepalived/keepalived.conf //keepalived主配置文件
global_defs { //定义全局参数
router_id lvs_01 //热备组的设备名称需要不一样
}
vrrp_instance vi_1 { //定义VRRP热备实例参数
state MASTER //指定热备状态，主为master，备位backup
interface ens33 //指定承载VIP的物理接口
virtual_router_id 51 //指定虚拟路由器的ID号，每个热备组保持一致
priority 110 //优先级，数值越大越优先
advert_int 1
authentication {
auth_type PASS
auth_pass 6666
}
virtual_ipaddress { //指定集群的VIP地址
192.168.20.12
}
}
virtual_server 192.168.20.12 80 { //指定虚拟服务器的vip、端口
lb_algo rr //使用轮询
lb_kind DR //指定集群荣作模式，直接路由DR
persistence_timeout 6 //健康检查间隔时间
protocol TCP //服务所用的协议
real_server 192.168.20.22 80 { //web节点的地址和端口
weight 1 //节点权重
TCP_CHECK {
connect_port 80 //目标端口
connect_timeout 3 //连接超时
nb_get_retry 3 //添加重试次数
delay_before_retry 3 //重试间隔
}
}
real_server 192.168.20.33 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
[root@lvs1 opt]# ifconfig //这里ens33：0现在可以看见，但是keepalived开启后虚拟地址将会绑定到ens33里
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.20.11 netmask 255.255.255.0 broadcast 192.168.20.255
inet6 fe80::a127:8d6a:5ef8:bcc6 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:7e:9c:75 txqueuelen 1000 (Ethernet)
RX packets 188472 bytes 270987677 (258.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 32976 bytes 2387421 (2.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.20.12 netmask 255.255.255.255 broadcast 192.168.20.12
ether 00:0c:29:7e:9c:75 txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 84 bytes 9828 (9.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 84 bytes 9828 (9.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:6b:93:3d txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@lvs1 opt]# systemctl start keepalived.service //开启服务
[root@lvs1 opt]# systemctl status keepalived.service //查看是否开启
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running)since 二 2021-07-27 17:30:39 CST; 44s ago
Process: 26413 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 26414 (keepalived)
[root@lvs1 opt]# ifconfig //ens33：0消失
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.20.11 netmask 255.255.255.0 broadcast 192.168.20.255
inet6 fe80::a127:8d6a:5ef8:bcc6 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:7e:9c:75 txqueuelen 1000 (Ethernet)
RX packets 188636 bytes 271002234 (258.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 33131 bytes 2405899 (2.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 84 bytes 9828 (9.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 84 bytes 9828 (9.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:6b:93:3d txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@lvs1 opt]# ip addr //使用ip addr可以看到ens33里有vip
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:7e:9c:75 brd ff:ff:ff:ff:ff:ff
inet 192.168.20.11/24 brd 192.168.20.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.20.12/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::a127:8d6a:5ef8:bcc6/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@lvs1 opt]# sh -x dr.sh //开启ipvsadm
+ ipvsadm -C
+ ipvsadm -A -t 192.168.20.12:80 -s rr
+ ipvsadm -a -t 192.168.20.12:80 -r 192.168.20.22:80 -g
+ ipvsadm -a -t 192.168.20.12:80 -r 192.168.20.33:80 -g
+ ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP localhost.localdomain:http rr
-> 192.168.20.22:http Route 1 0 0
-> 192.168.20.33:http Route 1 0 0
[root@lvs2 ~]# cd /etc/sysconfig/network-scripts/ //配置VIP地址
[root@lvs2 network-scripts]# cp -p ifcfg-ens33 ifcfg-ens33:0
[root@lvs2 network-scripts]# vim ifcfg-ens33:0
DEVICE=ens33:0
ONBOOT=yes
IPADDR=192.168.20.12
NETMASK=255.255.255.255
[root@lvs2 network-scripts]# ifup ens33:0 //这里用ifup会报错，所有重启就可以
ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Error, some other host (00:0C:29:7E:9C:75) already uses address 192.168.20.12.
[root@lvs2 network-scripts]# systemctl restart network
[root@lvs2 network-scripts]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.20.44 netmask 255.255.255.0 broadcast 192.168.20.255
inet6 fe80::9e09:b371:574b:cbef prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:47:b8:6e txqueuelen 1000 (Ethernet)
RX packets 15872 bytes 19775513 (18.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7835 bytes 583242 (569.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.20.12 netmask 255.255.255.255 broadcast 192.168.20.12
ether 00:0c:29:47:b8:6e txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 420 bytes 35880 (35.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 420 bytes 35880 (35.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:d1:f1:86 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@lvs2 network-scripts]# vim /etc/sysctl.conf //关闭内核的重定向响应参数，不充当路由
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
[root@lvs2 network-scripts]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
[root@lvs2 network-scripts]# modprobe ip_vs //开启ipvsadm
[root@lvs2 network-scripts]# cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@lvs2 network-scripts]# ipvsadm-save > /etc/sysconfig/ipvsadm //配置负载分配策略
[root@lvs2 network-scripts]# systemctl start ipvsadm.service //开启ipvsadm
[root@lvs2 network-scripts]# vim /opt/dr.sh //配置启动脚本
#!/bin/bash
ipvsadm -C
ipvsadm -A -t 192.168.20.12:80 -s rr
ipvsadm -a -t 192.168.20.12:80 -r 192.168.20.22:80 -g
ipvsadm -a -t 192.168.20.12:80 -r 192.168.20.33:80 -g
ipvsadm
[root@lvs2 network-scripts]# ipvsadm -Lnc //查看ipvsadm策略
IPVS connection entries
pro expire state source virtual destination
[root@lvs2 network-scripts]# vim /etc/keepalived/keepalived.conf //配置keepalived主配置文件
global_defs {
router_id lvs_02 //不能和lvs1一样
}
vrrp_instance vi_1 {
state BACKUP //改为备
interface ens33
virtual_router_id 51 //ID需要一样
priority 100 //优先级越高越优先
advert_int 1
authentication {
auth_type PASS
auth_pass 6666
}
virtual_ipaddress {
192.168.20.12
}
}
virtual_server 192.168.20.12 80 {
lb_algo rr
lb_kind DR
persistence_timeout 6
protocol TCP
real_server 192.168.20.22 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.20.33 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
[root@lvs2 network-scripts]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.20.44 netmask 255.255.255.0 broadcast 192.168.20.255
inet6 fe80::9e09:b371:574b:cbef prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:47:b8:6e txqueuelen 1000 (Ethernet)
RX packets 19240 bytes 20064697 (19.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10906 bytes 1121133 (1.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.20.12 netmask 255.255.255.255 broadcast 192.168.20.12
ether 00:0c:29:47:b8:6e txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 420 bytes 35880 (35.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 420 bytes 35880 (35.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:d1:f1:86 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@lvs2 network-scripts]# systemctl start keepalived.service //开启服务
[root@lvs2 network-scripts]# systemctl status keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running) since 二 2021-07-27 17:30:48 CST; 8s ago
Process: 76730 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 76731 (keepalived)
[root@lvs2 network-scripts]# ip a // 可以看到现在并没有ens33：0
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:47:b8:6e brd ff:ff:ff:ff:ff:ff
inet 192.168.20.44/24 brd 192.168.20.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::9e09:b371:574b:cbef/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@lvs2 ]# cd /opt
[root@lvs2 opt]# sh +x dr.sh //执行脚本
Zero port specified for non-persistent service
Service not defined
Service not defined
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@lvs2 opt]# sh -x dr.sh
+ ipvsadm -C
+ ipvsadm -A -t 192.168.20.12:80 -s rr
+ ipvsadm -a -t 192.168.20.12:80 -r 192.168.20.22:80 -g
+ ipvsadm -a -t 192.168.20.12:80 -r 192.168.20.33:80 -g
+ ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.20.12:http rr
-> 192.168.20.22:http Route 1 0 0
-> 192.168.20.33:http Route 1 0 0
再用WIN10验证是否轮询
[root@localhost opt]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.20.12:80 rr
-> 192.168.20.22:80 Route 1 1 2
-> 192.168.20.33:80 Route 1 1 1
- lvs1
[root@lvs1 opt]# systemctl stop keepalived.service //现在把lvs1的keeplived服务关闭
[root@lvs1 opt]# ip a
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:7e:9c:75 brd ff:ff:ff:ff:ff:ff
inet 192.168.20.11/24 brd 192.168.20.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::a127:8d6a:5ef8:bcc6/64 scope link noprefixroute
valid_lft forever preferred_lft forever
- lvs2
[root@lvs2 opt]# ip a //在lvs2上观察vip已经漂移过来
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:47:b8:6e brd ff:ff:ff:ff:ff:ff
inet 192.168.20.44/24 brd 192.168.20.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.20.12/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::9e09:b371:574b:cbef/64 scope link noprefixroute
valid_lft forever preferred_lft forever
- 再用win10访问依然能成功
[root@localhost opt]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.20.12:80 rr
-> 192.168.20.22:80 Route 1 1 1
-> 192.168.20.33:80 Route 1 1 2
总结
- lvs是一种四层的负载均衡策略，常用在大型的服务器环境，lvs集成在内核之中，所以相对的比较消耗资源
- lvs有三种模式
nat
TUN
DR - 四种分流算法
轮询
加权轮询
最小链接
加权最小连接 - 做nat模式时
1.首先要确保LVS负载调度器添加了一块网卡，并进入/etc/sysconfig/network-scripts/ifcfg-ens37 配置但是不要设置dns和网关（ens33也不设置dns和网关）
2.确保所有机器不要设置dns
3.如果全部做完但是没有找到页面，回到两台web服务器，ping-ens37ip查看是否能ping通，检查网关（这里的网关应该指的时LVS负载调度器）
4.如果ping通了但是还是找不到就把配置的策略全部删掉重新做一次
ipvsadm -d -t 192.168.26.100:80 -r 192.168.20.11:80 -m -w 1 #删除群集中某一节点服务器
ipvsadm -D -t 192.168.26.100:80 #删除整个虚拟服务器
systemctl stop ipvsadm #停止服务（清除策略）
systemctl start ipvsadm #启动服务（重建规则）
ipvsadm-restore > /etc/sysconfig/ipvsadm #恢复LVS策略
- keepalived
vrrp热备协议，主要针对对象为router路由器，同一个热备组中的ID不可相同，VIP作为提供服务的IP地址，只会漂移再优先级最高的设备中，如果主宕机，那么主优先级会下降，备服务器提升为主。
扫一扫
2065
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
