LVS

啊风z

已于 2022-09-28 20:33:08 修改

阅读量314

点赞数

分类专栏： linux 文章标签： lvs http 服务器

于 2022-09-28 20:32:35 首次发布

本文链接：https://blog.csdn.net/m0_67900916/article/details/127096539

版权

linux 专栏收录该内容

21 篇文章 0 订阅

订阅专栏

lvs

lvs概述

LVS，是Linux Virtual Server的简称，也就是Linux虚拟服务器, 是一个由章文嵩博士发起的自由软件项目。LVS由用户空间的ipvsadm和内核空间的IPVS组成，ipvsadm用来定义规则，IPVS利用ipvsadm定义的规则工作。现在LVS已经是 Linux标准内核的一部分，在Linux2.4内核以前，使用LVS时必须要重新编译内核以支持LVS功能模块，但是从Linux2.4内核以后，已经完全内置了LVS的各个功能模块，无需给内核打任何补丁，可以直接使用LVS提供的各种功能。

LVS特点

通过LVS提供的负载均衡技术和Linux操作系统实现一个高性能、高可用的服务器群集，它具有良好可靠性、可扩展性和可操作性。从而以低廉的成本实现最优的服务性能。LVS的主要特点有以下几个方面：

高并发连接：LVS基于内核网络层面工作，有超强的承载能力和并发处理能力。单台LVS负载均衡器，可支持上万并发连接。稳定性强：是工作在网络4层之上仅作分发之用，这个特点也决定了它在负载均衡软件里的性能最强，稳定性最好，对内存和cpu资源消耗极低。
成本低廉：硬件负载均衡器少则十几万，多则几十万上百万，LVS只需一台服务器和就能免费部署使用，性价比极高。
配置简单：LVS配置非常简单，仅需几行命令即可完成配置，也可写成脚本进行管理。
支持多种算法：支持多种论调算法，可根据业务场景灵活调配进行使用
支持多种工作模型：可根据业务场景，使用不同的工作模式来解决生产环境请求处理问题。
应用范围广：因为LVS工作在4层，所以它几乎可以对所有应用做负载均衡，包括http、数据库、DNS、ftp服务等等
缺点：工作在4层，不支持7层规则修改，机制过于庞大，不适合小规模应用。

LVS常见术语

名称	解释
ipvsadm	用户空间的命令行工具，用于管理集群服务及集群服务上的RS等；
IPVS	工作于内核上的netfilter INPUT HOOK之上的程序，可根据用户定义的集群实现请求转发；
VS	Virtual Server ,虚拟服务
Director, Balancer	负载均衡器、分发器
RS	Real Server 后端请求处理服务器
CIP	Client IP,客户端IP
VIP	Director Virtual IP,负载均衡器虚拟IP
DIP	Director IP,负载均衡器IP
RIP	Real Server IP,后端请求处理服务器IP

LVS组成

LVS 由2部分程序组成，包括 ipvs 和 ipvsadm。

IPVS(ip virtual server)：一段代码工作在内核空间，叫IPVS，是真正生效实现调度的代码。IPVS的总体结构主要由IP包处理、负载均衡算法、系统配置与管理三个模块及虚拟服务器与真实服务器链表组成。
ipvsadm：另外一段是工作在用户空间，叫ipvsadm，即IPVS管理器，负责为ipvs内核框架编写规则，定义谁是集群服务，而谁是后端真实的服务器(Real Server)。

LVS基本原理

在这里插入图片描述

当用户向负载均衡调度器（Director Server）发起请求，调度器将请求发往至内核空间。
PREROUTING链首先会接收到用户请求，判断目标IP确定是本机IP，将数据包发往INPUT链。
IPVS是工作在INPUT链上的，当用户请求到达INPUT时，IPVS会将用户请求和自己已定义好的集群服务进行比对，如果用户请求的就是定义的集群服务，那么此时IPVS会强行修改数据包里的目标IP地址及端口，并将新的数据包发往POSTROUTING链。
POSTROUTING链接收数据包后发现目标IP地址刚好是自己的后端服务器，那么此时通过选路，将数据包最终发送给后端的服务器

配置lvs-nat模式的httpd负载集群—http

环境说明：

主机名称	网卡信息
Client客户端	192.168.159.134
ZWL DIP:	192.168.159.167
RS1 RIP:	192.168.159.167
RS2 RIP:	192.168.159.168

给ZWL主机添加一个仅主机的网卡
[root@ZWL ~]# ifconfig     //查看到仅主机网卡的名字为 ens38
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.159.167  netmask 255.255.255.0  broadcast 192.168.91.255
        inet6 fe80::20c:29ff:feb8:3224  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:b8:32:24  txqueuelen 1000  (Ethernet)
        RX packets 299161  bytes 120054672 (114.4 MiB)
        RX errors 0  zwlopped 0  overruns 0  frame 0
        TX packets 382902  bytes 85603867 (81.6 MiB)
        TX errors 0  zwlopped 0 overruns 0  carrier 0  collisions 0

ens38: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:0c:29:b8:32:2e  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  zwlopped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  zwlopped 0 overruns 0  carrier 0  collisions 0
//查看虚拟网络编辑器种仅主机的网段为192.168.237.0
[root@ZWL ~]# nmcli connection add con-name ens38 ifname ens38 type  ethernet 
Connection 'ens38' (9b003222-efb6-4d19-8043-a625b3a9c154) successfully added.
[root@ZWL ~]# nmcli connection 
NAME    UUID                                  TYPE      DEVICE 
ens33   af4d3903-2150-4bda-9723-f37666535088  ethernet  ens33  
ens38   9b003222-efb6-4d19-8043-a625b3a9c154  ethernet  ens38  
virbr0  95dd368f-e449-44b6-8fb2-cd0cbbb50c2f  bridge    virbr0 
[root@ZWL ~]# nmcli connection modify ens38 ipv4.adzwlesses 192.168.159.129/24 ipv4.method manual autoconnect yes 
[root@ZWL ~]# systemctl restart NetworkManager
[root@ZWL ~]# nmcli connection up ens38 
Connection successfully activated (D-Bus active path:/org/freedesktop/NetworkManager/ActiveConnection/5)




ZWL、RS1、RS2三台主机都关闭防火墙和selinux
[root@ZWL ~]# systemctl stop firewalld
[root@ZWL ~]# systemctl disable firewalld
[root@ZWL ~]# sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config 
[root@ZWL ~]# setenforce 0

 
[root@RS1 ~]# systemctl stop firewalld
[root@RS1 ~]# systemctl disable firewalld
[root@RS1 ~]# sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config
[root@RS1 ~]# setenforce 0

 

[root@RS2 ~]# systemctl stop firewalld
[root@RS2 ~]# systemctl disable firewalld
[root@RS2 ~]# sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config
[root@RS2 ~]# setenforce 0
配置ip信息
//ZWL:
[root@ZWL ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
IPADZWL=192.168.159.167
PREFIX=24
GATEWAY=192.168.91.2
DNS1=8.8.8.8

//RS1:
[root@RS1 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
.....
IPADZWL=192.168.159.167
PREFIX=24
GATEWAY=192.168.159.167
DNS1=8.8.8.8



//RS2:
[root@RS2 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
IPADZWL=192.168.159.168
PREFIX=24
GATEWAY=192.168.159.167
DNS1=8.8.8.8


//后端RS1和RS2部署WEB服务器
RS1:
[root@RS1 ~]# yum -y install httpd
[root@RS1 ~]# echo RS1 > /var/www/html/index.html
[root@RS1 ~]# systemctl restart httpd
[root@RS1 ~]# systemctl enable httpd

//RS2:
[root@RS2 ~]# yum -y install httpd
[root@RS2 ~]# echo RS2 > /var/www/html/index.html
[root@RS2 ~]# systemctl restart httpd
[root@RS2 ~]# systemctl enable httpd

//配置ZWL
（1）开启IP转发功能
[root@ZWL ~]# vim /etc/sysctl.con
net.ipv4.ip_forward = 1
[root@ZWL ~]# sysctl -p
net.ipv4.ip_forward = 1


//安装ipvsadm并添加规则
[root@ZWL ~]# yum -y install ipvsadm
[root@ZWL ~]# ipvsadm -A -t 192.168.159.129:80 -s rr
[root@ZWL ~]# ipvsadm -a -t 192.168.159.129:80 -r 192.168.159.167:80 -m
[root@ZWL ~]# ipvsadm -a -t 192.168.159.129:80 -r 192.168.159.168:80 -m
[root@ZWL ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAdzwless:Port Scheduler Flags
  -> RemoteAdzwless:Port           Forward Weight ActiveConn InActConn
TCP  192.168.159.129:80 rr
  -> 192.168.159.167:80            Masq    1      0          0         
  -> 192.168.159.168:80            Masq    1      0          0

[root@ZWL ~]# ipvsadm -Sn > /etc/sysconfig/ipvsadm
[root@ZWL ~]# systemctl restart ipvsadm.service 
[root@ZWL ~]# systemctl enable ipvsadm.service 

//客户端测试
[root@Client ~]# curl http://192.168.159.129
RS2
[root@Client ~]# curl http://192.168.159.129
RS1
[root@Client ~]# curl http://192.168.159.129
RS2
[root@Client ~]# curl http://192.168.159.129
RS1

配置lvs-nat模式的httpd负载集群—https
//在ZWL中生成一对密钥
[root@ZWL ~]# mkdir  -p  /etc/pki/CA/private
[root@ZWL ~]# cd /etc/pki/CA/
[root@ZWL CA]# (umask 077;openssl genrsa -out private/cakey.pem 2048)
Generating RSA private key, 2048 bit long modulus (2 primes)
........................................................................................+++++
..........+++++
e is 65537 (0x010001)

[root@ZWL CA]# openssl rsa -in private/cakey.pem -pubout
writing RSA key
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwzoaJdd6iyCTOe97L7jg
sd3I7TKZuADMwRWKYYwyt5x2QuBuI7FaJ0gtP6sSRn9UmOpxixXDKOX2wpv27Ld+
N0L45eX/cDcMNJtMLCm4eVxzFegJagiE60gt6paTn6JX70AK6RM8iIClAwQMPbc3
lUooeSDcRoWW7LU85+QU36p3RpNIKcNOow5GNvuHe/GQhqArA50gxXsKqkFDsZVm
7xLZVWyBJ5WImhHgjV9wEhjk+/fM+8i05KOS3+WPf01I58zmehh3REohMi1X4Knz
RAS25s4pU6Shs2XAj6nHRLrxPUslEE5ZS9Uc9hKLUizLUeWDTo37yv4CJkVi50XV
TwIDAQAB
-----END PUBLIC KEY-----

[root@ZWL CA]# openssl req -new -x509 -key private/cakey.pem -out cacert.pem -days 1024
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:hubei
Locality Name (eg, city) [Default City]:wuhan
Organization Name (eg, company) [Default Company Ltd]:runtime   
Organizational Unit Name (eg, section) []:linux
Common Name (eg, your name or your server's hostname) []:zwl
Email Adzwless []:zwl@1314.com

[root@ZWL CA]# touch index.txt && echo 01 > serial


在RS1中生成证书签署请求，并发送给CA
[root@RS1 ~]# yum -y install mod_ssl
[root@RS1 ~]# mkdir /etc/httpd/ssl
[root@RS1 ~]# cd /etc/httpd/ssl/
[root@RS1 ssl]# (umask 077;openssl genrsa -out httpd.key 2048)
Generating RSA private key, 2048 bit long modulus (2 primes)
..................................+++++
............................................................................+++++
e is 65537 (0x010001)
[root@RS1 ssl]# openssl req -new -key httpd.key -days 1024 -out httpd.csr
Ignoring -days; not generating a certificate
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:hubei
Locality Name (eg, city) [Default City]:wuhan
Organization Name (eg, company) [Default Company Ltd]:runtime
Organizational Unit Name (eg, section) []:linux
Common Name (eg, your name or your server's hostname) []:zwl  
Email Adzwless []:zwl@1314.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@RS1 ssl]# 
[root@RS1 ssl]# ls
httpd.csr  httpd.key
[root@RS1 ssl]# scp httpd.csr root@192.168.159.167:/root/
The authenticity of host '192.168.159.167 (192.168.159.167)' can't be established.
ECDSA key fingerprint is SHA256:Z3HMzqS6THCLCxpluX/FENh3Ag0hppqEQar7Klpf2LU.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.159.167' (ECDSA) to the list of known hosts.
root@192.168.159.167's password: 
httpd.csr               100% 1025   280.9KB/s   00:00                                     


//在ZWL中查看
[root@ZWL ~]# ls
anaconda-ks.cfg  cmake  Desktop  Documents  Downloads  httpd.csr  initial-setup-ks.cfg

//CA签署证书并发给RS1
[root@ZWL ~]# mkdir /etc/pki/CA/newcerts
[root@ZWL ~]# touch /etc/pki/CA/index.txt
[root@ZWL ~]# echo "01" > /etc/pki/CA/serial
[root@ZWL ~]# openssl ca -in httpd.csr -out httpd.crt -days 1024
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 1 (0x1)
        Validity
            Not Before: Sep 27 06:57:45 2022 GMT
            Not After : Jul 17 06:57:45 2025 GMT
        Subject:
            countryName               = CN
            stateOrProvinceName       = hubei
            organizationName          = runtime
            organizationalUnitName    = linux
            commonName                = zwl
            emailAdzwless              = zwl@1314.com
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Comment: 
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier: 
                2F:0E:AD:05:5C:72:ED:18:44:EF:9B:CC:D4:8A:FA:98:E7:5E:9A:43
            X509v3 Authority Key Identifier: 
                keyid:BC:33:75:57:47:33:5A:3E:EB:17:8E:7B:37:E2:80:B1:BB:B2:4D:5E

Certificate is to be certified until Jul 17 06:57:45 2025 GMT (1024 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

[root@ZWL ~]# ls
anaconda-ks.cfg  Desktop    Downloads  httpd.csr             Music     Public     Videos
cmake            Documents  httpd.crt  initial-setup-ks.cfg  Pictures  Templates

//将CA签署的证书httpd.crt和服务器的证书cacert.pem发送给RS1
[root@ZWL ~]# scp httpd.crt root@192.168.159.167:/etc/httpd/ssl   
[root@ZWL ~]# scp /etc/pki/CA/cacert.pem root@192.168.159.167:/etc/httpd/ssl

//RS2配置https
[root@RS2 ~]# yum -y install mod_ssl
[root@RS2 ~]# mkdir /etc/httpd/ssl
//RS1中把RS1的证书和密钥发送给RS2
[root@RS1 ~]# cd /etc/httpd/ssl/
[root@RS1 ssl]# scp cacert.pem httpd.crt httpd.key root@192.168.159.168:/etc/httpd/ssl

//在RS1中修改https的配置文件
[root@RS1 ssl]# vim /etc/httpd/conf.d/ssl.conf
SSLCertificateFile /etc/httpd/ssl/httpd.crt
SSLCertificateKeyFile /etc/httpd/ssl/httpd.key
SSLCACertificateFile /etc/httpd/ssl/cacert.pem
[root@RS1 ssl]# systemctl restart httpd.service 
[root@RS1 ssl]# ss -anlt |grep 443
LISTEN 0      128                *:443             *:*


//在RS2中修改https的配置文件
[root@RS2 ~]# vim /etc/httpd/conf.d/ssl.conf
SSLCertificateFile /etc/httpd/ssl/httpd.crt
SSLCertificateKeyFile /etc/httpd/ssl/httpd.key
SSLCACertificateFile /etc/httpd/ssl/cacert.pem
[root@RS2 ~]# systemctl restart httpd.service 
[root@RS2 ~]# ss -anlt|grep 443
LISTEN 0      128                *:443             *:*

//在ZWL中添加规则
[root@ZWL ~]# ipvsadm -A -t 192.168.159.129:443 -s rr
[root@ZWL ~]# ipvsadm -a -t 192.168.159.129:443 -r 192.168.159.167 -m
[root@ZWL ~]# ipvsadm -a -t 192.168.159.129:443 -r 192.168.159.168 -m
[root@ZWL ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAdzwless:Port Scheduler Flags
  -> RemoteAdzwless:Port           Forward Weight ActiveConn InActConn
TCP  192.168.159.129:80 rr
  -> 192.168.159.167:80            Masq    1      0          0         
  -> 192.168.159.168:80            Masq    1      0          0         
TCP  192.168.159.129:443 rr
  -> 192.168.159.167:443           Masq    1      0          0         
  -> 192.168.159.168:443           Masq    1      0          0 
  
//客户端测试
[root@Client ~]# curl -k https://192.168.159.129:443
RS1
[root@Client ~]# curl -k https://192.168.159.129:443
RS2
[root@Client ~]# curl -k https://192.168.159.129:443
RS1
[root@Client ~]# curl -k https://192.168.159.129:443
RS2
配置lvs-zwl模式的httpd负载集群
Lvs服务器（ZWL） DIP 192.168.159.167 VIP 192.168.91.100
Apache服务器（RS1） RIP 192.168.159.167 VIP 192.168.91.100
Apache服务器（RS2） RIP 192.168.159.168 VIP 192.168.91.100
client客户端 IP 192.168.159.134
//配置httpd
RS1：
关闭防火墙和selinux
[root@rs1 ~]# systemctl stop firewalld
[root@rs1 ~]# systemctl disable firewalld
[root@rs1 ~]# sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config

安装httpd
[root@rs1 ~]# yum -y install httpd
[root@rs1 ~]# echo "RS1" > /var/www/html/index.html
[root@rs1 ~]# systemctl restart httpd
[root@rs1 ~]# systemctl enable httpd


RS2：
关闭防火墙和selinux
[root@RS2 ~]# systemctl stop firewalld
[root@RS2 ~]# systemctl disable firewalld
[root@RS2 ~]# sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config 


安装httpd
[root@RS2 ~]# yum -y install httpd
[root@RS2 ~]# echo "RS2" > /var/www/html/index.html
[root@RS2 ~]# systemctl restart httpd
[root@RS2 ~]# systemctl enable httpd


LVS上配置ip：
ZWL：
//关闭防火墙和selinux
[root@ZWL ~]# systemctl stop firewalld
[root@ZWL ~]# systemctl disable firewalld
[root@ZWL ~]# sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config 

//临时生效
[root@ZWL ~]# ifconfig lo 192.168.91.100 broadcast 192.168.91.100 netmask 255.255.255.255 up
//永久生效
[root@ZWL ~]# vim /etc/rc.d/rc.local    
ifconfig lo 192.168.91.100 broadcast 192.168.91.100 netmask 255.255.255.255 up
[root@ZWL ~]# chmod +x /etc/rc.d/rc.local
[root@ZWL ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 192.168.91.100/32 brd 192.168.91.100 scope global lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:b8:32:24 brd ff:ff:ff:ff:ff:ff
    inet 192.168.159.167/24 brd 192.168.91.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:feb8:3224/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

//RS上配置arp内核参数
RS1和RS2上都需要操作
 vim /etc/sysctl.conf
#将对应网卡设置为只回应目标IP为自身接口地址的ARP请求
net.ipv4.conf.all.arp_ignore = 1
#将ARP请求的源IP设置为所有接口的IP，也就是RIP
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2

//RS1
[root@RS1 ~]# vim /etc/sysctl.conf
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
[root@RS1 ~]# sysctl -p
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

//RS2
[root@RS2 ~]# vim /etc/sysctl.conf
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
[root@RS2 ~]# sysctl -p
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2


//RS上配置VIP
一定要先配置好内核参数，再配置VIP，如果先配置VIP，VIP配置好后会立即通告给所有人，而修改内核参数就是为了不通告。
//LVS服务器的ens33网卡的ip：192.168.91.100作为VIP
两台RS都要做
RS1:
[root@RS1 ~]# ifconfig lo 192.168.91.100 broadcast 192.168.91.100 netmask 255.255.255.255 up
[root@RS1 ~]# ip a
[root@RS1 ~]# ifconfig lo 192.168.91.100 broadcast 192.168.91.100 netmask 255.255.255.255 up
[root@RS1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 192.168.91.100/32 brd 192.168.91.100 scope global lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:70:9e:3b brd ff:ff:ff:ff:ff:ff
    inet 192.168.159.167/24 brd 192.168.91.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe70:9e3b/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
       
[root@RS2 ~]# ifconfig lo 192.168.91.100 broadcast 192.168.91.100 netmask 255.255.255.255 up
[root@RS2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 192.168.91.100/32 brd 192.168.91.100 scope global lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:33:c1:e3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.159.168/24 brd 192.168.91.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe33:c1e3/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

//添加路由信息
RS1：
[root@RS1 ~]# route add -host 192.168.100.100/32 dev lo

RS2：
[root@RS2 ~]# route add -host 192.168.100.100/32 dev lo

//添加并保存规则
[root@ZWL ~]# ipvsadm -A -t 192.168.91.100:80 -s rr
[root@ZWL ~]# ipvsadm -a -t 192.168.91.100:80  -r  192.168.159.167:80 -g
[root@ZWL ~]# ipvsadm -a -t 192.168.91.100:80  -r  192.168.159.168:80 -g
[root@ZWL ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAdzwless:Port Scheduler Flags
  -> RemoteAdzwless:Port           Forward Weight ActiveConn InActConn
TCP  192.168.91.100:80 rr
  -> 192.168.159.167:80            Route   1      0          0         
  -> 192.168.159.168:80            Route   1      0          0
[root@ZWL ~]# ipvsadm -Sn > /etc/sysconfig/ipvsadm
[root@ZWL ~]# cat /etc/sysconfig/ipvsadm
-A -t 192.168.91.100:80 -s rr
-a -t 192.168.91.100:80 -r 192.168.159.167:80 -g -w 1
-a -t 192.168.91.100:80 -r 192.168.159.168:80 -g -w 1
[root@ZWL ~]# systemctl restart ipvsadm.service 
[root@ZWL ~]# systemctl enable ipvsadm.service

//客户端验证
[root@Client ~]# curl http://192.168.91.100
RS1
[root@Client ~]# curl http://192.168.91.100
RS2
[root@Client ~]# curl http://192.168.91.100
RS1
[root@Client ~]# curl http://192.168.91.100
RS2
TUN模式
ZWL：
//关闭防火墙和selinux
修改内核参数，开启IP转发
[root@ZWL ~]# vim /etc/sysctl.conf 
net.ipv4.ip_forward = 1
[root@ZWL ~]# sysctl -p
net.ipv4.ip_forward = 1
[root@ZWL ~]# yum -y install ipvsadm
[root@ZWL ~]# ifconfig tunl0 192.168.91.55 broadcast 192.168.91.55 netmask 255.255.255.255 up
[root@ZWL ~]# ip a
.....
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
  link/ipip 0.0.0.0 brd 0.0.0.0
  inet 192.168.91.55/32 brd 192.168.91.55 scope global tunl0
     valid_lft forever preferred_lft forever
     
     
//RS1和RS2（两台主机操作一样）
关闭防火墙和selinux，部署httpd

//启用ipip模块
RS1
[root@rs1 ~]# modprobe ipip
[root@rs1 ~]# ifconfig tunl0 192.168.91.55 broadcast 192.168.91.55 netmask 255.255.255.255 up

RS2
[root@rs2 ~]# modprobe ipip
[root@rs2 ~]# ifconfig tunl0 192.168.91.55 broadcast 192.168.91.55 netmask 255.255.255.255 up

//修改内核参数为
RS1:
[root@RS1 ~]# vim /etc/sysctl.conf 
[root@RS1 ~]# sysctl -p
net.ipv4.conf.tunl0.arp_ignore = 1
net.ipv4.conf.tunl0.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.tunl0.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0

RS2:
[root@RS2 ~]# vim /etc/sysctl.conf 
[root@RS2 ~]# sysctl -p
net.ipv4.conf.tunl0.arp_ignore = 1
net.ipv4.conf.tunl0.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.tunl0.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0

//ZWL上添加规则：
[root@ZWL ~]# ipvsadm -A -t 192.168.91.55:80 -s rr
[root@ZWL ~]# ipvsadm -a -t 192.168.91.55:80 -r 192.168.159.167 -i
[root@ZWL ~]# ipvsadm -a -t 192.168.91.55:80 -r 192.168.159.168 -i
[root@ZWL ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAdzwless:Port Scheduler Flags
-> RemoteAdzwless:Port           Forward Weight ActiveConn InActConn
TCP  192.168.91.55:80 rr
-> 192.168.159.167:80            Tunnel  1      0          0         
-> 192.168.159.168:80            Tunnel  1      0          0         
[root@ZWL ~]# ipvsadm -Sn > /etc/sysconfig/ipvsadm
[root@ZWL ~]# systemctl restart ipvsadm.service

//客户端验证：
[root@Client ~]# curl http://192.168.91.55
RS1
[root@Client ~]# curl http://192.168.91.55
RS2
[root@Client ~]# curl http://192.168.91.55
RS1
[root@Client ~]# curl http://192.168.91.55
RS2