day65:DR模式搭建及keepalived+LVS

1、LVS  DR模式搭建:准备工作:也是目前使用最多的模式:

  在生产环境中用的比较多的是DR模式,NAT模式有瓶颈,好在节省公网IP,对小公司来说公网IP也是要花钱的:

如果采用DR模式是配置多台机器,每天机器都要配置公网IP也是要花钱的:而在当下的IP也越来越少:

而另一种方案:搭建内部的lvs,全部都用到内网,包括vip也用内网,用一个公网IP+80端口对内网的VIP地址+80端口做一个映射:

准备三台机器:一般是调度器和RS均用内网的IP,然后只需要一个公网IP(VIP),然后做内网端口映射则可以了,公网的80端口映射到内网80端口:

调度器(director):192.168.149.129

real   server  1(RS1):192.168.149.131

real  server   2(RS2):192.168.149.132

 VIP : 192.168.149.254

1:首先编写调度器dir的配置脚本:  /usr/local/sbin/lvs_dr.sh

[root@localhost_02 ~]# vim /usr/local/sbin/lvs_dr.sh 
#! /bin/bash
echo 1 > /proc/sys/net/ipv4/ip_forward
#开启路由转发:
ipv=/usr/sbin/ipvsadm
vip=192.168.149.254
rs1=192.168.149.131	
rs2=192.168.149.132
ifdown eth0
ifup eth0
#在此重启网卡的目的是避免重复设置命令行提供的IP:
ifconfig eth0:2 $vip broadcast $vip netmask 255.255.255.255 up
#绑定VIP到dir的虚拟网卡ens33:2
route add -host $vip dev eth0:2
#添加网关
$ipv -C
$ipv -A -t $vip:80 -s wrr
$ipv -a -t $vip:80 -r $rs1:80 -g -w 1
$ipv -a -t $vip:80 -r $rs2:80 -g -w 1
#设置ipvsadm规则,-g=gateway:使用默认网关(DR模式)

注释:查看dr的网卡,发现vip地址绑定到eth0上面: 

[root@localhost_02 ~]# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:37:3b:d9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.149.129/24 brd 192.168.149.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.149.254/32 brd 192.168.149.254 scope global eth0:2
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe37:3bd9/64 scope link 
       valid_lft forever preferred_lft forever

2、配置real  server(RS):需要分别在RS1和RS2上执行:      /usr/local/sbin/lvs_rs.sh

[root@localhost_03 ~]# vim /usr/local/sbin/lvs_rs.sh
#/bin/bash
vip=192.168.149.254
#把vip绑定在lo上,是为了实现rs直接把结果返回给客户端
ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up
route add -host $vip lo:0
#以下操作为更改arp内核参数,目的是为了让rs顺利发送mac地址给客户端
#参考文档www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

第二RS:
[root@localhost_04 network-scripts]# cat /usr/local/sbin/lvs_rs.sh 
#/bin/bash
vip=192.168.149.254
#把vip绑定在lo上,是为了实现rs直接把结果返回给客户端
ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up
route add -host $vip lo:0
#以下操作为更改arp内核参数,目的是为了让rs顺利发送mac地址给客户端
#参考文档www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
[root@localhost_04 network-scripts]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.149.2   0.0.0.0         UG    100    0        0 eth0
192.168.149.0   0.0.0.0         255.255.255.0   U     100    0        0 eth0
192.168.149.254 0.0.0.0         255.255.255.255 UH    0      0        0 lo

注释:查看其路由网关地址:

注释:更改arp内核参数:参考文档www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html

第二RS:
[root@localhost_04 network-scripts]# cat /usr/local/sbin/lvs_rs.sh 
#/bin/bash
vip=192.168.149.254
#把vip绑定在lo上,是为了实现rs直接把结果返回给客户端
ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up
route add -host $vip lo:0
#以下操作为更改arp内核参数,目的是为了让rs顺利发送mac地址给客户端
#参考文档www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

3、查看IP信息和VIP信息,发现其绑定在lo网卡上: 

[root@localhost_03 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 192.168.149.254/32 brd 192.168.149.254 scope global lo:0
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

4、测试,在浏览器上测试:192.168.149.254     访问页面会在RS1和RS2跳转:

824aeb40fa9ad450c2152e4ac9a6fd5d153.jpg

注释:curl命令访问这个vip(curl测试vip在rs上不太好用的,因为在本机绑定了这个vip,若是访问vip,等于访问自己),但是直接在A机器上去访问vip会发现失败,只能再开一个虚拟机来测试,不过用 ipvsadm -ln 命令,会看到ActiveConn都会有变化,表示实验成功:

[root@localhost_02 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.149.254:80 wrr
  -> 192.168.149.131:80           Route   1      0          4         
  -> 192.168.149.132:80           Route   1      0          5  

然后我们再开一个虚拟机来测试:

[root@localhost_01 ~]# curl  192.168.149.254
The is real server_01 !!!
[root@localhost_01 ~]# curl  192.168.149.254
The is real server_02 !!!
[root@localhost_01 ~]# curl  192.168.149.254
The is real server_01 !!!
[root@localhost_01 ~]# curl  192.168.149.254
The is real server_02 !!!
[root@localhost_01 ~]# curl  192.168.149.254
The is real server_01 !!!
[root@localhost_01 ~]# curl  192.168.149.254
The is real server_02 !!!
[root@localhost_01 ~]# curl  192.168.149.254

注释:

arp_ignore:定义对目标地址为本地IP的ARP询问不同的应答模式0 

0 - (默认值): 回应任何网络接口上对任何本地IP地址的arp查询请求 

1 - 只回答目标IP地址是来访网络接口本地地址的ARP查询请求 

2 -只回答目标IP地址是来访网络接口本地地址的ARP查询请求,且来访IP必须在该网络接口的子网段内 

3 - 不回应该网络界面的arp请求,而只对设置的唯一和连接地址做出回应 

4-7 - 保留未使用 

8 -不回应所有(本地地址)的arp查询:

arp_announce:对网络接口上,本地IP地址的发出的,ARP回应,作出相应级别的限制: 确定不同程度的限制,宣布对来自本地源IP地址发出Arp请求的接口 

0 - (默认) 在任意网络接口(eth0,eth1,lo)上的任何本地地址 

1 -尽量避免不在该网络接口子网段的本地地址做出arp回应. 当发起ARP请求的源IP地址是被设置应该经由路由达到此网络接口的时候很有用.此时会检查来访IP是否为所有接口上的子网段内ip之一.如果改来访IP不属于各个网络接口上的子网段内,那么将采用级别2的方式来进行处理. 

2 - 对查询目标使用最适当的本地地址.在此模式下将忽略这个IP数据包的源地址并尝试选择与能与该地址通信的本地地址.首要是选择所有的网络接口的子网中外出访问子网中包含该目标IP地址的本地地址. 如果没有合适的地址被发现,将选择当前的发送网络接口或其他的有可能接受到该ARP回应的网络接口来进行发送.

设置参数的时候将arp_ignore 设置为1,意味着当别人的arp请求过来的时候,如果接收的设备上面没有这个ip,就不做出响应,默认是0,只要这台机器上面任何一个设备上面有这个ip,就响应arp请求,并发送mac地址:

汇总:lvs不管是nat还是dr模式,配置过程都不是很复杂,需要注意是修改内核参数,端口转发,另外NAT模式比较重要的是RS的网关要设置dir的IP地址:

2、keepalived+lvs  dr模式的集合

完整的架构需要两台角色为DR(分发器)的服务器,分别安装keepalived服务,目的实现高可用:

keepalived内置的ipvsadm功能,所以不再需要安装ipvsadm这个包,也不用编写和执行lvs_dr.sh那个脚本了:

四台机器分别如下

dir_01:192.168.149.129

dir_02:192.168.149.130

rs_01:192.168.149.131

rs_02:192.168.149.132

1、编辑配置文件/etc/keepalived/keepalived.conf   #keepalived配置文件:

两台rs上都需要执行/usr/local/sbin/lvs_rs.sh

keepalived有一个好的功能,可以在一台rs宕机时,不再把请求转发过去

注释:为什么要在lvs中加入了keepalived功能:

1:因为lvs他又个关键角色,就是dir分发器,如果其中一台分发器挂了,那所有的访问请求都会终止,因为所有的流量入口都在分发器这里,所以需要给分发器做一个高可用,用keepalived实现高可用,并且keepalived还有负载均衡的功能:

2:在使用lvs时,如果其中一台RS挂了,lvs还是会转发数据到这台挂了RS上,会出现无法访问的情况,而如果使用了keepalived的话,web还能正常访问的,一般会是两台keepalived的设备:

因为keepalived内置了ipvsadm功能,所以不需要在安装ipvsadm了,也不需要执行lvs_dir.sh这个脚本:

准备四台机器分别如下:

dir_01-A:192.168.149.129  (需要安装keepalived软件):

dir_02-B:192.168.149.130

rs_01:192.168.149.131

rs_02:192.168.149.132

在两台dir上A和B修改配置文件内容:/etc/keepalived/keepalived.conf      #keepalived配置文件:

A机器修改配置并启动:        systemctl     start        keepalived

dir_A机器修改配置:
[root@localhost_01 ~]# cat /etc/keepalived/keepalived.conf 
vrrp_instance VI_1 {
    #备用服务器上为 BACKUP
    state MASTER
    #绑定vip的网卡为eth0,你的网卡可能不一样,这里需要你改一下
    interface eth0
    virtual_router_id 50
    #备用服务器上为90
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass aminglinux
    }
    virtual_ipaddress {
        192.168.149.254
    }
}
virtual_server 192.168.149.254 80 {
    #(每隔10秒查询realserver状态)
    delay_loop 10
    #(lvs 算法)
    lb_algo wlc
    #(DR模式)
    lb_kind DR
    #(同一IP的连接60秒内被分配到同一台realserver)
    persistence_timeout 0
    #(用TCP协议检查realserver状态)
    protocol TCP

    real_server 192.168.149.131 80 {
        #(权重)
        weight 100
        TCP_CHECK {
        #(10秒无响应超时)
        connect_timeout 10
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
        }
    }
    real_server 192.168.149.132 80 {
        weight 100
        TCP_CHECK {
        connect_timeout 10
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
        }
     }
}

启动keepalived:  systemctl      start     keepalived

[root@localhost_01 ~]# systemctl start keepalived
[root@localhost_01 ~]# ps aux |grep keepalived
root       1363  0.0  0.1 118652  1392 ?        Ss   22:02   0:00 /usr/sbin/keepalived -D
root       1364  0.0  0.3 127520  3336 ?        S    22:02   0:00 /usr/sbin/keepalived -D
root       1365  0.0  0.2 127388  2612 ?        S    22:02   0:00 /usr/sbin/keepalived -D
root       1383  0.0  0.0 112720   972 pts/0    R+   22:05   0:00 grep --color=auto keepalived

B机器(bakup)修改配置:并启动keepalibved:

B机器修改配置:
[root@localhost_02 ~]# cat /etc/keepalived/keepalived.conf 
vrrp_instance VI_1 {
    #主用服务器上为 MASTER
    state BACKUP
    #绑定vip的网卡为eth0,你的网卡可能不一样,这里需要你改一下
    interface eth0
    virtual_router_id 50
    #备用服务器上为90
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass aminglinux
    }
    virtual_ipaddress {
        192.168.149.254
    }
}
virtual_server 192.168.149.254 80 {
    #(每隔10秒查询realserver状态)
    delay_loop 10
    #(lvs 算法)
    lb_algo wlc
    #(DR模式)
    lb_kind DR
    #(同一IP的连接60秒内被分配到同一台realserver)
    persistence_timeout 0
    #(用TCP协议检查realserver状态)
    protocol TCP

    real_server 192.168.149.131 80 {
        #(权重)
        weight 100
        TCP_CHECK {
        #(10秒无响应超时)
        connect_timeout 10
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
        }
    }
    real_server 192.168.149.132 80 {
        weight 100
        TCP_CHECK {
        connect_timeout 10
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
        }
     }
}

启动keepalived:   systemctl       start     keepalived

[root@localhost_02 ~]# systemctl start keepalived
[root@localhost_02 ~]# ps aux |grep keep
root       2810  0.0  0.0 118608  1380 ?        Ss   22:01   0:00 /usr/sbin/keepalived -D
root       2811  0.0  0.1 127472  3336 ?        S    22:01   0:00 /usr/sbin/keepalived -D
root       2812  0.0  0.1 127340  2612 ?        S    22:01   0:00 /usr/sbin/keepalived -D
root       2833  0.0  0.0 112676   984 pts/0    S+   22:06   0:00 grep --color=auto keep

2、分别启动RS_01和RS_02的nginx服务:         systemctl     start     nginx

RS_01
[root@localhost_03 ~]# systemctl start nginx
[root@localhost_03 ~]# ps aux |grep nginx
root       1032  0.0  0.2 120752  2260 ?        Ss   17:50   0:00 nginx: master process /usr/sbin/nginx
nginx      1033  0.0  0.3 121136  3588 ?        S    17:50   0:00 nginx: worker process
root       1233  0.0  0.0 112676   984 pts/0    R+   22:08   0:00 grep --color=auto nginx

RS_02
[root@localhost_04 sbin]# systemctl start nginx
[root@localhost_04 sbin]# ps aux |grep nginx
root       1021  0.0  0.2 120752  2256 ?        Ss   17:51   0:00 nginx: master process /usr/sbin/nginx
nginx      1022  0.0  0.3 121136  3588 ?        S    17:51   0:00 nginx: worker process
root       1249  0.0  0.0 112676   984 pts/0    S+   22:09   0:00 grep --color=auto nginx

注释:两台RS上需要执行:    /usr/local/lvs_rs.sh      

[root@localhost_03 ~]# vim /usr/local/sbin/lvs_rs.sh
#/bin/bash
vip=192.168.149.254
#把vip绑定在lo上,是为了实现rs直接把结果返回给客户端
ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up
route add -host $vip lo:0
#以下操作为更改arp内核参数,目的是为了让rs顺利发送mac地址给客户端
#参考文档www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

3、首先我们查看dir_02(主keepalived)这台的机器的虚拟IP信息:发现192.168.149.254存在

[root@localhost_01 ~]# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:81:f4:4b brd ff:ff:ff:ff:ff:ff
    inet 192.168.149.130/24 brd 192.168.149.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.149.254/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe81:f44b/64 scope link 
       valid_lft forever preferred_lft forever

注释:查看规则:

[root@localhost_01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.149.254:80 wlc
  -> 192.168.149.131:80           Route   100    0          0         
  -> 192.168.149.132:80           Route   100    0          0  

注释:此时在dir_02(备keepalived)上是查询不到虚拟IP以及规则信息:并且默认状态下keeplived备机时不工作的,只有主keepalived宕机后才能工作:

测试:首先在另一台测试机测试,然后下面测试再分两步:

[root@ceshiji_01 ~]# curl 192.168.149.254                                     
The is real server_01 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_02 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_01 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_02 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_01 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_02 !!!

1、测试keepalive的高可用性: 通过宕掉A(手动关闭主的keepalive服务),然后看是否会由B(Bkeepalive)服务:

判定标准:看虚拟IP是否切换到B(原备keepalive)上:

A(主keepalive)操作: 关闭keepalive服务:                  #systemctl      stop     keepalived

[root@localhost_01 ~]# systemctl stop keepalived
[root@localhost_01 ~]# ps aux |grep keepalived
root       1445  0.0  0.0 112720   976 pts/0    S+   22:34   0:00 grep --color=auto keepalived

B(备keepalive)查看虚拟IP是否切换过来, 然后看到网站是否可以正常访问:

[root@localhost_02 ~]# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:37:3b:d9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.149.129/24 brd 192.168.149.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.149.254/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe37:3bd9/64 scope link 
       valid_lft forever preferred_lft forever
[root@localhost_02 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.149.254:80 wlc
  -> 192.168.149.131:80           Route   100    0          2         
  -> 192.168.149.132:80           Route   100    0          3 

然后在测试机上测试:

[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_01 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_02 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_01 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_02 !!!

注释:说明A(主keepalive)宕机后,虚拟IP会自动切换到B(备keepalive)上,正好验证可dr分发器的高可用,并且当访问网站时,也是分别向两台服务器发起请求,体现了负载均衡性,不影响正常的网站访问:

测试:测试业务的负载均衡性:当宕了一台RS服务(rs_03),也不会影响网站的正常访问:

RS_03:关闭nginx服务:

[root@localhost_03 ~]# systemctl stop nginx
[root@localhost_03 ~]# ps aux |grep nginx
root       1250  0.0  0.0 112676   984 pts/0    S+   22:50   0:00 grep --color=auto nginx

然后再次访问:测试机上:         curl    192.168.1449.254

[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_02 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_02 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_02 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_02 !!!
[root@ceshiji_01 ~]# curl 192.168.149.254
The is real server_02 !!!

注释:keepalive有一个比较好的功能,可以在一台rs宕机的时候,及时把他踢出 ipvsadm 集群,将不再发送数据包给,也就很好的避免的访问无连接的情况发送:

注释:dir上:需要打开echo 1 > /proc/sys/net/ipv4/ip_forward //打开端口转发:

 

转载于:https://my.oschina.net/yuanhaohao/blog/2046440

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值