Lvs下使用Docker扩容实验

Lvs下使用Docker扩容实验

 

拓扑

搭环境

LVS

         1. 安装组件：keepalived和lvs

注意，keepalived apt-get安装的版本有bug，在自动测试rs时，一直报出错，不能自动探测，下载1.2.19版本的源码编译安装后觖决。

apt-getupdate

apt-getinstall ipvsadm

apt-getinstall keepalived

 

         2. 配置

                   主要是/etc/keepalived/keepalived.conf文件。

root@ubuntu:~/dev/keepalived-1.2.19#cat /etc/keepalived/keepalived.conf

#GlobalConfiguration

global_defs{

     lvs_id lvs_m

}

 

#VRRPConfiguration

vrrp_instanceZGYE_LVS {

     state MASTER          #LVS_S上修发成BACKUP

     interface eth0

     virtual_router_id 60

     priority 100                #LVS_S上修发成50,小于LVS_M，以确定哪个为主

     advert_int 5

     authentication {

              auth_type PASS

              auth_pass zgye_test

     }

     virtual_ipaddress {

              172.18.2.100

     }

 

     #Virtual Server Configu - for WWW service

     virtual_server 172.18.2.100 80 {

              delay_loop 1

              lb_algo rr

              lb_kind DR

              persistence_timerout 60

              protocol TCP

              #Real Server 1 Config - web1

              real_server 172.18.2.83 80 {

                       weight 1

                       TCP_CHECK {

                                 connection_timeout10

                                 nb_get_retry 3

                                 delay_before_retry3

                                 connect_port 80

                       }

              }

              real_server 172.18.2.85 80 {

                       weight 1

                       TCP_CHECK {

                                 connection_timeout10

                                 nb_get_retry 3

                                 delay_before_retry3

                                 connect_port 80

                       }

              }

              real_server 172.18.2.125 80 {

                       weight 1

                       TCP_CHECK {

                                 connection_timeout10

                                 nb_get_retry 3

                                 delay_before_retry3

                                 connect_port 80

                       }

              }

     }

}

 

RS

VM

1.        安装组件: nginx

apt-get install nginx

2.        配置nginx

         主要是修改 /usr/share/nginx/html/index.html，以区分哪个rs做了应答。

3.        增加vip

ifconfig eth0:0 172.18.2.100 broadcast172.18.2.100 netmask 255.255.255.255 up

route add -host 172.18.2.100 deveth0:0

4.        修改arp过滤

echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore

echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce

echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

 

DOCKER

1.        安装docker

apt-key adv --keyserverhkp://keyserver.ubuntu.com:80 --recv-keys36A1D7869245C8950F966E92D8576A8BA88D21E9

sh -c "echo debhttps://get.docker.com/ubuntu docker main >/etc/apt/sources.list.d/docker.list"

apt-get update

apt-get install -y lxc-docker gitmake

source/etc/bash_completion.d/docker

2.        下载nginx镜像

docker pull nginx

3.        运行nginx容器

docker run -it --net=none--name=nginx2 nginx /bin/bash

4.        配置固定ip

使用脚本配置：

./docker_static_ip.sh a2ad572f9ca2172.18.2.125 255.255.255.128 172.18.2.1 nginx_2

其中脚本为：

root@OA-72:~# catdocker_static_ip.sh

#/bin/bash

BR_DEV=br0

if [ -z $1 ] || [ -z $2 ] || [ -z$3 ] || [ -z $4 ] || [ -z $5 ];

then

        echo "*****Input the necessaryparameters: CONTAINERID IP MASK GATEWAY ETHNAME"

        echo "*****Call the script like:sh manual_con_static_ip.sh  b0e18b6a4432192.168.5.123 24 192.168.5.1 deth0"

        exit

fi

 

CONTAINERID=$1

SETIP=$2

SETMASK=$3

GATEWAY=$4

ETHNAME=$5

 

ifconfig $ETHNAME > /dev/null2>&1

if [ $? -eq 0 ]; then

   read -p "$ETHNAME exist,do you want delelte it? y/n " del

   if [[ $del == 'y' ]]; then

   ip link del $ETHNAME

   else

   exit

   fi

fi

#

pid=`docker inspect -f'{{.State.Pid}}' $CONTAINERID`

mkdir -p /var/run/netns

find -L /var/run/netns -type l-delete

 

if [ -f /var/run/netns/$pid ]; then

   rm -f /var/run/netns/$pid

fi

ln -s /proc/$pid/ns/net/var/run/netns/$pid

#

ip link add $ETHNAME type veth peername B

brctl addif $BR_DEV $ETHNAME

ip link set $ETHNAME up

ip link set B netns $pid

#delete eth0 in docker

ip netns exec $pid ip link del eth0> /dev/null 2>&1

#set eth0 in docker

ip netns exec $pid ip link set devB name eth0

ip netns exec $pid ip link set eth0up

ip netns exec $pid ip addr add$SETIP/$SETMASK dev eth0

ip netns exec $pid ip route adddefault via $GATEWAY

5.        配置vip

参考脚本中的配置ip命令：

ip netns exec 7397 ip addr add 172.18.2.100/25dev eth0

6.        启动nginx服务

docker exec -d nginx2 service nginxstart

数据报文

         Client:发请求到vip上。

        

         LVS:只接收请求，并修改请求发到rs，由rs做应答。

RS：收到第一个syn的smac为lvs的，但回的syn\ack的dmac变成了client的mac。

 

测试

在cilent上通过

root@ubuntu:~# for ((i=0; i<100; i++)); do curlhttp://172.18.2.100:80; done

访问来测试均衡情况

 

在113上可以看到：

root@ubuntu:~# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  ->RemoteAddress:Port           ForwardWeight ActiveConn InActConn

TCP 172.18.2.100:80 rr

  ->172.18.2.83:80               Route   1     0          33       

  ->172.18.2.85:80               Route   1     0          33       

  ->172.18.2.125:80              Route   1     0          34