lvs和Nginx部署说明

LVS

安装

su

yum install popt-devel libnl-devel popt-static install openssl-devel libnfnetlink-devel gcc

tar -zxvf ipvsadm-1.26.tar.gz

tar -zxvf keepalived-1.2.24.tar.gz

cd ipvsadm-1.26

make && make install

cd keepalived-1.2.24

./configure && make && make install

 

cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/

cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived/

cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/

cp /usr/local/sbin/keepalived /usr/sbin/

chown appdeploy:mwopr /etc/keepalived/ -R

配置

参考SIT环境: 10.202.94.75/10.202.94.76

vi /etc/keepalived/keepalived.conf

 

! Configuration File for keepalived

global_defs {

   router_id LVS_DEVEL

}

vrrp_instance VI_1 {

    state MASTER // 备机改成BACKUP

    interface eth0

    virtual_router_id 66 需要修改

    priority 100 // 备机改成99

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        10.202.95.25 // VIP

    }

}

virtual_server 10.202.95.25 443 { // VIP Port

    delay_loop 6

    lb_algo wlc

    lb_kind DR

    persistence_timeout 180

    protocol TCP

 

    real_server 10.202.95.26 443 { // Real Server IP Port

        weight 10

        TCP_CHECK {

            connect_timeout 10

            nb_get_retry 3

            delay_before_retry 3

            connect_port 443

        }

    }

      

       real_server 10.202.95.27 443 { // Real Server IP Port

        weight 10

        TCP_CHECK {

            connect_timeout 10

            nb_get_retry 3

            delay_before_retry 3

            connect_port 443

        }

    }

}

 

Log日志查看在/var/log/messages

Nginx

安装

1. 上传openresty-11.2.1.tar.gz, openssl-1.0.2h.tar.gz;
2. 依次执行以下命令:

su

yum install readline-devel pcre-devel openssl-devel gcc

tar -zxvf openresty-1.11.2.1.tar.gz

tar -zxvf openssl-1.0.2h.tar.gz

cd openresty-1.11.2.1

(./configure --prefix=/app/openresty --with-http_v2_module --with-openssl=/app/backup/openssl-1.0.2h/ && make && make install)

(注：提示undefined reference to 'pcre_free_study' 的问题及解决, 在末尾加上 --with-ld-opt="-L /usr/local/lib", 这个路径是pcre的安装路径)

chown appdeploy:mwopr -R /app/openresty/

cp /app/openresty/nginx/sbin/nginx /usr/sbin/nginx

修改nginx.conf

worker_processes  auto;

 

events {

    worker_connections  10240;

}

 

http {

worker_processes  auto;

 

events {

    worker_connections  10240;

}

 

http {

worker_processes  auto;

 

events {

    worker_connections  10240;

}

 

http {

    gzip  on;

    gzip_types text/plain application/x-javascript text/css application/json application/xml text/javascript;

    client_max_body_size 10m;

 

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

                      '$status $body_bytes_sent "$http_referer" '

                      '"$http_user_agent" "$http_x_forwarded_for" '

                      '$request_time $upstream_addr $upstream_response_time';

    access_log  logs/access.log  main;

 

    upstream nginx_wa {

        ip_hash;

        server 10.202.39.164:8080;  改成内部网关LVS

        server 10.202.39.165:8080;

        keepalive 1024;

    }

 

    server {

        listen       443 ssl http2 default_server;

        server_name  localhost;

 

        ssl_certificate      /app/ssl/SGS-GW-CORE-LVS.pem;

        ssl_certificate_key  /app/ssl/SGS-GW-CORE-LVS.key;

 

        location / {

            proxy_pass https://nginx_wa;

            proxy_http_version 1.1;

            proxy_set_header Connection "";

            proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;

        }

    }

}

 

配置

参考SIT环境

外网NGX_WEB: 10.202.94.78/10.202.94.79

内网NGX_WA:  10.202.38.100/10.202.38.101

 

 

Lvs-rs.sh 配置

在外网NGX_WEB(10.202.94.78、10.202.94.79)app下面新建lvs目录

mkdir lvs

vi lvs-rs.sh

 

#!/bin/bash

VIP=10.202.94.77 vip的IP

# . /etc/rc.d/init.d/functions

case "$1" in

start)

  /sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP

  /sbin/route add -host $VIP dev lo:0

  echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

  echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

  echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

  echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

  echo "1" >/proc/sys/net/ipv4/conf/eth0/arp_ignore

  echo "2" >/proc/sys/net/ipv4/conf/eth0/arp_announce

  sysctl -p >/dev/null 2>&1

  echo "RealServer Start OK"

  ;;

stop)

  /sbin/ifconfig lo:0 down

  /sbin/route del $VIP >/dev/null 2>&1

  echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

  echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

  echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

  echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

  echo "0" >/proc/sys/net/ipv4/conf/eth0/arp_ignore

  echo "0" >/proc/sys/net/ipv4/conf/eth0/arp_announce

  echo "RealServer Stoped"

  ;;

status)

  # Status of LVS-DR real server.

  islothere=`/sbin/ifconfig lo:0 | grep $VIP`

  isrothere=`netstat -rn | grep "lo:0" | grep $VIP`

  if [ ! "$islothere" -o ! "isrothere" ];then

  # Either the route or the lo:0 device not found.

  echo "LVS-DR real server Stopped."

  else

  echo "LVS-DR Running."

  fi

  ;;

*)

  echo "Usage: $0 {start|status|stop}"

  exit 1

esac

exit 0

 

启动

放在LVS后面执行

 

问题排查

lvs和realserver的IP必须在同一个网段

haproxy可以跨网段

 

依次访问以下地址验证是否返回{"status":"ok"}:

1. https://sgs-gw-lvs.sit.sf-express.com:5008/index.html
2. https://10.202.94.77/index.html
3. https://10.202.94.78/index.html

https://10.202.94.79/index.html

1. https://10.202.38.100:8080/index.html

https://10.202.38.101:8080/index.html

(因为sit环境使用的SSL自签名证书, 所以都会有安全警告, 忽略即可)

 

1. 如果a地址访问不通, b地址ok, 那应该是域名解析或者防火墙有问题, 这个我们自己无法解决, 只能找基础架构同事.(目前还未出现过)
2. 如果ab地址访问不通, c是ok的, 那应该是lvs出问题了, 重启lvs服务验证.(一般这里可能出问题多一些, 通常都是因为有人重启lvs服务器了)
3. 如果abc地址访问不通, d是ok的, 那可能是lvs或者外网nginx出问题了, 先重启外网nginx, 验证ok以后再继续排查是否lvs有问题
4. 如果abcd均访问不通, 那可能是内网nginx出问题了, 先重启内网nginx, 验证ok以后再继续依次排查外网nginx和lvs.

LVS重启

10.202.94.75(主)/10.202.94.76(备)

 

查看状态:

sudo service keepalived status

停止:

sudo service keepalived stop

启动:

sudo service keepalived start

 

验证:

sudo ipvsadm –Ln(注：需要手动输入才可以ipvsadm -Ln)

看到有连接数即ok

 

如果看不到连接数,去重启一下外网nginx上的real server（/app/lvs/lvs-rs.sh start

）服务, 再重启一遍lvs验证.

外网Nginx重启

10.202.94.78/10.202.94.79

 

查看nginx服务是否存活:

ps -ef | grep nginx

停止nginx:

sudo nginx -s stop

启动nginx:

sudo nginx

 

查看lvs real server服务:

sudo /app/lvs/lvs-rs.sh status

停止lvs real server服务:

sudo /app/lvs/lvs-rs.sh stop

启动lvs real server服务:

sudo /app/lvs/lvs-rs.sh start

 

内网Nginx重启

10.202.38.100/10.202.38.101

 

查看nginx服务是否存活:

ps -ef | grep nginx

停止nginx:

cd /app/openresty/nginx/sbin/

./nginx -s stop

启动nginx:

cd /app/openresty/nginx/sbin/

./nginx