LVS
安装
su
yum install popt-devel libnl-devel popt-static install openssl-devel libnfnetlink-devel gcc
tar -zxvf ipvsadm-1.26.tar.gz
tar -zxvf keepalived-1.2.24.tar.gz
cd ipvsadm-1.26
make && make install
cd keepalived-1.2.24
./configure && make && make install
cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/
cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
mkdir /etc/keepalived/
cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
cp /usr/local/sbin/keepalived /usr/sbin/
chown appdeploy:mwopr /etc/keepalived/ -R
配置
参考SIT环境: 10.202.94.75/10.202.94.76
vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER // 备机改成BACKUP
interface eth0
virtual_router_id 66 需要修改
priority 100 // 备机改成99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.202.95.25 // VIP
}
}
virtual_server 10.202.95.25 443 { // VIP Port
delay_loop 6
lb_algo wlc
lb_kind DR
persistence_timeout 180
protocol TCP
real_server 10.202.95.26 443 { // Real Server IP Port
weight 10
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 443
}
}
real_server 10.202.95.27 443 { // Real Server IP Port
weight 10
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 443
}
}
}
Log日志查看在/var/log/messages
Nginx
安装
- 上传openresty-11.2.1.tar.gz, openssl-1.0.2h.tar.gz;
- 依次执行以下命令:
su
yum install readline-devel pcre-devel openssl-devel gcc
tar -zxvf openresty-1.11.2.1.tar.gz
tar -zxvf openssl-1.0.2h.tar.gz
cd openresty-1.11.2.1
(./configure --prefix=/app/openresty --with-http_v2_module --with-openssl=/app/backup/openssl-1.0.2h/ && make && make install)
(注:提示undefined reference to 'pcre_free_study' 的问题及解决, 在末尾加上 --with-ld-opt="-L /usr/local/lib", 这个路径是pcre的安装路径)
chown appdeploy:mwopr -R /app/openresty/
cp /app/openresty/nginx/sbin/nginx /usr/sbin/nginx
修改nginx.conf
worker_processes auto;
events {
worker_connections 10240;
}
http {
worker_processes auto;
events {
worker_connections 10240;
}
http {
worker_processes auto;
events {
worker_connections 10240;
}
http {
gzip on;
gzip_types text/plain application/x-javascript text/css application/json application/xml text/javascript;
client_max_body_size 10m;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'$request_time $upstream_addr $upstream_response_time';
access_log logs/access.log main;
upstream nginx_wa {
ip_hash;
server 10.202.39.164:8080; 改成内部网关LVS
server 10.202.39.165:8080;
keepalive 1024;
}
server {
listen 443 ssl http2 default_server;
server_name localhost;
ssl_certificate /app/ssl/SGS-GW-CORE-LVS.pem;
ssl_certificate_key /app/ssl/SGS-GW-CORE-LVS.key;
location / {
proxy_pass https://nginx_wa;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
}
}
}
配置
参考SIT环境
外网NGX_WEB: 10.202.94.78/10.202.94.79
内网NGX_WA: 10.202.38.100/10.202.38.101
Lvs-rs.sh 配置
在外网NGX_WEB(10.202.94.78、10.202.94.79)app下面新建lvs目录
mkdir lvs
vi lvs-rs.sh
#!/bin/bash
VIP=10.202.94.77 vip的IP
# . /etc/rc.d/init.d/functions
case "$1" in
start)
/sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP
/sbin/route add -host $VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/eth0/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/eth0/arp_announce
sysctl -p >/dev/null 2>&1
echo "RealServer Start OK"
;;
stop)
/sbin/ifconfig lo:0 down
/sbin/route del $VIP >/dev/null 2>&1
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/eth0/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/eth0/arp_announce
echo "RealServer Stoped"
;;
status)
# Status of LVS-DR real server.
islothere=`/sbin/ifconfig lo:0 | grep $VIP`
isrothere=`netstat -rn | grep "lo:0" | grep $VIP`
if [ ! "$islothere" -o ! "isrothere" ];then
# Either the route or the lo:0 device not found.
echo "LVS-DR real server Stopped."
else
echo "LVS-DR Running."
fi
;;
*)
echo "Usage: $0 {start|status|stop}"
exit 1
esac
exit 0
启动
放在LVS后面执行
问题排查
lvs和realserver的IP必须在同一个网段
haproxy可以跨网段
依次访问以下地址验证是否返回{"status":"ok"}:
- https://sgs-gw-lvs.sit.sf-express.com:5008/index.html
- https://10.202.94.77/index.html
- https://10.202.94.78/index.html
https://10.202.94.79/index.html
https://10.202.38.101:8080/index.html
(因为sit环境使用的SSL自签名证书, 所以都会有安全警告, 忽略即可)
- 如果a地址访问不通, b地址ok, 那应该是域名解析或者防火墙有问题, 这个我们自己无法解决, 只能找基础架构同事.(目前还未出现过)
- 如果ab地址访问不通, c是ok的, 那应该是lvs出问题了, 重启lvs服务验证.(一般这里可能出问题多一些, 通常都是因为有人重启lvs服务器了)
- 如果abc地址访问不通, d是ok的, 那可能是lvs或者外网nginx出问题了, 先重启外网nginx, 验证ok以后再继续排查是否lvs有问题
- 如果abcd均访问不通, 那可能是内网nginx出问题了, 先重启内网nginx, 验证ok以后再继续依次排查外网nginx和lvs.
LVS重启
10.202.94.75(主)/10.202.94.76(备)
查看状态:
sudo service keepalived status
停止:
sudo service keepalived stop
启动:
sudo service keepalived start
验证:
sudo ipvsadm –Ln(注:需要手动输入才可以ipvsadm -Ln)
看到有连接数即ok
如果看不到连接数,去重启一下外网nginx上的real server(/app/lvs/lvs-rs.sh start
)服务, 再重启一遍lvs验证.
外网Nginx重启
10.202.94.78/10.202.94.79
查看nginx服务是否存活:
ps -ef | grep nginx
停止nginx:
sudo nginx -s stop
启动nginx:
sudo nginx
查看lvs real server服务:
sudo /app/lvs/lvs-rs.sh status
停止lvs real server服务:
sudo /app/lvs/lvs-rs.sh stop
启动lvs real server服务:
sudo /app/lvs/lvs-rs.sh start
内网Nginx重启
10.202.38.100/10.202.38.101
查看nginx服务是否存活:
ps -ef | grep nginx
停止nginx:
cd /app/openresty/nginx/sbin/
./nginx -s stop
启动nginx:
cd /app/openresty/nginx/sbin/
./nginx