Nginx + keepalived 高可用配置文档
解决的问题:
解决服务器网线松动等网络故障
服务器硬件故障发生损坏现象而崩溃
Nginx服务死掉 ---> 用脚本保障
准备工作:
三步走:host文件,时间同步,关闭iptables
环境:
Centos6.2(64位)、 Nginx -1.2.6 、keepalived-1.2.7
主nginx负载均衡器:192.168.10.46
辅nginx负载均衡器:192.168.10.97
vip:192.168.10.14
第一部分、先安装Nginx负载均衡器及相关脚本 (nginx服务配置跳过,按默认服务器上配置)
yum -y install gcc gcc+ gcc-c++ openssl openssl-devel
groupadd www
useradd -g www www
mkdir -p /data/logs/
chown -R www:www /data/logs
1、安装Nginx -1.2.6(最新稳定版本)
先要装pcre
cd /usr/src/pcre-8.31/
./configure && make && make install
cd /usr/src/nginx-1.2.6/
./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module
make && make install
2、配置nginx负载均衡器的配置文件vim /usr/local/nginx/conf/nginx.conf (纯80转发)
user www www;
worker_processes 16;
pid /usr/local/nginx/logs/nginx.pid;
worker_rlimit_nofile 65535;
events
{
use epoll;
worker_connections 65535;
}
http{
log_format access '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $http_x_forwarded_for';
include mime.types;
default_type application/octet-stream;
server_names_hash_bucket_size 128;
client_header_buffer_size 32k;
large_client_header_buffers 4 32k;
client_max_body_size 8m;
sendfile on;
tcp_nopush on;
keepalive_timeout 60;
tcp_nodelay on;
fastcgi_connect_timeout 300;
fastcgi_send_timeout 300;
fastcgi_read_timeout 300;
fastcgi_buffer_size 64k;
fastcgi_buffers 4 64k;
fastcgi_busy_buffers_size 128k;
fastcgi_temp_file_write_size 128k;
gzip on;
gzip_min_length 1k;
gzip_buffers 4 16k;
gzip_http_version 1.0;
gzip_comp_level 2;
gzip_types text/plain application/x-javascript text/css application/xml;
gzip_vary on;
upstream backend
{
ip_hash;
server 192.168.126.100:80;
server 192.168.126.101:80;
}
server {
listen 80;
server_name www.wondershare.cn;
location / {
root /var/www/html;
index index.jsp index.htm index.html;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://backend;
}
location /nginx {
access_log on;
auth_basic "NginxStatus";
auth_basic_user_file /usr/local/nginx/htpasswd;
}
access_log /data/logs/access.log access;
}
}
/usr/local/nginx/sbin/nginx
##可能问题1##
nginx启动error while loading shared libraries: libpcre.so.1
从错误看出是缺少lib文件导致,进一步查看下
ldd $(which /usr/local/nginx/sbin/nginx)
可以看出 libpcre.so.1 => not found 并没有找到,
解决方法:
32位系统 [root@sever lib]# ln -s /usr/local/lib/libpcre.so.1 /lib
64位系统 [root@sever lib]# ln -s /usr/local/lib/libpcre.so.1 /lib64
##可能问题2##
nginx: [warn] the "log_format" directive may be used only on "http" level in......
将/usr/local/nginx/conf/nginx.conf 里server段里的下面代码移出放到该server段的前面即可;
logformat需要在nginx.conf的http层定义,然后在分域名下面就不用定义log_format,直接引用即可
第二部分:安装Keepalived,让其分别作web及Nginx的HA
主备DR上安装keepalived 并将其做成服务模式,方便以后调试。
# yum install popt-devel net-snmp* *libnl* -y (两台在编译前先安装相关的依赖性)
#编译参数
./configure --prefix=/usr/local/keepalived
Keepalived configuration
------------------------
Keepalived version : 1.2.7
Compiler : gcc
Compiler flags : -g -O2
Extra Lib : -lpopt -lssl -lcrypto -lnl
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
IPVS use libnl : Yes
Use VRRP Framework : Yes
Use VRRP VMAC : Yes
SNMP support : No
Use Debug flags : No
# 特殊编译
./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-220.el6.x86_64/ --with-kernel-version=2.6 --enable-snmp --enable-profile
Keepalived configuration
------------------------
Keepalived version : 1.2.7
Compiler : gcc
Compiler flags : -g -O2 -pg
Extra Lib : -Wl,-z,relro -Wl,-z,now -L/usr/lib64 -lnetsnmpagent -lnetsnmphelpers -lnetsnmpmibs -lnetsnmp -Wl,-E -Wl,-rpath,/usr/lib64/perl5/CORE -lpopt -lssl -lcrypto -lnl
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
IPVS use libnl : Yes
Use VRRP Framework : Yes
Use VRRP VMAC : Yes
SNMP support : Yes
Use Debug flags : No
make && make install
下面的步骤是主备DR都要做的
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/ #拷贝服务脚本
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ #拷贝全局配置文件
mkdir /etc/keepalived
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ #拷贝主配置文件
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/ #拷贝命令
/etc/init.d/keepalived restart --看服务是否可以启动,能启动表示OK
第二步:修改主备DR的配置文件
主DR 配置文件
Master DR->vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
wuchao@wondershare.cn
18026935656@189.cn
}
notification_email_from root@wondershare.cn
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
mcast_src_ip 192.168.10.46
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.10.14/24
}
}
#备机上配置:
! Configuration File for keepalived
global_defs {
notification_email {
wuchao@wondershare.cn
18026935656@189.cn
}
notification_email_from root@wondershare.cn
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP # 从
interface eth0
virtual_router_id 51 # 与主要一至
priority 99 # 比主要小
mcast_src_ip 192.168.10.97 #
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.10.14/24
}
}
启动keepalived
第三部分:针对Keepalived的不足,用Nginx_pid.sh来监控nginx进程,实现真正意义上的负载均衡高可用。
vim /wuchao/nginx_pid.sh
#!/bin/bash
while :
do
nginxpid=`ps -C nginx --no-header | wc -l`
if [ $nginxpid -eq 0 ];then
/usr/local/nginx/sbin/nginx
sleep 5
nginxpid=`ps -C nginx --no-header | wc -l`
if [ $nginxpid -eq 0 ];then
/etc/init.d/keepalived stop
fi
fi
sleep 5
done
置于(主)后台运行:
nohup /bin/bash /wuchao/nginx_pid.sh &