Nginx + keepalived 高可用配置文档

Nginx + keepalived 高可用配置文档

解决的问题：

解决服务器网线松动等网络故障

服务器硬件故障发生损坏现象而崩溃

Nginx服务死掉   ---> 用脚本保障

 

 

准备工作：

三步走：host文件,时间同步，关闭iptables

 

 

环境：

Centos6.2(64位)、 Nginx -1.2.6 、keepalived-1.2.7  

主nginx负载均衡器：192.168.10.46

辅nginx负载均衡器：192.168.10.97

vip：192.168.10.14

 

 

 

第一部分、先安装Nginx负载均衡器及相关脚本 (nginx服务配置跳过，按默认服务器上配置)

yum -y install gcc gcc+ gcc-c++ openssl openssl-devel

groupadd www

useradd -g www www

mkdir -p /data/logs/

chown -R www:www /data/logs

 

 

1、安装Nginx -1.2.6(最新稳定版本)

先要装pcre

cd /usr/src/pcre-8.31/

./configure && make && make install

 

cd /usr/src/nginx-1.2.6/

./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module

make && make install

 

 

 

 

2、配置nginx负载均衡器的配置文件vim /usr/local/nginx/conf/nginx.conf (纯80转发)

 

user www www; 

worker_processes 16; 

 

pid /usr/local/nginx/logs/nginx.pid; 

worker_rlimit_nofile 65535; 

 

events 

{ 

use epoll; 

worker_connections 65535; 

} 

http{

 

log_format access '$remote_addr - $remote_user [$time_local] "$request" ' 

'$status $body_bytes_sent "$http_referer" ' 

'"$http_user_agent" $http_x_forwarded_for'; 

 

include       mime.types; 

default_type application/octet-stream; 

server_names_hash_bucket_size 128; 

client_header_buffer_size 32k; 

large_client_header_buffers 4 32k; 

client_max_body_size 8m; 

sendfile on; 

tcp_nopush     on; 

keepalive_timeout 60; 

tcp_nodelay on; 

fastcgi_connect_timeout 300; 

fastcgi_send_timeout 300; 

fastcgi_read_timeout 300; 

fastcgi_buffer_size 64k; 

fastcgi_buffers 4 64k; 

fastcgi_busy_buffers_size 128k; 

fastcgi_temp_file_write_size 128k; 

gzip on; 

gzip_min_length 1k; 

gzip_buffers     4 16k; 

gzip_http_version 1.0; 

gzip_comp_level 2; 

gzip_types       text/plain application/x-javascript text/css application/xml; 

gzip_vary on; 

 

upstream backend 

{ 

ip_hash;

server 192.168.126.100:80; 

server 192.168.126.101:80; 

} 

server { 

listen 80; 

server_name www.wondershare.cn; 

location / { 

root /var/www/html; 

index index.jsp index.htm index.html; 

proxy_redirect off; 

proxy_set_header Host $host; 

proxy_set_header X-Real-IP $remote_addr; 

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 

proxy_pass http://backend; 

} 

 

location /nginx { 

access_log on; 

auth_basic "NginxStatus"; 

auth_basic_user_file /usr/local/nginx/htpasswd; 

} 

 

access_log /data/logs/access.log access; 

 

} 

}

 

/usr/local/nginx/sbin/nginx

 

 

 

##可能问题1##

nginx启动error while loading shared libraries: libpcre.so.1

从错误看出是缺少lib文件导致，进一步查看下

ldd $(which /usr/local/nginx/sbin/nginx)

可以看出 libpcre.so.1 => not found 并没有找到，

解决方法：

32位系统 [root@sever lib]# ln -s /usr/local/lib/libpcre.so.1 /lib

64位系统 [root@sever lib]# ln -s /usr/local/lib/libpcre.so.1 /lib64

##可能问题2##

nginx: [warn] the "log_format" directive may be used only on "http" level in......

将/usr/local/nginx/conf/nginx.conf 里server段里的下面代码移出放到该server段的前面即可;

logformat需要在nginx.conf的http层定义，然后在分域名下面就不用定义log_format,直接引用即可

 

 

 

第二部分：安装Keepalived，让其分别作web及Nginx的HA

 

主备DR上安装keepalived 并将其做成服务模式，方便以后调试。

# yum install popt-devel net-snmp* *libnl* -y  (两台在编译前先安装相关的依赖性)

#编译参数

./configure --prefix=/usr/local/keepalived

Keepalived configuration

------------------------

Keepalived version       : 1.2.7

Compiler                 : gcc

Compiler flags           : -g -O2

Extra Lib                : -lpopt -lssl -lcrypto  -lnl

Use IPVS Framework       : Yes

IPVS sync daemon support : Yes

IPVS use libnl           : Yes

Use VRRP Framework       : Yes

Use VRRP VMAC            : Yes

SNMP support             : No

Use Debug flags          : No

 

# 特殊编译

./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-220.el6.x86_64/ --with-kernel-version=2.6 --enable-snmp --enable-profile

Keepalived configuration

------------------------

Keepalived version       : 1.2.7

Compiler                 : gcc

Compiler flags           : -g -O2 -pg

Extra Lib                : -Wl,-z,relro -Wl,-z,now -L/usr/lib64 -lnetsnmpagent -lnetsnmphelpers -lnetsnmpmibs -lnetsnmp -Wl,-E -Wl,-rpath,/usr/lib64/perl5/CORE -lpopt -lssl -lcrypto  -lnl

Use IPVS Framework       : Yes

IPVS sync daemon support : Yes

IPVS use libnl           : Yes

Use VRRP Framework       : Yes

Use VRRP VMAC            : Yes

SNMP support             : Yes

Use Debug flags          : No

 

make && make install

 

 

下面的步骤是主备DR都要做的

  cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/ #拷贝服务脚本

  cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/  #拷贝全局配置文件

  mkdir /etc/keepalived

  cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ #拷贝主配置文件

  cp /usr/local/keepalived/sbin/keepalived /usr/sbin/  #拷贝命令

 

         /etc/init.d/keepalived   restart   --看服务是否可以启动，能启动表示OK

 

 

第二步：修改主备DR的配置文件

 

 

主DR 配置文件

 

Master DR->vim /etc/keepalived/keepalived.conf

 

! Configuration File for keepalived

global_defs {

   notification_email {

     wuchao@wondershare.cn

     18026935656@189.cn

   }

   notification_email_from root@wondershare.cn

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id LVS_DEVEL

}

 

vrrp_instance VI_1 {

    state MASTER

    interface eth0

    virtual_router_id 51

    priority 100

    mcast_src_ip 192.168.10.46

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.10.14/24

    }

}

#备机上配置：

! Configuration File for keepalived

global_defs {

   notification_email {

     wuchao@wondershare.cn

     18026935656@189.cn

   }

   notification_email_from root@wondershare.cn

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id LVS_DEVEL

}

 

vrrp_instance VI_1 {

    state BACKUP          # 从

    interface eth0

    virtual_router_id 51  # 与主要一至

    priority 99           # 比主要小

    mcast_src_ip 192.168.10.97  #

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.10.14/24

    }

}

 

启动keepalived

 

 

 

第三部分：针对Keepalived的不足，用Nginx_pid.sh来监控nginx进程，实现真正意义上的负载均衡高可用。

vim /wuchao/nginx_pid.sh  

#!/bin/bash 

while  : 

do 

 nginxpid=`ps -C nginx --no-header | wc -l` 

 if [ $nginxpid -eq 0 ];then 

  /usr/local/nginx/sbin/nginx 

  sleep 5 

  nginxpid=`ps -C nginx --no-header | wc -l`

   if [ $nginxpid -eq 0 ];then 

   /etc/init.d/keepalived stop 

   fi 

 fi 

 sleep 5   

done

 

置于(主)后台运行：

nohup /bin/bash /wuchao/nginx_pid.sh &