nginx做调度(自身用dns轮询保证高可用),varnish做后端的cache

Nginx + varnish 构建高可用CDN节点集群

作者:随风 发布于:2012-6-19 10:29 Tuesday 分类:linux运维

OSAPUB 出品的集群方案,旨在抛砖引玉,给广大朋友提供高可用CDN节点集群方案
转载请标明出去:http://bbs.osapub.com/thread-3370-1-1.html

一、        环境描述

Linux server A  (CentOS release 5.8 Final)  实IP:192.168.4.97  虚IP:192.168.4.96
Linux server B  (CentOS release 5.8 Final)  实IP:192.168.4.99  虚IP:192.168.4.98

域名环境(DNS轮询解析到虚IP):
server.osapub.com 192.168.4.96
server.osapub.com 192.168.4.98

二、        简单架构示意图
22.jpg
描述:前端两台NGINX做DNS轮询,通过虚IP漂移+HA监控脚本相结合,实现前端两台NGINX高可用,利用NGINX反向代理功能对后端varnish实现高可用集群。

三、软件环境搭建
3.1 编译安装nginx

001 ?
002 #!/bin/bash
003 ####nginx 环境安装脚本,注意环境不同可能导致脚本运行出错,如果环境不同建议手工一条一条执行指令。
004  
005 #创建工作目录
006 mkdir -p /dist/{dist,src}
007 cd /dist/dist
008  
009 #下载安装包
011 wget http://bbs.osapub.com/down/libunwind-0.99.tar.gz &> /dev/null
012 wget http://bbs.osapub.com/down/pcre-8.01.tar.gz &> /dev/null
013 wget http://bbs.osapub.com/down/nginx-1.0.5.tar.gz &> /dev/null
014  
015 #------------------------------------------------------------------------
016 # 使用Google的开源TCMalloc库,忧化性能
017  
018 cd /dist/src
019 tar zxf ../dist/libunwind-0.99.tar.gz
020 cd libunwind-0.99/
021  
022 ## 注意这里不能加其它 CFLAGS加速编译参数
023  
024 CFLAGS=-fPIC ./configure
025 make clean
026 make CFLAGS=-fPIC
027 make CFLAGS=-fPIC install
028 if [ "$?" == "0" ]; then
029         echo "libunwind-0.99安装成功." >> ./install_log.txt
030 else
031         echo "libunwind-0.99安装失败." >> ./install_log.txt
032         exit 1
033 fi
034  
035 ##----------------------------------------------------------
036 ## 使用Google的开源TCMalloc库,提高MySQL在高并发情况下的性能
037  
038 cd /dist/src
039 tar zxf ../dist/google-perftools-1.8.3.tar.gz
040 cd google-perftools-1.8.3/
041  
042 CHOST="x86_64-pc-linux-gnu" CFLAGS="-march=nocona -O2 -pipe" CXXFLAGS="-march=nocona -O2 -pipe" \
043 ./configure
044  
045 make clean
046 make && make install
047 if [ "$?" == "0" ]; then
048         echo "google-perftools-1.8.3安装成功." >> ./install_log.txt
049 else
050         echo "google-perftools-1.8.3安装失败." >> ./install_log.txt
051         exit 1
052 fi
053  
054 echo "/usr/local/lib" > /etc/ld.so.conf.d/usr_local_lib.conf
055 /sbin/ldconfig
056  
057 ################ 安装nginx ##########################
058 #安装Nginx所需的pcre库
059  
060 cd /dist/src
061 tar zxvf ../dist/pcre-8.01.tar.gz
062 cd pcre-8.01/
063  
064 CHOST="x86_64-pc-linux-gnu" CFLAGS="-march=nocona -O2 -pipe" CXXFLAGS="-march=nocona -O2 -pipe" \
065 ./configure
066 make && make install
067 if [ "$?" == "0" ]; then
068         echo "pcre-8.01安装成功." >> ./install_log.txt
069 else
070         echo "pcre-8.01安装失败." >> ./install_log.txt
071         exit 1
072 fi
073 cd ../
074  
075 ## 安装Nginx
076 ## 为优化性能,可以安装 google 的 tcmalloc ,之前己经安装过了
077 ## 所以我们编译 Nginx 时,加上参数 --with-google_perftools_module
078 ## 然后在启动nginx前需要设置环境变量 export LD_PRELOAD=/usr/local/lib/libtcmalloc.so
079 ## 加上 -O2 参数也能优化一些性能
080 ##
081 ## 默认的Nginx编译选项里居然是用 debug模式的(-g参数),在 auto/cc/gcc 文件最底下,去掉那个 -g 参数
082 ## 就是将  CFLAGS="$CFLAGS -g"  修改为   CFLAGS="$CFLAGS"   或者直接删除这一行
083  
084 cd /dist/src
085 rm -rf nginx-1.0.5
086 tar zxf ../dist/nginx-1.0.5.tar.gz
087 cd nginx-1.0.5/
088  
089 sed -i 's#CFLAGS="$CFLAGS -g"#CFLAGS="$CFLAGS "#' auto/cc/gcc
090  
091 make clean
092  
093 CHOST="x86_64-pc-linux-gnu" CFLAGS="-march=nocona -O2 -pipe" CXXFLAGS="-march=nocona -O2 -pipe" \
094 ./configure --user=www --group=www \
095 --prefix=/usr/local/nginx \
096 --with-http_stub_status_module \
097 --with-google_perftools_module
098  
099 make && make install
100 if [ "$?" == "0" ]; then
101         echo "nginx-1.0.5安装成功." >> ./install_log.txt
102 else
103         echo "nginx-1.0.5安装失败." >> ./install_log.txt
104         exit 1
105 fi
106 cd ../
107  
108 #创建Nginx日志目录
109 mkdir -p /data/logs
110 chmod +w /data/logs
111 chown -R www:www /data/logs
112  
113 cd /usr/local/nginx/
114 mv conf conf_bak
115 ln -s /data/conf/nginx/ conf
116  
117 echo 'export LD_PRELOAD=/usr/local/lib/libtcmalloc.so' > /root/nginx_start
118 echo 'ulimit -SHn 51200' >> /root/nginx_start
119 echo '/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf' >> /root/nginx_start
120  
121 echo '/usr/local/nginx/sbin/nginx -t' > /root/nginx_reload
122 echo 'kill -HUP `cat /usr/local/nginx/logs/nginx.pid`' >> /root/nginx_reload
123  
124 chmod 700 /root/nginx_*

3.2        编译安装varnish

01 #!/bin/bash
02  
03 #进入工作目录
04 cd /dist/dist
05  
06 #下载安装包
07 wget <a href="\"http://bbs.osapub.com/down/varnish-3.0.0.tar.gz\"" target="\"_blank\"">http://bbs.osapub.com/down/varnish-3.0.0.tar.gz</a> &> /dev/null
08 cd /dist/src
09 rm -fr varnish-3.0.0
10 export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
11 tar zxvf ../dist/varnish-3.0.0.tar.gz
12 cd varnish-3.0.0
13 #编译参数可以根据自己需要定制
14 ./configure -prefix=/usr/local/varnish -enable-debugging-symbols -enable-developer-warnings -enable-dependency-tracking
15 make && make install
16 if [ "$?" == "0" ]; then
17         echo "varnish-3.0.0安装成功." >> ./install_log.txt
18 else
19         echo "varnish-3.0.0安装失败." >> ./install_log.txt
20         exit 1
21 fi
22  
23 #设置启动、重启脚本
24 cat > /root/varnish_restart.sh <<EOF
25 #!/bin/bash
26 pkill varnish
27 pkill varnish
28 ulimit -SHn 51200
29 /usr/local/varnish/sbin/varnishd -n /data/varnish -f /usr/local/varnish/vcl.conf -a 0.0.0.0:81 -s malloc,12G -g www -u www -w 4000,12000,10 -T 127.0.0.1:3500 -p http_max_hdr=256 -p http_req_hdr_len=8192
30 EOF
31  
32 cat > /root/varnish_start.sh <<EOF
33 #!/bin/bash
34 ulimit -SHn 51200
35 /usr/local/varnish/sbin/varnishd -n /data/varnish -f /usr/local/varnish/vcl.conf -a 0.0.0.0:81 -s malloc,12G -g www -u www -w 4000,12000,10 -T 127.0.0.1:3500 -p http_max_hdr=256 -p http_req_hdr_len=8192
36 EOF
37  
38 chmod 755 /root/varnish*


三、        配置varnish一些参数说明:
Backend servers
Varnish有后端(或称为源)服务器的概念。后端服务器是指Varnish提供加速服务的那台,通常提供内容。
第一件要做的事情是告诉Varnish,哪里能找到要加速的内容。
vcl_recv
vcl_recv是在请求开始时调用的。完成该子程序后,请求就被接收并解析了。用于确定是否需要服务请求,怎么服务,如果可用,使用哪个后端。
在vcl_recv中,你也可以修改请求。通常你可以修改cookie,或添加/移除请求头信息。
注意在vcl_recv中,只可以使用请求对象req。

vcl_fetch
vcl_fetch是在文档从后端被成功接收后调用的。通常用于调整响应头信息,触发ESI处理,万一请求失败就换个后端服务器。
在vcl_fecth中,你还可以使用请求对象req。还有个后端响应对象beresp。Beresp包含了后端的HTTP头信息。

varnish 3.X 配置参考文档:http://anykoro.sinaapp.com/?p=261

编辑:/usr/local/varnish/vcl.conf ,文件不存在则创建。注意:Server A 与server B 配置一致!

配置详情如下:

001 ?
002 ###########后台代理服务器########
003  
004 backend server_osapub_com{
005         .host = "192.168.4.97";
006         .port = "82";
007 }
008  
009 acl purge {
010         "localhost";
011                 "127.0.0.1";
012 }
013  
014 #############################################
015  
016 sub vcl_recv {
017  
018      #################BAN##########################begin
019          if (req.request == "BAN") {
020                 # Same ACL check as above:
021                 if (!client.ip ~ purge) {
022                         error 405 "Not allowed.";
023                 }
024                 ban("req.http.host == " + req.http.host +
025                       "&& req.url == " + req.url);
026                 error 200 "Ban added";
027         }
028      #################BAN##########################end
029  
030          ###############配置域名##################################
031           if(req.http.host ~ "^server.osapub.com" ||req.http.host ~ ".osapub.com" ) {
032  
033                 #使用哪一组后台服务器
034                         set req.backend = server_osapub_com;
035  
036                 }
037  
038                 ##################################################
039  
040         if (req.restarts == 0) {
041         if (req.http.x-forwarded-for) {
042            set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
043          }
044          else {
045             set req.http.X-Forwarded-For = client.ip;
046          }
047      }
048  
049         if (req.http.Accept-Encoding) {
050         if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$") {
051             remove req.http.Accept-Encoding;
052         } elsif (req.http.Accept-Encoding ~ "gzip") {
053             set req.http.Accept-Encoding = "gzip";}
054           elsif (req.http.Accept-Encoding ~ "deflate") {
055             set req.http.Accept-Encoding = "deflate";}
056         else {
057          remove req.http.Accept-Encoding;
058       }
059     }
060      if (req.http.Cache-Control ~ "no-cache") {
061          return (pass);
062      }
063      if (req.request != "GET" &&
064        req.request != "HEAD" &&
065        req.request != "PUT" &&
066        req.request != "POST" &&
067        req.request != "TRACE" &&
068        req.request != "OPTIONS" &&
069        req.request != "DELETE") {
070          return (pipe);
071      }
072  
073      if (req.request != "GET" && req.request != "HEAD") {
074                         return(pass);
075                 }              
076  
077          if (req.http.Authorization || req.http.Cookie ||req.http.Authenticate) {
078                  return (pass);
079      }        
080  
081         if (req.request == "GET" && req.url ~ "(?i)\.php($|\?)"){
082                 return (pass);
083              }
084  
085         if (req.url ~ "\.(css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$") {
086                 unset req.http.Cookie;
087         }           
088  
089         return (lookup);
090  
091 }
092  
093 sub vcl_fetch {
094  
095         set beresp.grace = 5m;
096  
097         if (beresp.status == 404 || beresp.status == 503 || beresp.status == 500 || beresp.status == 502) {
098                 set beresp.http.X-Cacheable = "NO: beresp.status";
099                 set beresp.http.X-Cacheable-status = beresp.status;
100                 return (hit_for_pass);
101         }
102  
103         #决定哪些头不缓存
104  
105     if (req.url ~ "\.(php|shtml|asp|aspx|jsp|js|ashx)$") {
106                 return (hit_for_pass);
107          }
108  
109         if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$") {
110                 unset beresp.http.set-cookie;
111         }
112  
113         if (beresp.ttl <= 0s) {
114                 set beresp.http.X-Cacheable = "NO: !beresp.cacheable";
115                 return (hit_for_pass);
116         }
117         else {
118                 unset beresp.http.expires;
119         }
120  
121         return (deliver);
122  
123 }
124  
125 sub vcl_deliver {
126  
127 if (resp.http.magicmarker) {
128  
129      unset resp.http.magicmarker;
130  
131      set resp.http.age = "0";
132  }
133  
134 # add cache hit data
135 if (obj.hits > 0) {
136  
137    set resp.http.X-Cache = "HIT";
138    set resp.http.X-Cache-Hits = obj.hits;
139 }
140 else {
141      set resp.http.X-Cache = "MISS";
142 }
143  
144 # hidden some sensitive http header returning to client, when the cache server received from backend server response
145 remove resp.http.X-Varnish;
146 remove resp.http.Via;
147 remove resp.http.Age;
148 remove resp.http.X-Powered-By;
149 remove resp.http.X-Drupal-Cache;
150  
151 return (deliver);
152 }
153  
154 sub vcl_error {
155  
156  if (obj.status == 503 && req.restarts < 5) {
157    set obj.http.X-Restarts = req.restarts;
158    return (restart);
159  }
160  
161 }
162  
163 sub vcl_hit {
164  
165 if (req.http.Cache-Control ~ "no-cache") {
166  
167   if (! (req.http.Via || req.http.User-Agent ~ "bot|MSIE")) {
168       set obj.ttl = 0s;
169       return (restart);
170    }
171 }
172   return(deliver);
173 }

配置完成后执行:/root/varnish_start.sh 如果启动成功则表示配置无误!
33.jpg

四、        配置nginx编辑: /usr/local/nginx/conf/nginx.conf ,nginx 的配置方法请参考网上文档资料。
Server A(192.168.4.97) nginx配置如下:
001 ?
002 user  www www;
003 worker_processes 16;
004 error_log /logs/nginx/nginx_error.log crit;
005  
006 #Specifies the value for maximum file descriptors that can be opened by this process.
007 worker_rlimit_nofile 65500;
008  
009 events
010 {
011         use epoll;
012  
013         worker_connections 65500;
014 }
015  
016 http
017 {
018         include       mime.types;
019         default_type  application/octet-stream;
020  
021         server_names_hash_bucket_size 128;
022         client_header_buffer_size 64k;
023         large_client_header_buffers 4 64k;
024         client_max_body_size 10m;
025  
026         server_tokens off;
027  
028         expires       1h;
029  
030         sendfile on;
031         tcp_nopush     on;
032         keepalive_timeout 60;
033         tcp_nodelay on;
034  
035     fastcgi_connect_timeout 200;
036     fastcgi_send_timeout 300;
037     fastcgi_read_timeout 600;
038     fastcgi_buffer_size 128k;
039     fastcgi_buffers 4 64k;
040     fastcgi_busy_buffers_size 128k;
041     fastcgi_temp_file_write_size 128k;
042     fastcgi_temp_path /dev/shm;
043         gzip on;
044         gzip_min_length  2048;
045         gzip_buffers     4 16k;
046         gzip_http_version 1.1;
047         gzip_types  text/plain  text/css  application/xml application/x-javascript ;
048  
049         log_format  access  '$remote_addr - $remote_user [$time_local] "$request" '
050                           '$status $body_bytes_sent "$http_referer" '
051                           '"$http_user_agent" $http_x_forwarded_for';
052  
053 #################  include  ###################
054  
055          upstream varnish_server {
056                 server 127.0.0.1:81 weight=2 max_fails=3 fail_timeout=30s;
057                     server 192.168.4.99:81 weight=2 max_fails=3 fail_timeout=30s;
058                 ip_hash;
059         }
060  
061                 server
062         {
063                 listen       80;
064                 server_name  192.168.4.96 192.168.4.98;
065                              index index.html index.htm index.php index.aspx;
066                 location /
067                 {
068                         proxy_set_header   Host             $host;
069                         proxy_set_header   X-Real-IP        $remote_addr;
070                         proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
071  
072                         proxy_connect_timeout 200;
073                         proxy_send_timeout 300;
074                         proxy_read_timeout 500;
075                         proxy_buffer_size 256k;
076                         proxy_buffers 4 128k;
077                         proxy_busy_buffers_size 256k;
078                         proxy_temp_file_write_size 256k;
079                         proxy_temp_path  /dev/shm;
080  
081                         proxy_pass  http://varnish_server;
082  
083                         expires off;
084  
085                         access_log  /logs/nginx/192.168.4.96.log  access;
086                 }
087         }
088  
089                 server
090         {
091               listen       82;
092               server_name  192.168.4.97;
093               index index.html index.htm index.php;
094               root  /data/web/awstats/www;
095  
096               location ~ .*\.php$
097               {
098                    include fcgi.conf;
099                    fastcgi_pass  127.0.0.1:10080;
100                    fastcgi_index index.php;
101                    expires off;
102               }
103               access_log  /logs/nginx/awstats.osapub.com.log  access;
104         }              
105  
106 }

server B (192.168.4.99)配置如下:

001 ?
002 user  www www;
003 worker_processes 16;
004 error_log /logs/nginx/nginx_error.log crit;
005  
006 #Specifies the value for maximum file descriptors that can be opened by this process.
007 worker_rlimit_nofile 65500;
008  
009 events
010 {
011         use epoll;
012  
013         worker_connections 65500;
014 }
015  
016 http
017 {
018         include       mime.types;
019         default_type  application/octet-stream;
020  
021         server_names_hash_bucket_size 128;
022         client_header_buffer_size 64k;
023         large_client_header_buffers 4 64k;
024         client_max_body_size 10m;
025  
026         server_tokens off;
027  
028         expires       1h;
029  
030         sendfile on;
031         tcp_nopush     on;
032         keepalive_timeout 60;
033         tcp_nodelay on;
034  
035     fastcgi_connect_timeout 200;
036     fastcgi_send_timeout 300;
037     fastcgi_read_timeout 600;
038     fastcgi_buffer_size 128k;
039     fastcgi_buffers 4 64k;
040     fastcgi_busy_buffers_size 128k;
041     fastcgi_temp_file_write_size 128k;
042     fastcgi_temp_path /dev/shm;
043         gzip on;
044         gzip_min_length  2048;
045         gzip_buffers     4 16k;
046         gzip_http_version 1.1;
047         gzip_types  text/plain  text/css  application/xml application/x-javascript ;
048  
049         log_format  access  '$remote_addr - $remote_user [$time_local] "$request" '
050                           '$status $body_bytes_sent "$http_referer" '
051                           '"$http_user_agent" $http_x_forwarded_for';
052  
053 #################  include  ###################
054  
055          upstream varnish_server {
056                 server 127.0.0.1:81 weight=2 max_fails=3 fail_timeout=30s;
057                     server 192.168.4.97:81 weight=2 max_fails=3 fail_timeout=30s;
058                 ip_hash;
059         }
060  
061                 server
062         {
063                 listen       80;
064                 server_name  192.168.4.96 192.168.4.98;
065                              index index.html index.htm index.php index.aspx;
066                 location /
067                 {
068                         proxy_set_header   Host             $host;
069                         proxy_set_header   X-Real-IP        $remote_addr;
070                         proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
071  
072                         proxy_connect_timeout 200;
073                         proxy_send_timeout 300;
074                         proxy_read_timeout 500;
075                         proxy_buffer_size 256k;
076                         proxy_buffers 4 128k;
077                         proxy_busy_buffers_size 256k;
078                         proxy_temp_file_write_size 256k;
079                         proxy_temp_path  /dev/shm;
080  
081                         proxy_pass  http://varnish_server;
082  
083                         expires off;
084  
085                         access_log  /logs/nginx/192.168.4.98.log  access;
086                 }
087         }
088  
089                 server
090         {
091               listen       82;
092               server_name  192.168.4.99;
093               index index.html index.htm index.php;
094               root  /data/web/awstats/www;
095  
096               location ~ .*\.php$
097               {
098                    include fcgi.conf;
099                    fastcgi_pass  127.0.0.1:10080;
100                    fastcgi_index index.php;
101                    expires off;
102               }
103               access_log  /logs/nginx/awstats.osapub.com.log  access;
104         }              
105  
106 }

启动nginx:
/root/nginx_start
启动正常会监听80,82端口!
五、        运行nginx ha脚本

server A 的脚本下载地址:http://bbs.osapub.com/down/server_a_ha.tar.gz
解压力后得到三个脚本:
nginx_watchdog.sh
nginxha.sh
nginx_ha1.sh

server B 的脚本下载地址:http://bbs.osapub.com/down/server_b_ha.tar.gz
解压力后得到三个脚本:
nginx_watchdog.sh
nginxha.sh
nginx_ha2.sh

注意:脚本建议放到:/data/sh/ha 目录下,否则需要修改nginxha.sh  里面的程序路径!

六、        测试1,测试之前先把nginx 负载去掉,可以在前面加#号暂时注释,只保留本机解析,确保测试结果准确。
44.jpg

2,修改本机host文件
添加:192.168.4.99 server.osapub.com 到末尾,保存后访问。
55.jpg

3,正常结果显示如下:
66.jpg

到这里整个架构基本能运行起来了,根据大家的实际需求,对配置文件进行调优,HA脚本也可以进一步调优,关于报警,请参考社区自动安装mutt报警的脚本!

没有更多推荐了,返回首页