varnish 2.1.5 安装应用实战

先给个varnish官网的地址吧：http://www.varnish-cache.org/

 

说说背景，我们的应用架构是 nginx + apache + php

 

前端请求连接数并发1万以上。前端两台nginx 请求基本吃掉，后端两台apache，每台响应只能支持到150左右。但数据库支持的并发连接更少。所以在1万并发请求下应用基本瘫痪。于是考虑在nginx 和 apache之间加一层 varnish。来吧，说干就干。

 

看中文的文档比较全的是张宴的07年的一篇，但是下了varnish 2.1.5版本安装完成后，发现vcl配置语法不一样了。于是翻看官方手册，完了配置完成。

 

 

一、前期准备：我用的是CentOS 5.5

 

               先yum 相关包：

To build Varnish on a Red Hat or CentOS system you need the following packages installed:

• automake
• autoconf
• libtool
• ncurses-devel
• libxslt
• groff
• pcre-devel
• pkgconfig

当然如果你一开始就yum了最新的包，这个肯定没问题。 我们是先yum update 所有包，然后再卸载掉不用的，下面是我们同事的一份清理不用包的脚本，在执行完yum update后运行。

 

#!/bin/sh rpm -qa|grep httpd rpm -e httpd rpm -qa|grep mysql rpm -e mysql rpm -qa|grep php rpm -e php yum -y remove httpd yum -y remove httpd* yum -y remove mysql* yum -y remove php* yum -y install yum-fastestmirror yum -y remove httpd yum -y update #Synchronization time rm -rf /etc/localtime ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime yum install -y ntp ntpdate -d cn.pool.ntp.org date yum -y install patch make gcc gcc-c++ gcc-g77 flex bison file yum -y install libtool libtool-libs autoconf kernel-devel yum -y install libjpeg libjpeg-devel libpng libpng-devel libpng10 libpng10-devel gd gd-devel yum -y install freetype freetype-devel libxml2 libxml2-devel zlib zlib-devel yum -y install glib2 glib2-devel bzip2 bzip2-devel libevent libevent-devel yum -y install ncurses ncurses-devel curl curl-devel e2fsprogs yum -y install e2fsprogs-devel krb5 krb5-devel libidn libidn-devel yum -y install openssl openssl-devel vim-minimal nano sendmail yum -y install fonts-chinese gettext gettext-devel yum -y install ncurses-devel yum -y install gmp-devel pspell-devel yum -y install unzip

 

 

 

 

二、下载安装

 

前期都可以参考 张宴的安装 ，我这里装的是 http://repo.varnish-cache.org/source/varnish-2.1.5.tar.gz

 

1、创建www用户和组，以及Varnish缓存文件存放目录（/var/vcache）：

 

/usr/sbin/groupadd www -g 48
/usr/sbin/useradd -u 48 -g www www
mkdir -p /var/vcache
chmod +w /var/vcache
chown -R www:www /var/vcache

　　2、创建Varnish日志目录（/var/logs/）：

 

mkdir -p /var/logs
chmod +w /var/logs
chown -R www:www /var/logs

　　3、编译安装varnish：

 

wget http://repo.varnish-cache.org/source/varnish-2.1.5.tar.gz
tar zxvf varnish-2.1.5.tar.gz
cd varnish-2.1.5
./configure --prefix=/usr/local/varnish
make && make install

    4、配置文件

 

这里新版的和老版本语法不一样了。看官方的默认配置的例子即可。并附上vcl语法说明官方地址 ：

http://www.varnish-cache.org/trac/wiki/VCL

 

vi /usr/local/varnish/default.vcl

 

# This is a basic VCL configuration file for varnish. See the vcl(7) # man page for details on VCL syntax and semantics. # # Default backend definition. Set this to point to your content # server. # backend phpserver { #定义后端服务器名 .host = "10.10.10.101"; #定义后端服务器IP .port = "80"; # .probe = { # .timeout = 15s; # } } sub vcl_recv { if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For ", " client.ip; } else { set req.http.X-Forwarded-For = client.ip; } } if (req.http.host ~ "^blog.csdn.com" ){ set req.backend = phpserver; if (req.request != "GET" && req.request != "HEAD") { return (pipe); } else { return (lookup); } }else { error 404 "Not Found"; } } sub vcl_pipe { return (pipe); } sub vcl_pass { return (pass); } sub vcl_hash { set req.hash += req.url; if (req.http.host) { set req.hash += req.http.host; } else { set req.hash += server.ip; } return (hash); } sub vcl_hit { if (!obj.cacheable) { return (pass); } return (deliver); } sub vcl_miss { return (fetch); } sub vcl_fetch { if (!beresp.cacheable) { return (pass); } if (beresp.http.Set-Cookie) { return (pass); } if (req.request == "GET" && req.url ~ "/.(js|css|mp3|jpg|png|gif|swf|jpeg|ico)$") { set beresp.ttl = 7d; } else{ set beresp.ttl = 3600s; } return (deliver); } sub vcl_deliver { return (deliver); } sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; synthetic {""}; return (deliver); } 　

 

 

具体每一行的意义，官方也有详细说明，稍后附上中文翻译

 

How VCL works

Your VCL file is more than just a config file. Your Varnish config is in fact a mini program, a program that is actually compiled and linked in to Varnish at runtime. There are several steps to how Varnish handles each request and in your VCL file you have the option to customize the behavior of each one.

There are actually 8 subroutines that control how Varnish behaves, and that you can change in your Varnish config. They are:

 

 

vcl_recv()

Called after a request is received from the browser, but before it is processed.

 

就是说请求过来先调用这个方法块。匹配后端主机啥的都是在这里配置

vcl_pipe()

Called when a request must be forwarded directly to the backend with minimal handling by Varnish (think HTTP CONNECT)

 

这个方法块，就是请求过来后不使用本地缓存，直接透过请求后端应用处理。所以如果不想适用缓存的请求可调用这个方法，并且这个方法的返回不会缓存。

vcl_hash()

Called to determine the hash key used to look up a request in the cache.

 

在本地查找时，即调用look up时使用，检查hash key

vcl_hit()

Called after a cache lookup when the object requested has been found in the cache.

 

在调用 lookup 后 调用，查找命中时调用

vcl_miss()

Called after a cache lookup when the object requested was not found in the cache.

 

在调用lookup后调用，查找没命中时调用

vcl_pass()

Called when the request is to be passed to the backend without looking it up in the cache.

 

在请求不需要缓存查缓存时，调用这个方法，这个方法返回值也不会在本地缓存

vcl_fetch()

Called when the request has been sent to the backend and a response has been received from the backend.

 

这个就是处理请求发往后端返回的响应，并做缓存策略的方法块。

vcl_deliver()

Called before a response object (from the cache or the web server) is sent to the requesting client.

 

这个可以给返回客户端的响应头里加信息头使。

That seems like a lot, but as I mentioned before Varnish has pretty reasonable defaults, so you only need to override a few of these.

Strictly speaking, Varnish doesn't actually let you replace its defaults. Your definitions of the above routines simply run before the builtin versions of those same routines. Fortunately, we can prevent Varnish from proceeding on to the builtin versions if we wish by returning the appropriate value within our version of the routine. If that seems confusing, don't worry. It will become clear when we start looking at our config.

The routines we are most interested in are vcl_recv, which handles the incoming request from the browser, and vcl_fetch, which is where we will decide whether the object just retrieved should go into the cache.

 

这一段就不翻译了。

 

其实说白了，用的上的就上面几个方法，并且配置文件语法很强大，可以根据需要发挥吧。

 

 

 

5、启动Varnish

启动需要根据不同场景，适当放大几个参数。

其中 -w参数 在2.x版本中和 1.x版本改动较大，需要根据帮助调整最适合自己的。还就是缓存文件的大小，我这里设的是50G。因为开始设的2G ，结果没多久，varnish就崩溃了。

 

# /usr/local/varnish/sbin/varnishd -n /var/vcache -f /usr/local/varnish/default.vcl -a 0.0.0.0:80 -s file,/data/vcache/varnish_cache.data,50G -u www -w2,65536,60 -T 127.0.0.1:3600 -p thread_pool_min=200 -p thread_pool_max=4000 -p thread_pools=4 -p thread_pool_add_delay=2 -p listen_depth=4096 -p lru_interval=20

　　6、启动varnishncsa用来将Varnish访问日志写入日志文件：

/usr/local/varnish/bin/varnishncsa -n /var/vcache -w /var/logs/varnish.log &

 

 

 往后的步骤就是加默认开机启动，看命中情况，清缓存等。以及更细致的配置参数可以参考官网的文档。

 

 

 

附上我们应用的命中情况，基本都缓存了，top看Cpu负载为0 ，呵呵