Varnish安装
-
源码安装Varnish
安装pcre:
# wget -c http://downloads.sourceforge.net/project/pcre/pcre/8.32/pcre-8.32.tar.bz2 # tar jxvf pcre-8.32.tar.bz2 # cd pcre-8.32 # ./configure && make && make install
源码安装Varnish:
# wget -c http://repo.varnish-cache.org/source/varnish-3.0.3.tar.gz # tar zxvf varnish-3.0.3.tar.gz # cd varnish-3.0.3 # export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig # ./configure --prefix=/usr/local/varnish # make && make install
Varnish配置与管理
-
varnish启动配置
# cp /data/install/varnish-3.0.3/redhat/varnish.sysconfig /etc/sysconfig/varnish # cp /data/install/varnish-3.0.3/redhat/varnish.initrc /etc/init.d/varnish # cp /data/install/varnish-3.0.3/redhat/varnish_reload_vcl /usr/local/varnish/bin/ # groupadd varnish # useradd -g varnish -s /sbin/nologin varnish
编辑varnish启动配置:
# vi /etc/sysconfig/varnish # cat /etc/sysconfig/varnish
## Alternative 3, Advanced configuration # # See varnishd(1) for more information. # # # Main configuration file. You probably want to change it #VARNISH_VCL_CONF=/etc/varnish/default.vcl VARNISH_VCL_CONF=/usr/local/varnish/etc/varnish/fdfs.vcl # # Default address and port to bind to # # Blank address means all IPv4 and IPv6 interfaces, otherwise specify # # a host name, an IPv4 dotted quad, or an IPv6 address in brackets. # VARNISH_LISTEN_ADDRESS= #VARNISH_LISTEN_PORT=6081 VARNISH_LISTEN_PORT=80 # # # Telnet admin interface listen address and port VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 VARNISH_ADMIN_LISTEN_PORT=6082 # # # Shared secret file for admin interface #VARNISH_SECRET_FILE=/etc/varnish/secret VARNISH_SECRET_FILE=/usr/local/varnish/etc/varnish/secret # # # The minimum number of worker threads to start VARNISH_MIN_THREADS=50 # # # The Maximum number of worker threads to start VARNISH_MAX_THREADS=1000 # # # Idle timeout for worker threads VARNISH_THREAD_TIMEOUT=120 # # # Cache file location VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin # # # Cache file size: in bytes, optionally using k / M / G / T suffix, # # or in percentage of available disk space using the % suffix. VARNISH_STORAGE_SIZE=150M # # # Backend storage specification #VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}" VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}" # # # Default TTL used when the backend does not specify one VARNISH_TTL=120 # # # DAEMON_OPTS is used by the init script. If you add or remove options, make # # sure you update this section, too. DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ -f ${VARNISH_VCL_CONF} \ -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ -t ${VARNISH_TTL} \ -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \ -u varnish -g varnish \ -S ${VARNISH_SECRET_FILE} \ -s ${VARNISH_STORAGE}" # ## Alternative 4, Do It Yourself. See varnishd(1) for more information. # # DAEMON_OPTS=""
编辑varnish启动管理脚本:
# vi /etc/init.d/varnish # cat /etc/init.d/varnish
... pidfile=/var/run/varnish.pid exec="/usr/local/varnish/sbin/varnishd" reload_exec="/usr/local/varnish/bin/varnish_reload_vcl" ...
编辑动态加载VCL配置的脚本:
# vi /usr/local/varnish/bin/varnish_reload_vcl # cat /usr/local/varnish/bin/varnish_reload_vcl
... # Done parsing, set up command VARNISHADM="/usr/local/varnish/bin/varnishadm $secret -T $VARNISH_ADMIN_LISTEN_ADDRESS:$VARNISH_ADMIN_LISTEN_PORT" ...
注:也可以将/usr/local/varnish/bin添加到系统的PATH中,这样就不需要编辑varnish_reload_vcl
生成varnish管理秘钥:
# /usr/bin/uuidgen > /usr/local/varnish/etc/varnish/secret # chmod 644 /usr/local/varnish/etc/varnish/secret
-
VCL配置
FastDFS部署的前端Varnish缓存的vcl配置列子:
# cat /usr/local/varnish/etc/varnish/fdfs.vcl
# backend health checking probe health { .url = "/favicon.ico"; .timeout = 60ms; .interval = 2s; .window = 5; .threshold = 3; } backend server00 { .host = "192.168.1.190"; .port = "8080"; .probe = health; } backend server01 { .host = "192.168.1.191"; .port = "80"; .probe = health; } backend server02 { .host = "192.168.1.192"; .port = "80"; .probe = health; } # round-robin loadblancing director dr1 round-robin { { .backend = server01; } { .backend = server02; } } # Access Control acl internal { "localhost"; "192.168.1.0"/24; ! "192.168.1.1"; } sub vcl_recv { # Backend selection if (req.url ~ "^/M00") { set req.backend = dr1; } else { set req.backend = server00; } # purge request if (req.request == "PURGE") { if (!client.ip ~ internal) { error 405 "Not allowed."; } return (lookup); } # Use anonymous, cached pages if all backends are down if (!req.backend.healthy) { unset req.http.Cookie; } /* Bypass cache for large files. The x-pipe header is set in vcl_fetch when a too large file is detected.*/ if (req.http.x-pipe && req.restarts > 0) { unset req.http.x-pipe; return (pipe); } # Do not cache these paths if (req.url ~ "^/admin$" || req.url ~ "^/admin/.*$") { return (pass); } # Allow the backend to serve up stale content if it is responding slowly if (req.backend.healthy) { set req.grace = 30s; } else { set req.grace = 6h; } # Always cache the following file types if (req.url ~ "(?i)\.(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)(\?.*)?$") { unset req.http.Cookie; } } sub vcl_hit { if (req.request == "PURGE") { purge; error 200 "Purged."; } } sub vcl_miss { if (req.request == "PURGE") { purge; error 404 "Not in cache."; } } sub vcl_pass { if (req.request == "PURGE") { error 502 "PURGE on a passed object"; } } # Set a header to track a cache HIT/MISS. sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Varnish-Cache = "HIT"; } else { set resp.http.X-Varnish-Cache = "MISS"; } } sub vcl_fetch { # Don't cache files larger than 10MB if (beresp.http.Content-Length ~ "[0-9]{8,}") { set req.http.x-pipe = "1"; return (restart); } # override the default time to live of a cached object if ( req.url ~ "(?i)\.(png|jpg|jpeg|gif)$") { set beresp.ttl = 30d; unset beresp.http.Set-Cookie; } else { set beresp.ttl = 30s; } # Allow items to be stale if needed set beresp.grace = 6h; } -
启动/停止varnish
检查VCL配置是否正确:
# /etc/init.d/varnish configtest
或
# /usr/local/varnish/sbin/varnishd -C -f /usr/local/varnish/etc/varnish/fdfs.vcl
启动varnish:
# service varnish start
查看varnish状态:
# /etc/init.d/varnish status
动态加载VCL配置:
# /etc/init.d/varnish reload
停止varnish:
# /etc/init.d/varnish stop
查看当前varnish监听的80端口:
# netstat -plan | grep :80 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3068/varnishd tcp 1 0 192.168.1.190:44391 192.168.1.191:80 CLOSE_WAIT 3068/varnishd tcp 0 0 :::80 :::* LISTEN 3068/varnishd
查看varnish进程:
# ps -ef | grep varnishd | grep -v grep root 3067 1 0 15:06 ? 00:00:00 /usr/local/varnish/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /usr/local/varnish/etc/varnish/fdfs.vcl -T 127.0.0.1:6082 -t 120 -w 50,1000,120 -u varnish -g varnish -S /usr/local/varnish/etc/varnish/secret -s malloc,150M varnish 3068 3067 0 15:06 ? 00:00:03 /usr/local/varnish/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /usr/local/varnish/etc/varnish/fdfs.vcl -T 127.0.0.1:6082 -t 120 -w 50,1000,120 -u varnish -g varnish -S /usr/local/varnish/etc/varnish/secret -s malloc,150M
-
Varnish访问日志
varnishncsa可以使用NCSA通用日志格式(NCSA Common Log Format)将HTTP请求记录到日志文件.
# cp /data/install/varnish-3.0.3/redhat/varnishncsa.initrc /etc/init.d/varnishncsa # chmod +x /etc/init.d/varnishncsa # mkdir -p /usr/local/varnish/logs
编辑varnishncsa启动配置:
# vi /etc/init.d/varnishncsa # cat /etc/init.d/varnishncsa
... logfile="/usr/local/varnish/logs/varnishncsa.log" exec="/usr/local/varnish/bin/varnishncsa" ...
启动varnishncsa：
# /etc/init.d/varnishncsa start
使用logrotate轮询日志文件(每天轮询,保留最近30天):
# vi /etc/logrotate.d/varnish # cat /etc/logrotate.d/varnish
/usr/local/varnish/logs/varnishncsa.log { missingok notifempty sharedscripts daily rotate 30 dateext dateformat .%Y-%d-%m delaycompress postrotate /bin/kill -HUP `cat /var/run/varnishncsa.pid 2>/dev/null` 2> /dev/null || true endscript }日志轮询debug测试:
# logrotate -df /etc/logrotate.d/varnish
-
配置开机自动启动
# chkconfig --add varnish # chkconfig varnish on # chkconfig --add varnishncsa # chkconfig varnishncsa on
-
Varnish辅助工具
- varnishtop 显示客户端最频繁请求的URL的持续更新列表:
# varnishtop -i RxURL
显示最频繁命中后端的URL的持续更新列表:
# varnishtop -b -i TxURL
- varnishstat
varnishstat实时查看缓存统计信息(如:连接数和命中率):# varnishstat
仅查看特定项统计信息:
# varnishstat -f client_conn,client_drop,client_req,cache_hit,cache_hitpass,cache_miss,backend_conn,backend_fail,backend_reuse,n_lru_nuked,n_wrk_lqueue,n_wrk_queued,n_wrk_drop
- varnishlog
Debug缓存操作(ReqStart:后面为某客户端请求的IP):# varnishlog -c -m "ReqStart:192.168.1.120"
仅显示发送到后端的URLs(如:没命中缓存和内容还没被缓存):
# varnishlog -O -i TxURL
将varnishlog写入到文件,分析访问特定页面的客户端请求:
# varnishlog -w /usr/local/varnish/logs/varnish.log -d # varnishlog -r /usr/local/varnish/logs/varnish.log -c -m 'RxURL:^/test/c.php$'
- varnishadm
查看varnishadm提供的命令:# varnishadm -T 127.0.0.1:6082 -S /usr/local/varnish/etc/varnish/secret help
help [command] ping [timestamp] auth response quit banner status start stop vcl.load vcl.inline <quoted_VCLstring> vcl.use vcl.discard vcl.list vcl.show param.show [-l] [] param.set panic.show panic.clear storage.list backend.list backend.set_health matcher state ban.url ban [&& ]... ban.list
查看后端服务器健康状况:
# varnishadm debug.health Backend server01 is Healthy Current states good: 4 threshold: 3 window: 5 Average responsetime of good probes: 0.001339 Oldest Newest ================================================================ 4444444444444444444444444444444444444444444444444444444444444444 Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-RRRR Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-HHHH Happy Backend server02 is Healthy Current states good: 5 threshold: 3 window: 5 Average responsetime of good probes: 0.001387 Oldest Newest ================================================================ 4444444444444444444444444444444444444444444444444444444444444444 Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-RRRRRRRRRRRR Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-HHHHHHHHHHHH Happy Backend server00 is Healthy Current states good: 5 threshold: 3 window: 5 Average responsetime of good probes: 0.000331 Oldest Newest ================================================================ 4444444444444444444444444444444444444444444444444444444444444444 Good IPv4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good Recv HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy
或
# varnishadm backend.list Backend name Refs Admin Probe server01(192.168.1.191,,80) 3 probe Healthy 5/5 server02(192.168.1.192,,80) 3 probe Healthy 5/5 server00(192.168.1.190,,8080) 2 probe Healthy 5/5
- varnishtop 显示客户端最频繁请求的URL的持续更新列表:
Varnish进阶使用
-
VCL流程简介
VCL是用于定义Varnish缓存策略的配置语言(VCL参考手册).VCL配置大部分是写到子程序,并遵循一个预先定义的请求和响应阶段的流程.
处理HTTP请求的子程序(subroutine):
- vcl_recv:当有HTTP请求进来的时候,vcl_recv是第一个被执行的子程序,它决定是否处理/如何处理请求.在这里可正常化URL,修改HTTP头,调整/删除cookies,定义使用哪个后端backend或director,控制访问...
- vcl_hash:它是vcl_recv子程序执行之后才执行的一个子程序.它的作用是生成hash作为存储对象在内存中映射的key,对获得高命中率发挥重要作用.
- vcl_pipe:当一个pipe指令在vcl_recv执行的之后(接下来的VCL code将不被进行处理),它就会将客户端请求直接传输给后端服务器.客户端和后端服务器之间直接来回传输数据,直至有一方关闭连接.
- vcl_pass:它会将客户端请求传输给后端服务器,后端服务器的响应数直接传回给客户端,但不进行缓存.
处理HTTP响应的子程序:
- vcl_fetch:它是第一个处理响应阶段的子程序,对缓存策略和ESI发挥重要作用.它是在从后端服务器成功获得内容之后被调用.
- vcl_deliver:它是在缓存对象发送给客户端之前调用.可以用于隐藏服务器信息,添加debug头部信息,最后清理http响应头部信息等.由于vcl_deliver是在对象进入缓存之后才调用的,所有在vcl_deliver内的操作都不会被持久化.
- vcl_error:当遇到错误的时候调用,显式地或隐式地由于后端或是内部的错误.也可以用于拒绝请求访问或将请求导向新的地址,或是用于系统维护提示页面.
建议在写自己的VCL代码之前,先阅读Varnish默认内嵌的VCL配置文件default.vcl:
# cat /usr/local/varnish/etc/varnish/default.vcl
-
Varnish负载均衡
Varnish负载均衡的director有random,client,hash,round-robin,dns,fallback等.
下面是两种最常用的director的负载均衡.director使用round-robin的VCL:
... director dr1 round-robin { { .backend = server01; } { .backend = server02; } } sub vcl_recv { if (req.http.host ~ "(?i)^images.zrwm.com$"){ set req.backend = dr1; } } ...director使用client的VCL:
... director dr2 client { .retries = 2; { .backend = server01; .weight = 1; } { .backend = server02; .weight = 1; } } if (req.http.host ~ "(?i)^images.zrwm.com$"){ set req.backend = dr2; /* Load balance by client IP, this is the default */ set client.identity = client.ip; /* ## Load balance by clients' real IP if other services in front of Varnish Cache set client.identity = req.http.X-Forwarded-For; ## Load balance by URL set client.identity = req.url; ## Load balance by user agent set client.identity = req.http.user-agent; */ } ... -
HTTP缓存头部
HTTP1.1引入了cache-control响应头部来替换HTTP 1.0的expires头部.两者的主要区别在于expires是使用一个日期时间值,而cache-control可以接收一个age值.
Expires: Fri, 1 Oct 2012 14:19:41 GMT Cache-Control: max-age=3600
一个使用PHP处理HTTP缓存(缓存10秒)的例子:
<?php header('Cache-Control: public, must-revalidate, max-age=10, s-maxage=10'); header('Last-Modified: ' . gmdate("D, d M Y H:i:s") . ' GMT'); header('Expires: ' . gmdate ('D, d M Y H:i:s', time() + 10). ' GMT'); echo "content from php"; ?>Varnish会遵守这些缓存头部(cache headers)的值,除非Varnish被告知不需要遵守.因此,当内容应该被缓存而实际没缓存的时候,可检查响应头部(response headers)是否有no-store/no-cache或是一个已经过期的时间值.
-
清除Varnish缓存内容
清除Varnish缓存内容有三种方式:
- Purge:每次从缓存中清除一个对象(清除所有Vary:-对象版本),需要在vcl_hit和vcl_miss两个子程序中实现.
- Ban:可以通过正则表达式一次清除大量的内容.因此清除大量的缓存内容的时候,Ban方法会比Purge快很多.
- 使缓存不命中:在vcl_recv子程序中
set req.hash_always_miss = true;这样会使Varnish每次都去缓存中寻找对象,但会忽略任何发现的对象副本.
注意:Purge方式会马上将内容从缓存中删除,并释放内存.Ban方式不会马上删除内容,也不释放内存,而是创建一个ban列表,每次请求进来的时候都会检查这个列表.
Purge只能在VCL代码中实现,参考上面的fdfs.vcl.
下面是使用Purge方式清理缓存的一个PHP脚本:<?php function purge($ip, $url) { $fp = fsockopen($ip, 80, $errno, $errstr, 2); if (!$fp) { echo "$errstr ($errno)", PHP_EOL; } else { $out = "PURGE $url HTTP/1.0\r\n"; $out .= "HOST:www.zrwm.com\r\n"; $out .= "Connection: Close\r\n\r\n"; fwrite($fp, $out); while (!feof($fp)) { echo fgets($fp, 128); } fclose($fp); } } purge('127.0.0.1', '/M00/00/00/wKgBv1FSXxGADRiEAAO1XLFK2Tw732.jpg');Ban方式既可以在VCL代码vcl_recv子程序中实现,也可以通过CLI方式的varnishadm管理客户端来操作.
下面是通过varnishadm使用ban方式清除缓存的几个例子:
清除单个文件c.php的缓存:# varnishadm ban.url c.php
# varnishadm ban "req.http.host ~ a.qingluobo.com && req.url ~ c.php"
清除/M00目录下所有文件的缓存:# varnishadm ban.url "^/M00"
清除所有缓存:
# varnishadm ban.url "^/.*$"
查看所有的ban列表:
# varnishadm ban.list
-
HTTP压缩
文本类型的响应内容可以进行压缩,比如(html,css,js,xml).
在对象进入缓存之前进行压缩再存储:sub vcl_fetch { ... if (beresp.http.content-type ~ "text") { set beresp.do_gzip = true; } ... }
1585
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
