查看Apache访问日志的user_agent：

[root@centos6 apache2]# cd logs/

[root@centos6 logs]# ls

access_log  test.com-access_20170114_log  test.com-error_log

error_log   test.com-access_20170115_log

httpd.pid   test.com-access_log

[root@centos6 logs]#

[root@centos6 logs]# ls

access_log  test.com-access_20170114_log  test.com-error_log

error_log   test.com-access_20170115_log

httpd.pid   test.com-access_log

[root@centos6 logs]# tail test.com-access_20170115_log

……

192.168.147.1 - - [15/Jan/2017:01:39:33 +0800] "GET /data/info.php HTTP/1.1" 403 215 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"

192.168.147.1 - - [15/Jan/2017:01:39:33 +0800] "GET /favicon.ico HTTP/1.1" 200 5558 "http://www.test.com/data/info.php" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"

127.0.0.1 - - [15/Jan/2017:01:40:13 +0800] "GET http://www.test.com/data/info.php HTTP/1.1" 200 20 "-" "curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"

可以禁止指定user_agent访问我们的网站，尤其是浏览器爬虫偷跑流量。编辑虚拟主机配置文件：

[root@centos6 logs]# vim /usr/local/apache2/conf/extra/httpd-vhosts.conf

加入如下内容：禁止curl和chrome

……

<IfModule mod_rewrite.c>

RewriteEngine on

RewriteCond %{HTTP_HOST} ^www.aaa.com$ [OR]

RewriteCond %{HTTP_HOST} ^www.bbb.com$

RewriteRule ^/(.*)$ http://www.test.com/$1 [R=301,L]

      RewriteCond %{HTTP_USER_AGENT} ^.*curl.* [NC,OR]

       RewriteCond %{HTTP_USER_AGENT} ^.*chrome* [NC]

       RewriteRule .* - [F]

</IfModule>

<Directory /data/www/important/>

AllowOverride AuthConfig

AuthName "username"

AuthType Basic

……

[root@centos6 apache2]# apachectl -t

Syntax OK

[root@centos6 apache2]# apachectl graceful

禁用后chrome浏览器不再能访问我们的网站，

wKiom1h8fC7ArC5JAADPDKffqTI378.png-wh_50

但是其他浏览器仍可以访问：IE浏览器可以访问

wKioL1h8fC_xJYNSAAD3pw6ZUKI712.png-wh_50

360浏览器也可以访问

wKioL1h8fC_iTpZzAADITSHsIMg478.png-wh_50

curl测试返回值也都是403，

[root@centos6 apache2]# curl -x127.0.0.1:80 www.test.com

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<html><head>

<title>403 Forbidden</title>

</head><body>

<h1>Forbidden</h1>

<p>You don't have permission to access /

on this server.</p>

</body></html>

[root@centos6 apache2]# curl -x127.0.0.1:80 www.test.com/forum.php

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<html><head>

<title>403 Forbidden</title>

</head><body>

<h1>Forbidden</h1>

<p>You don't have permission to access /forum.php

on this server.</p>

</body></html>

[root@centos6 apache2]# curl -x127.0.0.1:80 www.test.com/data/info.php

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<html><head>

<title>403 Forbidden</title>

</head><body>

<h1>Forbidden</h1>

<p>You don't have permission to access /data/info.php

on this server.</p>

</body></html>

[root@centos6 apache2]#

正常情况下的curl能找到的页面返回200，找不到的页面返回404，禁止curl之后都返回403。这里我们还需要使用curl进行测试，先去掉curl的禁用：

[root@centos6 apache2]# vim /usr/local/apache2/conf/extra/httpd-vhosts.conf

……

<IfModule mod_rewrite.c>

RewriteEngine on

RewriteCond %{HTTP_HOST} ^www.aaa.com$ [OR]

RewriteCond %{HTTP_HOST} ^www.bbb.com$

RewriteRule ^/(.*)$ http://www.test.com/$1 [R=301,L]

   #RewriteCond %{HTTP_USER_AGENT} ^.*curl.* [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^.*chrome* [NC]

RewriteRule .* - [F]

</IfModule>

……

[root@centos6 apache2]# apachectl -t

Syntax OK

[root@centos6 apache2]# apachectl graceful

[root@centos6 apache2]# curl -x192.168.147.132:80 www.test.com/forum.php -I

HTTP/1.1 200 OK

Date: Sat, 14 Jan 2017 19:38:46 GMT

Server: Apache/2.2.9 (Unix) PHP/5.4.36

X-Powered-By: PHP/5.4.36

Set-Cookie: sTi8_2132_saltkey=QmAybXxa; expires=Mon, 13-Feb-2017 19:38:46 GMT; path=/; httponly

Set-Cookie: sTi8_2132_lastvisit=1484419126; expires=Mon, 13-Feb-2017 19:38:46 GMT; path=/

Set-Cookie: sTi8_2132_sid=MVss2L; expires=Sun, 15-Jan-2017 19:38:46 GMT; path=/

Set-Cookie: sTi8_2132_checkpatch=1; expires=Sat, 14-Jan-2017 19:39:46 GMT; path=/

Set-Cookie: sTi8_2132_lastact=1484422726%09forum.php%09; expires=Sun, 15-Jan-2017 19:38:46 GMT; path=/

Set-Cookie: sTi8_2132_onlineusernum=1; expires=Sat, 14-Jan-2017 19:43:46 GMT; path=/

Set-Cookie: sTi8_2132_sid=MVss2L; expires=Sun, 15-Jan-2017 19:38:46 GMT; path=/

Cache-Control: max-age=0

Expires: Sat, 14 Jan 2017 19:38:46 GMT

Content-Type: text/html; charset=gbk

使用-A选项模拟user_agent，不含"chrome"时可以访问，加入"chrome"后不能访问

[root@centos6 apache2]# curl -A "asdfghjkl" -x192.168.147.132:80 www.test.com/forum.php -I

HTTP/1.1 200 OK

Date: Sat, 14 Jan 2017 19:40:17 GMT

Server: Apache/2.2.9 (Unix) PHP/5.4.36

X-Powered-By: PHP/5.4.36

Set-Cookie: sTi8_2132_saltkey=ZggZv4O6; expires=Mon, 13-Feb-2017 19:40:17 GMT; path=/; httponly

Set-Cookie: sTi8_2132_lastvisit=1484419217; expires=Mon, 13-Feb-2017 19:40:17 GMT; path=/

Set-Cookie: sTi8_2132_sid=RG77fb; expires=Sun, 15-Jan-2017 19:40:17 GMT; path=/

Set-Cookie: sTi8_2132_lastact=1484422817%09forum.php%09; expires=Sun, 15-Jan-2017 19:40:17 GMT; path=/

Set-Cookie: sTi8_2132_onlineusernum=1; expires=Sat, 14-Jan-2017 19:45:17 GMT; path=/

Set-Cookie: sTi8_2132_sid=RG77fb; expires=Sun, 15-Jan-2017 19:40:17 GMT; path=/

Cache-Control: max-age=0

Expires: Sat, 14 Jan 2017 19:40:17 GMT

Content-Type: text/html; charset=gbk

[root@centos6 apache2]# curl -A "asdchromefghjkl" -x192.168.147.132:80 www.test.com/forum.php -I

HTTP/1.1 403 Forbidden

Date: Sat, 14 Jan 2017 19:41:33 GMT

Server: Apache/2.2.9 (Unix) PHP/5.4.36

Content-Type: text/html; charset=iso-8859-1