防止屏蔽网络蛛蛛网络爬虫配置

新建立的网站，未必需要流量，各个方面还在优化，也为了节省资源和网站安全，需要屏蔽一些东西的

location / {
                       if ($http_user_agent ~* "robot|qihoobot|Baiduspider|Googlebot|Googlebot-Mobile|Googlebot-Image|Mediapartners-Google|Adsbot-Google|Feedfetcher-Google|Yahoo! Slurp|Yahoo! Slurp China|YoudaoBot|Sosospider|Sogou spider|Sogou web spider|MSNBot|ia_archiver|Tomato Bot") {  
                return 403;  
        }  
                   }

测试 ...

curl -I -A "qihoobot" www.xxx.com