搜索引擎访问模拟(百度、谷歌、雅虎…)

WebRequest.UserAgent = "Googlebot/2.1 (+http://www.google.com/bot.html)";

Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/;)

msnbot/1.0 (+http://search.msn.com/msnbot.htm)

转载于:https://www.cnblogs.com/ayxiaopan/archive/2013/05/10/3071622.html