勤劳的网络蜘蛛

列举一下这一年来勤劳的网络蜘蛛（web spider）：

1.	Sosospider+(+http://help.soso.com/webspider.htm)	腾讯搜搜
2.	Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )	有道搜索1
3.	Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)	搜狗搜索1
4.	Baiduspider+(+http://www.baidu.com/search/spider.htm)	百度搜索
5.	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.net)	奇虎搜索1
6.	Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)	cuil搜索1
7.	Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)	谷歌搜索
8.	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.yodao.com/help/webmaster/spider/;)	有道搜索2
9.	Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )	新有道搜索
10.	Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)	搜狗搜索2
11.	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
12.	Mediapartners-Google
13.	Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)	有道搜索3
14.	Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)
15.	Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)	yahoo搜索
16.	Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)	cuil搜索2
17.	Mozilla/5.0 (compatible; ia_archiver +http://www.greaterera.com/)
18.	Yeti/1.0 (+http://help.naver.com/robots/)	naver搜索
19.	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
20.	Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)	majestic12
21.	psbot/0.1 (+http://www.picsearch.com/bot.html)	picsearch
22.	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
23.	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 1.7)
24.	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; CollapsarWEB qihoobot@qihoo.net)	奇虎搜索2
25.	Mozilla/3.0 (compatible; Indy Library)
26.	sogou in spider(+http://www.sogou.com/docs/help/webmasters.htm#07)	搜狗搜索3
27.	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
28.	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
29.	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
30.	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16
31.	Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)	majestic12
32.	Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)	yahoo搜索
33.	Sogou head spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)	搜狗搜索4

除了那些中国人都知道的搜索引擎之外，cuil.com是今年一个google前员工推出的，据说网页量很大。

naver.com是韩国的搜索引擎，majestic12是一个英国的搜索引擎？

picsearch.com是一个新兴的图片搜索引擎。

转载于:https://www.cnblogs.com/chuncn/archive/2009/03/27/1423269.html