新浪IaskSpider

今天在日志中发现了新浪IaskSpider的踪迹：
219.142.78.210 - - [10/Dec/2005:00:00:36 +0800] "GET /phpMan.php/man/crond/8/ HTTP/1.0" 200 3926 "-" "iaskspider" 219.142.78.2
10.114661134144036588
219.142.78.210 - - [10/Dec/2005:00:00:36 +0800] "GET /phpMan.php/man/cytune/8/ HTTP/1.0" 200 8639 "-" "iaskspider" 219.142.78.
210.302381134144036603
219.142.118.65 - - [10/Dec/2005:00:00:50 +0800] "GET /phpMan.php/info/pathchk/ HTTP/1.0" 200 3136 "-" "iaskspider" 219.142.118
.65.3811134144050430
219.142.78.85 - - [10/Dec/2005:00:00:58 +0800] "GET /phpMan.php/man/Config::myconfig/ HTTP/1.0" 200 1998 "-" "iaskspider" 219.
142.78.85.302351134144058586

IaskSpider的访问来源和频度分布：
grep -i iaskspider *|awk '{print $1}' |awk -F ':' '{print $2}' |sort|uniq -c|sort -rn |head -10
490 219.142.78.87
444 219.142.118.62
429 219.142.78.210
390 219.142.118.63
355 219.142.118.65
352 219.142.78.59
320 219.142.78.85
306 219.142.118.64
304 219.142.78.60
247 219.142.78.179

和国内外其他主流搜索引擎的SPIDER不一样：Sohu和新浪的都不抓取robots.txt文件，这样就有可能存在一些不希望被搜索引擎收录的内容被公开的潜在威胁。

附：AWStats中增加针对Sohu Agent和IaskSpider的统计
Index: robots.pm
===================================================================
RCS file: /cvsroot/awstats/awstats/wwwroot/cgi-bin/lib/robots.pm,v
retrieving revision 1.40
diff -r1.40 robots.pm
214a215
> 'iaskspider',
321a323
> 'sohu',
718a721
> 'iaskspider','Sina Iask',
724a728
> 'sohu','SohuSpider',

国内搜索引擎的定义：
===================================================================
RCS file: /cvsroot/awstats/awstats/wwwroot/cgi-bin/lib/search_engines.pm,v
retrieving revision 1.35
diff -r1.35 search_engines.pm
94c94,96
< 'baidu\.com','search\.sina\.com','search\.sohu\.com',
---
> 'baidu\.com','search\.sohu\.com','iask\.com',
> '\.3721\.com','\.163\.com','search\.tom\.com','search\.21cn\.com','\.qq\.com',
> 'yisou\.com','search\.china\.com','zhongsou\.com', 'search\.online\.', 'sogou\.com',
203c205
< 'search\.sina\.com','sina',
---
> 'iask\.com','iask',
204a207,216
> '\.3721\.com','3721',
> '\.163\.com','netease',
> 'search\.tom\.com','tom',
> 'search\.21cn\.com','21cn',
> '\.qq\.com','qq',
> 'yisou\.com','yisou',
> 'search\.china\.com','china',
> 'zhongsou\.com','zhongsou',
> 'search\.online','shonline',
> 'sogou\.com','sogou',
360c372,374
< 'baidu','word=', 'sina', 'word=', 'sohu','word=',
---
> 'baidu','(bs|wd|word)=', 'iask','(_searchkey|k)', 'sohu','(query|key_word|word)=',
> '3721','(p|name)=', 'netease','q=', 'tom','word=', '21cn','word=', 'qq','(word|w)=',
> 'yisou','p=', 'china','query=', 'zhongsou','word=', 'shonline','word=', 'sogou', 'query=',
473c487,490
< 'baidu','Baidu', 'sina','Sina', 'sohu','Sohu',
---
> 'baidu','Baidu', 'iask','Iask', 'sohu','Sohu',
> '3721','3721', 'tom','Tom', 'netease', 'NetEase', '21cn','21CN',
> 'qq','QQ', 'yisou','YiSou', 'zhongsou','ZhougSou', 'shonline','SHOnline',

作者： 车东 发表于：2005-12-12 09:12 最后更新于：2007-04-15 19:04
版权声明：可以转载，转载时请务必以超链接形式标明文章 新浪IaskSpider 的原始出处和作者信息及 本版权声明。
http://www.chedong.com/blog/archives/001086.html

« Web 2.0服务的成本分析：低成本，做你喜欢的事儿 | (回到Blog入口)|(回到首页) | 被Yahoo收购的del.icio.us » [再编辑]