Heritrix3.1.1是老外写的爬虫,可配置性非常好,但是有一点不好,老外很懂礼貌,所以这个爬虫也很懂礼貌,爬起来非常的慢,1万多个链接,一天也爬不完。
仔细研究它的源代码和文档,网上的文档非常的少,基本要看源代码。
根据网上的这篇文档https://webarchive.jira.com/wiki/display/Heritrix/Politeness+parameters
这个爬虫对网站的礼貌主要是单线程,对同一网站只有一个线程。。。好吧,这个我们没什么办法了,因为它是单线的。
第二个可以设置的地方是politeness参数,这个线程取一个url,处理完成以后,在处理队列里的下一个url之前,先要睡一小觉(snooze),而睡多长时间,是由DispositionProcessor这个bean来决定的,里面有好几个参数,基本上是根据前一个uri的抓取需要的时间,乘上delayFactor来决定。。。缺省的设置是5倍,就是说,如果前一个uri花了1秒来获取,那么这个线程就要睡5秒。
处理的代码片断如下:
/**
* Update any scheduling structures with the new information in this
* CrawlURI. Chiefly means make necessary arrangements for no other URIs at
* the same host to be visited within the appropriate politeness window.
*
* @param curi
* The CrawlURI
* @return millisecond politeness delay
*/
protected long politenessDelayFor(CrawlURI curi) {
long durationToWait = 0;
Map<String,Object> cdata = curi.getData();
if (cdata.containsKey(A_FETCH_BEGAN_TIME)
&& cdata.containsKey(A_FETCH_COMPLETED_TIME)) {
long completeTime = curi.getFetchCompletedTime();
long durationTaken = (completeTime - curi.getFetchBeginTime());
durationToWait = (long)(getDelayFactor() * durationTaken);
long minDelay = getMinDelayMs();
if (minDelay > durationToWait) {
// wait at least the minimum
durationToWait = minDelay;
}
long maxDelay = getMaxDelayMs();
if (durationToWait > maxDelay) {
// wait no more than the maximum
durationToWait = maxDelay;
}
long respectThreshold = getRespectCrawlDelayUpToSeconds() * 1000;
if (durationToWait<respectThreshold) {
// may need to extend wait
CrawlServer s = getServerCache().getServerFor(curi.getUURI());
String ua = curi.getUserAgent();
if (ua == null) {
ua = metadata.getUserAgent();
}
Robotstxt rep = s.getRobotstxt();
if (rep != null) {
long crawlDelay = (long)(1000 * rep.getDirectivesFor(ua).getCrawlDelay());
crawlDelay =
(crawlDelay > respectThreshold)
? respectThreshold
: crawlDelay;
if (crawlDelay > durationToWait) {
// wait at least the directive crawl-delay
durationToWait = crawlDelay;
}
}
}
long now = System.currentTimeMillis();
int maxBandwidthKB = getMaxPerHostBandwidthUsageKbSec();
if (maxBandwidthKB > 0) {
// Enforce bandwidth limit
ServerCache cache = this.getServerCache();
CrawlHost host = cache.getHostFor(curi.getUURI());
long minDurationToWait = host.getEarliestNextURIEmitTime()
- now;
float maxBandwidth = maxBandwidthKB * 1.024F; // kilo factor
long processedBytes = curi.getContentSize();
host
.setEarliestNextURIEmitTime((long)(processedBytes / maxBandwidth)
+ now);
if (minDurationToWait > durationToWait) {
durationToWait = minDurationToWait;
}
}
}
return durationToWait;
}
仔细看一下上面的代码,发现maxDelayMs这个参数可以直接设置为0。。。哈哈,改了一下再试,这个线程就没法睡觉了。。。睡啥啊,干完活想睡多久睡多久
<bean id="disposition" class="org.archive.crawler.postprocessor.DispositionProcessor">
<!-- <property name="delayFactor" value="5.0" /> -->
<!-- <property name="minDelayMs" value="3000" /> -->
<!-- <property name="respectCrawlDelayUpToSeconds" value="300" /> -->
<property name="maxDelayMs" value="0" />
<!-- <property name="maxPerHostBandwidthUsageKbSec" value="0" /> -->
</bean>
实战结果统计为:
0.57 URIs/sec (0.64 avg); 35 KB/sec (28 avg)
而之前这个值为:
0.1 URIs/sec (0.07 avg) 6 KB/sec(4 avg)
提升了5倍左右