Larbin learnin(3)——how to limit the scope of crawling

1 In  larbin.conf, close to crawle external sites except  in the list of startURL by setting noExternalLinks.

# do you want to follow external links,

noExternalLinks

2 The way to improve the crawlling speed

【引】http://hi.baidu.com/hustwk/blog/item/fd3325dde12598dc8c1029ef.html

    1、将larbin.conf里面的waitDuration设置为1，这里不再考虑polite^_^, 设置为1大多数网站其实还能忍受；

    2、将types.h里面的maxUrlsBySite修改为254；

    3、将main.cc里面的代码做如下修改：

   // see if we should read again urls in fifowait
if (( global::now % 30) == 0 ) {
    global::readPriorityWait = global::URLsPriorityWait->getLength();
    global::readWait = global::URLsDiskWait->getLength();
}
if ((global::now % 30) == 15) {
    global::readPriorityWait = 0;
    global::readWait = 0;
}