Larbin Source Code Analysis 3——Class global(updating……)

最新推荐文章于 2017-11-14 14:39:00 发布

Harold

最新推荐文章于 2017-11-14 14:39:00 发布

阅读量828

点赞数

分类专栏： Network Crawler Analysis 文章标签： class destructor profiling signal constructor debugging

本文链接：https://blog.csdn.net/hero7935/article/details/6398074

版权

Network Crawler Analysis 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

#include

Collaboration diagram for global:

Class global is the head, the heart, the core of the Larbin, so its complexity is unavoidable and releatively acceptable(Don’t tell me you have been imaging that a webcrawler’s main class should be VERY little and small).

YES, you must be compaining in front of your lovely computer and saying “Damn! can’t you make it clearer?"

But the fact is You DON’T need to see it clearly. All you have to do is know how complexity the relationsip, because we will dig into it later, don’t worry!

_________________________________________________________________________________________________

0. in main.cc

/*
* author: Ailleret
* comments by: Harold Wang
* http://blog.csdn.net/hero7935
*/

///
// If this thread terminates, the whole program exits
int main (int argc, char *argv[]) {
// create all the structures
global glob(argc, argv); /*initialize all we need*/
#ifdef PROF
signal (2, handler); /*setup signal handler for SIGINT*/
#endif // PROF

#ifndef NOWEBSERVER
// launch the webserver if needeed
if (global::httpPort != 0)
startThread(startWebserver, NULL);
/*if we need a webserver to see the state and condition of the larbin*/
#endif // NOWEBSERVER

// Start the search
printf("%s is starting its search/n", global::userAgent);
time_t old = global::now;

for (;;)
{                  /*Ok, Never end!*/
    // update time
    global::now = time(NULL);
    if (old != global::now) {
      // this block is called every second
      old = global::now;
      cron();
/*a lot of stats and profiling things. like shall we stop? check if timeouts? etc.*/
    }
    stateMain(-count); /*where is the main loop(just for debugging)*/
    waitBandwidth(&old); /*wait to limit bandwidth usage*/
    stateMain(1);
    for (int i=0; i      global::ansPoll[i] = 0; /*initialize I/O multiplexing*/
/*register the I/O multiplexing fds set which we have intrest in*/
    for (uint i=0; i      global::ansPoll[global::pollfds[i].fd] = global::pollfds[i].revents;
    global::posPoll = 0; /*initial the pos of the max used field in pollfds to 0*/
    stateMain(2);
    input(); /*if we have inputfiles to handle*/
    stateMain(3);
/*put urls from the queue(including global::URLsPriority || global::URLsPriorityWait || *global::URLsDisk || global::URLsDiskWait) to the queue(global::namedSiteList[u->hostHashCode()]) */
    sequencer();
    stateMain(4);
/*get website from dnsSites to start dns request and receive the website’s robot.txt*/
    fetchDns();
    stateMain(5);
/*get a namesite from OKSite and go on crawling its urls*/
    fetchOpen();
    stateMain(6);
/*check all the connections’s state, I do not finish this item’s comment yet.*/
    checkAll();
    // select
    stateMain(count++);
/*put pollfds to the device waiting queue, waiting for any events in the next 10 millisecond*/
    poll(global::pollfds, global::posPoll, 10); /*posPoll know how many fds we have used*/
    stateMain(7);
}
}