org.archive.crawler.framework.ToeThread

[b]1、controller.getFetchChain().process(curi,this);[/b]
1.1、org.archive.crawler.prefetch.Preselector,
1.2、org.archive.crawler.prefetch.PreconditionEnforcer,
1.3、org.archive.modules.fetcher.FetchDNS,
//httpclient
1.4、org.archive.modules.fetcher.FetchHTTP,
1.5、org.archive.modules.extractor.ExtractorHTTP,
1.6、org.archive.modules.extractor.ExtractorHTML,
1.7、org.archive.modules.extractor.ExtractorCSS,
1.8、org.archive.modules.extractor.ExtractorJS,
1.9、org.archive.modules.extractor.ExtractorSWF

[b]2、controller.getFrontier().beginDisposition(curi);[/b]
BdbFrontier.beginDisposition(curi)

[b]3、controller.getDispositionChain().process(curi,this);[/b]
3.1、org.archive.modules.writer.MirrorWriterProcessor
//getCandidateChain().process(prereq, null);//add seeds
3.2、org.archive.crawler.postprocessor.CandidatesProcessor
//run isInScope(curi)
3.2.1、org.archive.crawler.prefetch.CandidateScoper
3.2.2、org.archive.crawler.prefetch.FrontierPreparer
// set schedulingDirective
3.2.2.1、curi.setSchedulingDirective(getSchedulingDirective(curi));
// set canonicalized version
3.2.2.2、curi.setCanonicalString(canonicalize(curi));
// set queue key
3.2.2.3、curi.setClassKey(getClassKey(curi));
// set cost
3.2.2.4、curi.setHolderCost(getCost(curi));
// set URI precedence
3.2.2.5、getUriPrecedencePolicy().uriScheduled(curi);
3.3、org.archive.crawler.postprocessor.DispositionProcessor