2021SC@SDUSC
根据上一章的分析,“bin/nutch fetch crawl/segments/*”这条命令最终会调用org.apache.nutch.fetcher.Fetcher的main函数。
public static void main(String[] args) throws Exception {
int res = ToolRunner.run(NutchConfiguration.create(), new Fetcher(), args);
System.exit(res);
}
ToolRunner的run函数进而调用Fetcher的run函数。
Fetcher::run
public int run(String[] args) throws Exception {
Path segment = new Path(args[0]);
int threads = getConf().getInt("fetcher.threads.fetch", 10);
for (int i = 1; i < args.length; i++) {
if (args[i].equals("-threads")) {
threads = Integer.parseInt(args[++i]);
}
}
getConf().setInt("fetcher.threads.fetch", threads);
fetch(segment, threads);
return 0;
}
获取抓取网页的线程数threads,默认为10,segment为crawl/segments/2*的目录路径,最后调用fetch函数。
Fetcher::run->fetch
public void fetch(Path segment, int threads) throws IOException {
checkConfiguration();
JobConf job = new NutchJob(getConf());
job.setJobName("fetch " + segment);
job.setInt("fetcher.threads.fetch", threads);
job.set(Nutch.SEGMENT_NAME_KEY, segment.getName());
job.setSpeculativeExecution(false);
FileInputFormat.addInputPath(job, new Path(segment,
CrawlDatum.GENERATE_DIR_NAME));
job.setInputFormat(InputFormat.class);
job.setMapRunnerClass(Fetcher.class);
FileOutputFormat.setOutputPath(job, segment);
job.setOutputFormat(FetcherOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(NutchWritable.class);
JobClient.runJob(job);
}
checkConfiguration在配置文件中检查是否配置了http.agent.name属性,如果没有设置则抛出异常。接下来创建hadoop的Job,输入为crawl/segments/2下的crawl_generate目录,由命令generate生成,处理函数为Fetcher中的run函数,输出也为crawl/segments/2目录下,FetcherOutputFormat定义了最后如何输出。
Fetcher::run
public void run(RecordReader<Text, CrawlDatum> input,
OutputCollector<Text, NutchWritable> output, Reporter reporter)
throws IOException {
...
feeder = new QueueFeeder(input, fetchQueues, threadCount
* queueDepthMuliplier);
feeder.start();
for (int i = 0; i < threadCount; i++) {
FetcherThread t = new FetcherThread(getConf(), getActiveThreads(), fetchQueues,
feeder, spinWaiting, lastRequestStart, reporter, errors, segmentName,
parsing, output, storingContent, pages, bytes);
fetcherThreads.add(t);
t.start();
}
...
}
Fetcher的run函数首先创建一个共享队列QueueFeeder,然后创建QueueFeeder(feeder),用于读取crawl/crawldb/2*下的url和CrawlDatum,把它们放到共享队列FetchItemQueues(fetchQueues)中。
然后创建FetcherThread,并调用其start函数开始抓取网页。
Fetcher::run->QueueFeeder::run
public void run() {
boolean hasMore = true;
while (hasMore) {
...
int feed = size - queues.getTotalSize();
if (feed <= 0) {
Thread.sleep(1000);
continue;
} else {
while (feed > 0 && hasMore) {
Text url = new Text();
CrawlDatum datum = new CrawlDatum();
hasMore = reader.next(url, datum);
if (hasMore) {
queues.addFetchItem(url, datum);
feed--;
}
}
}
}
}
feed变量表示共享队列FetchItemQueues中是否有空闲位置可以插入待抓取的url和CrawlDatum,如果feed小于0,表示空间不足,就需要进程睡眠等待,如果feed大于0,表示空间足够,此时通过RecordReader(reader)的next函数从crawl/crawldb/2*/crawl_generate文件夹中依次读取url和CrawlDatum,调用addFetchItem函数将其封装成FetchItem并添加到共享队列中。