【无标题】

2021SC@SDUSC

Fetcher::run->FetcherThread::run

public void run() {

FetchItem fit = null;
try {
  while (true) {

    ...

    fit = ((FetchItemQueues) fetchQueues).getFetchItem();

    ...

    try {
      do {
        Protocol protocol = this.protocolFactory.getProtocol(fit.url
            .toString());
        BaseRobotRules rules = protocol.getRobotRules(fit.url, fit.datum);

        ...

        ProtocolOutput output = protocol.getProtocolOutput(fit.url,
            fit.datum);
        ProtocolStatus status = output.getStatus();
        Content content = output.getContent();
        ParseStatus pstatus = null;
        ((FetchItemQueues) fetchQueues).finishFetchItem(fit);
        String urlString = fit.url.toString();
        switch (status.getCode()) {

        ...

        case ProtocolStatus.SUCCESS:
          pstatus = output(fit.url, fit.datum, content, status,
              CrawlDatum.STATUS_FETCH_SUCCESS, fit.outlinkDepth);
          ...
          break;

        ...

        }

        ...

      } while (redirecting && (redirectCount <= maxRedirect));

    } catch (Throwable t) {

    }
  }

} catch (Throwable e) {

} finally {

}

}

protocolFactory为ProtocolFactory，根据url的协议（例如http、ftp等），从插件库中获取协议类org.apache.nutch.protocol.http.Http。Http的getRobotRules函数用于抓取对应url下的robots.txt文件，robots.txt是爬虫协议，搜索引擎应该根据该协议设定抓取策略，例如哪些页面可以或不可以抓取，这里假设为null。
接下来通过Http的getProtocolOutput函数从url对应的地址读取内容并返回一个ProtocolOutput，内部封装了从url读取出的信息以及状态码。
最后调用output将刚刚得到的数据写入文件中。

Fetcher::run->FetcherThread::run->Http::getProtocolOutput

public ProtocolOutput getProtocolOutput(Text url, CrawlDatum datum) {

String urlString = url.toString();
try {
  URL u = new URL(urlString);

  long startTime = System.currentTimeMillis();
  Response response = getResponse(u, datum, false);

  if (this.responseTime) {
    int elapsedTime = (int) (System.currentTimeMillis() - startTime);
    datum.getMetaData().put(RESPONSE_TIME, new IntWritable(elapsedTime));
  }

  int code = response.getCode();
  datum.getMetaData().put(Nutch.PROTOCOL_STATUS_CODE_KEY,
    new Text(Integer.toString(code)));

  byte[] content = response.getContent();
  Content c = new Content(u.toString(), u.toString(),
      (content == null ? EMPTY_CONTENT : content),
      response.getHeader("Content-Type"), response.getHeaders(), this.conf);

  if (code == 200) { 
    return new ProtocolOutput(c);
  }

  ...
} catch (Throwable e) {

}

}

getProtocolOutput的主要任务是从url地址处根据协议获取信息（一般情况下就是下载url对应的网页），并刷新CrawlDatum中信息，最后创建ProtocolOutput并返回，ProtocolOutput内保存了两个重要的内容，一是url地址处对应的内容（例如网站html文件内的内容），二是保存了本次请求的状态信息，例如是否成功等等。

Fetcher::run->FetcherThread::run->output

private ParseStatus output(Text key, CrawlDatum datum, Content content,
ProtocolStatus pstatus, int status, int outlinkDepth) {

datum.setStatus(status);
datum.setFetchTime(System.currentTimeMillis());
if (pstatus != null)
  datum.getMetaData().put(Nutch.WRITABLE_PROTO_STATUS_KEY, pstatus);

ParseResult parseResult = null;
if (content != null) {
  Metadata metadata = content.getMetadata();

  if (content.getContentType() != null)
    datum.getMetaData().put(new Text(Metadata.CONTENT_TYPE),
        new Text(content.getContentType()));

  metadata.set(Nutch.SEGMENT_NAME_KEY, segmentName);
  try {
    scfilters.passScoreBeforeParsing(key, datum, content);
  } catch (Exception e) {

  }

  ...

  content.getMetadata().add(Nutch.FETCH_STATUS_KEY,
      Integer.toString(status));
}

try {
  output.collect(key, new NutchWritable(datum));
  if (content != null && storingContent)
    output.collect(key, new NutchWritable(content));

  ...

} catch (IOException e) {

}
return null;

}

省略的代码和解析有关，放在下一章分析。output函数向CrawlDatum中记录各个信息，然后通过collect收集CrawlDatum和Content，最后由FetcherOutputFormat定义了如何输出。

FetcherOutputFormat::getRecordWriter

public RecordWriter<Text, NutchWritable> getRecordWriter(final FileSystem fs,
final JobConf job, final String name, final Progressable progress)
throws IOException {

Path out = FileOutputFormat.getOutputPath(job);
final Path fetch = new Path(new Path(out, CrawlDatum.FETCH_DIR_NAME), name);
final Path content = new Path(new Path(out, Content.DIR_NAME), name);

final CompressionType compType = SequenceFileOutputFormat
    .getOutputCompressionType(job);

Option fKeyClassOpt = MapFile.Writer.keyClass(Text.class);
org.apache.hadoop.io.SequenceFile.Writer.Option fValClassOpt = SequenceFile.Writer.valueClass(CrawlDatum.class);
org.apache.hadoop.io.SequenceFile.Writer.Option fProgressOpt = SequenceFile.Writer.progressable(progress);
org.apache.hadoop.io.SequenceFile.Writer.Option fCompOpt = SequenceFile.Writer.compression(compType);

final MapFile.Writer fetchOut = new MapFile.Writer(job,
    fetch, fKeyClassOpt, fValClassOpt, fCompOpt, fProgressOpt);

return new RecordWriter<Text, NutchWritable>() {
  private MapFile.Writer contentOut;
  private RecordWriter<Text, Parse> parseOut;

  {
    if (Fetcher.isStoringContent(job)) {
      Option cKeyClassOpt = MapFile.Writer.keyClass(Text.class);
      org.apache.hadoop.io.SequenceFile.Writer.Option cValClassOpt = SequenceFile.Writer.valueClass(Content.class);
      org.apache.hadoop.io.SequenceFile.Writer.Option cProgressOpt = SequenceFile.Writer.progressable(progress);
      org.apache.hadoop.io.SequenceFile.Writer.Option cCompOpt = SequenceFile.Writer.compression(compType);
      contentOut = new MapFile.Writer(job, content,
          cKeyClassOpt, cValClassOpt, cCompOpt, cProgressOpt);
    }

    if (Fetcher.isParsing(job)) {
      parseOut = new ParseOutputFormat().getRecordWriter(fs, job, name,
          progress);
    }
  }

  public void write(Text key, NutchWritable value) throws IOException {

    Writable w = value.get();

    if (w instanceof CrawlDatum)
      fetchOut.append(key, w);
    else if (w instanceof Content && contentOut != null)
      contentOut.append(key, w);
    else if (w instanceof Parse && parseOut != null)
      parseOut.write(key, (Parse) w);
  }

  public void close(Reporter reporter) throws IOException {
    fetchOut.close();
    if (contentOut != null) {
      contentOut.close();
    }
    if (parseOut != null) {
      parseOut.close(reporter);
    }
  }

};

}

FETCH_DIR_NAME和DIR_NAME常量分别为crawl_fetch和content，getRecordWriter在crawl/segments/2*目录下创建这两个目录和对应的输出流。getRecordWriter函数最后返回RecordWriter，对应的write函数根据数据类型输出到不同的文件中。