【无标题】

2021SC@SDUSC

“bin/nutch parse crawl/segments/*”这条命令最终会调用org.apache.nutch.parse.ParseSegment的main函数。
ParseSegment::main

public static void main(String[] args) throws Exception {
int res = ToolRunner.run(NutchConfiguration.create(), new ParseSegment(),
args);
System.exit(res);
}

ToolRunner的run函数最终调用ParseSegment的run函数。

ParseSegment::run

public int run(String[] args) throws Exception {
Path segment = new Path(args[0]);
parse(segment);
return 0;
}

public void parse(Path segment) throws IOException {

SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
long start = System.currentTimeMillis();

JobConf job = new NutchJob(getConf());
job.setJobName("parse " + segment);

FileInputFormat.addInputPath(job, new Path(segment, Content.DIR_NAME));
job.set(Nutch.SEGMENT_NAME_KEY, segment.getName());
job.setInputFormat(SequenceFileInputFormat.class);
job.setMapperClass(ParseSegment.class);
job.setReducerClass(ParseSegment.class);

FileOutputFormat.setOutputPath(job, segment);
job.setOutputFormat(ParseOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(ParseImpl.class);

JobClient.runJob(job);

}

ParseSegment的parse函数创建一个Job，设置输入为crawl/segments/2*下的content目录，处理函数为ParseSegment的map和reduce函数，最后的输出使用ParseOutputFormat处理。其中，reduce函数没有任何功能，因此下面主要看map和ParseOutputFormat的处理函数。

ParseSegment::map

public void map(WritableComparable<?> key, Content content,
OutputCollector<Text, ParseImpl> output, Reporter reporter)
throws IOException {

...

parseUtil = new ParseUtil(getConf());
ParseResult parseResult = parseUtil.parse(content);

for (Entry<Text, Parse> entry : parseResult) {
  Text url = entry.getKey();
  Parse parse = entry.getValue();
  ParseStatus parseStatus = parse.getData().getStatus();

  ...

  parse.getData().getContentMeta()
      .set(Nutch.SEGMENT_NAME_KEY, getConf().get(Nutch.SEGMENT_NAME_KEY));
  byte[] signature = SignatureFactory.getSignature(getConf()).calculate(
      content, parse);
  parse.getData().getContentMeta()
      .set(Nutch.SIGNATURE_KEY, StringUtil.toHexString(signature));

  ...

  output.collect(
      url,
      new ParseImpl(new ParseText(parse.getText()), parse.getData(), parse
          .isCanonical()));
}

}

传入的参数Content为网页内容的封装，map函数创建ParseUtil并调用其parse函数对所有网页进行解析，解析的结果封装在ParseResult结构中。然后遍历ParseResult结构，将结果封装为ParseImpl并写入文件中。

ParseSegment::map->ParseUtil::parse

public ParseResult parse(Content content) throws ParseException {
Parser[] parsers = null;

parsers = this.parserFactory.getParsers(content.getContentType(),
      content.getUrl() != null ? content.getUrl() : "");

for (int i = 0; i < parsers.length; i++) {
  ParseResult parseResult = runParser(parsers[i], content);

  if (parseResult != null && !parseResult.isEmpty())
    return parseResult;
}

}

getParsers最后获得org.apache.nutch.parse.html.HtmlParser（根据抓取的文件不同调用不同的Parser），然后调用runParser解析并返回ParseResult。

ParseSegment::map->ParseUtil::parse->runParser

private ParseResult runParser(Parser p, Content content) {
ParseCallable pc = new ParseCallable(p, content);
Future task = executorService.submit(pc);
ParseResult res = null;
try {
res = task.get(maxParseTime, TimeUnit.SECONDS);
} catch (Exception e) {
task.cancel(true);
} finally {
pc = null;
}
return res;
}

public ParseResult call() throws Exception {
return p.getParse(content);
}

runParser针对每个Parser创建线程进行解析，ParseCallable的call函数如下，

public ParseResult call() throws Exception {
return p.getParse(content);
}
1
2
3
因此每个线程最终调用HtmlParser的getParse函数进行解析并返回ParseResult。

HtmlParser::getParse

public ParseResult getParse(Content content) {
HTMLMetaTags metaTags = new HTMLMetaTags();

URL base = new URL(content.getBaseUrl());

String text = "";
String title = "";
Outlink[] outlinks = new Outlink[0];
Metadata metadata = new Metadata();

DocumentFragment root;

byte[] contentInOctets = content.getContent();
InputSource input = new InputSource(new ByteArrayInputStream(
      contentInOctets));

EncodingDetector detector = new EncodingDetector(conf);
detector.autoDetectClues(content, true);
detector.addClue(sniffCharacterEncoding(contentInOctets), "sniffed");
String encoding = detector.guessEncoding(content, defaultCharEncoding);

metadata.set(Metadata.ORIGINAL_CHAR_ENCODING, encoding);
metadata.set(Metadata.CHAR_ENCODING_FOR_CONVERSION, encoding);

input.setEncoding(encoding);
root = parse(input);

HTMLMetaProcessor.getMetaTags(metaTags, root, base);
if (!metaTags.getNoIndex()) {
  StringBuffer sb = new StringBuffer();
  utils.getText(sb, root);
  text = sb.toString();
  sb.setLength(0);
  utils.getTitle(sb, root);
  title = sb.toString().trim();
}

if (!metaTags.getNoFollow()) {
  ArrayList<Outlink> l = new ArrayList<Outlink>();
  URL baseTag = utils.getBase(root);
  utils.getOutlinks(baseTag != null ? baseTag : base, l, root);
  outlinks = l.toArray(new Outlink[l.size()]);
}

ParseStatus status = new ParseStatus(ParseStatus.SUCCESS);
ParseData parseData = new ParseData(status, title, outlinks,
    content.getMetadata(), metadata);
ParseResult parseResult = ParseResult.createParseResult(content.getUrl(),
    new ParseImpl(text, parseData));

ParseResult filteredParse = this.htmlParseFilters.filter(content,
    parseResult, metaTags, root);
return filteredParse;

}

sniffCharacterEncoding取出内容的前面一部分byte，和guessEncoding一起获取html文件的编码，例如UTF-8。接下来的parse函数最终调用parseNeko函数通过NekoHtml解析html文件，返回根节点。
getMetaTags获取文档的meta信息，DOMContentUtils的getText函数获取html文件中的文本内容，getTitle函数获取标题，getOutlinks获取网页内的连接，最后创建ParseResult封装这些信息并返回，下面一一看这些函数。

HtmlParser::getParse->parse->parseNeko

private DocumentFragment parseNeko(InputSource input) throws Exception {
DOMFragmentParser parser = new DOMFragmentParser();
try {
parser
.setFeature(
“http://cyberneko.org/html/features/scanner/allow-selfclosing-iframe”,
true);
parser.setFeature(“http://cyberneko.org/html/features/augmentations”,
true);
parser.setProperty(
“http://cyberneko.org/html/properties/default-encoding”,
defaultCharEncoding);
parser
.setFeature(
“http://cyberneko.org/html/features/scanner/ignore-specified-charset”,
true);
parser
.setFeature(
“http://cyberneko.org/html/features/balance-tags/ignore-outside-content”,
false);
parser.setFeature(
“http://cyberneko.org/html/features/balance-tags/document-fragment”,
true);
parser.setFeature(“http://cyberneko.org/html/features/report-errors”,
LOG.isTraceEnabled());
} catch (SAXException e) {
}

HTMLDocumentImpl doc = new HTMLDocumentImpl();
doc.setErrorChecking(false);
DocumentFragment res = doc.createDocumentFragment();
DocumentFragment frag = doc.createDocumentFragment();
parser.parse(input, frag);
res.appendChild(frag);

while (true) {
  frag = doc.createDocumentFragment();
  parser.parse(input, frag);
  if (!frag.hasChildNodes())
    break;
  res.appendChild(frag);
}
return res;

}

NekoHTML会扫描解析html文件并作适当的修正，例如当缺少META标签时会添加等等。具体如何处理该html文件由DOMFragmentParser的setFeature和setFeature界定，最后调用DOMFragmentParser的parse函数解析文档（input）生成的DocumentFragment代表文档的某一部分，然后将其添加到根DocumentFragment中并返回。