2021SC@SDUSC
“bin/nutch parse crawl/segments/*”这条命令最终会调用org.apache.nutch.parse.ParseSegment的main函数。
ParseSegment::main
public static void main(String[] args) throws Exception {
int res = ToolRunner.run(NutchConfiguration.create(), new ParseSegment(),
args);
System.exit(res);
}
ToolRunner的run函数最终调用ParseSegment的run函数。
ParseSegment::run
public int run(String[] args) throws Exception {
Path segment = new Path(args[0]);
parse(segment);
return 0;
}
public void parse(Path segment) throws IOException {
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
long start = System.currentTimeMillis();
JobConf job = new NutchJob(getConf());
job.setJobName("parse " + segment);
FileInputFormat.addInputPath(job, new Path(segment, Content.DIR_NAME));
job.set(Nutch.SEGMENT_NAME_KEY, segment.getName());
job.setInputFormat(SequenceFileInputFormat.class);
job.setMapperClass(ParseSegment.class);
job.setReducerClass(ParseSegment.class);
FileOutputFormat.setOutputPath(job, segment);
job.setOutputFormat(ParseOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(ParseImpl.class);
JobClient.runJob(job);
}
ParseSegment的parse函数创建一个Job,设置输入为crawl/segments/2*下的content目录,处理函数为ParseSegment的map和reduce函数,最后的输出使用ParseOutputFormat处理。其中,reduce函数没有任何功能,因此下面主要看map和ParseOutputFormat的处理函数。
ParseSegment::map
public void map(WritableComparable<?> key, Content content,
OutputCollector<Text, ParseImpl> output, Reporter reporter)
throws IOException {
...
parseUtil = new ParseUtil(getConf());
ParseResult parseResult = parseUtil.parse(content);
for (Entry<Text, Parse> entry : parseResult) {
Text url = entry.getKey();
Parse parse = entry.getValue();
ParseStatus parseStatus = parse.getData().getStatus();
...
parse.getData().getContentMeta()
.set(Nutch.SEGMENT_NAME_KEY, getConf().get(Nutch.SEGMENT_NAME_KEY));
byte[] signature = SignatureFactory.getSignature(getConf()).calculate(
content, parse);
parse.getData().getContentMeta()
.set(Nutch.SIGNATURE_KEY, StringUtil.toHexString(signature));
...
output.collect(
url,
new ParseImpl(new ParseText(parse.getText()), parse.getData(), parse
.isCanonical()));
}
}
传入的参数Content为网页内容的封装,map函数创建ParseUtil并调用其parse函数对所有网页进行解析,解析的结果封装在ParseResult结构中。然后遍历ParseResult结构,将结果封装为ParseImpl并写入文件中。
ParseSegment::map->ParseUtil::parse
public ParseResult parse(Content content) throws ParseException {
Parser[] parsers = null;
parsers = this.parserFactory.getParsers(content.getContentType(),
content.getUrl() != null ? content.getUrl() : "");
for (int i = 0; i < parsers.length; i++) {
ParseResult parseResult = runParser(parsers[i], content);
if (parseResult != null && !parseResult.isEmpty())
return parseResult;
}
}
getParsers最后获得org.apache.nutch.parse.html.HtmlParser(根据抓取的文件不同调用不同的Parser),然后调用runParser解析并返回ParseResult。
ParseSegment::map->ParseUtil::parse->runParser
private ParseResult runParser(Parser p, Content content) {
ParseCallable pc = new ParseCallable(p, content);
Future task = executorService.submit(pc);
ParseResult res = null;
try {
res = task.get(maxParseTime, TimeUnit.SECONDS);
} catch (Exception e) {
task.cancel(true);
} finally {
pc = null;
}
return res;
}
public ParseResult call() throws Exception {
return p.getParse(content);
}
runParser针对每个Parser创建线程进行解析,ParseCallable的call函数如下,
public ParseResult call() throws Exception {
return p.getParse(content);
}
1
2
3
因此每个线程最终调用HtmlParser的getParse函数进行解析并返回ParseResult。
HtmlParser::getParse
public ParseResult getParse(Content content) {
HTMLMetaTags metaTags = new HTMLMetaTags();
URL base = new URL(content.getBaseUrl());
String text = "";
String title = "";
Outlink[] outlinks = new Outlink[0];
Metadata metadata = new Metadata();
DocumentFragment root;
byte[] contentInOctets = content.getContent();
InputSource input = new InputSource(new ByteArrayInputStream(
contentInOctets));
EncodingDetector detector = new EncodingDetector(conf);
detector.autoDetectClues(content, true);
detector.addClue(sniffCharacterEncoding(contentInOctets), "sniffed");
String encoding = detector.guessEncoding(content, defaultCharEncoding);
metadata.set(Metadata.ORIGINAL_CHAR_ENCODING, encoding);
metadata.set(Metadata.CHAR_ENCODING_FOR_CONVERSION, encoding);
input.setEncoding(encoding);
root = parse(input);
HTMLMetaProcessor.getMetaTags(metaTags, root, base);
if (!metaTags.getNoIndex()) {
StringBuffer sb = new StringBuffer();
utils.getText(sb, root);
text = sb.toString();
sb.setLength(0);
utils.getTitle(sb, root);
title = sb.toString().trim();
}
if (!metaTags.getNoFollow()) {
ArrayList<Outlink> l = new ArrayList<Outlink>();
URL baseTag = utils.getBase(root);
utils.getOutlinks(baseTag != null ? baseTag : base, l, root);
outlinks = l.toArray(new Outlink[l.size()]);
}
ParseStatus status = new ParseStatus(ParseStatus.SUCCESS);
ParseData parseData = new ParseData(status, title, outlinks,
content.getMetadata(), metadata);
ParseResult parseResult = ParseResult.createParseResult(content.getUrl(),
new ParseImpl(text, parseData));
ParseResult filteredParse = this.htmlParseFilters.filter(content,
parseResult, metaTags, root);
return filteredParse;
}
sniffCharacterEncoding取出内容的前面一部分byte,和guessEncoding一起获取html文件的编码,例如UTF-8。接下来的parse函数最终调用parseNeko函数通过NekoHtml解析html文件,返回根节点。
getMetaTags获取文档的meta信息,DOMContentUtils的getText函数获取html文件中的文本内容,getTitle函数获取标题,getOutlinks获取网页内的连接,最后创建ParseResult封装这些信息并返回,下面一一看这些函数。
HtmlParser::getParse->parse->parseNeko
private DocumentFragment parseNeko(InputSource input) throws Exception {
DOMFragmentParser parser = new DOMFragmentParser();
try {
parser
.setFeature(
“http://cyberneko.org/html/features/scanner/allow-selfclosing-iframe”,
true);
parser.setFeature(“http://cyberneko.org/html/features/augmentations”,
true);
parser.setProperty(
“http://cyberneko.org/html/properties/default-encoding”,
defaultCharEncoding);
parser
.setFeature(
“http://cyberneko.org/html/features/scanner/ignore-specified-charset”,
true);
parser
.setFeature(
“http://cyberneko.org/html/features/balance-tags/ignore-outside-content”,
false);
parser.setFeature(
“http://cyberneko.org/html/features/balance-tags/document-fragment”,
true);
parser.setFeature(“http://cyberneko.org/html/features/report-errors”,
LOG.isTraceEnabled());
} catch (SAXException e) {
}
HTMLDocumentImpl doc = new HTMLDocumentImpl();
doc.setErrorChecking(false);
DocumentFragment res = doc.createDocumentFragment();
DocumentFragment frag = doc.createDocumentFragment();
parser.parse(input, frag);
res.appendChild(frag);
while (true) {
frag = doc.createDocumentFragment();
parser.parse(input, frag);
if (!frag.hasChildNodes())
break;
res.appendChild(frag);
}
return res;
}
NekoHTML会扫描解析html文件并作适当的修正,例如当缺少META标签时会添加等等。具体如何处理该html文件由DOMFragmentParser的setFeature和setFeature界定,最后调用DOMFragmentParser的parse函数解析文档(input)生成的DocumentFragment代表文档的某一部分,然后将其添加到根DocumentFragment中并返回。