2021SC@SDUSC
Selector::map->DefaultFetchSchedule::shouldFetch
public boolean shouldFetch(Text url, CrawlDatum datum, long curTime) {
if (datum.getFetchTime() - curTime > (long) maxInterval * 1000) {
if (datum.getFetchInterval() > maxInterval) {
datum.setFetchInterval(maxInterval * 0.9f);
}
datum.setFetchTime(curTime);
}
if (datum.getFetchTime() > curTime) {
return false;
}
return true;
}
再往下,如果url对应的信息中设置了WRITABLE_GENERATE_TIME_KEY对应的值,即不为null,且上一次抓取该url的时间没有过期,过期的默认时间genDelay为7天,则不进行抓取。
再往下的generatorSortValue会给url进行打分,CrawlDatum的evaluate函数会对CrawlDatum中的数据进行Jexl解析,这里就不往下看了。map函数再往下检查状态、打分、抓取间隔是否满足指定要求。
Selector的map函数的执行结果,会通过Selector的getPartition函数分发到Reducer上。
Selector::getPartition
public int getPartition(FloatWritable key, Writable value,
int numReduceTasks) {
return partitioner.getPartition(((SelectorEntry) value).url, key,
numReduceTasks);
}
partitioner默认为URLPartitioner,传入的参数numReduceTasks表示有多少任务需要执行。URLPartitioner的getPartition函数最终返回的整型指定哪一个Reducer处理Maper的输出结果。
Selector::getPartition->URLPartitioner::getPartition
public int getPartition(Text key, Writable value, int numReduceTasks) {
String urlString = key.toString();
URL url = null;
int hashCode = urlString.hashCode();
urlString = normalizers.normalize(urlString, URLNormalizers.SCOPE_PARTITION);
url = new URL(urlString);
hashCode = url.getHost().hashCode();
if (mode.equals(PARTITION_MODE_DOMAIN) && url != null)
hashCode = URLUtil.getDomainName(url).hashCode();
else if (mode.equals(PARTITION_MODE_IP)) {
InetAddress address = InetAddress.getByName(url.getHost());
hashCode = address.getHostAddress().hashCode();
}
hashCode ^= seed;
return (hashCode & Integer.MAX_VALUE) % numReduceTasks;
}
getPartition函数根据mode的不同,从url中提取出主机名、域名或主机IP地址,然后计算其hash值,最后模以numReduceTasks计算出需要哪个Reducer进行处理。
最后看一下Selector的reduce函数。
Selector::reduce
public void reduce(FloatWritable key, Iterator<SelectorEntry> values,
OutputCollector<FloatWritable, SelectorEntry> output, Reporter reporter)
throws IOException {
while (values.hasNext()) {
if (count == limit) {
if (currentsegmentnum < maxNumSegments) {
count = 0;
currentsegmentnum++;
} else
break;
}
SelectorEntry entry = values.next();
Text url = entry.url;
String urlString = url.toString();
URL u = null;
String hostordomain = null;
if (normalise && normalizers != null) {
urlString = normalizers.normalize(urlString, URLNormalizers.SCOPE_GENERATE_HOST_COUNT);
}
u = new URL(urlString);
if (byDomain) {
hostordomain = URLUtil.getDomainName(u);
} else {
hostordomain = new URL(urlString).getHost();
}
hostordomain = hostordomain.toLowerCase();
if (maxCount > 0) {
int[] hostCount = hostCounts.get(hostordomain);
if (hostCount == null) {
hostCount = new int[] { 1, 0 };
hostCounts.put(hostordomain, hostCount);
}
hostCount[1]++;
while (segCounts[hostCount[0] - 1] >= limit
&& hostCount[0] < maxNumSegments) {
hostCount[0]++;
hostCount[1] = 0;
}
if (hostCount[1] >= maxCount) {
if (hostCount[0] < maxNumSegments) {
hostCount[0]++;
hostCount[1] = 0;
} else {
continue;
}
}
entry.segnum = new IntWritable(hostCount[0]);
segCounts[hostCount[0] - 1]++;
} else {
entry.segnum = new IntWritable(currentsegmentnum);
segCounts[currentsegmentnum - 1]++;
}
output.collect(key, entry);
count++;
}
}
reduce函数的核心功能就是计算该url属于哪一个segment中,不同时间获取到的不同url地址会存入crawl/segments下不同的文件夹中。
Generator::run->generate
第二部分
public Path[] generate(Path dbDir, Path segments, int numLists, long topN,
long curTime, boolean filter, boolean norm, boolean force,
int maxNumSegments, String expr) throws IOException {
...
List<Path> generatedSegments = new ArrayList<Path>();
FileStatus[] status = fs.listStatus(tempDir);
for (FileStatus stat : status) {
Path subfetchlist = stat.getPath();
if (!subfetchlist.getName().startsWith("fetchlist-"))
continue;
Path newSeg = partitionSegment(fs, segments, subfetchlist, numLists);
generatedSegments.add(newSeg);
}
if (getConf().getBoolean(GENERATE_UPDATE_CRAWLDB, false)) {
Path tempDir2 = new Path(getConf().get("mapred.temp.dir", ".")
+ "/generate-temp-" + java.util.UUID.randomUUID().toString());
job = new NutchJob(getConf());
job.setJobName("generate: updatedb " + dbDir);
job.setLong(Nutch.GENERATE_TIME_KEY, generateTime);
for (Path segmpaths : generatedSegments) {
Path subGenDir = new Path(segmpaths, CrawlDatum.GENERATE_DIR_NAME);
FileInputFormat.addInputPath(job, subGenDir);
}
FileInputFormat.addInputPath(job, new Path(dbDir, CrawlDb.CURRENT_NAME));
job.setInputFormat(SequenceFileInputFormat.class);
job.setMapperClass(CrawlDbUpdater.class);
job.setReducerClass(CrawlDbUpdater.class);
job.setOutputFormat(MapFileOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(CrawlDatum.class);
FileOutputFormat.setOutputPath(job, tempDir2);
JobClient.runJob(job);
CrawlDb.install(job, dbDir);
fs.delete(tempDir2, true);
}
LockUtil.removeLockFile(fs, lock);
fs.delete(tempDir, true);
Path[] patharray = new Path[generatedSegments.size()];
return generatedSegments.toArray(patharray);
}
tempDir目录保存了前面Reducer的输出结果,依次遍历tempDir目录下文件名为“fetchlist-”开头的子目录。partitionSegment函数内部利用hadoop将tempDir文件夹下的输出结果写入crawl/segments中,并返回目录路径。
接下来的if语句是合并crawlDb中和crawl_generate文件夹中的数据,本章不仔细往下看了。
最后删除锁文件,删除临时文件夹tempDir,最后返回crawl/segments下的目录路径,例如“crawl/segments/2016*/crawl_generate”。
Generator::run->generate->partitionSegment
private Path partitionSegment(FileSystem fs, Path segmentsDir, Path inputDir,
int numLists) throws IOException {
Path segment = new Path(segmentsDir, generateSegmentName());
Path output = new Path(segment, CrawlDatum.GENERATE_DIR_NAME);
NutchJob job = new NutchJob(getConf());
job.setJobName("generate: partition " + segment);
job.setInt("partition.url.seed", new Random().nextInt());
FileInputFormat.addInputPath(job, inputDir);
job.setInputFormat(SequenceFileInputFormat.class);
job.setMapperClass(SelectorInverseMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(SelectorEntry.class);
job.setPartitionerClass(URLPartitioner.class);
job.setReducerClass(PartitionReducer.class);
job.setNumReduceTasks(numLists);
FileOutputFormat.setOutputPath(job, output);
job.setOutputFormat(SequenceFileOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(CrawlDatum.class);
job.setOutputKeyComparatorClass(HashComparator.class);
JobClient.runJob(job);
return segment;
}
generateSegmentName函数根据当前时间生成字符串。
partitionSegment在segmentsDir指向的crawl/segments文件夹下创建generateSegmentName生成的时间为名称的文件夹,然后再在该文件夹下创建crawl_generate文件夹。
inputDir指向tempDir文件夹下的fetchlist-*文件夹。
接下来创建Job,SelectorInverseMapper的map函数什么也没做。URLPartitioner的getPartition函数前面分析过了。PartitionReducer的reduce函数从SelectorEntry中提取出CrawlDatum结构,最终写入到crawl_generate文件夹中。
总结一下,nutch的generate命令其实就是把crawl/crawldb中保存的url地址处理、过滤,最终存入crawl/segments/时间/crawl_generate文件夹中。