Nutch开源搜索引擎的crawl日志分析及工作目录说明

看了nutch关于crawl的源码后，我将crawl的日志分析了一下，主要是熟悉一下整个下载、分析、索引的过程。nutch在整个过程中都是通过Hadoop的MapReduce来实现的。 
可以通过nutch来深入学习Hadoop编程，都是比较横的代码。这一块待以后研究完毕后，blog出来。 

crawl通过nutch-default.xml参数来控制运行过程，另外需要修改crawl-urlfilter.txt和nutch-site.xml来配合运行，这一点在之前的文章中都提到过，就不再赘述了。 

下面是我用来crawl的命令 
> bin/nutch crawl urls -dir crawl10 -depth 10 -threads 10 >& nohup.out 

crawl started in: crawl10                               //表明网目录络蜘蛛的名称 
rootUrlDir = urls //待下载数据的列表文件或列表
threads = 10 //下载线程为10个 
depth = 10 //深度是10层 
Injector: starting //注入下载列表
Injector: crawlDb: crawl10/crawldb 
Injector: urlDir: urls 
Injector: Converting injected urls to crawl db entries. //根据注入的列表生成待下载的地址库 
Injector: Merging injected urls into crawl db. //执行merge
Injector: done 
Generator: Selecting best-scoring urls due for fetch. //判断网页重要性，决定下载顺序 
Generator: starting 
Generator: segment: crawl10/segments/20080904102201 //生成下载结果存储的数据段 
Generator: filtering: false 
Generator: topN: 2147483647 //没有指定topN大小，nutch会取默认值
Generator: Partitioning selected urls by host, for politeness.  //将url下载列表按hadoop的中配置文件slaves中定义的datanode来分配。 
Generator: done. 
Fetcher: starting 
Fetcher: segment: crawl10/segments/20080904102201 //下载指定网页内容到segment中去 
Fetcher: done 
CrawlDb update: starting //下载完毕后，更新下载数据库，增加新的下载 
CrawlDb update: db: crawl10/crawldb 
CrawlDb update: segments: [crawl10/segments/20080904102201] 
CrawlDb update: additions allowed: true 
CrawlDb update: URL normalizing: true 
CrawlDb update: URL filtering: true 
CrawlDb update: Merging segment data into db. 
CrawlDb update: done 

//循环执行下载 
Generator: Selecting best-scoring urls due for fetch. 
Generator: starting 
Generator: segment: crawl10/segments/20080904102453 
Generator: filtering: false 
Generator: topN: 2147483647 
Generator: Partitioning selected urls by host, for politeness. 
Generator: done. 
Fetcher: starting 
Fetcher: segment: crawl10/segments/20080904102453 
Fetcher: done 
CrawlDb update: starting 
CrawlDb update: db: crawl10/crawldb 
CrawlDb update: segments: [crawl10/segments/20080904102453] 
CrawlDb update: additions allowed: true 
CrawlDb update: URL normalizing: true 
CrawlDb update: URL filtering: true 
CrawlDb update: Merging segment data into db. 
CrawlDb update: done 

...... //一共循环10次，Nutch的局域网模式采用了广度优先策略，把二级页面抓取完成以后，进行三级页面抓取。 

LinkDb: starting //进行网页链接关系分析 
LinkDb: linkdb: crawl10/linkdb 
LinkDb: URL normalize: true //规范化 
LinkDb: URL filter: true //根据crawl-urlfilter.txt来过滤 
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904102201
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904102453 
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904102841 
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904104322 
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904113511 
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904132510 
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904153615 
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904175052 
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904194724 
LinkDb: adding segment: /user/nutch/crawl10/segments/20080904211956 
LinkDb: done //链接分析完毕 
Indexer: starting //开始创建索引 
Indexer: linkdb: crawl10/linkdb 
Indexer: adding segment: /user/nutch/crawl10/segments/20080904102201 
Indexer: adding segment: /user/nutch/crawl10/segments/20080904102453 
Indexer: adding segment: /user/nutch/crawl10/segments/20080904102841 
Indexer: adding segment: /user/nutch/crawl10/segments/20080904104322 
Indexer: adding segment: /user/nutch/crawl10/segments/20080904113511 
Indexer: adding segment: /user/nutch/crawl10/segments/20080904132510 
Indexer: adding segment: /user/nutch/crawl10/segments/20080904153615 
Indexer: adding segment: /user/nutch/crawl10/segments/20080904175052 
Indexer: adding segment: /user/nutch/crawl10/segments/20080904194724 
Indexer: adding segment: /user/nutch/crawl10/segments/20080904211956 
Indexer: done //索引创建完毕 
Dedup: starting //风页去重 
Dedup: adding indexes in: crawl10/indexes
Dedup: done 
merging indexes to: crawl10/index //索引合并 
Adding /user/nutch/crawl10/indexes/part-00000 
Adding /user/nutch/crawl10/indexes/part-00001 
Adding /user/nutch/crawl10/indexes/part-00002 
Adding /user/nutch/crawl10/indexes/part-00003 
done merging //合并完毕 
crawl finished: crawl10 //入口注入、循环下载、链接分析、建立索引、去重、合并 


抓取程序自动在用户根目录(/user/nutch)下面建立了crawl10目录，可以看到crawldb,segments,index,indexs,linkdb目录, 
1)crawldb目录下面存放下载的URL,以及下载的日期，用来页面更新检查时间。 
2)linkdb目录存放URL的关联关系，是下载完成后分析时创建的，通过这个关联关系可以实现类似google的pagerank功能。 
3)segments目录存储抓取的页面，下面子目录的个数与获取页面的层数有关系，我指定-depth是10层，这个目录下就有10层。 
  里面有6个子目录 
  content,下载页面的内容 
  crawl_fetch,下载URL的状态内容 
  crawl_generate,待下载的URL的集合，在generate任务生成时和下载过程中持续分析出来 
  crawl_parse,存放用来更新crawldb的外部链接库 
  parse_data,存放每个URL解析出来的外部链接和元数据 
  parse_text,存放每个解析过的URL的文本内容 
4)index目录存放符合lucene格式的索引目录，是indexs里所有的索引内容合并后的完整内容,看了一下这里的索引文件和用lucenedemo做出来的文件名称都不一样，待进一步研究 
5)indexs目录存放每次下载的索引目录，存放part-0000到part-0003 

crawl流程和工作目录搞清楚了，大家可以进一步拜读一下nutch源码。进一步搞清楚各个目录下面的文件作用，以及文件格式。待续！