nutch index路径找不到问题记录

最新推荐文章于 2022-04-04 14:47:33 发布

xishanxinyue

最新推荐文章于 2022-04-04 14:47:33 发布

阅读量551

点赞数

分类专栏：搜索引擎 hadoop 文章标签： nutch 网页

本文链接：https://blog.csdn.net/xishanxinyue/article/details/46698415

版权

hadoop 同时被 2 个专栏收录

5 篇文章 0 订阅

订阅专栏

搜索引擎

3 篇文章 0 订阅

订阅专栏

nutch 在抓取网页的时代报如下错误：

2015-06-30 16:25:53,420 DEBUG security.UserGroupInformation - PrivilegedActionException as:hadoop (auth:SIMPLE) cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_data/crawl_fetch
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_data/crawl_parse
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_data/parse_data
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_data/parse_text
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/content/crawl_fetch
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/content/crawl_parse
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/content/parse_data
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/content/parse_text
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_generate/crawl_fetch
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_generate/crawl_parse
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_generate/parse_data
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_generate/parse_text
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_parse/crawl_fetch
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_parse/crawl_parse
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_parse/parse_data
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_parse/parse_text
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_text/crawl_fetch
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_text/crawl_parse
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_text/parse_data
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/parse_text/parse_text
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/crawl_fetch
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/crawl_parse
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/parse_data
Input path does not exist: file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/parse_text

路径找到不到。该问题在升级nutch 到1.9的时候出现。
通过定位，要找的该路径不对，file:/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/parse_text ，每个路径后面多了一个层冗余路径。正确的路径应该是：
/app/apache-nutch-1.9/data/segments/20150630152657/crawl_fetch/ 到20150630152657时间下的一层就可以了。那肯定是路径寻找出问题了。

nutch index的时候找segment路径，是IndexingJob的如下代码实现

 for (int i = 1; i < args.length; i++) {
            if (args[i].equals("-linkdb")) {
                linkDb = new Path(args[++i]);
                //这里开始解析传入的参数路径
            } else if (args[i].equals("-dir")) {
                Path dir = new Path(args[++i]);
                FileSystem fs = dir.getFileSystem(getConf());
                //这里列出了所有的子目录，
                FileStatus[] fstats = fs.listStatus(dir,
                        HadoopFSUtil.getPassDirectoriesFilter(fs));
                Path[] files = HadoopFSUtil.getPaths(fstats);
                for (Path p : files) {
                    segments.add(p);
                }
            } else if (args[i].equals("-noCommit")) {
                noCommit = true;
            } else if (args[i].equals("-deleteGone")) {
                deleteGone = true;
            } else if (args[i].equals("-filter")) {
                filter = true;
            } else if (args[i].equals("-normalize")) {
                normalize = true;
            } else if (args[i].equals("-params")) {
                params = args[++i];
            } else {
                segments.add(new Path(args[i]));
            }
        }

实际上对调用程序传入的 segment path参数又找了一次子目录，而crawl脚本程序传入的路径已经到20150630152657时间这一层了，导致路径多了一层。
crawl脚本如下：

  echo "Indexing $SEGMENT on SOLR index -> $SOLRURL"
  "$bin/nutch" index -D solr.server.url=$SOLRURL "$CRAWL_PATH"/crawldb -linkdb "$CRAWL_PATH"/linkdb "$CRAWL_PATH"/segments/$SEGMENT  ---$SEGMENT即是生成的时间路径

   //IndexerMapReduce 初始化job类负责加上最后的 crawl_fetch、crawl_parse 、parse_data、parse_text 路径

  public static void initMRJob(Path crawlDb, Path linkDb,
                           Collection<Path> segments,
                           JobConf job) {

    LOG.info("IndexerMapReduce: crawldb: " + crawlDb);

    if (linkDb!=null)
      LOG.info("IndexerMapReduce: linkdb: " + linkDb);

    for (final Path segment : segments) {
      LOG.info("IndexerMapReduces: adding segment: " + segment);
      //
      FileInputFormat.addInputPath(job, new Path(segment, CrawlDatum.FETCH_DIR_NAME));
      FileInputFormat.addInputPath(job, new Path(segment, CrawlDatum.PARSE_DIR_NAME));
      FileInputFormat.addInputPath(job, new Path(segment, ParseData.DIR_NAME));
      FileInputFormat.addInputPath(job, new Path(segment, ParseText.DIR_NAME));
    }

    FileInputFormat.addInputPath(job, new Path(crawlDb, CrawlDb.CURRENT_NAME));

    if (linkDb!=null)
      FileInputFormat.addInputPath(job, new Path(linkDb, LinkDb.CURRENT_NAME));

    job.setInputFormat(SequenceFileInputFormat.class);

    job.setMapperClass(IndexerMapReduce.class);
    job.setReducerClass(IndexerMapReduce.class);

    job.setOutputFormat(IndexerOutputFormat.class);
    job.setOutputKeyClass(Text.class);
    job.setMapOutputValueClass(NutchWritable.class);
    job.setOutputValueClass(NutchWritable.class);
  }