nutch搜索引擎架设 之 抓取模块

一、准备工作

jdk：http://www.oracle.com/technetwork/java/javase/downloads/index.html

tomcat：http://tomcat.apache.org/

eclipse：http://www.eclipse.org/downloads/

Cygwin：http://www.cygwin.com/  下载里面的setup.exe （用于模拟linux环境，nutch 是适应于linux平台的,）

nutch0.9：http://www.kuaipan.cn/file/id_16410726440632359.htm（开源的基于java的搜索引擎）

缺省jar下载：rtf-parser.jar：http://www.kuaipan.cn/file/id_16410726440632361.htm

jid3lib-0.5.1.jar：http://www.kuaipan.cn/file/id_16410726440632360.htm

二、安装

将下面下载到的（除了缺省jar）进行安装

jdk 安装：点击下载下来的安装包进行安装，别忘了配环境变量，这里不再详述

tomcat 安装：安装好以后，启动startup.bat  在浏览器端测试  http://localhost:8080/  如出现欢迎界面即安装成功，如果安装不成功  在cmd 里运行startup.bat 查看错误信息，如果一闪而过，可手动修改startup.bat     将里面的这句  call "%EXECUTABLE%" start %CMD_LINE_ARGS%  

 修改为  call "%EXECUTABLE%" run %CMD_LINE_ARGS%

然后再在cmd里运行startup根据提供的具体错误信息来处理。

eclipse：这个没什么好说的。

cygwin：按提示一步一步install即可

nutch 0.9：解压到你的工程根目录里

缺省jar：将rtf-parser.jar 放到nutch目录  nutch-0.9\src\plugin\parse-rtf\lib 里面  将 jid3lib-0.5.1.jar 放到  nutch-0.9\src\plugin\parse-mp3\lib 里面

三、配置与加载nutch

首先打开Cygwin 会出现一个类似cmd的命令行的窗口，其命令格式与dos几乎差不多，不懂得可以输入help看详细的命令说明

然后进入你的nutch-0.9 目录

然后运行nutch 命令：

bin/nutch

如图所示会出现关于nutch命令的说明 即表示nutch运行环境搭建成功。

接着开始导入和配置nutch-0.9：

导入：

首先将nutch-0.9 作为工程项目倒入eclipse中。

打开eclipse，新建java项目，添加nutch-0.9 目录作为项目源

然后不要急着点完成

在Source选项卡 配置默认输出文件路径my_build

在Libraries选项卡将nutch目录的conf文件添加进来

在Order and Export选项卡 将conf文件置顶

刷新项目工程，这下基本将nutch完整地导入到eclipse中了

配置：

1.修改nutch-default.xml文件

在eclipse打开刚才倒入的项目 -〉 Referenced Libraries -〉conf 找到其中的nutch-default.xml文件

找到<name>plugin.folders</name>这个属性  将下面以行的value值修改为<value>./src/plugin</value>

2.配置nutch抓取策略

在nutch-0.9 工程根目录建立weburl.txt 用以存放搜索引擎要抓取内容得网址，里面填入要抓取的网址（例如：http://www.buct.edu.cn/）

然后在conf文件中找到crawl_urlfilter.txt   

找到  ：

# accept hosts in MY.DOMAIN.NAME

# +^http://([a-z0-9]*\.)*MY.DOMAIN.NAME/

修改为：

# accept hosts in MY.DOMAIN.NAME

# +^http://([a-z0-9]*\.)*MY.DOMAIN.NAME/

+^http://www.buct.edu.cn/

即配置仅抓取此网站的内容

最后配置conf里的nutch-site.xml文件

将下面一段内容完全覆盖nutch-site.xml文件中的内容：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->




<configuration>
<property>
<name>http.agent.name</name>
<value>ahathinkingSpider</value>
<description>My Search Engine</description>
</property>

<property>
<name>http.agent.description</name>
<value>My web</value>
<description>Further description of our bot- this text is used in
User-Agent header. It appears in parenthesis after the agent name.</description>
</property>

<property>
<name>http.agent.url</name>
<value>ahathinking.com</value>
<description>A URL to advertise in the User-Agent header. This will
appear in parenthesis after the agent name. Custom dictates that this
should be a URL of a page explaining the purpose and behavior. of this
</description>
</property>

<property>
<name>http.agent.email</name>
<value>yx.coding@gmail.com</value>
<description>An email address to advertise in the HTTP 'From' request
header and User-Agent header. A good practice is to mangle this
address (e.g. 'info at example dot com') to avoid spamming.
</description>
</property>

</configuration>

3.配置运行参数：

选取 nutch 项目，在运行栏里选择 Run Configurations... 修改相关运行参数

修改如下：

注意修改完后点击Apply

注意修改完后点击Apply

最后点击Run运行

crawl started in: Local
rootUrlDir = weburl.txt
threads = 5
depth = 5
topN = 100
Injector: starting
Injector: crawlDb: Local/crawldb
Injector: urlDir: weburl.txt
Injector: Converting injected urls to crawl db entries.
Injector: Merging injected urls into crawl db.
Injector: done
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: Local/segments/20121022110809
Generator: filtering: false
Generator: topN: 100
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls by host, for politeness.
Generator: done.
Fetcher: starting
Fetcher: segment: Local/segments/20121022110809
Fetcher: threads: 5
fetching http://www.buct.edu.cn/
Fetcher: done
CrawlDb update: starting
CrawlDb update: db: Local/crawldb
CrawlDb update: segments: [Local/segments/20121022110809]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: done
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: Local/segments/20121022110815
Generator: filtering: false
Generator: topN: 100
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls by host, for politeness.
Generator: done.
Fetcher: starting
Fetcher: segment: Local/segments/20121022110815
Fetcher: threads: 5
fetching http://www.buct.edu.cn/kxyj/index.htm
fetching http://www.buct.edu.cn/zjbh/ldjt/index.htm
fetching http://www.buct.edu.cn/bhxy/index.htm
fetching http://www.buct.edu.cn/zjbh/xycf/index.htm
fetching http://www.buct.edu.cn/zjbh/lrld/index.htm
fetching http://www.buct.edu.cn/xywh/index.htm
fetching http://www.buct.edu.cn/tjtp/25314.htm
fetching http://www.buct.edu.cn/xysz/index.htm
fetching http://www.buct.edu.cn/zsjy/index.htm
fetching http://www.buct.edu.cn/kslj/23583.htm
fetching http://www.buct.edu.cn/js/jquery.flexslider-min.js
fetching http://www.buct.edu.cn/rcpy/index.htm
............

出现上述信息即表示nutch的抓取模块已架设成功。

备注：

nutch架设 常见错误：

1.1.1   Crawl抓取出现hadoop出错提示

配置完成nutch在cygwin中运行nutch的crawl命令时：

[Fatal Error] hadoop-site.xml:15:7: The content of elements must consist of well

-formed character data or markup.

Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParseExcep

tion: The content of elements must consist of well-formed character data or mark

up.

问题解决：

hadoop-site.xml、hadoop-site.xml：其中一个标签</property>前面多了一个尖括号

1.1.2   运行crawl报错Job failed

Exception in thread "main" java.io.IOException: Job failed!

        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)

        at org.apache.nutch.indexer.DeleteDuplicates.dedup(DeleteDuplicates.java

:439)

        at org.apache.nutch.crawl.Crawl.main(Crawl.java:135)

问题解决：

此多为crawl-urlfilter.txt：MY.DOMAIN.NAME的修改不正确

1.1.3   又一个Job failed

Exception in thread "main" java.io.IOException: Job failed!

        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)

        at org.apache.nutch.indexer.DeleteDuplicates.dedup(DeleteDuplicates.java

:439)

        at org.apache.nutch.crawl.Crawl.main(Crawl.java:135)

问题解决：

多为crawl-urlfilter.txt的MY.DOMAIN.NAME修改不正确

1.1.4   Eclipse中运行nutch：Job failed

Exception in thread "main" java.io.IOException: Job failed!

       at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)

       at org.apache.nutch.crawl.Injector.inject(Injector.java:162)

       at org.apache.nutch.crawl.Crawl.main(Crawl.java:115)

问题解决：

此问题是eclipse的java版本设置问题，解决方法：

如原来使用java1.4，需要改为1.6

project-》properties-》java compiler

右 jdk compliance

compiler compliance level：改为6.0

1.1.5当修改配置文件指定索引库.( nutch-site.xml）

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
         <name>http.agent.name</name>
         <value>HD nutch agent</value>
     </property>
     <property>
       <name>http.agent.version</name>
       <value>1.0</value>
      </property>
   <property> 
<name>urlfilter.order</name> 
<value>org.apache.nutch.urlfilter.regex.RegexURLFilter</value> 
<description>The order by which url filters are applied. 
If empty, all available url filters (as dictated by properties 
plugin-includes and plugin-excludes above) are loaded and applied in system 
defined order. If not empty, only named filters are loaded and applied 
in given order. For example, if this property has value: 
org.apache.nutch.urlfilter.regex.RegexURLFilter org.apache.nutch.urlfilter.prefix.PrefixURLFilter 
then RegexURLFilter is applied first, and PrefixURLFilter second. 
Since all filters are AND'ed, filter ordering does not have impact 
on end result, but it may have performance implication, depending 
on relative expensiveness of filters. 
</description> 
</property>
</configuration>

复制上述配置文件时，如果出现下列错误，是因为复制文件时带有空格或编码格式，重敲一遍即可: java.io.UTFDataFormatException: Invalid byte 1 of 1-byte UTF-8 sequence

总的来说常见错误多见于配置文件配置不当……

参考文章：

http://www.ahathinking.com/archives/140.html

http://hi.baidu.com/orminknckdbkntq/item/020a8bc6e4d275000bd93a2d