Nutch2.3.1在Linux上部署

1、下载Nutch2.3.1，使用Ant构造Eclipse工程

2、在Idea中导入Nutch

3、配置nutch-default.xml文件

<property>
  <name>plugin.folders</name>
  <value>/usr/local/nutch/nutch-2.3.1/src/plugin</value>
  <description>Directories where nutch plugins are located.  Each
  element may be a relative or absolute path.  If absolute, it is used
  as is.  If relative, it is searched for on the classpath.</description>
</property>

4、配置nutch-site.xml文件，在configuration标签中加入，红色部分需要修改

<property>

<name>http.agent.name</name>

<value>My Nutch Spider</value>

</property>

<property>

<name>storage.data.store.class</name>

<value>org.apache.gora.mongodb.store.MongoStore</value>

<description>Default class for storing data</description>

</property>

<property>

<name>generate.batch.id</name>

<value>*</value>

</property>

<property>

<name>io.serializations</name>

<value>org.apache.hadoop.io.serializer.WritableSerialization</value>

<description>A list of serialization classes that can be used for

obtaining serializers and deserializers.</description>

</property>

<property>

<name>plugin.includes</name>

<value>protocol-httpclient|urlfilter-regex|index-(basic|more)|query-(basic|site|url|lang)|indexer-solr|nutch-extensionpoints|protocol-httpclient|urlfilter-regex|parse-(text|html|msexcel|msword|mspowerpoint|pdf)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)protocol-http|urlfilter-regex|parse-(html|tika|metatags)|index-(basic|anchor|more|metadata)</value>

</property>

<property>

<name>plugin.folders</name>

<value> /usr/local/nutch/nutch-2.3.1/src/plugin</value>

<description>Directories where nutch plugins are located. Each

element may be a relative or absolute path. If absolute, it is used

as is. If relative, it is searched for on the classpath.</description>

</property>

<!-- utf-8 -->

<property>

<name>http.accept.language</name>

<value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value>

<description>Value of the “Accept-Language” request header field.

This

allows selecting non-English language as default one to retrieve.

It

is a useful setting for search engines build for certain national

group.

</description>

</property>

<property>

<name>parser.character.encoding.default</name>