nutch导入到Eclipse总结

安装PPT: http://download.csdn.net/detail/hero_quan/4673329 

1.       首先，要下载Cygwin，下载地址：http://www.cygwin.com/setup.exe。安装完成后，一定要配置系统变量，如下所示：

   C:/cygwin/bin

2.

·        File > New > Project > Javaproject > click Next

·        Name the project (Nutch_Trunk forinstance)

·        Select "Create project fromexisting source" and use the location where you downloaded Nutch

·        Click on Next, and wait while Eclipseis scanning the folders

·        Add the folder "conf" tothe classpath (Right-click on the project, select "properties" then"Java Build Path" tab (left menu) and then the "Libraries"tab. Click "Add Class Folder..." button, and select "conf"from the list)

·        Go to "Order and Export"tab, find the entry for added "conf" folder and move it to the top(by checking it and clicking the "Top" button). This is required soEclipse will take config (nutch-default.xml, nutch-final.xml, etc.) resourcesfrom our "conf" folder and not from somewhere else.

·        Eclipse should have guessed all theJava files that must be added to your classpath. If that's not the case, add"src/java", "src/test" and all plugin "src/java"and "src/test" folders to your source folders. Also add all jars in"lib" and in the plugin lib folders to your libraries

·        Click the "Source" tab andset the default output folder to "Nutch_Trunk/bin/tmp_build". (Youmay need to create the tmp_build folder.)

·        Click the "Finish" button

·        DO NOT add "build" toclasspath

·        在工程中找到nutch-site.xml（在conf目录）在<configuration></configuration>中间加入以下代码

<property> 
<name>http.agent.name</name> 
<value>tangmiSpider</value> 
<description>My Search Engine</description> 
</property>

<property> 
<name>http.agent.description</name> 
<value>search</value> 
<description>Further description of our bot- this text is used in 
the User-Agent header. It appears in parenthesis after the agent name. 
</description> 
</property>

<property> 
<name>http.agent.url</name> <value>http://hi.baidu.com/ebyy</value> <description>A URL to advertisein the User-Agent header. This will 
appear in parenthesis after the agent name. Custom dictates that this 
should be a URL of a page explaining the purpose and behavior. of this 
crawler. 
</description> 
</property>

<property> 
<name>http.agent.email</name> 
<value>dl_tangmi@163.com</value> 
<description>An email address to advertise in the HTTP 'From' request 
header and User-Agent header. A good practice is to mangle this 
address (e.g. 'info at example dot com') to avoid spamming. 
</description> 
</property>

在工程中找到nutch-default.xml（在conf目录）这里基本不用做什么改动就可以 在这里<name>http.agent.name</name>的value里面加上你的名字，随便写啦，就是你爬虫的名字，如果你的nutch-site.xml里写了这里就不用写了. 还有这里要千万写对   

 

·        导入成功测试方法：

选择Run->Run As->JavaApplication在弹出的SelectJava Application上选择Crawl-org.apache.nutch.crawl.
接下来，选择 Run->Run Configurations…在左边的Java Application下面会有Crawl这一项，选择它，
切换到Arguments，Program Arguments的内容就是要设置的参数，填上urls -dir crawl -depth 3 -topN50（这里视自己的具体情况而定，urls为链接）
在VM arguments下面填上-Dhadoop.log.dir=logs-Dhadoop.log.file=hadoop.log.

3．一些错误：

Exception in thread "main" java.io.IOException: Job failed!

atorg.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)

atorg.apache.nutch.crawl.Injector.inject(Injector.java:211)

atorg.apache.nutch.crawl.Crawl.main(Crawl.java:124)

有两个原因导致这个错误（目前只发现两个原因）。

一个是官方wiki里提到的 Java Heap Size problem。

解决方法是：

Eclipse -> Window -> Preferences -> Java ->Installed JREs -> edit -> Default VM arguments

I've set mine to

 

-Xms5m -Xmx150m

because I have like 200MB RAM left after running all apps

-Xms (minimum ammount of RAM memory for runningapplications) -Xmx (maximum)

第2个是plugin的路径错误。

解决方法是：

<property>

  <name>plugin.folders</name>

  <value>plugins</value>

  <description>Directories where nutch plugins are located. Each

  element may be a relative or absolutepath. If absolute, it is used

  as is. If relative, it is searched for on the classpath.</description>

</property>

Value的值一定是文件路径中plugins文件夹的路径，写错了就悲剧了。