nutch-2.1、mysql整合

这次主要介绍下nutch2.1和mysql和elasticsearch的整合，是在单机上运行，并不是分布式部署。

1、下载nutch2.1

    nutch下载地址：http://labs.mop.com/apache-mirror/nutch/2.1/apache-nutch-2.1-src.tar.gz

    

2、配置nutch使用mysql作为数据存储，修改nutch根目录/ivy/ivy.xml文件

    将这行的注释取消<dependency org=”mysql” name=”mysql-connector-java” rev=”5.1.18″ conf=”*->default”/>

    修改nutch根目录/conf/gora.properties文件，把默认存储配置改成如下 

# MySQL properties # ############################### gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver gora.sqlstore.jdbc.url=jdbc:mysql://host:3306/nutch? useUnicode=true&characterEncoding=utf8&autoReconnect=true&zeroDateTimeBehavior=convertToNull gora.sqlstore.jdbc.user=root gora.sqlstore.jdbc.password=root

3、修改conf的nutch-site.xml

<? xml version = "1.0" ?> <? xml-stylesheet type = "text/xsl" href = "configuration.xsl" ?>   < configuration > < property > < name >http.agent.name</ name > < value >My Spider</ value > </ property >   < property > < name >http.accept.language</ name > < value >ja-jp,zh-cn,en-us,en-gb,en;q=0.7,*;q=0.3</ value > </ property >   < property > < name >parser.character.encoding.default</ name > < value >utf-8</ value > < description >The character encoding to fall back to when no other information is available</ description > </ property >   < property > < name >storage.data.store.class</ name > < value >org.apache.gora.sql.store.SqlStore</ value > </ property >   < property > < name >plugin.includes</ name > < value >protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|urlnormalizer-(pass|regex|basic)|scoring-opic</ value > </ property >   </ configuration >

4、使用ant编译源码。

     nutch根目录下输入ant

5、设置待抓取的网站

    cd nutch根目录/runtime/local 

    创建urls目录

    mkdir -p urls 

    创建初始爬取网站列表

    echo 'http://www.iie.ac.cn/' > urls/seed.txt

 

6、创建数据库和表

CREATE DATABASE nutch DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;    CREATE TABLE `webpage` (`id` varchar (767) CHARACTER SET latin1 NOT NULL , `headers` blob, `text` mediumtext DEFAULT NULL , `status` int (11) DEFAULT NULL , `markers` blob, `parseStatus` blob, `modifiedTime` bigint (20) DEFAULT NULL , `score` float DEFAULT NULL , `typ` varchar (32) CHARACTER SET latin1 DEFAULT NULL , `baseUrl` varchar (512) CHARACTER SET latin1 DEFAULT NULL , `content` mediumblob, `title` varchar (2048) DEFAULT NULL , `reprUrl` varchar (512) CHARACTER SET latin1 DEFAULT NULL , `fetchInterval` int (11) DEFAULT NULL , `prevFetchTime` bigint (20) DEFAULT NULL , `inlinks` mediumblob, `prevSignature` blob, `outlinks` mediumblob, `fetchTime` bigint (20) DEFAULT NULL , `retriesSinceFetch` int (11) DEFAULT NULL , `protocolStatus` blob, `signature` blob, `metadata` blob, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;

 千万要注意：typ不能写成type，否则会报下面的错误：

2016-03-31 04:21:06,234 INFO  fetcher.FetcherJob - -activeThreads=0
2016-03-31 04:21:06,281 WARN  mapred.FileOutputCommitter - Output path is null in cleanup
2016-03-31 04:21:06,282 WARN  mapred.LocalJobRunner - job_local_0003
java.io.IOException: java.sql.BatchUpdateException: Unknown column 'typ' in 'field list'
        at org.apache.gora.sql.store.SqlStore.flush(SqlStore.java:340)
        at org.apache.gora.sql.store.SqlStore.close(SqlStore.java:185)
        at org.apache.gora.mapreduce.GoraRecordWriter.close(GoraRecordWriter.java:55)
        at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:579)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
Caused by: java.sql.BatchUpdateException: Unknown column 'typ' in 'field list'
        at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2016)
        at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1452)
        at org.apache.gora.sql.store.SqlStore.flush(SqlStore.java:328)
        ... 6 more

7、执行爬行操作：

./nutch crawl ../urls -depth 3

执行完在mysql中即可以查看到爬虫抓取的内容