CentOS 6.4环境下的Apache Nutch 1.7 + Solr 4.4.0安装笔记

本人原创,转载请注明出处:http://blog.csdn.net/panjunbiao/article/details/12171147

Nutch安装

参考文档:http://wiki.apache.org/nutch/NutchTutorial

安装必要程序:
yum update
yum list java* 
yum install java-1.7.0-openjdk-devel.x86_64 

找到java的安装路径:
参考:http://serverfaullt.com/questions/50883/what-is-the-value-of-java-home-for-centos
设置JAVA_HOME:
参考:http://www.cnblogs.com/zhoulf/archive/2013/02/04/2891608.html

vi + /etc/profile
JAVA_HOME=/usr/lib/jvm/java
JRE_HOME=/usr/lib/jvm/java/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export JAVA_HOME JRE_HOME PATH CLASSPATH
使profile文件立即生效:
source /etc/profile

下载二进制包文件:
curl -O http://apache.fayea.com/apache-mirror/nutch/1.7/apache-nutch-1.7-bin.tar.gz

解包:
tar -xvzf apache-nutch-1.7-bin.tar.gz
 
检验运行文件
cd apache-nutch-1.7
bin/nutch
此时会出现用法帮助,表示安装成功了。

修改文件conf/nutch-site.xml,设置HTTP请求中agent的名字: 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property>
    <name>http.agent.name</name>
    <value>Friendly Crawler</value>
  </property>
</configuration>

创建种子文件夹
mkdir -p urls
 
执行第一次爬虫任务:
bin/nutch crawl urls -dir crawl
solrUrl is not set, indexing will be skipped...
crawl started in: crawl
rootUrlDir = urls
threads = 10
depth = 5
solrUrl=null
Injector: starting at 2013-09-29 12:01:30
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: total number of urls rejected by filters: 0
Injector: total number of urls injected after normalization and filtering: 0
Injector: Merging injected urls into crawl db.
Injector: finished at 2013-09-29 12:01:33, elapsed: 00:00:03
Generator: starting at 2013-09-29 12:01:33
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: jobtracker is 'local', generating exactly one partition.
Generator: 0 records selected for fetching, exiting ...
Stopping at depth=0 - no more URLs to fetch.
No URLs to fetch - check your seed list and URL filters.
crawl finished: crawl
由于没有设置任何种子URL,所以爬虫什么都不做就退出了。

将种子URL写到文件urls/seed.txt中:
http://www.36kr.com/
vi conf/regex-urlfilter.txt
# accept anything else
# +.

# added by panjunbiao
+36kr.com

再次执行爬虫程序,发现有些种子网站被skip了:
bin/nutch crawl urls -dir crawl
solrUrl is not set, indexing will be skipped...
crawl started in: crawl
rootUrlDir = urls
threads = 10
depth = 5
solrUrl=null
Injector: starting at 2013-09-29 12:10:24
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: total number of urls rejected by filters: 0
Injector: total number of urls injected after normalization and filtering: 1
Injector: Merging injected urls into crawl db.
Injector: finished at 2013-09-29 12:10:27, elapsed: 00:00:03
Generator: starting at 2013-09-29 12:10:27
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: crawl/segments/20130929121029
Generator: finished at 2013-09-29 12:10:30, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2013-09-29 12:10:30
Fetcher: segment: crawl/segments/20130929121029
Using queue mode : byHost
Fetcher: threads: 10
Fetcher: time-out divisor: 2
QueueFeeder finished: total 1 records + hit by time limit :0
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
fetching http://www.36kr.com/ (queue crawl delay=5000ms)
-finishing thread FetcherThread, activeThreads=8
-finishing thread FetcherThread, activeThreads=7
-finishing thread FetcherThread, activeThreads=6
-finishing thread FetcherThread, activeThreads=5
-finishing thread FetcherThread, activeThreads=4
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2013-09-29 12:10:32, elapsed: 00:00:02
ParseSegment: starting at 2013-09-29 12:10:32
ParseSegment: segment: crawl/segments/20130929121029
http://www.36kr.com/ skipped. Content of size 67099 was truncated to 59363
ParseSegment: finished at 2013-09-29 12:10:33, elapsed: 00:00:01
CrawlDb update: starting at 2013-09-29 12:10:33
CrawlDb update: db: crawl/crawldb
CrawlDb update: segments: [crawl/segments/20130929121029]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: 404 purging: false
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2013-09-29 12:10:34, elapsed: 00:00:01
Generator: starting at 2013-09-29 12:10:34
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: jobtracker is 'local', generating exactly one partition.
Generator: 0 records selected for fetching, exiting ...
Stopping at depth=1 - no more URLs to fetch.
LinkDb: starting at 2013-09-29 12:10:35
LinkDb: linkdb: crawl/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: internal links will be ignored.
LinkDb: adding segment: file:/root/apache-nutch-1.7/crawl/segments/20130929121029
LinkDb: finished at 2013-09-29 12:10:36, elapsed: 00:00:01
crawl finished: crawl
为什么呢?用tcpdump或者wireshark抓包发现,该网站的页面内容采用truncate的方式分段返回,而nutch的默认设置是不处理这种方式的,需要打开之,修改conf/nutch-site.xml,在里面增加一个 parser.skip.truncated 属性:
<property> 
  <name>parser.skip.truncated</name> 
  <value>false</value> 
</property>
参考:http://lucene.472066.n3.nabble.com/Content-Truncation-in-Nutch-2-1-MySQL-td4038888.html

修改后再次执行爬虫任务,已经能够正常抓取了:
bin/nutch crawl urls -dir crawl

Solr安装

下载安装文件
curl -O http://mirrors.cnnic.cn/apache/lucene/solr/4.4.0/solr-4.4.0.tgz

tar -xvzf solr-4.4.0.tgz

cd solr-4.4.0/example
java -jar start.jar

验证Solr安装(假设安装在本机)
http://localhost:8983/solr/

集成Nutch与Solr

vi + /etc/profile
NUTCH_RUNTIME_HOME=/root/apache-nutch-1.7APACHE_SOLR_HOME=/root/solr-4.4.0export JAVA_HOME JRE_HOME PATH CLASSPATH NUTCH_RUNTIME_HOME APACHE_SOLR_HOME
source /etc/profile

mkdir ${APACHE_SOLR_HOME}/example/solr/conf
cp ${NUTCH_RUNTIME_HOME}/conf/schema.xml ${APACHE_SOLR_HOME}/example/solr/conf/

重新启动solr的start程序
java -jar start.jar

建立索引:
bin/nutch crawl urls  -dir crawl -depth 2 -topN 5 -solr http://localhost:8983/solr/
索引出错:
Active IndexWriters :
SOLRIndexWriter
     solr.server.url : URL of the SOLR instance (mandatory)
     solr.commit.size : buffer size when sending to SOLR (default 1000)
     solr.mapping.file : name of the mapping file for fields (default solrindex-mapping.xml)
     solr.auth : use authentication (default false)
     solr.auth.username : use authentication (default false)
     solr.auth : username for authentication
     solr.auth.password : password for authentication


Exception in thread "main" java.io.IOException: Job failed!
     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
     at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:123)
     at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:81)
     at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:65)
     at org.apache.nutch.crawl.Crawl.run(Crawl.java:155)
     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
     at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
检查Solr的日志:
2859895 [qtp1478922764-16] INFO  org.apache.solr.update.processor.LogUpdateProcessor  ? [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {} 0 1
2859902 [qtp1478922764-16] ERROR org.apache.solr.core.SolrCore  ? org.apache.solr.common.SolrException: ERROR: [doc=http://www.36kr.com/] unknown field 'host'
     at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:174)
     at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:73)
     at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:210)
     at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
     at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
     at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:556)
     at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:692)
     at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:435)
     at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
     at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
     at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
     at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
     at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
     at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
     at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
     at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
     at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
     at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
     at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
     at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
     at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
     at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
     at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
     at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
     at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
     at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
     at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
     at org.eclipse.jetty.server.Server.handle(Server.java:368)
     at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
     at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
     at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
     at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
     at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
     at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
     at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
     at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
     at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
     at java.lang.Thread.run(Thread.java:724)

2859920 [qtp1478922764-16] INFO  org.apache.solr.update.processor.LogUpdateProcessor  ? [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {} 0 5
2859921 [qtp1478922764-16] ERROR org.apache.solr.core.SolrCore  ? org.apache.solr.common.SolrException: ERROR: [doc=http://www.36kr.com/] unknown field 'host'
     at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:174)
     at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:73)
     at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:210)
     at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
     at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
     at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:556)
     at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:692)
     at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:435)
     at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
     at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
     at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
     at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
     at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
     at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
     at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
     at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
     at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
     at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
     at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
     at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
     at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
     at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
     at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
     at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
     at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
     at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
     at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
     at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
     at org.eclipse.jetty.server.Server.handle(Server.java:368)
     at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
     at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
     at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
     at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
     at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
     at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
     at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
     at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
     at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
     at java.lang.Thread.run(Thread.java:724)
这个问题可以参考 http://stackoverflow.com/questions/13429481/error-while-indexing-in-solr-data-crawled-by-nutch
类似的还有其他一些字段需要补充,方法是编辑 ~/solr-4.4.0/example/solr/collection1/conf/schema.xml,在<field>…</fields>中增加以下的字段:
<fields>   <field name="host" type="string" stored="false" indexed="true"/>
   <field name="digest" type="string" stored="true" indexed="false"/>
   <field name="segment" type="string" stored="true" indexed="false"/>
   <field name="boost" type="float" stored="true" indexed="false"/>

   <field name="tstamp" type="date" stored="true" indexed="false"/>

   <field name="anchor" type="string" stored="true" indexed="true"
            multiValued="true"/>
   <field name="cache" type="string" stored="true" indexed="false"/>

</fields> 


安装验证

重新开始抓取:
[root@localhost apache-nutch-1.7]# rm crawl/ -Rf
[root@localhost apache-nutch-1.7]# bin/nutch crawl urls  -dir crawl -depth 2 -topN 5 -solrhttp://localhost:8983/solr/
crawl started in: crawl
rootUrlDir = urls
threads = 10
depth = 2
solrUrl=http://localhost:8983/solr/
topN = 5
Injector: starting at 2013-09-29 15:27:35
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: total number of urls rejected by filters: 0
Injector: total number of urls injected after normalization and filtering: 1
Injector: Merging injected urls into crawl db.
Injector: finished at 2013-09-29 15:27:38, elapsed: 00:00:02
Generator: starting at 2013-09-29 15:27:38
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 5
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: crawl/segments/20130929152740
Generator: finished at 2013-09-29 15:27:41, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2013-09-29 15:27:41
Fetcher: segment: crawl/segments/20130929152740
Using queue mode : byHost
Fetcher: threads: 10
Fetcher: time-out divisor: 2
QueueFeeder finished: total 1 records + hit by time limit :0
Using queue mode : byHost
Using queue mode : byHost
fetching http://www.36kr.com/ (queue crawl delay=5000ms)
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
-finishing thread FetcherThread, activeThreads=8
-finishing thread FetcherThread, activeThreads=7
-finishing thread FetcherThread, activeThreads=6
-finishing thread FetcherThread, activeThreads=5
-finishing thread FetcherThread, activeThreads=4
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2013-09-29 15:27:43, elapsed: 00:00:02
ParseSegment: starting at 2013-09-29 15:27:43
ParseSegment: segment: crawl/segments/20130929152740
Parsed (25ms):http://www.36kr.com/
ParseSegment: finished at 2013-09-29 15:27:45, elapsed: 00:00:02
CrawlDb update: starting at 2013-09-29 15:27:45
CrawlDb update: db: crawl/crawldb
CrawlDb update: segments: [crawl/segments/20130929152740]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: 404 purging: false
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2013-09-29 15:27:47, elapsed: 00:00:01
Generator: starting at 2013-09-29 15:27:47
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 5
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: crawl/segments/20130929152749
Generator: finished at 2013-09-29 15:27:50, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2013-09-29 15:27:50
Fetcher: segment: crawl/segments/20130929152749
Using queue mode : byHost
Fetcher: threads: 10
Fetcher: time-out divisor: 2
QueueFeeder finished: total 5 records + hit by time limit :0
Using queue mode : byHost
Using queue mode : byHost
fetching http://www.36kr.com/category/breaking (queue crawl delay=5000ms)
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4
* queue: http://www.36kr.com
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1380439675396
  now           = 1380439671274
  0. http://www.36kr.com/p/206589.html
  1. http://www.36kr.com/category/cn-news
  2. http://www.36kr.com/category/column
  3. http://www.36kr.com/guolee89
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4
* queue: http://www.36kr.com
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1380439675396
  now           = 1380439672275
  0. http://www.36kr.com/p/206589.html
  1. http://www.36kr.com/category/cn-news
  2. http://www.36kr.com/category/column
  3. http://www.36kr.com/guolee89
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4
* queue: http://www.36kr.com
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1380439675396
  now           = 1380439673277
  0. http://www.36kr.com/p/206589.html
  1. http://www.36kr.com/category/cn-news
  2. http://www.36kr.com/category/column
  3. http://www.36kr.com/guolee89
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4。。。-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1
* queue: http://www.36kr.com
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1380439690613
  now           = 1380439690291
  0. http://www.36kr.com/guolee89
fetching http://www.36kr.com/guolee89 (queue crawl delay=5000ms)
-finishing thread FetcherThread, activeThreads=9
-finishing thread FetcherThread, activeThreads=8
-finishing thread FetcherThread, activeThreads=7
-finishing thread FetcherThread, activeThreads=6
-finishing thread FetcherThread, activeThreads=5
-finishing thread FetcherThread, activeThreads=4
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2013-09-29 15:28:12, elapsed: 00:00:22
ParseSegment: starting at 2013-09-29 15:28:12
ParseSegment: segment: crawl/segments/20130929152749
Parsed (8ms):http://www.36kr.com/category/breaking
Parsed (6ms):http://www.36kr.com/category/cn-news
Parsed (6ms):http://www.36kr.com/category/column
Parsed (3ms):http://www.36kr.com/guolee89
Parsed (6ms):http://www.36kr.com/p/206589.html
ParseSegment: finished at 2013-09-29 15:28:14, elapsed: 00:00:02
CrawlDb update: starting at 2013-09-29 15:28:14
CrawlDb update: db: crawl/crawldb
CrawlDb update: segments: [crawl/segments/20130929152749]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: 404 purging: false
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2013-09-29 15:28:15, elapsed: 00:00:01
LinkDb: starting at 2013-09-29 15:28:15
LinkDb: linkdb: crawl/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: internal links will be ignored.
LinkDb: adding segment: file:/root/apache-nutch-1.7/crawl/segments/20130929152740
LinkDb: adding segment: file:/root/apache-nutch-1.7/crawl/segments/20130929152749
LinkDb: finished at 2013-09-29 15:28:16, elapsed: 00:00:01
Indexer: starting at 2013-09-29 15:28:16
Indexer: deleting gone documents: false
Indexer: URL filtering: false
Indexer: URL normalizing: false
Active IndexWriters :
SOLRIndexWriter
     solr.server.url : URL of the SOLR instance (mandatory)
     solr.commit.size : buffer size when sending to SOLR (default 1000)
     solr.mapping.file : name of the mapping file for fields (default solrindex-mapping.xml)
     solr.auth : use authentication (default false)
     solr.auth.username : use authentication (default false)
     solr.auth : username for authentication
     solr.auth.password : password for authentication


Indexer: finished at 2013-09-29 15:28:19, elapsed: 00:00:03
SolrDeleteDuplicates: starting at 2013-09-29 15:28:19
SolrDeleteDuplicates: Solr url: http://localhost:8983/solr/
SolrDeleteDuplicates: finished at 2013-09-29 15:28:20, elapsed: 00:00:01
crawl finished: crawl

检索抓取到的内容,用浏览器打开  http://localhost:8983/solr/#/collection1/query  


已标记关键词 清除标记
©️2020 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页