apache-nutch-1.10 安装笔记_apacheorg nutch-CSDN博客

本文链接：https://blog.csdn.net/liyuetao680/article/details/46877497

1、下载：

下载地址：http://www.apache.org/dyn/closer.cgi/nutch/
下载后解压
tar -zvxf apache-nutch-1.10-bin.tar.gz

2、配置：

2.1 java_home

使用的centos6.5，一般安装系统时，系统自带有openjdk环境，但须手动配置java_home环境变量
具体配置见下：

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

2.2 nutch-site-xml

对nutch-site-xml文件进行修改，具体修改见下

        <property>
                <name>http.agent.name</name>
                <value>My First Nutch</value>
                <description>HTTP 'User-Agent' request header. MUST NOT be empty
                please set this to a single word uniquely related to your organiz-
                ation.

                NOTE: You should also check other related properties:

                http.robots.agents
                http.agent.description
                http.agent.url
                http.agent.email
                http.agent.version

                and set their values appropriately.

                </description>
        </property>

2.3 reg-urlfilter.txt

使用命令：vi reg-urlfilter.txt
下面的两行经行修改

# accept anything else
+.

修改后

# accept anything else
#+.

在最后添加如下的内容
+^http://([a-z0-9]*.)*nutch.apache.org/

使用正则表达式是为了在爬虫是只爬nutch.apache.org域名，如果中间还会其他链接，则丢弃。

3、测试是否安装成功

bin/nutch
bin/crawl

4、爬取第一个网页

4.1 执行命令

bin/crawl urls data 5

要爬取网页的地址已经放在urls文件夹里url.txt里面了，这里url.txt的地址是爬取网页的url

命令执行完成后，会在data下产生三个文件夹

crawldb
linkdb
segments

crawldb ：是所有需要爬取的超链接
linkdb ：存放所有的超链接及每个连接的接入地址
segments ：存放的是抓取的界面，以爬取的时间命名，个数不多于爬取的深度，nutch爬取的策略是广优先，每一层URL生成一个文件夹，知道没有新的URL。

在segments下有6个文件

crawl_generate ：带爬取的url
crawl_fetch ：爬取的url状态
content ：页面内容
parse_text ：网页文本信息
parse_data ： url解析出的外链和与元数据
crawl_parse ：更新crawldb 的外链

因为爬下来的三个文件是无法显示的，要想查看爬取的网页需要安装一些工具，才能显示出来。但也可以借助一下指令进行查看。

4.2 命令：bin/nutch readdb youku/crawldb/ -stats

[root@hdmaster apache-nutch-1.10]# bin/nutch readdb youku/crawldb/ -stats
CrawlDb statistics start: youku/crawldb/
Statistics for CrawlDb: youku/crawldb/
TOTAL urls: 3453
retry 0:    3453
min score:  0.0
avg score:  7.390675E-4
max score:  1.434
status 1 (db_unfetched):    3385
status 2 (db_fetched):  67
status 7 (db_duplicate):    1
CrawlDb statistics: done

上述的结果查看URL总数，状态和评分

4.3 命令： bin/nutch readseg -list -dir youku/segments/

在命令行输入下指令
[root@hdmaster apache-nutch-1.10]# bin/nutch readseg -list -dir youku/segments/

为了便于显示，做成表格形式，结果如下

NAME	GENERATED	FETCHER START	FETCHER END	FETCHED	PARSED
20150714132454 7	2015-07-14T13:25:03	2015-07-14T13:26:02	7	7
20150714132616 15	2015-07-14T13:26:25	2015-07-14T13:27:52	15	15
20150714133316	3385	?	?	?	?
20150714132806	45	2015-07-14T13:28:16	2015-07-14T13:33:00	45	45
20150714132437	1	2015-07-14T13:24:41	2015-07-14T13:24:41	1	1

其中有NAME, GENERATED, FETCHER START, FETCHER END, FETCHED,PARSED这几个参数

NAME ： segements名称
GENERATED ：产生的页面数
FETCHER START：抓取开始时间
FETCHER END ：抓取结束时间
FETCHED ：抓取数
PARSED ：解析数

[root@hdmaster apache-nutch-1.10]# bin/nutch inject youku/crawldb urls
Injector: starting at 2015-07-14 16:54:51
Injector: crawlDb: youku/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: Total number of urls rejected by filters: 0
Injector: Total number of urls after normalization: 1
Injector: Merging injected urls into crawl db.
Injector: overwrite: false
Injector: update: false
Injector: URLs merged: 1
Injector: Total new urls injected: 0
Injector: finished at 2015-07-14 16:54:53, elapsed: 00:00:02

命令

bin/ntch generate youku/crawldb youku/segments

[root@hdmaster apache-nutch-1.10]# bin/ntch generate youku/crawldb youku/segments
bash: bin/ntch: No such file or directory
[root@hdmaster apache-nutch-1.10]# bin/nutch generate youku/crawldb youku/segments
Generator: starting at 2015-07-14 17:01:09
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: youku/segments/20150714170112
Generator: finished at 2015-07-14 17:01:13, elapsed: 00:00:04

命令：

bin/ntch generate youku/crawldb youku/segments

[root@hdmaster apache-nutch-1.10]# bin/ntch generate youku/crawldb youku/segments
bash: bin/ntch: No such file or directory
[root@hdmaster apache-nutch-1.10]# bin/nutch generate youku/crawldb youku/segments
Generator: starting at 2015-07-14 17:01:09
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: youku/segments/20150714170112
Generator: finished at 2015-07-14 17:01:13, elapsed: 00:00:04

未完待续。。。

-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=5, fetchQueues.getQueueCount=1
-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=5, fetchQueues.getQueueCount=1
-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=5, fetchQueues.getQueueCount=1
-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=5, fetchQueues.getQueueCount=1
-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=5, fetchQueues.getQueueCount=1
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5, fetchQueues.getQueueCount=1
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5, fetchQueues.getQueueCount=1
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5, fetchQueues.getQueueCount=1
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5, fetchQueues.getQueueCount=1
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=5, fetchQueues.getQueueCount=1
fetching http://nutch.apache.org/apidocs/apidocs-1.6/org/apache/nutch/indexer/anchor/package-frame.html (queue crawl delay=5000ms)
-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=4, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 1
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888986175
  now           = 1436888986414
  0. http://nutch.apache.org/apidocs/apidocs-1.1/org/apache/nutch/parse/html/package-summary.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html
  2. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  3. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888991781
  now           = 1436888987414
  0. http://nutch.apache.org/apidocs/apidocs-1.1/org/apache/nutch/parse/html/package-summary.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html
  2. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  3. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888991781
  now           = 1436888988415
  0. http://nutch.apache.org/apidocs/apidocs-1.1/org/apache/nutch/parse/html/package-summary.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html
  2. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  3. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888991781
  now           = 1436888989415
  0. http://nutch.apache.org/apidocs/apidocs-1.1/org/apache/nutch/parse/html/package-summary.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html
  2. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  3. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888991781
  now           = 1436888990416
  0. http://nutch.apache.org/apidocs/apidocs-1.1/org/apache/nutch/parse/html/package-summary.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html
  2. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  3. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=4, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888991781
  now           = 1436888991416
  0. http://nutch.apache.org/apidocs/apidocs-1.1/org/apache/nutch/parse/html/package-summary.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html
  2. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  3. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
fetching http://nutch.apache.org/apidocs/apidocs-1.1/org/apache/nutch/parse/html/package-summary.html (queue crawl delay=5000ms)
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888997240
  now           = 1436888992416
  0. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html
  1. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  2. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888997240
  now           = 1436888993418
  0. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html
  1. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  2. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888997240
  now           = 1436888994418
  0. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html
  1. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  2. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888997240
  now           = 1436888995419
  0. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html
  1. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  2. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=3, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888997240
  now           = 1436888996419
  0. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html
  1. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  2. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
fetching http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/protocol/Content.html (queue crawl delay=5000ms)
-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=2, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 1
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888997240
  now           = 1436888997419
  0. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=2, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 1
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888997240
  now           = 1436888998421
  0. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=2, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 1
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888997240
  now           = 1436888999422
  0. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=2, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 1
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888997240
  now           = 1436889000422
  0. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=2, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 1
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436888997240
  now           = 1436889001422
  0. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436889007172
  now           = 1436889002423
  0. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436889007172
  now           = 1436889003425
  0. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436889007172
  now           = 1436889004425
  0. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436889007172
  now           = 1436889005425
  0. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436889007172
  now           = 1436889006426
  0. http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html
  1. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
fetching http://nutch.apache.org/apidocs/apidocs-2.3/org/apache/nutch/host/package-frame.html (queue crawl delay=5000ms)
-activeThreads=10, spinWaiting=9, fetchQueues.totalSize=1, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 1
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436889007172
  now           = 1436889007426
  0. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436889012838
  now           = 1436889008428
  0. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436889012838
  now           = 1436889009428
  0. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436889012838
  now           = 1436889010428
  0. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436889012838
  now           = 1436889011429
  0. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1, fetchQueues.getQueueCount=1
* queue: http://nutch.apache.org
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 5000
  minCrawlDelay = 0
  nextFetchTime = 1436889012838
  now           = 1436889012429
  0. http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html
fetching http://nutch.apache.org/apidocs/apidocs-2.0/org/apache/nutch/util/DeflateUtils.html (queue crawl delay=5000ms)
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=9
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=8
Thread FetcherThread has no more work available
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=7
-finishing thread FetcherThread, activeThreads=6
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=5
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=4
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=3
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=2
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=1
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=1
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=0
-activeThreads=0
Fetcher: finished at 2015-07-14 23:50:15, elapsed: 06:41:09