Nutch-1.x学习笔记



Nutch单步操作
1、<创建种子url>
mkdir -p urls
cd urls
touch seed.txt
echo "http://www.qq.com/">>urls/seed.txt  #每行一个种子url
2、<inject>
bin/nutch inject crawl/crawldb urls
3、<generate>
bin/nutch generate crawl/crawldb crawl/segments
4、<fetch>
s1=`ls -d crawl/segments/2* | tail -1`
echo $s1
bin/nutch fetch $s1
5、<parse>
bin/nutch parse $s1
6、<updatedb>
bin/nutch updatedb crawl/crawldb $s1
7、多次操作3-6步
8、<invertlinks>
bin/nutch invertlinks crawl/linkdb -dir crawl/segments
9、<Indexing into Apache Solr>
bin/nutch solrindex http://localhost:8983/solr crawl/crawldb/ -linkdb crawl/linkdb/ crawl/segments/20131108063838/ -filter -normalize
10、<Deleting Duplicates>
/bin/nutch solrdedup http://localhost:8983/solr
11、<Cleaning Solr>
/bin/nutch solrclean crawl/crawldb/ http://localhost:8983/solr

Nutch脚本操作

     Usage: crawl [-i|--index] [-D "key=value"] <Seed Dir> <Crawl Dir> <Num Rounds>
        -i|--index      Indexes crawl results into a configured indexer
        -D              A Java property to pass to Nutch calls
        Seed Dir        Directory in which to look for a seeds file
        Crawl Dir       Directory where the crawl/link/segments dirs are saved
        Num Rounds      The number of rounds to run this crawl for
     Example: bin/crawl -i -D solr.server.url=http://localhost:8983/solr/ urls/ TestCrawl/  2

refer from : http://wiki.apache.org/nutch/NutchTutorial