java crawler4j github_GitHub - zhuoran/crawler4j: Open Source Simple Web Crawler for Java. Simple F...

#Crawler4j是使用JAVA开发的开源Web爬虫

###Crawler4j通过配置文件配置抓取任务,然后使用多线程进行抓取的Web爬虫.每个抓取任务使用独立线程上下文,支持在配置文件中同时配置多个抓取任务,复杂的抓取任务可通过扩展框架提供的基类实现,可以方便的将爬虫和其他解析存储程序进行集成.

####使用方法请参考 crawler4j-simple 模块

##It's composed of two parts:

crawler4j-core: crawler4j core module.

crawler4j-simple: a simple WEB crawler implementation base on crawler4j-core.

================================================================

Quick Start

0.Install the git and maven command line:

yum install git

or: apt-get install git

cd ~

wget http://www.apache.org/dist/maven/maven-3/3.0.4/binaries/apache-maven-3.0.4-bin.tar.gz

tar zxvf apache-maven-3.0.4-bin.tar.gz

vi .bash_profile

- edit: export PATH=$PATH:~/apache-maven-3.0.4/bin

source .bash_profile

1.Checkout the crawler4j source code:

cd ~

git clone https://github.com/zhuoran/crawler4j.git

cd crawler4j

git checkout -b crawler4j-0.1

git checkout master

2.Import the source code to eclipse project:

cd crawler4j/crawler4j-simple

mvn eclipse:eclipse

Eclipse -> Menu -> File -> Import -> Exsiting Projects to Workspace -> Browse -> Finish

Edit Config:

crawler4j-simple/src/main/resources/crawler4j.xml

3.Build the binary package:

cd ~/crawler4j

mvn clean install -Dmaven.test.skip

ll

4.Install the crawler4j simple demo:

cd ~/crawler4j/crawler4j-simple/target

tar zxvf crawler4j-simple-0.1-assembly.tar.gz

cd crawler4j-simple-0.1/bin

./start.sh

================================================================

Configuration

Crawler Config Example :

See crawler4j-simple/src/main/resources/crawler4j.xml

You can configure multiple crawling tasks in crawler4j.xml

UTF-8

Infoq-News

1000

div.box-content-3

me.zhuoran.crawler4j.simple.ParserDemo

me.zhuoran.crawler4j.simple.InfoqCrawler

[charset] For target html page charset,like UTF-8 GBK .

[name] Task name or thread name.

[delay] Each fetch of a html page delay some time, unit is ms.

[url] URL can be list page url or target page url.

URL can contain one or more list page url must use "," split.

URL can contain one or more target page url must use "|" split.

[extract_links_elementId] This tag can be a regex string or a jsoup query.

[parser] This is class full name of parser.

[crawler] This is class full name of cralwer.

================================================================

Feature

支持更多类型数据自动识别

完善HttpClient部分代码,支持更多抓取模式,支持gzip等格式解析

重构并优化架构

For more, please refer to:

[Wiki](http://www.zhuoran.me/crawler4j/wiki) 准备中

================================================================

Support

Twitter: @lopor

新浪微博: @王小然