简单介绍一些java爬虫框架

JsoupXpath

官方文档:http://jsoupxpath.wanghaomiao.cn/

githup:https://github.com/zhegexiaohuozi/JsoupXpath

maven:

<dependency>
    <groupId>cn.wanghaomiao</groupId>
    <artifactId>JsoupXpath</artifactId>
    <version>2.3.2</version>
</dependency>

示例:

String url = "https://www.qidian.com/rank/yuepiao?chn=21&page=1";
Document document = Jsoup.connect(url).get();
String xpath = ".//div[@class='book-mid-info']/h4/a/text()";
JXDocument underTest = JXDocument.create(document.toString());
List<Object> res = underTest.sel(xpath);
for (Object re : res) {
  System.err.println(re);
}

slennium

教程:https://www.cnblogs.com/TankXiao/p/5252754.html

maven

<!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>3.141.59</version>
</dependency>
<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-chrome-driver</artifactId>
</dependency>
<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-remote-driver</artifactId>
</dependency>
<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-api</artifactId>
</dependency>
<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-support</artifactId>
</dependency>
<dependency>
    <groupId>net.sourceforge.htmlcleaner</groupId>
    <artifactId>htmlcleaner</artifactId>
    <version>2.9</version>
</dependency>

示例:

// 可省略，若驱动放在其他目录需指定驱动路径
System.setProperty(
    "webdriver.chrome.driver",
    "D:\\Project\\java爬虫\\javapc01\\src\\test\\java\\com\\hw\\javapc\\chromedriver.exe");
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--headless");
ChromeDriver driver = new ChromeDriver(chromeOptions);
driver.get("https://www.qidian.com/rank/yuepiao?chn=21&page=1");
// 休眠1s,为了让js执行完
Thread.sleep(1000L);
// 网页源码
String pageSource = driver.getPageSource();
//使用xpath解析
HtmlCleaner hc = new HtmlCleaner();
TagNode tn = hc.clean(pageSource);
Object[] objects = tn.evaluateXPath("//div[@class='book-mid-info']/h4/a/text()");
for (Object object : objects) {
  System.out.println(object);
}

jsoup

文档:https://www.open-open.com/jsoup/ https://jsoup.org/apidocs/org/jsoup/nodes/Document.html

案例:https://www.cnblogs.com/sam-uncle/p/10922366.html

Document document = Jsoup.connect("https://www.qidian.com/rank/yuepiao?chn=21&page=1").get();
Elements elementsByClass = document.getElementsByClass("book-mid-info");
for (Element byClass : elementsByClass) {

  String text = byClass.select("h4 a[data-eid='qd_C40']").text();
  String user = byClass.select("p a[data-eid='qd_C41']").text();
  String type = byClass.select("p a[data-eid='qd_C42']").text();
  String trim = byClass.select("p[class='intro']").text().trim();
  log.info("小说名:{},作者:{},类型:{},介绍:{}",text,user,type,trim);
}

webmagic (有bug且maven仓库不更新)

类似python的scrapy.作者是参考scrapy写的,但是已经不怎么更新了

文档:http://webmagic.io/docs/zh/

githup:https://github.com/code4craft/webmagic

maven:

<dependency>
    <groupId>us.codecraft</groupId>
    <artifactId>webmagic-core</artifactId>
    <version>0.7.3</version>
</dependency>
<dependency>
    <groupId>us.codecraft</groupId>
    <artifactId>webmagic-extension</artifactId>
    <version>0.7.3</version>
</dependency>

java的生态还是非常不错的,什么类型的框架都有  python能弄的爬虫  java也一样可以的,  说到底语言不过是工具,真正用什么根据合适的情况考虑永不同的东西   简单爬虫就用jsoup和jsoupxpath就行了