首先刚开始的时候,遇到了个问题,java调用代码没法启动phantomjs。然后我重新下载了一波,冲配置环境变量,重新写了下。我是windows7,所以就是java + phantomjs ,环境windows7
至于为什么要用这个,因为,爬虫有时候爬到页面源代码后发现并不能获取到想要的数据,所以需要模拟浏览器,获取浏览器获取的内容里面的数据。
对了,我是在springmvc上面搞得数据处理,所以遇到个问题就是jar包冲突,所以我在maven里面的配置改成了如下:
<!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-api</artifactId>
<version>3.4.0</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-remote-driver</artifactId>
<version>3.4.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.github.detro.ghostdriver/phantomjsdriver -->
<dependency>
<groupId>com.codeborne</groupId>
<artifactId>phantomjsdriver</artifactId>
<version>1.4.3</version>
</dependency>
不过我发现一个问题,就是这个爬虫跑起来,很慢,而且如果不加加上driver.close();很快内存就要满了,电脑会卡到崩溃,爬虫也跑不下去。如果加上driver.close();就会多撑一会儿。
如果想让电脑不卡,内存不被撑满,就每次结束的时候用driver.quit();当然速度会慢很多,但是,不会把电脑搞崩掉。因为,这个相当于把浏览器关掉重启,而第一个前面的相当于关掉页面。
上代码了:
package com.tongdatech.webbug;
import java.util.concurrent.TimeUnit;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.phantomjs.PhantomJSDriver;
import org.openqa.selenium.phantomjs.PhantomJSDriverService;
import org.openqa.selenium.remote.DesiredCapabilities;
public class WebBugClass {
public StockInfo getInfo(String stock_url)
{
StockInfo stockInfo = new StockInfo();
stockInfo.setStock_url(stock_url);
String url = "http://finance.sina.com.cn/realstock/company/" + stock_url + "/nc.shtml";
//sh580017
DesiredCapabilities dcaps = new DesiredCapabilities();
dcaps.setCapability("acceptSslCerts", true);
dcaps.setCapability("takesScreenshot", false);
dcaps.setCapability("cssSelectorsEnabled", true);
dcaps.setJavascriptEnabled(true);
dcaps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY, "E:\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe");
PhantomJSDriver driver = new PhantomJSDriver(dcaps);
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
driver.get(url);
WebElement body = driver.findElement(By.tagName("body"));
String body_text = body.getText();
System.out.println(body_text);
try
{
WebElement price = driver.findElement(By.id("price"));
}catch(Exception e)
{
driver.quit();
stockInfo.setStock_price("已退市");
return stockInfo;
}
if("页面没有找到 5秒钟之后将会带您进入新浪首页!".equals(body_text))
{
System.out.println(body_text);
driver.quit();
stockInfo.setStock_price("已退市");
return stockInfo;
}
else
{
WebElement price1 = driver.findElement(By.id("price"));
// WebElement tital_share = driver.findElement(By.xpath("//*[@id='hqDetails']/tbody/tr[3]/td[2]"));
// WebElement pe = driver.findElement(By.xpath("//*[@id='hqDetails']/tbody/tr[4]/td[3]"));
Pattern p = Pattern.compile("(?<=市盈率TTM: ).*?(?=\\n)");
Pattern p1 = Pattern.compile("(?<=总市值: ).*?(?=亿)");
Matcher matcher = p.matcher(body_text);
while(matcher.find())
{
stockInfo.setPe(matcher.group());
System.out.println(matcher.group());
}
Matcher matcher1 = p1.matcher(body_text);
while(matcher1.find())
{
stockInfo.setTotalShare(matcher1.group());
System.out.println(matcher1.group());
}
String price_text = price1.getText();
stockInfo.setStock_price(price_text);
System.out.println(price_text);
// System.out.println(tital_share.getText());
// System.out.println(pe.getText());
// System.out.println(body.getText());
// System.setProperty("phantomjs.binary.path", "E:/phantomjs-2.1.1-windows/bin/phantomjs.exe");
// DesiredCapabilities desiredCapabilities = DesiredCapabilities.phantomjs();
//
// WebDriver driver = new PhantomJSDriver(desiredCapabilities);
// driver.get("http://www.baidu.com");
// System.out.println(driver.getCurrentUrl());
driver.quit();
return stockInfo;
}
}
}