需求:本地一批关键词,需要查询百度指数,5118一次只能查100条,5W不得手软,看接口0.013元1条,网上找了百度指数接口,都不怎么便宜,那就自己动手了,用webmagic也是很方便快捷的。
1、配置
webmagic两个包、打包成jar可以挂在服务器慢慢运行。
<dependencies>
<!-- webmagic核心包 -->
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-core</artifactId>
<version>0.7.4</version>
</dependency>
<!-- webmagic扩展包 -->
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-extension</artifactId>
<version>0.7.4</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.5</version>
<configuration>
<archive>
<manifest>
<mainClass>com.booy.task.OrderProcessor</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
log4j日志文件:
log4j.rootLogger=INFO,A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%-d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c]-[%p] %m%n
2、主类
public class BaiduZhishu implements PageProcessor {
@Override
public void process(Page page) {
Document doc = page.getHtml().getDocument();
String docStr = doc.toString();
Pattern statusP = Pattern.compile("status\\\":\\d*?(?=,)");
Matcher statusM = statusP.matcher(docStr);
statusM.find();
String statusGroup = statusM.group();
int i = Integer.parseInt(statusGroup.substring(8, statusGroup.length()));
String allZhishu = null