运行出现报错
java.lang.NullPointerException: null
理解为空指针异常;
问题出现在持久层
@Component
public class MyPipeline implements Pipeline {
@Autowired
HomerServer homerServer;
//保存数据 打印
@Override
public void process(ResultItems resultItems, Task task) {
List<HomeBean> list= resultItems.get("home");
this. homerServer.homInsert(list);
}
}
我把HomerServer以及 webmagic的三大组件都交给了springboot管理;
但是在Spider里却是创建对象的方式,完全与springboot背道而驰,就会出现空指针异常
//初始化pac
@Component
public class MySpider {
// private String url;
private Pipeline pipeline=new MyPipeline() ; //不能直接创建 应该使用注入
private Downloader downloader=new MyDownloader();
private PageProcessor processor=new MyPageProcessor() ;
public void start(String url){
//布隆防止重复
QueueScheduler scheduler=new QueueScheduler();
scheduler.setDuplicateRemover(new BloomFilterDuplicateRemover(1000000));
Spider.create(processor) //设置processor 组件
.setDownloader(downloader) //设置downloader 组件
.addPipeline(pipeline) //设置pipeline 组件
.setScheduler(scheduler) //设置url过滤
.addUrl(url) //设置起始url
.thread(1) //无头浏览器不支持多线程
.start(); //启动爬虫
}
}
应该全部使用注入的方式
//初始化pac
@Component
public class MySpider {
// private String url;
@Autowired
private Pipeline pipeline; //不能直接创建 应该使用注入
@Autowired
private Downloader downloader=new MyDownloader();
@Autowired
private PageProcessor processor=new MyPageProcessor() ;
public void start(String url){
//布隆防止重复
QueueScheduler scheduler=new QueueScheduler();
scheduler.setDuplicateRemover(new BloomFilterDuplicateRemover(1000000));
Spider.create(processor) //设置processor 组件
.setDownloader(downloader) //设置downloader 组件
.addPipeline(pipeline) //设置pipeline 组件
.setScheduler(scheduler) //设置url过滤
.addUrl(url) //设置起始url
.thread(1) //无头浏览器不支持多线程
.start(); //启动爬虫
}
}