使用Gecco爬虫时偶尔遇到如下莫名其妙的异常而停止:
Exception in thread "Spider-0" java.lang.NullPointerException
at java.util.regex.Matcher.getTextLength(Matcher.java:1283)
at java.util.regex.Matcher.reset(Matcher.java:309)
at java.util.regex.Matcher.<init>(Matcher.java:229)
at java.util.regex.Pattern.matcher(Pattern.java:1093)
at com.geccocrawler.gecco.utils.UrlMatcher.match(UrlMatcher.java:80)
at com.geccocrawler.gecco.spider.SpiderBeanFactory.matchSpider(SpiderBeanFactory.java:112)
at com.geccocrawler.gecco.spider.Spider.run(Spider.java:68)
at java.lang.Thread.run(Thread.java:745)
后来细心查看才发现(SpiderBeanFactory.java:112)中,空指针调用的url地址是来自 request
于是做了如下测试: HttpGetRequest start = new HttpGetRequest(null); GeccoEngine.create() .classpath("com.example.crawler") .start(start) .interval(2000) .run();
由此可知,是爬虫过程中如下的调用没有保证url不为null
HttpRequest sub = l.getRequest().subRequest(url); SchedulerContext.into(sub);
个人原创:爱我子月