2021SC@SDUSC
接上篇
完成对AllSort的注入后,我们需要对AllSort进行业务处理,这里我们不做分类信息持久化等处理,只对分类链接进行提取,进一步抓取商品列表信息。看代码:
@PipelineName("allSortPipeline")
public class AllSortPipeline implements Pipeline<AllSort> {
@Override
public void process(AllSort allSort) {
List<Category> categorys = allSort.getMobile();
for(Category category : categorys) {
List<HrefBean> hrefs = category.getCategorys();
for(HrefBean href : hrefs) {
String url = href.getUrl()+"&delivery=1&page=1&JL=4_10_0&go=0";
HttpRequest currRequest = allSort.getRequest();
SchedulerContext.into(currRequest.subRequest(url));
}
}
}
}
@PipelinName定义该pipeline的名称,在AllSort的@Gecco注解里进行关联,这样,gecco在抓取完并注入Bean后就会逐个调用@Gecco定义的pipeline了。为每个子链接增加"&delivery=1&page=1&JL=4_10_0&go=0"的目的是只抓取京东自营并且有货的商品。SchedulerContext.into()方法是将待抓取的链接放入队列中等待进一步抓取。AllSortPipeline已经将需要进一步抓取的商品列表信息的链接提取出来了,可以看到链接的格式是:http://list.jd.com/list.html?cat=9987,653,659&delivery=1&JL=4_10_0&go=0。因此我们建立商品列表的Bean——ProductList,代码如下:
@Gecco(matchUrl="http://list.jd.com/list.html?cat={cat}&delivery={delivery}&page={page}&JL={JL}&go=0", pipelines={"consolePipeline", "productListPipeline"})
public class ProductList implements HtmlBean {
private static final long serialVersionUID = 4369792078959596706L;
@Request
private HttpRequest request;
/**
* 抓取列表项的详细内容,包括titile,价格,详情页地址等
*/
@HtmlField(cssPath="#plist .gl-item")
private List<ProductBrief> details;
/**
* 获得商品列表的当前页
*/
@Text
@HtmlField(cssPath="#J_topPage > span > b")
private int currPage;
/**
* 获得商品列表的总页数
*/
@Text
@HtmlField(cssPath="#J_topPage > span > i")
private int totalPage;
public List<ProductBrief> getDetails() {
return details;
}
public void setDetails(List<ProductBrief> details) {
this.details = details;
}
public int getCurrPage() {
return currPage;
}
public void setCurrPage(int currPage) {
this.currPage = currPage;
}
public int getTotalPage() {
return totalPage;
}
public void setTotalPage(int totalPage) {
this.totalPage = totalPage;
}
public HttpRequest getRequest() {
return request;
}
public void setRequest(HttpRequest request) {
this.request = request;
}
}
currPage和totalPage是页面上的分页信息,为之后的分页抓取提供支持。ProductBrief对象是商品的简介,主要包括标题、预览图、详情页地址等。