scrapy高级用法之自动分页

最新推荐文章于 2024-08-03 19:27:22 发布

weixin_33910460

最新推荐文章于 2024-08-03 19:27:22 发布

阅读量306

点赞数 1

文章标签： python 前端 ViewUI

原文链接：https://my.oschina.net/u/2351685/blog/612940

版权

2019独角兽企业重金招聘Python工程师标准>>>

原理

根据规从页面中提取到“下一页”或“其他分页”链接

用到模块

  from pyquery import PyQuery as pq
  from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

完整代码

# -*- coding: utf-8 -*-
from pyquery import PyQuery as pq
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor


class SegSpider(CrawlSpider):
    name = "seg"
    allowed_domains = ["segmentfault.com"]
    start_urls = (
        'http://segmentfault.com/t/html5?type=newest&page=1',
    )

    rules = (
        Rule(SgmlLinkExtractor(allow=('\/t\/html5\?type=newest\&page=\d', ))),
        Rule(SgmlLinkExtractor(allow=('\/q\/\d+', )), callback='parse_item'),
    )

    def parse_item(self, response):
        html = response.body
        v = pq(html)
        item = dict()
        item['url'] = response.url
        item['title'] = v('title').text()
        yield item

转载于:https://my.oschina.net/u/2351685/blog/612940