使用scrapy框架爬取豆瓣影评

2019独角兽企业重金招聘Python工程师标准>>>

最近几天在看平凡岁月 感觉挺不错的一部生活剧，想看看豆瓣

对该据评论，使用scrapy爬取，基于python2.7进行实现

1. 创建scrapy项目

scrapy startproject doubanmovie

2.创建爬虫

scrapy genspider -t crawl dbm "movie.douban.com"

需要注意的是 爬虫名称不能和项目名称重复

3. 编写item

4.编写爬虫（spiders-->dbm.py）

# -*- coding: utf-8 -*-
import scrapy

from scrapy.linkextractors import LinkExtractor

from scrapy.spiders import CrawlSpider, Rule

from doubanmovie.items import DoubanmovieItem

class DbmSpider(CrawlSpider):
    name = 'dbm'
    allowed_domains = ['movie.douban.com']
    start_urls = ['https://movie.douban.com/subject/26795042/reviews?start=0']

    rules = (
        Rule(LinkExtractor(allow=r'start=\d+'), follow=True),
        Rule(LinkExtractor(allow=r'https://movie.douban.com/review/\d+/'), callback='parse_item', follow=False),
    )

    def parse_item(self, response):
        item = DoubanmovieItem()
        item['username'] = response.xpath("//header[@class='main-hd']/a/span/text()").extract()[0]
        title = response.xpath("//span[@property='v:summary']/text()").extract()[0]
        if len(title):
            item['title'] = title
        else:
            item['title'] = 'NULL'
        content = response.xpath("//div[@class='review-content clearfix']/p/text()").extract()
        if len(content):
            item['content'] = ''.join(content).strip()
        else:
            content = response.xpath("//div[@class='review-content clearfix']/text()").extract()
            if len(content):
                item['content'] = ''.join(content).strip()
            else:
                item['content'] = 'NULL'
        item['url'] = response.url
        yield item

5.编写管道文件(该案例使用json保存) doubanmovie->pipelines.py

import json
class YouyuanPipeline(object):
    def __init__(self):
        self.filename = open('youyuan.json','w')

    def process_item(self, item, spider):
        content = json.dumps(dict(item),ensure_ascii=False) + ',\n'
        self.filename.write(content.encode('utf-8'))
        return item

    def close_spider(self,spider):
        self.filename.close()

6.修改setting文件

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.54 Safari/536.5'  
DOWNLOAD_DELAY = 3
ITEM_PIPELINES = {
   'doubanmovie.pipelines.DoubanmoviePipeline': 300,
}

7- 执行爬虫

scrapy crawl dbm

转载于:https://my.oschina.net/u/3273360/blog/1579179