使用crawlspider自动爬取豆瓣TOP250电影信息

1.首先是创建一个crawlspider的爬虫项目

# cd 指定目录下
#  创建一个scrapy框架的项目,名字叫DOUBAN
# scrapy startproject DOUBAN
# cd DOUBAN/
# 进入到项目目录后,创建一个以crawl为模板的crawlspider,爬虫名叫db_crawl,并指定爬取的域:douban.com
# scrapy genspider -t crawl db_crawl douban.com

2.打开DOUBAN项目,首先打开items.py定义需要获取的信息

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class DoubanItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    """获取电影名,评分,简介等信息"""
    db_name = scrapy.Field()
    db_socer = scrapy.Field()
    db_info = scrapy.Field()
    db_detail = scrapy.Field()
    pass

3.打开主文件db_crawl.py,编写rule规则以及获取信息

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import