1.首先是创建一个crawlspider的爬虫项目
# cd 指定目录下
# 创建一个scrapy框架的项目,名字叫DOUBAN
# scrapy startproject DOUBAN
# cd DOUBAN/
# 进入到项目目录后,创建一个以crawl为模板的crawlspider,爬虫名叫db_crawl,并指定爬取的域:douban.com
# scrapy genspider -t crawl db_crawl douban.com
2.打开DOUBAN项目,首先打开items.py定义需要获取的信息
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html
import scrapy
class DoubanItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
"""获取电影名,评分,简介等信息"""
db_name = scrapy.Field()
db_socer = scrapy.Field()
db_info = scrapy.Field()
db_detail = scrapy.Field()
pass
3.打开主文件db_crawl.py,编写rule规则以及获取信息
# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import