爬虫实例四:通过使用scrapy框架爬取豆瓣电影top250字段

1-打开powshell 创建项目 执行命令scrapy startproject DouBanTop

scrapy startproject DouBanTop

2- 执行命令 cd DouBanTop

cd DouBanTop
scrapy genspider TouTiao movie.douban.com

3-使用pycharm或者vscode打开自己创建的项目名

(1)在settings.py文件里把ROBOTSTXT_OBEY = True改为False

ROBOTSTXT_OBEY = False

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 3  #下载延迟3秒
# The download delay setting will honor only one of:
CONCURRENT_REQUESTS_PER_DOMAIN = 16  #线程的并发数
#CONCURRENT_REQUESTS_PER_IP = 16

(2)在items.py中创建需要爬取豆瓣电影的字段 如电影标题 电影评分等等

import scrapy
class DoubanItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()  # 电影名字
    score = scrapy.Field() #电影评分
    count = scrapy.Field() #评价人数
    #introduce =scrapy.Field() #电影简介
    director = scrapy.Field() #电影导演

(3)在自己创建的DouBanTop.py文件中 对豆瓣电影进行爬取

# -*- coding: utf-8 -*-
import scrapy
from DouBan.items import DoubanItem

class DoubantopSpider(scrapy.Spider):
    name = 'DouBanTop'
    allowed_domains = ['movie.douban.com']
    start_urls = ['https://movie.douban.com/top250']

    def parse(self, response):
        print(response.request.headers,"###########################")
        #翻页并发
        #https://movie.douban.com/top250?start=50&filter=
        for index in range(10):
            param_url ='?start=%d&filter='%(index*25)
            #对每一页的Url进行拼接  获取每一页的url 分析可得每一页为start*25
            page_url = response.url+param_url
            index+=1
            yield scrapy.Request(url=page_url,callback=self.parse_page_url)

    # 每一页的25条电影信息
    def parse_page_url(self,response):

        """对每一页的25部电影并发"""
        #对每一页的url进行简单的判断 如果不存在 就返回
        if response.status !=