Scrapy框架 -- 中间件介绍

一、创建Scrapy项目zhongjj，进入zhongjj项目，创建爬虫文件zhongjjpc

scrapy startproject zhongjj
cd zhongjj
scrapy genspider zhongjjpc www.xxx.com

二、修改配置文件

ROBOTSTXT_OBEY = False
LOG_LEVEL = 'ERROR'

三、添加三个目标地址，其中最后一个地址是错误的url

start_urls = ["https://www.baidu.com/","https://www.sina.com.cn/","https://wwwwww.sohu.com/"]

四、修改中间件文件

1、删除爬虫中间件类ZhongjjSpiderMiddleware

2、修改拦截内容响应内容及异常内容

def process_request(self, request, spider):
        print(request.url+"我是requests")
        return None


    def process_response(self, request, response, spider):
        print(request.url+"我是response")
        return response


    def process_exception(self, request, exception, spider):
        print(request.url+"我是异常信息")
        pass

3、在settings文件里面开启中间件

DOWNLOADER_MIDDLEWARES = {
   "zhongjj.middlewares.ZhongjjDownloaderMiddleware": 543,
}

五、运行结果，三个函数都被调用

六、开发中间件

1、代理中间件

request.meta['proxy'] = 'https://ip:port'

2、UA中间件

request.headers['User-Agent'] = 'Mozilla/5.0 (Windows ......'

3、Cookies中间件

request.headers['cookie'] = 'xxx'
第二种方法
request.cookies = 'xxx'