【学习-scrapy】下载器中间件，ip代理设置

1、process_request(self.request,spider)

  参数：

         request、spider: 发送请求对象

  返回值： 

        返回none:

       返回Response:

2、process_response(self.request,spider)

  参数：

         request、response、sider: 发送请求对象

  返回值： 

        返回none:

       返回Response:

【例】随机请求头中间件：

def parse(self, response):
    print(response.text)

获取ip: http://httpbin.org/ip

》setting.py=修改

 

DOWNLOADER_MIDDLEWARES = {
   'baom.middlewares.UserAgen': 543,
}

》middlewares.py=中间件

import random

class UserAgen(object):
    User_Agens=[
        'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6776.400 QQBrowser/10.3.2577.400'
        'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6776.400 QQBrowser/10.3.2577.400'
        'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6776.400 QQBrowser/10.3.2577.400'

    ]
    #随机请求一个
    def process_request(self,request,spider):
        user_agent=random.choice(self.User_Agens)
        request.headers['User-Agent']=user_agent