Scrapy第十篇：IP代理中间件

我这里使用的是阿里云的IP代理服务。

1.编写工具类util.py，封装获取阿里云代理IP

import urllib.request
import json


def get_proxy_ip():
    host = 'http://zip.market.alicloudapi.com'
    path = '/devtoolservice/ipagency'
    method = 'GET'
    appcode = 'xxxxxxxxxxxx'
    querys = 'foreigntype=0&protocol=0'
    bodys = {}
    url = host + path + '?' + querys

    request = urllib.request.Request(url)
    request.add_header('Authorization', 'APPCODE ' + "xxxxxxxxx")
    response = urllib.request.urlopen(request)
    content = response.read()
    if (content):
        load = json.loads(str(content, encoding='utf8'))
        address_ = load['result'][0]['address']
        return address_

2.编写中间件：打开middlewares.py文件，新建中间件ProxyMiddleware

from announcement.util import get_proxy_ip


class ProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = get_proxy_ip()

3.开启中间件：打开settings.py

DOWNLOADER_MIDDLEWARES = {
    'announcement.middlewares.ProxyMiddleware': 541,
}

4.验证（略）