Scrapy添加IP的方法

版权声明:本文为博主原创文章,遵循 CC 4.0 by-sa 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/dongdong0071/article/details/79438032

1、settings.py中添加下载中间件

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 127,
    'lianjia.middlewares.MyproxiesSpiderMiddleware': 126,
}
2、middlewares.py中添加代码
from scrapy import signals
import random
from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware


class MyproxiesSpiderMiddleware(object):

    def __init__(self, ip=''):
        self.ip = ip

    def process_request(self, request, spider):
        thisip = random.choice(self.IPPOOL)
        print("this is ip:" + thisip["ipaddr"])
        request.meta["proxy"] = "http://" + thisip["ipaddr"]

    IPPOOL = [
        {"ipaddr": "66.81.206.241:3128"},
        {"ipaddr": "27.147.163.77:80"},
        {"ipaddr": "121.58.17.52:80"},
        {"ipaddr": "183.57.36.87:8888"},
        {"ipaddr": "47.52.222.165:80"},
        {"ipaddr": "47.52.44.31:8080"},
        {"ipaddr": "202.109.237.35:80"}
    ]


展开阅读全文

没有更多推荐了,返回首页