代理池分为4个模块:存储模块、获取模块、检测模块、接口模块。
模块 | 作用 |
---|---|
存储模块 | 存储模块使用Redis的有序集合,用来做dialing的去重和状态表示,同时它也是中心模块和基础模块,将其他模块串连起来 |
获取模块 | 获取模块定时从代理网站获取代理,将获取的代理传递给存储模块,并保存到数据库 |
检测模块 | 检测模块定时从存储模块获取所有代理,并对代理进行检测,根据不同的检测结果对代理设置不同的标识 |
接口模块 | 接口模块通过Web API接通的服务接口,接口通过连接数据库并通过Web形式返回可用的代理 |
检测模块的实现:
from multiprocessing import Process
import time
from api import app
from getter import Getter
from tester import Tester
TESTER_CYCLE = 20
GETTER_CYCLE = 20
TESTER_ENABLED = True
GETTER_ENABLED = True
API_ENABLED = True
class Scheduler():
"""调度模块"""
def schedule_tester(self, cycle=TESTER_CYCLE):
"""
定时测试代理
:param cycle:
:return:
"""
tester = Tester()
while True:
print("测试器开始运行")
tester.run()
time.sleep(cycle)
def schedule_getter(self, cycle=GETTER_CYCLE):
"""
定时获取代理
:param cycle:
:return:
"""
getter = Getter()
while True:
print("开始抓取代理")
getter.run()
time.sleep(cycle)
def schedule_api(self):
"""
开启api
:return:
"""
app.run()
def run(self):
print("代理池开始运行")
if TESTER_ENABLED:
test_process = Process(target=self.schedule_tester)
test_process.start()
if GETTER_ENABLED:
getter_process = Process(target=self.schedule_getter)
getter_process.start()
if API_ENABLED:
api_process = Process(target=self.schedule_api)
api_process.start()
if __name__ == '__main__':
a = Scheduler()
a.run()
API模块的实现:
from flask import Flask, g
from db import RedisClient
__all__ = ['app']
app = Flask(__name__)
def get_conn():
if not hasattr(g, 'redis'):
g.redis = RedisClient()
return g.redis
@app.route("/")
def index():
return '<h2>Welcom to Proxy Pool System </h2>'
@app.route('/random')
def get_proxy():
"""
获取随机可用代理
:return: 随机代理
"""
conn = get_conn()
return conn.random()
@app.route("/count")
def get_counts():
"""
获取代理池总量
:return: 代理池总量
"""
conn = get_conn()
return str(conn.count())
if __name__ == '__main__':
app.run()
调度模块的实现:
from multiprocessing import Process
import time
from api import app
from getter import Getter
from tester import Tester
TESTER_CYCLE = 20
GETTER_CYCLE = 20
TESTER_ENABLED = True
GETTER_ENABLED = True
API_ENABLED = True
class Scheduler():
"""调度模块"""
def schedule_tester(self, cycle=TESTER_CYCLE):
"""
定时测试代理
:param cycle:
:return:
"""
tester = Tester()
while True:
print("测试器开始运行")
tester.run()
time.sleep(cycle)
def schedule_getter(self, cycle=GETTER_CYCLE):
"""
定时获取代理
:param cycle:
:return:
"""
getter = Getter()
while True:
print("开始抓取代理")
getter.run()
time.sleep(cycle)
def schedule_api(self):
"""
开启api
:return:
"""
app.run()
def run(self):
print("代理池开始运行")
if TESTER_ENABLED:
test_process = Process(target=self.schedule_tester)
test_process.start()
if GETTER_ENABLED:
getter_process = Process(target=self.schedule_getter)
getter_process.start()
if API_ENABLED:
api_process = Process(target=self.schedule_api)
api_process.start()
if __name__ == '__main__':
a = Scheduler()
a.run()
获取模块、存储模块、公共模块见
爬虫:IP代理池的四个模块的介绍以及各个模块的实现(下)
https://blog.csdn.net/weixin_40576010/article/details/89366179
参考github : https://github.com/Python3WebSpider/ProxyPool