Celery 总体架构设计:
安装Celery
pip install celery
Task-Func-demo:
#-*-coding:utf-8-*-
#download.py
from celery import Celery
import urllib
app = Celery('download', broker='redis://localhost:6379/1')
# 图片url
img_url = ["http://img.hb.aicdn.com/3750dffcf16726889691f6d619671c51703a01db236961-TspqDv",
"http://img.hb.aicdn.com/91681f86e011c3b5dee27a75a0c353d418e5cf4226739-knqXah",
"http://img.hb.aicdn.com/8665e8c98a68a7677fea45fa0b2a7d426fe8b730814da-4R9hsY",
"http://img.hb.aicdn.com/605ca501843c37f602b781f83fb44709187c8e45988b5-bet9jM",
"http://img.hb.aicdn.com/af2c2f3add40743457e7f7bcf9260da85ac4a790aa6e1-NisN4l"
]
@app.task
def download(img_url):
fname = img_url.replace("http://img.hb.aicdn.com/", "")
urllib.urlretrieve(img_url, "%s.jpg" % fname)
if __name__ == "__main__":
# 将任务delay入队列
for url in img_url:
download.delay(url)
启动Redis-Server
./redis-server ./redis.conf
启动多个Celery-Worker
1.每个worker都连接到同一个redis-server
2.celery worker -A download -l info -c 5
3.-c 5 表示每个worker都开5个进程
4.现在Celery-Worker正在监听任务
执行download.py脚本,将任务delay进队列
1.python download.py
2.worker会自动从队列获取任务并执行