【Python】基于Celery的分布式应用

架构

---

安装Celery

• pip install celery

编写Task-Func

#-*-coding:utf-8-*-
#download.py

from celery import Celery
import urllib

app = Celery('download', broker='redis://localhost:6379/1')

# 图片url
img_url = ["http://img.hb.aicdn.com/3750dffcf16726889691f6d619671c51703a01db236961-TspqDv",
           "http://img.hb.aicdn.com/91681f86e011c3b5dee27a75a0c353d418e5cf4226739-knqXah",
           "http://img.hb.aicdn.com/8665e8c98a68a7677fea45fa0b2a7d426fe8b730814da-4R9hsY",
           "http://img.hb.aicdn.com/605ca501843c37f602b781f83fb44709187c8e45988b5-bet9jM",
           "http://img.hb.aicdn.com/af2c2f3add40743457e7f7bcf9260da85ac4a790aa6e1-NisN4l"
]

@app.task
def download(img_url):
    fname = img_url.replace("http://img.hb.aicdn.com/", "")
    urllib.urlretrieve(img_url, "%s.jpg" % fname)

if __name__ == "__main__":
    # 将任务delay入队列
    for url in img_url:
        download.delay(url)

---

启动Redis-Server

• ./redis-server ./redis.conf

启动多个Celery-Worker

• 每个worker都连接到同一个redis-server
• celery worker -A download -l info -c 5
• -c 5 表示每个worker都开5个进程
• 现在Celery-Worker正在监听任务
  

---

执行download.py脚本，将任务delay进队列

• python download.py
• worker会自动从队列获取任务并执行
  

---