- celery的简介
celery是一个异步分布式系统,专注于实时处理的任务队列,同时也支持任务调度
celery的最基础部分包括:
- 选择中间人(broker):celery需要一个发送和接收消息的解决方案,其通常以独立服务形式出现,称为消息中间人,中间人包括RabbitMQ(默认)、Redis、数据库和其他实验性的中间人
- 安装celery模块并创建一个任务(task)发送到broker
- 任务执行单元(celery worker)监控broker,有task就执行,把执行结果存到backend中
- celery追踪任务在不同状态间的迁移,并接收返回值(task execute success or faile)
- 安装celery模块
celery模块对python版本要求严格
![](https://i-blog.csdnimg.cn/blog_migrate/0ad76e0763990cfd56774d026fc0b6f5.png)
[root@web ~]# pip install celery //python2.6.6安装3.1及以下版本都无法使用命令,只能选择python2.7 安装celery最新版本
- celery小例子
- 3.1 创建一个应用
[root@web celery_py]# cat test.py #!/usr/bin/env python from celery import Celery #format: #redis://password@hostname:port/db_number br = 'redis://127.0.0.1:6379/0' //中间人 bd = 'redis://127.0.0.1:6379/1' //结果存储 app = Celery('test', broker=br, backend=bd) //第一个参数为应用名,第二个参数为中间人,第三个参数为结果存储方式 @app.task //固定写法 def add(x, y): return x+y
- 3.2 生成任务执行单元(worker)
[root@localhost sbin]# celery -A test worker -l info /usr/lib/python2.7/site-packages/celery/platforms.py:795: RuntimeWarning: You're running the worker with superuser privileges: this is absolutely not recommended! Please specify a different user using the -u option. User information: uid=0 euid=0 gid=0 egid=0 uid=uid, euid=euid, gid=gid, egid=egid, -------------- celery@localhost.localdomain v4.1.0 (latentcall) ---- **** ----- --- * *** * -- Linux-3.10.0-693.el7.x86_64-x86_64-with-centos-7.4.1708-Core 2017-12-21 23:41:46 -- * - **** --- - ** ---------- [config] - ** ---------- .> app: task:0x157b810 - ** ---------- .> transport: redis://127.0.0.1:6379/0 - ** ---------- .> results: redis://127.0.0.1:6379/1 - *** --- * --- .> concurrency: 1 (prefork) -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker) --- ***** ----- -------------- [queues] .> celery exchange=celery(direct) key=celery [tasks] . task.add [2017-12-21 23:41:47,010: INFO/MainProcess] Connected to redis://127.0.0.1:6379/0 [2017-12-21 23:41:47,016: INFO/MainProcess] mingle: searching for neighbors [2017-12-21 23:41:48,032: INFO/MainProcess] mingle: all alone [2017-12-21 23:41:48,053: INFO/MainProcess] celery@localhost.localdomain ready.
- 3.3 创建异步任务
创建异步任务,直接丢给中间人,任务算完成
worker监控中间人,发现属于自己的任务就会开始执行
In [1]: import os In [2]: os.chdir('/usr/local/sbin/') In [3]: from test import add In [4]: add.delay(2, 2) Out[4]: <AsyncResult: 400207f8-17bf-4a8e-b89f-1456b42a54fe> //返回一个AsyncResult实例,可用于检查任务状态,获取任务结果等 In [5]: result = add.delay(2, 2) -------------------------------------------------> //worker从broker中读取任务并执行 [2017-12-22 00:01:19,827: INFO/MainProcess] Received task: test.add[7010fecd-ccea-407e-a76a-1493f8140fd2] [2017-12-22 00:01:19,830: INFO/ForkPoolWorker-1] Task test.add[7010fecd-ccea-407e-a76a-1493f8140fd2] succeeded in 0.00122961999978s: 4 ------------------------------------------------> //在redis中 127.0.0.1:6379[1]> get "celery-task-meta-7010fecd-ccea-407e-a76a-1493f8140fd2" "{\"status\": \"SUCCESS\", \"traceback\": null, \"result\": 4, \"task_id\": \"7010fecd-ccea-407e-a76a-1493f8140fd2\", \"children\": []}"
- 3.4 AsyncResult实例向backend查询任务执行结果和状态
也就是使用实例的各种方法
![](https://i-blog.csdnimg.cn/blog_migrate/f42d1655d557273ee5a540fa93e3f964.png)
![](https://i-blog.csdnimg.cn/blog_migrate/6b2c446a996d4484a2d0ec2c939936de.png)
![](https://i-blog.csdnimg.cn/blog_migrate/3fb460ab80ac804b4f3cbb8bf1511f08.png)
![](https://i-blog.csdnimg.cn/blog_migrate/f8a8287844b3104ef03a7569271cc1c1.png)