Celery 实战指南

Celery 是

一个简单、灵活和可靠的分布式系统框架，多用作跨进程、多机器协同工作的场景。

场景

如何控制不预取

命令行增加 -O fair 配置文件supervisor中增加

;Late ack means the task messages will be acknowledged after the task has been executed, not just before (the default behavior).
task_acks_late = True
;How many messages to prefetch at a time multiplied by the number of concurrent processes. The default is 4 (four messages for each process). The default setting is usually a good choice, however – if you have very long running tasks waiting in the queue and you have to start the workers, note that the first worker to start will receive four times the number of messages initially. Thus the tasks may not be fairly distributed to the workers.
;To disable prefetching, set worker_prefetch_multiplier to 1. Changing that setting to 0 will allow the worker to keep consuming as many messages as it wants.
worker_prefetch_multiplier = 1

实际在使用时发现后面的无效，可以在命令行中直接写入 prefetch_multiplier=1

Celery 使用 阿里云 RabbitMQ 作为 Broker

阿里云的 RabbitMQ 只支持 PLAIN 认证； 强迫celery使用broker_login_method='PLAIN'

版本

Celery 4.4.0 官网介绍

特性

简单

Celery 使用非常简单，甚至可以无需配置文件。

from celery import Celery

app = Celery('hello', broker='amqp://guest@localhost//')

@app.task
def hello():
    return 'hello world'

高可用

Workers 和 Clients 遇到丢失连接或者连接失败时会自动重试。

快速

单个 Celery 进程每分钟可以处理数百万个任务，具有亚毫秒的往返延迟(使用RabbitMQ、librabbitmq并且优化设置)。

灵活

高度灵活，几乎所有模块都可自定义扩展使用。

支持

Brokers

Concurrency (并发)

Result Stores (任务结果存储)

Serialization (序列化)

Monitoring (监控)

Work-flows(工作流)

Time & Rate Limits (限流)

Scheduling (定时任务调度)

Resource Leak Protection(资源泄露保护)

User Components (用户自定义组件)

任务状态

结果	说明
PENDING	没有执行 or 状态未知
RECEIVED	已接收
STARTED	已启动
SUCCESS	成功
FAILURE	失败
REVOKED	已撤销
REJECTED	被拒绝
RETRY	重试中