celery 笔记 [ things should be known when talking about celery ]

从我的 blog 转过来的，希望多多交流啦

http://litaotao.github.io/hello-celery/

1. hello, celery

three questions:
- what is celery?
- why do we need celery?
- what features does celery have?

1.1 what is celery?

• First, celery is a distributed task queue [ a distributed system to process messages].

• Second, it provides operations with the tools required to maintain such a system.

  • flower

  • jobtastic

• Third, it was built for Dijango Apps in the early, but scaled since version 0.8.

• Fourth, it focus on real-time processing, while also supporting task scheduling.

1.2 why do we need celery?

• run something in the background.

• run something after the web request is done.

• making something is done by executing it asynchronously and using retries.

• schedule periodic task.

• distributed computing.

• parallel executing.

1.3 what features does celery have?

• simple: write in python, easily configure.

• high available: retry mechanism, independent broker HA.

• fast: process millions of tasks one minute.

• flexible: Almost every part of Celery can be extended or used on its own, Custom pool implementations, serializers, compression schemes, logging, schedulers, consumers, producers, autoscalers, broker transports and much more.

• monitoring: A stream of monitoring events is emitted by workers and is used by built-in and external tools to tell you what your cluster is doing – in real-time.

• auto reloading: In development workers can be configured to automatically reload source code as it changes.

• workflows: Simple and complex workflows can be composed using a set of powerful primitives we call the “canvas”, including grouping, chaining, chunking and more.

• auto scaling: Dynamically resizing the worker pool depending on load, or custom metrics specified by the user, used to limit memory usage in shared hosting/cloud environments or to enforce a given quality of service.

• Time & Rate Limits: You can control how many tasks can be executed per second/minute/hour, or how long a task can be allowed to run, and this can be set as a default, for a specific worker or individually for each task type.

• Scheduling: You can specify the time to run a task in seconds or a datetime, or or you can use periodic tasks for recurring events based on a simple interval, or crontab expressions supporting minute, hour, day of week, day of month, and month of year.

• Resource leak protection:

• User components:

2 how does celery work?

2.1 some terms in message queue and celery

• celery task: a serialized task info which will be scheduled to done by celery,

1. it contains some meta data, like bellow:

{"id": "4cc7438e-afd4-4f8f-a2f3-f46567e7ca77",
 "task": "celery.task.PingTask",
 "args": [],
 "kwargs": {},
 "retries": 0,
 "eta": "2009-11-17T12:30:56.527191"
}

• producer: the one who send task messages to the broker,

1. it is a celery client.

• consumer: fetch task messages from queues and process them.

• broker: the one who receives task messages from producer and route different

1. to different queues.

• exchange: exchange, or a router, the one who really do message receiving from producer

1. dispatching message to queues. two things should be taken in mind about exchange:

   • one message from producer can be routed to zero or more queues;

   • you can set route rules for the exchange, called bindings;

• queue: buffer for store task messages.

2.2 the celery workflow

2.3 the workflow deployment

3. install celery

two possible ways to install celery.

• install the latest stable celery and its dependences

chenshan@mac007:~/Desktop$sudo pip install -U celery
...
...
...
Installing collected packages: celery, kombu, billiard, amqp

the celery dependences info is listed here: Does Celery have many dependencies

• install celery with bundles

Celery also defines a group of bundles that can be used to install Celery and the dependencies for a given feature.

You can specify these in your requirements or on the pip comand-line by using brackets. Multiple bundles can be specified by separating them by commas. for example, using the command pip install "celery[librabbitmq,redis,auth,msgpack]" to install celery with librabbitmq, redis, auth, msgpack libriaries.

all bundles command can be find here: bundles

参考文档

• celery official site

• http://abhishek-tiwari.com/post/amqp-rabbitmq-and-celery-a-visual-guide-for-dummies

• Async Celery by Example: Why and How

• How Instagram Feeds Work: Celery and RabbitMQ

• stackoverflow: Celery parallel distributed task with multiprocessing

• Scaling Celery Sending Tasks To Remote Machines!

• Celery Messaging at Scale at Instagram

• CELERY - BEST PRACTICES

• CELERY - BEST PRACTICES slide

• CELERY - BEST PRACTICES 中文

• 从一次celery踩坑中谈谈Queryset的懒加载

• stackoverflow: Send log messages from all celery tasks to a single file