django celery 记录

django celery 记录

dvadmin-celery

Django+Django-Celery+Celery的整合实战

https://cloud.tencent.com/developer/article/1445252

https://blog.csdn.net/wowocpp/article/details/131475484

https://docs.celeryq.dev/en/latest/django/first-steps-with-django.html

https://docs.celeryq.dev/en/latest/django/index.html

http://docs.celeryproject.org/en/latest/

https://github.com/pylixm/celery-examples

https://pylixm.cc/posts/2015-12-03-Django-celery.html
中文
https://www.celerycn.io/ru-men/celery-chu-ci-shi-yong

中文
https://docs.jinkan.org/docs/celery/

Python Django构建简易CMDB
https://blog.51cto.com/u_15352934/3819825

简约而不简单的 Django 新手图文教程
https://zhuanlan.zhihu.com/p/393724439

牛哄哄的celery
https://www.cnblogs.com/pyedu/p/12461819.html

example

https://github.com/celery/celery

Backends and Brokers
Brokers 消息中间件 消息转发

Celery的架构由三部分组成，消息中间件（message broker），任务执行单元（worker）和任务执行结果存储（task result store）组成。

消息中间件

Celery本身不提供消息服务，但是可以方便的和第三方提供的消息中间件集成。包括，RabbitMQ, Redis等等

任务执行单元

Worker是Celery提供的任务执行的单元，worker并发的运行在分布式的系统节点中。

任务结果存储

Task result store用来存储Worker执行的任务的结果，Celery支持以不同方式存储任务的结果，包括AMQP, redis等

另外， Celery还支持不同的并发和序列化的手段

并发：Prefork, Eventlet, gevent, threads/single threaded
序列化：pickle, json, yaml, msgpack. zlib, bzip2 compression， Cryptographic message signing 等等

第一个例子：

python3 -m venv demo1_venv
source demo1_venv/bin/activate
mkdir work
cd work/
pip install -U Celery
pip install redis
mkdir CeleryTask
vi celery_task.py

import celery
import time
backend='redis://127.0.0.1:6379/1'
broker='redis://127.0.0.1:6379/2'
cel=celery.Celery('test',backend=backend,broker=broker)
@cel.task
def send_email(name):
    print("向%s发送邮件..."%name)
    time.sleep(5)
    print("向%s发送邮件完成"%name)
    return "ok"

celery -A celery_task worker -l info

(demo1_venv) mike@ubuntu:~/work/CeleryTask$ celery  -A celery_task worker -l info
 
 -------------- celery@ubuntu v5.3.1 (emerald-rush)
--- ***** ----- 
-- ******* ---- Linux-5.15.0-76-generic-x86_64-with-glibc2.29 2023-07-03 02:40:28
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         test:0x7f985fa979a0
- ** ---------- .> transport:   redis://127.0.0.1:6379/2
- ** ---------- .> results:     redis://127.0.0.1:6379/1
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . celery_task.send_email

[2023-07-03 02:40:28,783: WARNING/MainProcess] /home/mike/demo1_venv/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:498: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2023-07-03 02:40:28,806: INFO/MainProcess] Connected to redis://127.0.0.1:6379/2
[2023-07-03 02:40:28,808: WARNING/MainProcess] /home/mike/demo1_venv/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:498: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2023-07-03 02:40:28,811: INFO/MainProcess] mingle: searching for neighbors
[2023-07-03 02:40:29,827: INFO/MainProcess] mingle: all alone
[2023-07-03 02:40:29,870: INFO/MainProcess] celery@ubuntu ready.

第二个例子

mkdir app
vi tasks.py

from celery import Celery
backend='redis://127.0.0.1:6379/3'
broker='redis://127.0.0.1:6379/4'
app = Celery('tasks', broker=broker, backend=backend)
@app.task
def add(x, y):
        return x + y

启动实例任务celery worker （消费者）

celery -A tasks worker --loglevel=info
等待 redis中 来消息之后，执行任务

调用实例任务 （生产者）

给 消息队列发送消息（redis）

可以
celery worker --help

Usage: celery worker [OPTIONS]

  Start worker instance.

  Examples
  --------

  $ celery --app=proj worker -l INFO
  $ celery -A proj worker -l INFO -Q hipri,lopri
  $ celery -A proj worker --concurrency=4
  $ celery -A proj worker --concurrency=1000 -P eventlet
  $ celery worker --autoscale=10,0

Worker Options:
  -n, --hostname HOSTNAME         Set custom hostname (e.g., 'w1@%%h').
                                  Expands: %%h (hostname), %%n (name) and %%d,
                                  (domain).
  -D, --detach                    Start worker as a background process.
  -S, --statedb PATH              Path to the state database. The extension
                                  '.db' may be appended to the filename.
  -l, --loglevel [DEBUG|INFO|WARNING|ERROR|CRITICAL|FATAL]
                                  Logging level.
  -O, --optimization [default|fair]
                                  Apply optimization profile.
  --prefetch-multiplier <prefetch multiplier>
                                  Set custom prefetch multiplier value for
                                  this worker instance.

Pool Options:
  -c, --concurrency <concurrency>
                                  Number of child processes processing the
                                  queue.  The default is the number of CPUs
                                  available on your system.
  -P, --pool [prefork|eventlet|gevent|solo|processes|threads|custom]
                                  Pool implementation.
  -E, --task-events, --events     Send task-related events that can be
                                  captured by monitors like celery events,
                                  celerymon, and others.
  --time-limit FLOAT              Enables a hard time limit (in seconds
                                  int/float) for tasks.
  --soft-time-limit FLOAT         Enables a soft time limit (in seconds
                                  int/float) for tasks.
  --max-tasks-per-child INTEGER   Maximum number of tasks a pool worker can
                                  execute before it's terminated and replaced
                                  by a new worker.
  --max-memory-per-child INTEGER  Maximum amount of resident memory, in KiB,
                                  that may be consumed by a child process
                                  before it will be replaced by a new one.  If
                                  a single task causes a child process to
                                  exceed this limit, the task will be
                                  completed and the child process will be
                                  replaced afterwards. Default: no limit.

Queue Options:
  --purge, --discard
  -Q, --queues COMMA SEPARATED LIST
  -X, --exclude-queues COMMA SEPARATED LIST
  -I, --include COMMA SEPARATED LIST

Features:
  --without-gossip
  --without-mingle
  --without-heartbeat
  --heartbeat-interval INTEGER
  --autoscale <MIN WORKERS>, <MAX WORKERS>

Embedded Beat Options:
  -B, --beat
  -s, --schedule-filename, --schedule TEXT
  --scheduler TEXT

Daemonization Options:
  -f, --logfile TEXT  Log destination; defaults to stderr
  --pidfile TEXT
  --uid TEXT
  --gid TEXT
  --umask TEXT
  --executable TEXT

Options:
  --help  Show this message and exit.

celery --help

Usage: celery [OPTIONS] COMMAND [ARGS]...

  Celery command entrypoint.

Options:
  -A, --app APPLICATION
  -b, --broker TEXT
  --result-backend TEXT
  --loader TEXT
  --config TEXT
  --workdir PATH
  -C, --no-color
  -q, --quiet
  --version
  --skip-checks          Skip Django core checks on startup.
  --help                 Show this message and exit.

Commands:
  amqp     AMQP Administration Shell.
  beat     Start the beat periodic task scheduler.
  call     Call a task by name.
  control  Workers remote control.
  events   Event-stream utilities.
  graph    The ``celery graph`` command.
  inspect  Inspect the worker at runtime.
  list     Get info from broker.
  logtool  The ``celery logtool`` command.
  migrate  Migrate tasks from one broker to another.
  multi    Start multiple worker instances.
  purge    Erase all messages from all known task queues.
  report   Shows information useful to include in bug-reports.
  result   Print the return value for a given task id.
  shell    Start shell session with convenient access to celery symbols.
  status   Show list of workers that are online.
  upgrade  Perform upgrade between versions.
  worker   Start worker instance.

backend 任务结果后端

Celery 进阶使用

https://www.celerycn.io/ru-men/celery-jin-jie-shi-yong

在 work目录下创建proj目录

celery.py

from __future__ import absolute_import, unicode_literals
from celery import Celery

backend='redis://127.0.0.1:6379/5'
broker='redis://127.0.0.1:6379/6'


app = Celery('proj',
             broker=broker,
             backend=backend,
             include=['proj.tasks'])

# Optional configuration, see the application user guide.
app.conf.update(
    result_expires=3600,
)

if __name__ == '__main__':
    print("hello")
    app.start()

vi tasks.py

from __future__ import absolute_import, unicode_literals
from .celery import app

@app.task
def add(x, y):
    return x + y

@app.task
def mul(x, y):
    return x * y

@app.task
def xsum(numbers):
    return sum(numbers)

测试的话
在work目录下面执行
celery -A proj worker -l info

work目录下面执行：

python3
>>> from proj.tasks import add
>>> add.delay(2,2)
<AsyncResult: 32a194e0-1e91-49bf-8436-b215beb96e16>