django异步通信框架Celery

Celery 是一个简单、灵活且可靠的，处理大量消息的 分布式系统。
专注于实时处理的 异步任务队列。
同时也支持 任务调度。

1.安装

pip install Django-celery==3.3.1 # django3以上的需要安装 
pip install Django-celery==3.2.2 # django2

pip install Django-redis  

pip install redis==2.10.6 
# 如果redis版本过高后面会出现错误下面这个
#'str' object has no attribute 'items' (ven)

pip install celery==4.4.7 # python3.9
如果是python3.6 安装pip install celery==4.2.1

2.setting.py配置

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',
    # 异步
    'djcelery',
]

3.在settings.py同级下创建celery.py和tasks.py

celery：celery的配置
tasks：异步任务
celery.py

from celery import Celery, platforms
from django.conf import settings
import os

# 设置celery的默认工作目录
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "xxxxx.settings")
# 启动celery项目(要加入broker，如果用的是redis的话，不然默认开启的是rabbitmq-server)
app = Celery(broker='redis://localhost:6379/3')
# 项目加载配置
app.config_from_object("django.conf:settings")
# 加载djcelery
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
# 支持以root启动
platforms.C_FORCE_ROOT = True

tsaks.py

from celery import task
import time

@task
def task1():
    """无参方法"""
    print("task1...begin...")
    time.sleep(10)  # 模拟耗时操作
    print("task1...end...")

@task
def task2(a, b):
    """有参方法"""
    print("task2...begin...a={},b={}".format(a, b))
    time.sleep(10)  # 模拟耗时操作
    print("task2...end...")
    return a + b

4.在views.py调用

from .tasks import task2
task2.delay(5,5)

5.确保app在Django启动时被加载

需要在django_proj/proj/__init__.py模块中导入这个celery实例app。

from .celery import app as celery_app

__all__ = ('celery_app',)

6.控制台启动celery

启动服务的命令：
xxxx 为项目名称
celery -A xxxx beat -l info 定时任务
celery -A xxxx worker -l info 异步任务

守护进程方式启动，日志记录在celerylog.log里
celery multi start w1 -A xxxx -l info --logfile="./celerylog.log" --pidfile="./celerypid.pid"
停止：celery multi stop w1 -A xxxx -l info
重启：celery multi restart w1 -A xxxx -l info

 - myproject/
   - manage.py
   - project/
     - __init__.py # 修改这个文件
     - celery.py # 新增这个文件
     - asgi.py
     - settings.py
     - urls.py
     - wsgi.py

注意：

需要先运行django后再运行celery，修改异步任务后需要重新启动celery，不支持热加载

出现的错误

1.SyntaxError: invalid syntax

报错：python3将async这个单词定义为关键词，所以不可以作为模块名称，需要对源码修改。
修改kombu中的async文件名为asynchronous
然后修改报错的文档中所有的async为asynchronoushttps://blog.csdn.net/weixin_44420527/article/details/108877096

2.consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 61] Con

celery.py文件中配置

app = Celery(broker='redis://localhost:6379/3')

3.'str' object has no attribute 'items' (ven)

redis版本太高

pip uninstall redis
pip install redis==2.10.6

4.No module named 'celery.five'

celery降级到4.4.7

使用命令卸载：
pip uninstall celery
重新安装：
pip install celery==4.4.7

5.ImportError: cannot import name 'render_to_response' from 'django.shortcuts' (/opt/homebrew/lib/python3.9/site-packages/django/shortcuts.py)

原因 Django 3.0 已经将 render_to_response 移除了,

pip install Django-celery==3.3.1

6.anyjson>=0.3.3

ERROR: Cannot install celery because these package versions have conflicting dependencies.

The conflict is caused by:
    kombu 3.0.37 depends on anyjson>=0.3.3
    kombu 3.0.36 depends on anyjson>=0.3.3
    kombu 3.0.35 depends on anyjson>=0.3.3
    kombu 3.0.34 depends on anyjson>=0.3.3
    kombu 3.0.33 depends on anyjson>=0.3.3
    kombu 3.0.32 depends on anyjson>=0.3.3
    kombu 3.0.31 depends on anyjson>=0.3.3
    kombu 3.0.30 depends on anyjson>=0.3.3
    kombu 3.0.29 depends on anyjson>=0.3.3
    kombu 3.0.28 depends on anyjson>=0.3.3
    kombu 3.0.27 depends on anyjson>=0.3.3
    kombu 3.0.26 depends on anyjson>=0.3.3
    kombu 3.0.25 depends on anyjson>=0.3.3
    kombu 3.0.24 depends on anyjson>=0.3.3
    kombu 3.0.23 depends on anyjson>=0.3.3

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

需要安装依赖anyjson=0.3.3

ERROR: Command errored out with exit status 1:
   command: /home/projects/bbsvenv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vln5yijj/anyjson_6fd979a2629347a5a24a5ce49539ce5c/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vln5yijj/anyjson_6fd979a2629347a5a24a5ce49539ce5c/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-yolnm3e_
       cwd: /tmp/pip-install-vln5yijj/anyjson_6fd979a2629347a5a24a5ce49539ce5c/
  Complete output (3 lines):
  /home/projects/bbsvenv/lib/python3.6/site-packages/setuptools/dist.py:720: UserWarning: Usage of dash-separated 'index-url' will not be supported in future versions. Please use the underscore name 'index_url' instead
    % (opt, underscore_opt)
  error in anyjson setup command: use_2to3 is invalid.
  ----------------------------------------
WARNING: Discarding http://mirrors.cloud.aliyuncs.com/pypi/packages/c3/4d/d4089e1a3dd25b46bebdb55a992b0797cff657b4477bc32ce28038fdecbc/anyjson-0.3.3.tar.gz#sha256=37812d863c9ad3e35c0734c42e0bf0320ce8c3bed82cd20ad54cb34d158157ba (from http://mirrors.cloud.aliyuncs.com/pypi/simple/anyjson/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement anyjson==0.3.3 (from versions: 0.1, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.3, 0.3.1, 0.3.2, 0.3.3)
ERROR: No matching distribution found for anyjson==0.3.3

pip install --upgrade setuptools==57.5.0

pip install anyjson==0.3.3 

7.TypeError: config_from_object() got an unexpected keyword argument 'namespace'

celery版本问题

pip install celery==4.4.7

8."Redis transport requires redis-py versions 3.2.0 or later. You have 2.10.6"

redis版本问题

pip install redis==3.2.0

9.TypeError: expected str, bytes or os.PathLike object, not NoneType

启动命令路径错误

celery multi start w1 -A djbbs -l info --logfile="./celerylog.log" --pidfile="./celerypid.pid"

10.consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 24.00 seconds... (12/100)

celery与python版本冲突

python3.6的安装pip install celery==4.2.1

python3.9的安装pip install celery==4.4.7

11.Error: Unable to load celery application. Module 'djearth' has no attribute

要在项目文件下运行 

参考文章:https://blog.csdn.net/sanyuedexuanlv/article/details/88052884https://blog.csdn.net/weixin_44420527/article/details/108877096https://blog.csdn.net/weixin_44110998/article/details/103498932https://blog.csdn.net/Coxhuang/article/details/86921407