Django+Celery+Redis 实现异步,定时任务

版本

• django-admin --version=>2.0.5
• celery --version=>4.2.0 (windowlicker)
• redis-cli --version=>redis-cli 3.2.12

开始

• 创建django项目
  django-admin startproject test
• 创建admin管理员
  python manage.py createsuperuser根据提示创建管理员
• 在django项目的settings.py文件引入django-celery-beat包

INSTALLED_APPS = [
    ...
    'django_celery_beat',
    ...
]

• 创建celery.py文件(与settings同级)

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'oaapi.settings')
app = Celery('test')
app.config_from_object('test.celery_setting', namespace='CELERY')
app.autodiscover_tasks()

• 同级目录创建celery_setting.py文件并编写celery配置

import kombu	# 我使用的的是kombu的Queue对象创建消息队列
HOST = '127.0.0.1'
PORT = '6379'
PASSWORD = 'PASSWORD'
DB = '2'
CELERY_BROKER_URL = 'redis://:%s@%s:%s/%s' % (PASSWORD, HOST, PORT, DB)
HIGH_QUEUE = kombu.Queue('high',  kombu.Exchange('high'), routing_key='high')
LOW_QUEUE = kombu.Queue('low', kombu.Exchange('low'), routing_key='low')
CELERY_TASK_DEFAULT_QUEUE = LOW_QUEUE.name
CELERY_TASK_QUEUES = (HIGH_QUEUE, LOW_QUEUE)
HIGH_PRIORITY = 0
LOW_PRIORITY = 1
BROKER_TRANSPORT_OPTIONS = {
  'priority_steps': (HIGH_PRIORITY, LOW_PRIORITY),
}
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TASK_RESULT_EXPIRES = 3600
CELERYD_TASK_TIME_LIMIT = 30
CELERYD_MAX_TASKS_PER_CHILD = 50000
CELERY_TIMEZONE = 'Asia/Shanghai'
CELERY_TASK_IGNORE_RESULT = False
CELERYD_PREFETCH_MULTIPLIER = 6
CELERYD_TASK_SOFT_TIME_LIMIT = 300  # 5min
DJANGO_CELERY_BEAT_TZ_AWARE = False

• 在同级的__init__.py引入celery

from .celery import app as celery_app
__all__ = ['celery_app']

• 创建项目app并文件目录中创建tasks.py编写需要执行的任务

# 发送邮件
import os
from django.core.mail import send_mail
os.environ['DJANGO_SETTINGS_MODULE'] = 'test.settings'
@shared_task(queue='low')	# 定时任务装饰器
def test_send_mail():
	send_mail()	# djang自带的发送邮件方法

• 迁移,运行django项目并前往django的admin页面使用之前创建的管理员账号登录
  

• 对定时任务进行配置
  

• 启动worker
  celery multi start 16 -A test -l info -Q:1-16 high -c 1 -Ofair --pidfile=/tmp/high%n.pid -f /var/log/celery_worker_log&&celery multi start 16 -A test -l info -Q:1-16 low -c 1 -Ofair --pidfile=/tmp/high%n.pid -f /var/log/celery_worker_log

• 启动定时任务
  celery -A test beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler

• 使用flower查看运行情况
  celery flower -A test --broker=redis://127.0.0.1:6379/2