Django|Redis|Celery 配置与部署

引言

在django项目中，经常需要使用异步执行定时任务，如定时发送邮件、清空数据等，celery是很好的选择。本文介绍了使用redis作为broker的celery的使用和部署。

安装

Django + Redis

django与celery相关的安装已经在之前写过，这里不再赘述。详见Django|Nginx|Uwsgi|Redis|Websocket配置、使用与部署

Celery

pip install celery-with-redis

配置

在settings.py 所在目录添加celery.py ，添加如下内容：

#coding:utf-8
from __future__ import absolute_import, unicode_literals

from celery import Celery
from django.conf import settings
import os

#获取当前文件夹名，即为该Django的项目名
project_name = os.path.split(os.path.abspath('.'))[-1]
project_settings = '%s.settings' % project_name

#设置环境变量
os.environ.setdefault('DJANGO_SETTINGS_MODULE', project_settings)

#实例化Celery
app = Celery(project_name)

#使用django的settings文件配置celery
app.config_from_object('django.conf:settings')

#Celery加载所有注册的应用
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

在settings.py 所在目录的__init__.py ，添加如下内容：

#coding:utf-8
from __future__ import absolute_import, unicode_literals

#引入celery实例对象
from .celery import app as celery_app 

在settings.py 中加入：

#celery settings
#celery中间人 redis://redis服务所在的ip地址:端口/数据库号
BROKER_URL = 'redis://localhost:6379/0'
#celery结果返回，可用于跟踪结果
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'

#celery内容等消息的格式设置
CELERY_ACCEPT_CONTENT = ['application/json',]
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

#celery时区设置，使用settings中TIME_ZONE同样的时区
CELERY_TIMEZONE = TIME_ZONE

使用

在你的任意app下添加tasks.py 文件，写入:

from __future__ import absolute_import
from proj.celery import app


@app.task(name='tasks.exampleName')
def example():
    ...
    ...

然后在settings.py中加入：

CELERYBEAT_SCHEDULE = {
    'anyName': {
        'task': 'tasks.exampleName',
        'schedule': timedelta(seconds=30),
        'args': ()
    },
}

其中'schedule'为执行间隔或时间，使用datetime.timedelta()用来设定执行间隔，或使用celery.schedules.crontab()来设定执行时间。

在本地运行时，只需要在项目路径执行：

celery -A proj worker -l info

celery -A proj beat -l info

即可。

部署

在部署到服务器上时，需要使celery作为守护进程执行。celery本身不会以守护进程运行，需要额外的工具。

generic

pip install generic

下载官方脚本放入/etc/init.d/下。

新建文件/etc/default/celeryd进行配置:

# Names of nodes to start
#   most will only start one node:
CELERYD_NODES="worker1"
#   but you can also start multiple and configure settings
#   for each in CELERYD_OPTS (see `celery multi --help` for examples).
#CELERYD_NODES="worker1 worker2 worker3"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="proj"
# or fully qualified:
#CELERY_APP="proj.tasks:app"

# Where to chdir at start.
CELERYD_CHDIR="/opt/Myproject/"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists, e.g. nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery"

# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

然后执行/etc/init.d/celeryd {start|stop|restart}即可开启／停止／重启celery.