【Local/Docker/K8S/Racher】Local/Docker/K8S/Racher/Docker-compose安装Celery并启动异步任务-20220820

i see the future

已于 2022-08-24 23:15:17 修改

阅读量735

点赞数

于 2022-08-20 17:41:27 首次发布

本文链接：https://blog.csdn.net/m0_46629123/article/details/126405866

版权

django 同时被 3 个专栏收录

210 篇文章 21 订阅

订阅专栏

docker

39 篇文章 1 订阅

订阅专栏

kubernetes

36 篇文章 1 订阅

订阅专栏

1.Local安装Celery

参考链接https://docs.celeryq.dev/en/master/django/first-steps-with-django.html#starting-the-worker-process
debug:Django使用celery启动出错Unable to load celery application. Module ‘project_name‘ has no attribute ‘celery‘https://blog.csdn.net/weixin_41660414/article/details/108295843

步骤0：安装好redis，并启用。

参考链接https://blog.csdn.net/m0_46629123/article/details/126377540

步骤1：安装好celery，并启用。

pip install celery
pip install eventlet
pip install “celery[redis,auth,msgpack]”
在这里插入图片描述

步骤2：proj/proj/celery.py

在这里插入图片描述
celery.py

from __future__ import absolute_import, unicode_literals

import os

from celery import Celery, shared_task

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'GAGA.settings')

app = Celery('GAGA')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

步骤3：proj/proj/init.py

__ init __.py

from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ('celery_app',)

步骤4：proj/proj/settings.py

settings.py

CELERY_BROKER_URL = 'redis://10.41.241.169:32621/1'
CELERY_RESULT_BACKEND = 'redis://10.41.241.169:32621/1'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Asia/Shanghai'
CELERYD_MAX_TASKS_PER_CHILD = 10
CELERYD_LOG_FILE = os.path.join(BASE_DIR, "logs", "celery_work.log")
CELERYBEAT_LOG_FILE = os.path.join(BASE_DIR, "logs", "celery_beat.log")

步骤5：proj/app/tasks.py

tasks.py

from __future__ import absolute_import, unicode_literals
from celery import shared_task
import time
from celery import Celery

# 第一个参数 是当前脚本的名称，第二个参数 是 broker 服务地址
app = Celery('tasks', backend='redis://10.41.241.169:32621/1', broker='redis://10.41.241.169:32621/1')

@shared_task
def async_action(x, y):
    print(x+y)
    time.sleep(10)
    print(x+y+1)
    return x + y

步骤6：proj/app/views.py中async_action(100, 1)改为async_action.delay(100, 1)

views.py

from .tasks import async_action

# dashboard
def dashboard(request):
    # 异步
    async_action.delay(100, 1)
    #async_action(100, 1)

步骤7：启动redis，在根目录下启动django、celery、flower。

启动django
在这里插入图片描述

启动celery（注意-A GAGA）

celery -A GAGA worker --loglevel=INFO -P eventlet

在这里插入图片描述

启动flower（注意-A GAGA）

celery -A GAGA flower --broker=redis://10.41.241.169:32621/1

在这里插入图片描述

2.Docker安装Celery(shell启动)

步骤0：安装好redis，并启用。

参考链接https://blog.csdn.net/m0_46629123/article/details/126377540

步骤1：更新requirement.txt

Django==3.2.7
openpyxl==3.0.9
Pillow==8.3.2
django-simpleui
beautifulsoup4==4.10.0
requests==2.26.0
python-keycloak==2.1.1
schedule==0.6.0
django_redis
celery
eventlet
celery[redis,auth,msgpack]

步骤2：启动应用后，进入容器后台shell启动celery。

输入celery -A GAGA worker --loglevel=INFO -P eventlet。
在这里插入图片描述

3.K8S安装Celery(shell启动)

步骤0：安装好redis，并启用。

参考链接https://blog.csdn.net/m0_46629123/article/details/126377540

步骤1：K8S部署yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gaga-dp4
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gaga-pod4
  template:
    metadata:
      labels:
        app: gaga-pod4
    spec:
      containers:
      - name: gaga-web4
        # env:
        #   - name: DJANGO_DEBUG
        #     value: 'False'
        image: seasonzhang/gaga_meeting:0.8
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 200m
            memory: 200Mi
        ports:
        - containerPort: 8000
          name: gaga-web-pod4
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: gaga-svc4
spec:
  type: LoadBalancer
  ports:
   - port: 8400
     targetPort: 8000
     protocol: TCP
  selector:
    app: gaga-pod4

步骤2：启动应用后，进入pod后台shell启动celery。

输入celery -A GAGA worker --loglevel=INFO -P eventlet。
在这里插入图片描述

4.Racher安装Celery(shell启动)

步骤0：安装好redis，并启用。

步骤1：Racher部署应用

在这里插入图片描述

步骤2：启动应用后，进入pod后台shell启动celery。

5.Docker-compose安装Celery，并启动异步任务

情景1：仅启动web和celery，但redis另外部署。

version: "3.2"
services:
  web:
    image: "seasonzhang/gaga_meeting:0.9"
    container_name: gaga_meeting-web
    entrypoint: ["/bin/sh","/code/package/start.sh"]
    ports:
      - "8000:8000"
  celery:
    image: "seasonzhang/gaga_meeting:0.9"
    container_name: gaga_meeting-celery
    entrypoint: ["/bin/sh","/code/package/start-celery.sh"]

在这里插入图片描述

情景2之后的settings.py（redis由container产生，不引用外部redis）

# settings.py

```python
"""
Django settings for GAGA project.

Generated by 'django-admin startproject' using Django 3.2.7.

For more information on this file, see
https://docs.djangoproject.com/en/3.2/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.2/ref/settings/
"""

from pathlib import Path
import os
from django.utils.translation import gettext_lazy as _


# 环境变量(含默认值)
REDIS_LOCATION = os.environ.get('REDIS_LOCATION',"redis://redis:6379/1")
#REDIS_LOCATION = os.environ.get('REDIS_LOCATION',"redis://10.41.241.169:32621/1")
# REDIS_LOCATION = os.environ.get('REDIS_LOCATION',"redis://106.52.14.84:32222/1")
# REDIS_PASSWORD = os.environ.get('REDIS_PASSWORD','redis@654321')
DJANGO_DEBUG = os.environ.get('DJANGO_DEBUG', True)
# REDIS_LOCATION = "redis://106.52.14.84:32222/1"
# REDIS_PASSWORD = "redis@654321"


# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
LOG_DIR = Path(__file__).resolve().parent.parent / 'log'

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'django-insecure-+=99f6x#@pq9f5=riae=8skj)6vu55vptpo9-in#yl*+2qfop@'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False
DEBUG = DJANGO_DEBUG

ALLOWED_HOSTS = ['*']





# Application definition

INSTALLED_APPS = [
    'simpleui',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'GAGA_meeting',
    'django_celery_beat',
]


MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.locale.LocaleMiddleware',### 多语言中间件
    'django.middleware.cache.UpdateCacheMiddleware',#redis中间件
    'django.middleware.common.CommonMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',#redis中间件
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'GAGA.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'GAGA.wsgi.application'



# Database
# https://docs.djangoproject.com/en/3.2/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        #'NAME': BASE_DIR / 'db.sqlite3',
        'NAME': Path(__file__).resolve().parent /'SQL' /'db.sqlite3',
    }
}


# Password validation
# https://docs.djangoproject.com/en/3.2/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

# 修改中文，Django内部设置zh_Hans方法指向中文 
LANGUAGES = [
    ('zh-hans', _('Chinese')),
    ('en', _('English')),
]

LANGUAGE_CODE = 'zh-hans' 
# LANGUAGE_CODE = 'en-us'

LOCALE_PATHS = (
    os.path.join(BASE_DIR, 'locale'),
)

# 修改中国时区
#TIME_ZONE = 'Asia/Shanghai'
#USE_TZ = False
USE_TZ = True
USE_I18N = True
USE_L10N = True



# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.2/howto/static-files/
# 通过url直接访问我在项目中的静态文件
STATIC_URL = '/static/'
# 部署静态文件时(pyhtonmanage.pycollectstatic)所有的静态文静聚合的目录
STATIC_ROOT = os.path.join(BASE_DIR, "/static/")
# STATICFILES_DIRS告诉django,首先到STATICFILES_DIRS里面寻找静态文件,其次再到各个app的static文件夹里面找
STATICFILES_DIRS = (
    os.path.join(BASE_DIR, "static"),
)

MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')



# Default primary key field type
# https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field

DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'



LOGIN_URL = '/user_login/'

DATE_FORMAT = 'Y-m-d'





LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    # 'formatters': {
    #     'simple': { # exact format is not important, this is the minimum information
    #         'format': '%(asctime)s %(name)-12s %(lineno)d %(levelname)-8s %(message)s',
    #     },
    # },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },

        'mail_admins': { # Add Handler for mail_admins for `warning` and above
            # 'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
        },
        'file': {
            'class': 'logging.FileHandler',
            'filename': os.path.join(LOG_DIR, 'admin.log'),
        },

    },

    'root': {
        'handlers': ['console', 'file','mail_admins'],
        'level': 'INFO',
    },
}



# Racher redis
# CACHES = {
#     "default": {
#         "BACKEND": "django_redis.cache.RedisCache",
#         "LOCATION": "redis://10.41.241.169:32621/1",
#         #"LOCATION": REDIS_LOCATION,
#         "OPTIONS": {
#             "CLIENT_CLASS": "django_redis.client.DefaultClient",
#             #"PASSWORD":"000415",
#             #"PASSWORD":REDIS_PASSWORD,
#             "SOCKET_CONNECT_TIMEOUT": 5,  # in seconds
#             "SOCKET_TIMEOUT": 5,  # r/w timeout in seconds
#         }
#     }
# }



# K8S redis
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": REDIS_LOCATION,
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            #"PASSWORD":"redis@654321",
            "SOCKET_CONNECT_TIMEOUT": 5,  # in seconds
            "SOCKET_TIMEOUT": 5,  # r/w timeout in seconds
        }
    }
}



# CELERY_BROKER_URL = 'redis://10.41.241.169:32621/1'
#CELERY_BROKER_URL =  "redis://106.52.14.84:32222/1"
CELERY_BROKER_URL = REDIS_LOCATION
# CELERY_RESULT_BACKEND = 'redis://10.41.241.169:32621/1'
#CELERY_RESULT_BACKEND =  "redis://106.52.14.84:32222/1"
CELERY_RESULT_BACKEND = REDIS_LOCATION
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Asia/Shanghai'
CELERYD_MAX_TASKS_PER_CHILD = 10
CELERYD_LOG_FILE = os.path.join(BASE_DIR, "logs", "celery_work.log")
CELERYBEAT_LOG_FILE = os.path.join(BASE_DIR, "logs", "celery_beat.log")


#### 情景2：启动web和celery、redis
```yaml
version: "3.2"
services:
  web:
    image: "seasonzhang/gaga_meeting:0.9"
    container_name: gaga_meeting-web
    entrypoint: ["/bin/sh","/code/package/start.sh"]
    # environment:
    #   - server_params=--settings=settings.local
    # volumes:
    #   - .:/data/recruitment
    #   - /data/logs/recruitment/
    ports:
      - "8000:8000"
    depends_on:
      - redis
      - celery
      # - flower
  redis:
    image: "redis:alpine"
    container_name: gaga_meeting-redis
    ports:
      - "6379:6379"
  celery:
    image: "seasonzhang/gaga_meeting:0.9"
    container_name: gaga_meeting-celery
    # volumes:
    #   - .:/data/recruitment
    #   - /data/logs/recruitment/
    entrypoint: ["/bin/sh","/code/package/start-celery.sh"]
    depends_on: 
      - redis
  # flower:
  #   image: "seasonzhang/gaga_meeting:0.9"
  #   container_name: gaga_meeting-flower
  #   ports:
  #     - "5555:5555"
  #   # volumes:
  #   #   - .:/data/recruitment
  #   #   - /data/logs/recruitment/
  #   entrypoint: ["/bin/sh","/code/package/start-flower.sh"]
  #   depends_on: 
  #     - redis

在这里插入图片描述
添加链接描述

情景3：启动web和celery、redis、flower（celery为4.4版本）-成功

Debug: Error: No such command ‘flower‘https://blog.csdn.net/zxz567/article/details/109324339/ >pip install celery4.4.7即可解决
Running Flower using Supervisor https://stackoverflow.com/questions/26985746/running-flower-using-supervisor >pip install flower0.9.7即可解决

version: "3.2"
services:
  web:
    image: "seasonzhang/gaga_meeting:0.9"
    container_name: gaga_meeting-web
    entrypoint: ["/bin/sh","/code/package/start.sh"]
    # environment:
    #   - server_params=--settings=settings.local
    # volumes:
    #   - .:/data/recruitment
    #   - /data/logs/recruitment/
    ports:
      - "8000:8000"
    depends_on:
      - redis
      - celery
      # - flower
  redis:
    image: "redis:alpine"
    container_name: gaga_meeting-redis
    ports:
      - "6379:6379"
  celery:
    image: "seasonzhang/gaga_meeting:0.9"
    container_name: gaga_meeting-celery
    # volumes:
    #   - .:/data/recruitment
    #   - /data/logs/recruitment/
    entrypoint: ["/bin/sh","/code/package/start-celery.sh"]
    depends_on: 
      - redis
  flower:
    image: "seasonzhang/gaga_meeting:0.9"
    container_name: gaga_meeting-flower
    ports:
      - "5555:5555"
    # volumes:
    #   - .:/data/recruitment
    #   - /data/logs/recruitment/
    entrypoint: ["/bin/sh","/code/package/start-flower.sh"]
    depends_on: 
      - redis
      - web

在这里插入图片描述

情景4：启动web和celery、redis、flower（celery为4.4版本）、volume-成功

version: "3.2"
services:
  web:
    image: "seasonzhang/gaga_meeting:0.9"
    container_name: gaga_meeting-web
    entrypoint: ["/bin/sh","/code/package/start.sh"]
    # environment:
    #   - server_params=--settings=settings.local
    volumes:
      - C:/Users/Season/Desktop/SQL:/code/package/GAGA/SQL
    ports:
      - "8000:8000"
    depends_on:
      - redis
      - celery
      # - flower
  redis:
    image: "redis:alpine"
    container_name: gaga_meeting-redis
    ports:
      - "6379:6379"
  celery:
    image: "seasonzhang/gaga_meeting:0.9"
    container_name: gaga_meeting-celery
    # volumes:
    #   - .:/code/package/GAGA/SQL
    entrypoint: ["/bin/sh","/code/package/start-celery.sh"]
    depends_on: 
      - redis
  flower:
    image: "seasonzhang/gaga_meeting:0.9"
    container_name: gaga_meeting-flower
    ports:
      - "5555:5555"
    # volumes:
    #   - .:/code/package/GAGA/SQL
    entrypoint: ["/bin/sh","/code/package/start-flower.sh"]
    depends_on: 
      - redis
      - web