生产者消费者模式
在实际的软件开发过程中，经常会碰到如下场景：某个模块负责产生数据，这些数据由另一个模块来负责处理（此处的模块是广义的，可以是类、函数、线程、进程等）。产生数据的模块，就形象地称为生产者；而处理数据的模块，就称为消费者。
单单抽象出生产者和消费者，还够不上是生产者消费者模式。该模式还需要有一个缓冲区处于生产者和消费者之间，作为一个中介。生产者把数据放入缓冲区，而消费者从缓冲区取出数据。
生产者消费者模式是通过一个容器来解决生产者和消费者的强耦合问题。生产者和消费者彼此之间不直接通讯，而通过消息队列（缓冲区）来进行通讯，所以生产者生产完数据之后不用等待消费者处理，直接扔给消息队列，消费者不找生产者要数据，而是直接从消息队列里取，消息队列就相当于一个缓冲区，平衡了生产者和消费者的处理能力。这个消息队列就是用来给生产者和消费者解耦的。
解耦
假设生产者和消费者分别是两个类。如果让生产者直接调用消费者的某个方法，那么生产者对于消费者就会产生依赖（也就是耦合）。将来如果消费者的代码发生变化，可能会影响到生产者。而如果两者都依赖于某个缓冲区，两者之间不直接依赖，耦合也就相应降低了。生产者直接调用消费者的某个方法，还有另一个弊端。由于函数调用是同步的（或者叫阻塞的），在消费者的方法没有返回之前，生产者只好一直等在那边。万一消费者处理数据很慢，生产者就会白白糟蹋大好时光。缓冲区还有另一个好处。如果制造数据的速度时快时慢，缓冲区的好处就体现出来了。当数据制造快的时候，消费者来不及处理，未处理的数据可以暂时存在缓冲区中。等生产者的制造速度慢下来，消费者再慢慢处理掉。
Celery
简介
Celery是基于Python开发的一个分布式任务队列框架，支持使用任务队列的方式在分布的机器/进程/线程上执行任务调度。
它采用典型的生产者-消费者模式，主要由三部分组成：broker（消息队列）、workers（消费者：处理任务）、backend（存储结果）。
实际应用中，用户从Web前端发起一个请求，我们只需要将请求所要处理的任务放入任务队列broker中，由空闲的worker去处理任务即可，处理的结果会暂存在后台数据库backend中。我们可以在一台机器或多台机器上同时起多个worker进程来实现分布式地并行处理任务。
Celery 用消息通信，通常使用中间人（Broker）在客户端和职程间斡旋。这个过程从客户端向队列添加消息开始，之后中间人把消息派送给职程，职程对消息进行处理。
Celery 系统可包含多个职程和中间人，以此获得高可用性和横向扩展能力。
消息中间件
Celery本身不提供消息服务，但是可以方便的和第三方提供的消息中间件集成。包括，RabbitMQ, Redis, MongoDB (experimental), Amazon SQS (experimental),CouchDB (experimental), SQLAlchemy (experimental),Django ORM (experimental), IronMQ。
任务执行单元
Worker是Celery提供的任务执行的单元，worker并发的运行在分布式的系统节点中。
任务结果存储
Task result store用来存储Worker执行的任务的结果，Celery支持以不同方式存储任务的结果，包括AMQP, Redis，memcached, MongoDB，SQLAlchemy, Django ORM，Apache Cassandra, IronCache。
任务队列
任务队列是一种在线程或机器间分发任务的机制。
消息队列
消息队列的输入是工作的一个单元，称为任务，独立的职程（Worker）进程持续监视队列中是否有需要处理的新任务。
安装
安装django
pip install django
安装celery
pip install celery
安装消息中间件（如Redis）
Django中celery实现
创建项目和app
django-admin.py startproject celeryProject cd celeryProject django-admin.py startapp mainapp
配置settings.py
"""
Django settings for qms project.
Generated by 'django-admin startproject' using Django 3.1.3.
For more information on this file, see
https://docs.djangoproject.com/en/3.1/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.1/ref/settings/
"""
import os
from pathlib import Path
from celery.schedules import crontab, timedelta # celery配置
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/3.1/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '!ixbyuwjq(my2f@%tnq2@@oq(iedrkd47mkh+^ekcfs@_rl&pa'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = ['*']
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'corsheaders',
'mainapp',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'corsheaders.middleware.CorsMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.middleware.common.CommonMiddleware'
]
ROOT_URLCONF = 'celeryProject.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [os.path.join(BASE_DIR, 'templates')]
,
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'celeryProject.wsgi.application'
# Database
# https://docs.djangoproject.com/en/3.1/ref/settings/#databases
DATABASE_ROUTERS = ['celeryProject.database_router.DatabaseAppsRouter']
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
},
}
DATABASE_APPS_MAPPING = {
'django_apscheduler': 'default',
'admin': 'default',
'auth': 'default',
'contenttypes': 'default',
'sessions': 'default',
'mainapp': 'default'
}
# Password validation
# https://docs.djangoproject.com/en/3.1/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/3.1/topics/i18n/
LANGUAGE_CODE = 'zh-Hans'
TIME_ZONE = 'Asia/Shanghai'
USE_I18N = True
USE_L10N = True
USE_TZ = False
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.1/howto/static-files/
STATIC_URL = '/static/'
MEDIA_ROOT = os.path.join(BASE_DIR, "media")
MEDIA_URL = "/media/"
STATIC_ROOT = os.path.join(BASE_DIR, "static")
UPLOAD_ROOT = os.path.join(BASE_DIR, "media/upload")
UPLOAD_ROOT_AUDIT = os.path.join(BASE_DIR, 'media/audit')
UPLOAD_ROOT_ZIP = os.path.join(BASE_DIR, 'media/zip')
CORS_ALLOW_CREDENTIALS = True
CORS_ORIGIN_ALLOW_ALL = True
CORS_ALLOW_METHODS = (
'DELETE',
'GET',
'OPTIONS',
'PATCH',
'POST',
'PUT',
'VIEW',
)
CORS_ALLOW_HEADERS = (
'XMLHttpRequest',
'X_FILENAME',
'accept-encoding',
'authorization',
'content-type',
'dnt',
'origin',
'user-agent',
'x-csrftoken',
'x-requested-with',
'Token'
)
APPEND_SLASH = True
# celery配置
CELERY_TIMEZONE = 'Asia/Shanghai'
CELERY_ENABLE_UTC = True
if os.name != 'nt':
CELERY_ACCEPT_CONTENT = ['application/json', ]
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
else:
CELERY_ACCEPT_CONTENT = ['pickle', ]
CELERY_TASK_SERIALIZER = 'pickle'
CELERY_RESULT_SERIALIZER = 'pickle'
CELERY_QUEUES = {
'default': {
'exchange': 'default',
'exchange_type': 'direct',
'routing_key': 'default',
},
'mianQueue': {
'exchange': 'main',
'exchange_type': 'direct',
'routing_key': 'main.#',
},
}
CELERY_ROUTES = {
'publicModule.tasks.get_plc_info': {
'queue': 'plcQueue',
'routing_key': 'plc_info'
},
'deviceModule.tasks.voice_broadcast': {
'queue': 'voiceQueue',
'routing_key': 'voice_broadcast'
},
}
# 定时任务配置
CELERYBEAT_SCHEDULE = {
'get_plc_info': {
'task': 'publicModule.tasks.get_plc_info',
'schedule': timedelta(seconds=60)
# 'schedule': crontab(minute=0, hour=1, day_of_month=1)
}
}新建celery.py（创建celery对象）
# celery.py
import os
from celery import Celery, platforms
from celeryProject import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'celeryProject.settings')
app = Celery('celeryProject', broker='redis://127.0.0.1:6379/1', backend='redis://127.0.0.1:6379/2',
include=['mainapp.tasks'])
app.config_from_object('django.conf:settings')
platforms.C_FORCE_ROOT = True
app.autodiscover_tasks(['mainapp', ])
# app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)新建tasks.py，必须建在各app的根目录下，且不能随意命名（创建task）
import os
from datetime import datetime
from HslCommunication import MelsecMcNet
from common.requests2server import approvalget
from deviceModule.models import Device, DeviceStateRecord
from publicModule.models import PLCInfoToEquipmentClause, PLCInfo, PLCBasicInfo, TransitServerInfo
from celeryProject.celery import app
@app.task()
def get_plc_info():
transit_server_infos = TransitServerInfo.objects.all()
for transit_server_info in transit_server_infos:
if transit_server_info.server_ip and transit_server_info.server_port:
res = approvalget(
f'http://{transit_server_info.server_ip}:{transit_server_info.server_port}/plcModule/queryDeviceState/')
if res and res.get('code') == 200:
device_states = res.get('data', [])
for device_state in device_states:
if device_state.get('device_id'):
devices = Device.objects.filter(id=device_state.get('device_id'))
if devices:
if devices[0].current_state not in [3, 4, 6] and \
devices[0].current_state != device_state.get('current_state'):
devices.update(current_state=device_state.get('current_state'))
DeviceStateRecord.objects.create(device_id=devices[0].id,
state=device_state.get('current_state'),
record_time=datetime.now())
else:
Device.objects.exclude(current_state__in=[3, 4, 6]).update(current_state=0)
plc_infos = PLCInfo.objects.exclude(transit_server_info_id=None)
for plc_info in plc_infos:
plc_basic_info = PLCBasicInfo.objects.filter(id=plc_info.plc_basic_info_id).first()
if plc_basic_info:
plc_info_to_equipment_clauses = PLCInfoToEquipmentClause.objects.filter(plc_info_id=plc_info.id)
if plc_info_to_equipment_clauses:
if plc_basic_info.plc_protocol == 1:
melsec_mc_net = MelsecMcNet(plc_info.plc_ip, plc_info.plc_port)
elif plc_basic_info.plc_protocol == 2:
pass
for plc_info_to_equipment_clause in plc_info_to_equipment_clauses:
devices = Device.objects.filter(id=plc_info_to_equipment_clause.equipment_clause_id)
if devices:
current_state = devices[0].current_state
if plc_basic_info.plc_protocol == 1:
run_signal = melsec_mc_net.ReadBool(plc_info_to_equipment_clause.run_signal)
wait_signal = melsec_mc_net.ReadBool(plc_info_to_equipment_clause.wait_signal)
alerting_signal = melsec_mc_net.ReadBool(plc_info_to_equipment_clause.alerting_signal)
if run_signal:
if wait_signal:
current_state = 1
else:
current_state = 2
else:
current_state = 5
elif plc_basic_info.plc_protocol == 2:
pass
if devices[0].current_state not in [3, 4, 6] and devices[0].current_state != current_state:
devices.update(current_state=current_state)
DeviceStateRecord.objects.create(device_id=devices[0].id,
state=current_state,
record_time=datetime.now())
@app.task()
def backups(source_path, target_path):
res = os.system(f'cp -r {source_path} {target_path}')
return res启动服务
python manage.py runserver
启动worker
celery -A celeryProject worker -l info --pool=solo # 开启worker的实质实际上就是执行app=Celery(...)语句，可以使用 --concurrency=个数 来限制每个消费者可以并行的线程数
启动beat
celery -A celeryProject beat -l info # 开启定时任务
扫一扫
465
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
