Celery是什么?
Celery是个异步分布式任务队列。
通过Celery在后台跑任务并不像用线程那么的简单,但是用Celery的话,能够使应用有较好的可扩展性,因为Celery是个分布式架构。下面介绍Celery的三个核心组件。
- 生产者(Celery client)。生产者(Celery client)发送消息。在Flask上工作时,生产者(Celery client)在Flask应用内运行。
- 消费者(Celery workers)。消费者用于处理后台任务。消费者(Celery client)可以是本地的也可以是远程的。我们可以在运行Flask的server上运行一个单一的消费者(Celery workers),当业务量上涨之后再去添加更多消费者(Celery workers)。
- 消息传递者(message broker)。生产者(Celery client)和消费者(Celery workers)的信息的交互使用的是消息队列(message queue)。Celery支持若干方式的消息队列,其中最常用的是RabbitMQ和Redis.
话不多说上代码先。
版本要球:
Flask==1.0.2
celery==3.1.21
redis==2.10.5
代码:
新建celery_app模块与app同级别 如图:
celery_app模块下有两个文件 tasks.py __init__.py:
tasks.py
#! /usr/bin/env python
# -*- coding:utf-8 -*-
"""
__title__ = ''
__author__ = 'BY'
__mtime__ = '2018/11/30'
"""
from app.models.article import ArticleSource
from app.user.services.services_article_Info import ServicesArticleInfo
from . import celery
import os
@celery.task
def a_task():
"""
:return:
"""
from app import create_app, db
app = create_app( os.getenv( 'development' ) or 'default' )
article_source = ArticleSource( name="116" )
db.session.add( article_source )
db.session.commit()
db.session.close()
return True
@celery.task
def create_article_task(crawl_id, operator_id, user_id ):
"""
:param crawl_id:
:param operator_id:
:param user_id:
:return:
"""
from app import create_app, db
from app.article.services.article_service import ArticleService
app = create_app( os.getenv( 'development' ) or 'default' )
article_service = ArticleService()
article_service.create_article_by_crawl(crawl_id=crawl_id, operator_id=operator_id, user_id=user_id)
return True
@celery.task
def delete_article_task(creator_id_list):
"""
:param creator_id_list:
:return:
"""
from app import create_app, db
app = create_app(os.getenv( 'development' ) or 'default' )
services_article_info = ServicesArticleInfo()
services_article_info.delete_article(creator_id_list)
return True
@celery.task
def delete_comments_task(commenter_id_list):
"""
:param commenter_id_list:
:return:
"""
from app import create_app, db
app = create_app(os.getenv( 'development' ) or 'default' )
services_article_info = ServicesArticleInfo()
services_article_info.delete_comments(commenter_id_list)
return True
__init__.py:
#! /usr/bin/env python
# -*- coding:utf-8 -*-
"""
__title__ = ''
__author__ = 'BY'
__mtime__ = '2018/11/30'
"""
from celery import Celery, platforms
from setting import redis_url
platforms.C_FORCE_ROOT = True
celery = Celery( 'celery_app',
broker=redis_url,
backend=redis_url,
include=['celery_app.tasks', "celery_app.user_tasks"],
C_FORCE_ROOT=True,
)
运行celery:
celery worker -A celery_app.celery --loglevel=info