Celery - Task 根据任务状态执行不同的操做，和重试操作的模拟实验

背景

实验背景，拥有一堆任务需要分组分发执行。但是，糟糕的情况是每个任务的执行，大概率都是失败的。那么，如果对失败任务进行数次重试，以及如何对保存失败和成功的异步任务信息。是需要搞定的问题。

实验要求：模拟子任务的数次失败。一堆任务按照分组进行执行，且每个分组之间需要顺次执行，不可以混乱执行。并且，保证每个失败任务拥有三次的重试操作。

实验

服务端源码:

# -*- utf-8 -*-
from __future__ import absolute_import, unicode_literals, print_function
from .config import app

import os
import time
import random
import fcntl
import json
from datetime import datetime, timedelta

from celery.utils.log import get_task_logger
from celery import Task, group

logger = get_task_logger(__name__)

def func(x):
    if x in [1, 2, 3, 4]:
        raise

def write_file(x):
    date = datetime.now()
    date = '%04d%02d%02d' % (date.year, date.month, date.day)
    filename = os.path.join('./', 'test_log_%s.log' % date)

    with open(filename, 'a') as fh:
        fcntl.flock(fh.fileno(), fcntl.LOCK_EX)
        fh.write(json.dumps(x, ensure_ascii = False) + '\n')

# 继承Task类，重写on_success, on_failure, on_retry方法
class MyTask(Task):
    def on_success(self, retval, task_id, *args, **kwargs):
        _info = 'save %s success' % args[0]
        print(_info)
        write_file(_info)

    def on_failure(self, exc, task_id, *args, **kwargs):
        _info = 'save %s failed, roll back' % args[0]
        print(_info)
        write_file(_info)

    def on_retry(self, exc, task_id, *args, **kwargs):
        _info = 'retry %s' % args[0]
        print(_info)

# 绑定任务上下文，并定义基类
@app.task(bind = True, base = MyTask)
def save(self, s):
    #logger.info(self.request.__dict__)
    try:
    	# 模拟任务的大概率异常报错
        x = random.randint(1, 5)
        func(x)
    except Exception as e:
    	# 添加任务的失败重试
        raise self.retry(exc = e, countdown = 1, max_retries = 3)

@app.task()
def distribution(indexs):
    for i in range(0, len(indexs), 5):
        start = i
        end = (i + 5) if (i + 5) <= len(indexs) else len(indexs)

        L = []
        for j in range(start, end):
            L.append(save.s(indexs[j]))

        res = group(L)()

        while not res.ready():
            time.sleep(1)

客户端源码:

from proj.tasks_1 import distribution
from celery import group

import time
from threading import Thread

def async(f):
    def wrapper(*args, **kwargs):
        thr = Thread(target = f, args = args, kwargs = kwargs)
        thr.start()
    return wrapper

@async
def func(indexs):
   res = distribution(indexs)

def middle_func(indexs):
   func(indexs)
   return 'hello world'

if __name__ == '__main__':
   indexs = [1, 2, 3, 4, 5, 6, 7, 8]
   res = middle_func(indexs)
   print(res)

实验结果:

cat test_log_20200715.log
"save [1] success"
"save [3] success"
"save [2] failed, roll back"
"save [4] failed, roll back"
"save [5] failed, roll back"
"save [6] success"
"save [7] success"
"save [8] success"