Celery - Example01处理指定数量的任务

背景

特定需求，对于某类任务一次性仅处理固定上线数量的任务。等待此类任务执行完成后，再做下一次的处理。

解决办法

对于同一进程中的task异步任务的处理，MainProcess会依次获取task交给worker执行。直至等待此任务执行完毕后，MainProcess才会提取获取下一个task交给worker执行。根据这一工作原理，可以顺次处理若干个任务且每个任务之间互斥，任意两个任务不会同时被worker执行。

那么，如果每次处理多个任务，则可以将多个任务利用Group进行打包。则MainProcess会依次获取Group并且将Group中的task分发给若干个worker并发执行。直至等待Group中的任务全部执行完毕，MainProcess才会提取下一个Group交给worker执行。同理，各个worker间的任务互斥。

所以， 每次处理指定数量的任务问题就解决了。

实践场景

做1到100的累加分别将1-10，11-20，…，91-100打包为1,2,…10组。交给worker并发执行，等待worker执行完成后执行下一组任务。

from proj.tasks import add, less, mul, exc
from celery import group

if __name__ == '__main__':
   for i in range(0, 100, 10):
       res = group(add.s(j, j) for j in range(i, i + 10))()

       while True:													# 等待任务执行完成
           if res.ready() == True:
               print(res.get())
               break

查看任务执行结果：

[2020-07-02 19:31:48,736: INFO/MainProcess] Received task: proj.tasks.add[b87713ab-0294-4976-95b1-c7ef14826509]
[2020-07-02 19:31:48,743: INFO/MainProcess] Received task: proj.tasks.add[857738a3-ea33-4c5b-9336-c6551447666c]
[2020-07-02 19:31:48,749: INFO/MainProcess] Received task: proj.tasks.add[02fcdc2a-753c-499f-a31a-7adca6ce7aeb]
[2020-07-02 19:31:48,755: INFO/MainProcess] Received task: proj.tasks.add[17ed681a-7316-4e69-bebe-cc69997fcded]
[2020-07-02 19:31:48,761: INFO/MainProcess] Received task: proj.tasks.add[a14f0377-cf5f-401f-a9a7-c1bca03d6a56]
[2020-07-02 19:31:48,764: INFO/MainProcess] Received task: proj.tasks.add[69b986b5-844b-4749-9e75-d5ef1ad9e1e0]
[2020-07-02 19:31:48,768: INFO/MainProcess] Received task: proj.tasks.add[6fb09c96-188a-4142-8666-fe0a65731f21]
[2020-07-02 19:31:48,771: INFO/MainProcess] Received task: proj.tasks.add[d3a341ca-68d2-407f-8eaa-69f9c5d56e0d]
[2020-07-02 19:31:48,774: INFO/MainProcess] Received task: proj.tasks.add[3323863d-2d4a-47bd-b2f2-ed89b42dc7d3]
[2020-07-02 19:31:48,778: INFO/MainProcess] Received task: proj.tasks.add[3b6cac53-49de-4ca7-b7d4-6c0860a044da]
[2020-07-02 19:31:53,742: INFO/ForkPoolWorker-10] Task proj.tasks.add[b87713ab-0294-4976-95b1-c7ef14826509] succeeded in 5.00497811s: 0
[2020-07-02 19:31:53,750: INFO/ForkPoolWorker-5] Task proj.tasks.add[857738a3-ea33-4c5b-9336-c6551447666c] succeeded in 5.005962508s: 2
[2020-07-02 19:31:53,755: INFO/ForkPoolWorker-3] Task proj.tasks.add[02fcdc2a-753c-499f-a31a-7adca6ce7aeb] succeeded in 5.005489749s: 4
[2020-07-02 19:31:53,761: INFO/ForkPoolWorker-2] Task proj.tasks.add[17ed681a-7316-4e69-bebe-cc69997fcded] succeeded in 5.004969764s: 6
[2020-07-02 19:31:53,768: INFO/ForkPoolWorker-1] Task proj.tasks.add[a14f0377-cf5f-401f-a9a7-c1bca03d6a56] succeeded in 5.005773889s: 8
[2020-07-02 19:31:53,771: INFO/ForkPoolWorker-4] Task proj.tasks.add[69b986b5-844b-4749-9e75-d5ef1ad9e1e0] succeeded in 5.005366905s: 10
[2020-07-02 19:31:53,774: INFO/ForkPoolWorker-6] Task proj.tasks.add[6fb09c96-188a-4142-8666-fe0a65731f21] succeeded in 5.005731325s: 12
[2020-07-02 19:31:53,778: INFO/ForkPoolWorker-8] Task proj.tasks.add[d3a341ca-68d2-407f-8eaa-69f9c5d56e0d] succeeded in 5.005707748s: 14
[2020-07-02 19:31:53,780: INFO/ForkPoolWorker-9] Task proj.tasks.add[3323863d-2d4a-47bd-b2f2-ed89b42dc7d3] succeeded in 5.005099613s: 16
[2020-07-02 19:31:53,784: INFO/ForkPoolWorker-7] Task proj.tasks.add[3b6cac53-49de-4ca7-b7d4-6c0860a044da] succeeded in 5.005365442s: 18
[2020-07-02 19:31:59,301: INFO/MainProcess] Received task: proj.tasks.add[82b9d9d5-3a2a-48ff-99f6-d32563b22b0e]
[2020-07-02 19:31:59,308: INFO/MainProcess] Received task: proj.tasks.add[f72babb8-6613-44b9-8a4f-e0ad4e938d34]
[2020-07-02 19:31:59,312: INFO/MainProcess] Received task: proj.tasks.add[70efe11a-7ddd-4326-b10c-9e00a801b32e]
[2020-07-02 19:31:59,318: INFO/MainProcess] Received task: proj.tasks.add[581859a6-282d-4d2d-acd7-6c9f9a1498b9]
[2020-07-02 19:31:59,324: INFO/MainProcess] Received task: proj.tasks.add[37474fdb-1085-4f9b-893b-a9b1e0478070]
[2020-07-02 19:31:59,328: INFO/MainProcess] Received task: proj.tasks.add[82cdaa8b-7a90-449d-bbb6-59122824568a]
[2020-07-02 19:31:59,331: INFO/MainProcess] Received task: proj.tasks.add[c491413d-45e4-49ce-a856-f0e2672235e0]
[2020-07-02 19:31:59,335: INFO/MainProcess] Received task: proj.tasks.add[dcb77bd3-cdbb-4c4a-bf80-dbcdf0e8e732]
[2020-07-02 19:31:59,338: INFO/MainProcess] Received task: proj.tasks.add[16c7a873-f639-47b0-aa60-d288db5099d9]
[2020-07-02 19:31:59,341: INFO/MainProcess] Received task: proj.tasks.add[08dc0ee8-e3f9-46d3-8e03-2783811df5d4]
[2020-07-02 19:32:04,309: INFO/ForkPoolWorker-10] Task proj.tasks.add[82b9d9d5-3a2a-48ff-99f6-d32563b22b0e] succeeded in 5.005744526s: 20
[2020-07-02 19:32:04,314: INFO/ForkPoolWorker-5] Task proj.tasks.add[f72babb8-6613-44b9-8a4f-e0ad4e938d34] succeeded in 5.005296645s: 22
[2020-07-02 19:32:04,318: INFO/ForkPoolWorker-3] Task proj.tasks.add[70efe11a-7ddd-4326-b10c-9e00a801b32e] succeeded in 5.004933472s: 24
[2020-07-02 19:32:04,324: INFO/ForkPoolWorker-2] Task proj.tasks.add[581859a6-282d-4d2d-acd7-6c9f9a1498b9] succeeded in 5.005076992s: 26
[2020-07-02 19:32:04,330: INFO/ForkPoolWorker-1] Task proj.tasks.add[37474fdb-1085-4f9b-893b-a9b1e0478070] succeeded in 5.005172691s: 28
[2020-07-02 19:32:04,334: INFO/ForkPoolWorker-4] Task proj.tasks.add[82cdaa8b-7a90-449d-bbb6-59122824568a] succeeded in 5.00527741s: 30
[2020-07-02 19:32:04,338: INFO/ForkPoolWorker-6] Task proj.tasks.add[c491413d-45e4-49ce-a856-f0e2672235e0] succeeded in 5.005129477s: 32
[2020-07-02 19:32:04,341: INFO/ForkPoolWorker-8] Task proj.tasks.add[dcb77bd3-cdbb-4c4a-bf80-dbcdf0e8e732] succeeded in 5.005620185s: 34
[2020-07-02 19:32:04,344: INFO/ForkPoolWorker-9] Task proj.tasks.add[16c7a873-f639-47b0-aa60-d288db5099d9] succeeded in 5.005512142s: 36
[2020-07-02 19:32:04,348: INFO/ForkPoolWorker-7] Task proj.tasks.add[08dc0ee8-e3f9-46d3-8e03-2783811df5d4] succeeded in 5.005837253s: 38

遗留的问题：

刚刚处理时，若不进行等待，发现服务端的处理并非十分有规律。这是什么原因引起的呢？