Celery实践步骤

Celery实践步骤

环境

Windows 10
Celery 5.1.2
Redis Desktop Manager: http://www.520xiazai.com/soft/redis-desktop-manager.html
Redis x64-3.2.100.msi for windows: https://github.com/microsoftarchive/redis/releases

Redis Server启动

“Run as Administrator” cmd.exe:
C:\Program Files\Redis>redis-server.exe ./redis.windows.conf
它可以作为消息代理(队列)，也可以用来存储task运行结果

Celery

import time
import sys
sys.path.append('.')
from celery import Celery

celery = Celery('tasks', broker='redis://localhost:6379/0')

@celery.task
def sendemail(mail):
    print('sending mail to %s' % mail['to'])
    time.sleep(2)
    print('mail sent')

启动Celery，windows下如果使用缺省的concurrency: 8 (prefork)作为并行机制，会报错
“[2021-07-25 15:56:51,568: ERROR/MainProcess] Task handler raised error: ValueError(‘not enough values to unpack (expected 3, got 0)’,)”
因此换成eventlet。（需要先pip install eventlet）

(venv) PS C:\doc\pythonscripts\celeryproj> celery.exe -A tasks worker -l info -P eventlet -E
 
 -------------- celery@CNENXIONGJ5L2C v5.1.2 (sun-harmonics)
--- ***** -----
-- ******* ---- Windows-10-10.0.18362-SP0 2021-07-25 15:59:53
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x16c2bb182e8
- ** ---------- .> transport:   redis://localhost:6379/0
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 8 (eventlet)
-- ******* ---- .> task events: ON
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


[tasks]
  . tasks.sendemail

[2021-07-25 16:00:59,681: INFO/MainProcess] Connected to redis://localhost:6379/0
[2021-07-25 16:02:15,486: INFO/MainProcess] mingle: searching for neighbors
[2021-07-25 16:03:37,077: INFO/MainProcess] mingle: all alone
[2021-07-25 16:04:13,477: INFO/MainProcess] pidbox: Connected to redis://localhost:6379/0.
[2021-07-25 16:04:46,094: INFO/MainProcess] celery@computername ready.
[2021-07-25 16:04:46,096: INFO/MainProcess] Task tasks.sendemail[71f6127a-00fa-4aa1-993b-fa382653af7e] received
[2021-07-25 16:04:46,099: WARNING/MainProcess] sending mail to abc@efg.com
[2021-07-25 16:04:46,100: WARNING/MainProcess]

在另一终端启动task即可：

(venv) PS C:\doc\pythonscripts\celeryproj> python.exe
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from tasks import sendemail
>>> sendmail.delay(dict(to='abc@efg.com'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'sendmail' is not defined
>>> sendemail.delay(dict(to='abc@efg.com'))
<AsyncResult: 71f6127a-00fa-4aa1-993b-fa382653af7e>

问题

1. prefork在windows下不工作，它和eventlet的区别
2. from xxx import yyy在层次化celeryproj时不工作
3. 没有试验返回result的task，并把result存储在redis中