Celery, Open the door! Part Ⅰ

Celery, Open the door! Part Ⅰ

1. celery一般启动方式

   • command-line方式

     celery -A <celery实例应用包路径> worker [options]

     该命令会默认自动搜索启动目录下celery.py文件，若没有，则路径需包含具体文件名，如：
     celery -A celery_test.main worker
     即启动celery_test目录下的main.py文件。

   • 程序内启动

     app.worker_main(["-app=celery_app", "--loglevel=info"])
     

2. celery启动后常见的报错

   [2019-03-04 19:13:17,567: ERROR/MainProcess] Received unregistered task of type 'celery_test1.test'.
   The message has been ignored and discarded.
   
   Did you remember to import the module containing this task?
   Or maybe you're using relative imports?
   
   Please see
   http://docs.celeryq.org/en/latest/internals/protocol.html
   for more information.
   
   The full contents of the message body was:
   b'[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]' (77b)
   Traceback (most recent call last):
   File "c:", line 558, in on_task_received
   strategy = strategies[type_]
   KeyError: 'celery_test1.test'
   

   解决方法：不要将异步任务函数的定义和调用写到同一个文件中。

   (该错误常常出现在task未注册的情况下)

3. celery中的任务注册

   • 使用include参数

     app = Celery("test", include=["base_task.tasks"])
     

   • autodiscover_tasks方法

     app.autodiscover_tasks([".baseTask"], related_name="tasks")`
     

     autodiscover_tasks会自动注册制定目录下related_name文件中的被@app.tasks或@shared_tasks所装饰的tasks

   • 在tasks所在目录启动

     celery -A celery_test.baseTask.tasks worker -l info

   • celery会根据注册task时设定的name去找到对应的task，如果在celery运行过程中改变name，则task不能被成功执行。

4. celery中定时任务的设置

   • celery beat方式

     • setup_periodic_tasks

       @app.on_after_configure.connect
       def setup_periodic_tasks(sender, **kwargs):
           # Calls test('hello') every 10 seconds.
           sender.add_periodic_task(10.0, test.s("hello"), name='test every 10')
           # 可使用crontab实现更复杂的定时任务
       

     • app setting

       app.conf.beat_schedule = {
           'add-every-30-seconds': {
               'task': 'tasks.add',
               'schedule': 30.0,
               'args': (16, 16)
           },
       }
       

     • 启动beat

       celery -A proj beat [options]

   • python第三方库BlockingScheduler

     def schedule_test():
         scheduler = BlockingScheduler()
         scheduler.add_job(job, "interval", seconds=10)
         scheduler.start()