celeryd mysql stop_python分布式调度celery无法启动或自动退出排查步骤

Celery是个好东西,但是不具体去分析,等出现错误会显得手忙脚乱

1) 启动调试方法

celery worker -c 2 -A project -B -l debug -f /var/log/project.log

日志基本没有,控制台就是一些debug和warning日志

[2017-01-16 13:06:35,449: INFO/Beat] beat: Starting...

[2017-01-16 13:06:35,461: INFO/MainProcess] Connected to amqp://username:**@127.0.0.1:5672/

[2017-01-16 13:06:35,482: DEBUG/Beat] Current schedule:

[2017-01-16 13:06:35,482: DEBUG/Beat] beat: Ticking with max interval->5.00 minutes

[2017-01-16 13:06:35,483: INFO/MainProcess] mingle: searching for neighbors

[2017-01-16 13:06:35,503: INFO/Beat] Scheduler: Sending due task 任务名称 (任务名称)

[2017-01-16 13:06:36,500: INFO/MainProcess] mingle: all alone

[2017-01-16 13:06:36,528: WARNING/MainProcess] celery@localhost ready.

[2017-01-16 13:06:36,559: DEBUG/MainProcess] Canceling task consumer...

[2017-01-16 13:06:36,561: INFO/MainProcess] beat: Shutting down...

[2017-01-16 13:06:37,587: DEBUG/MainProcess] Canceling task consumer...

[2017-01-16 13:06:37,588: DEBUG/MainProcess] Closing consumer channel...

然后怎么办,懵逼了...

这个时候,不能随便猜了,不信你可要猜一晚上试试

去git上下celery的源码,搜索关键字

Canceling task consumer

估计原因是在xx地方报错了,然后打印了2行日志出来.

找到代码在worker/consumer.py里面

def shutdown(self, c):

if c.task_consumer:

import traceback

debug("start shutdown traceback ====")

traceback.print_stack()

self.stop(c)

debug('Closing consumer channel...')

ignore_errors(c, c.task_consumer.close)

c.task_consumer = None

添加一个traceback,打印出从哪里调用进来的.

再启动一下celery,你就会发现不一样了

[2017-01-16 13:06:37,579: WARNING/MainProcess] File "/usr/local/bin/celery", line 9, in

[2017-01-16 13:06:37,580: WARNING/MainProcess] load_entry_point('celery==3.1.23', 'console_scripts', 'celery')()

[2017-01-16 13:06:37,580: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/__main__.py", line 30, in main

[2017-01-16 13:06:37,580: WARNING/MainProcess] main()

[2017-01-16 13:06:37,580: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main

[2017-01-16 13:06:37,581: WARNING/MainProcess] cmd.execute_from_commandline(argv)

[2017-01-16 13:06:37,581: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline

[2017-01-16 13:06:37,581: WARNING/MainProcess] super(CeleryCommand, self).execute_from_commandline(argv)))

[2017-01-16 13:06:37,581: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline

[2017-01-16 13:06:37,581: WARNING/MainProcess] return self.handle_argv(self.prog_name, argv[1:])

[2017-01-16 13:06:37,582: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 785, in handle_argv

[2017-01-16 13:06:37,582: WARNING/MainProcess] return self.execute(command, argv)

[2017-01-16 13:06:37,582: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 717, in execute

[2017-01-16 13:06:37,582: WARNING/MainProcess] ).run_from_argv(self.prog_name, argv[1:], command=argv[0])

[2017-01-16 13:06:37,583: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv

[2017-01-16 13:06:37,583: WARNING/MainProcess] return self(*args, **options)

[2017-01-16 13:06:37,583: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 274, in __call__

[2017-01-16 13:06:37,583: WARNING/MainProcess] ret = self.run(*args, **kwargs)

[2017-01-16 13:06:37,583: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run

[2017-01-16 13:06:37,584: WARNING/MainProcess] state_db=self.node_format(state_db, hostname), **kwargs

[2017-01-16 13:06:37,584: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 211, in start

[2017-01-16 13:06:37,584: WARNING/MainProcess] self.stop()

[2017-01-16 13:06:37,584: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 252, in stop

[2017-01-16 13:06:37,584: WARNING/MainProcess] self._shutdown(warm=True)

[2017-01-16 13:06:37,585: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 266, in _shutdown

[2017-01-16 13:06:37,585: WARNING/MainProcess] self.blueprint.stop(self, terminate=not warm)

[2017-01-16 13:06:37,585: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 183, in stop

[2017-01-16 13:06:37,585: WARNING/MainProcess] self.on_stopped()

[2017-01-16 13:06:37,585: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 164, in on_stopped

[2017-01-16 13:06:37,586: WARNING/MainProcess] self.consumer.shutdown()

[2017-01-16 13:06:37,586: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 306, in shutdown

[2017-01-16 13:06:37,586: WARNING/MainProcess] self.blueprint.shutdown(self)

[2017-01-16 13:06:37,586: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 171, in shutdown

[2017-01-16 13:06:37,586: WARNING/MainProcess] self.send_all(parent, 'shutdown')

[2017-01-16 13:06:37,587: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 155, in send_all

[2017-01-16 13:06:37,587: WARNING/MainProcess] fun(parent, *args)

[2017-01-16 13:06:37,587: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 628, in shutdown

[2017-01-16 13:06:37,587: WARNING/MainProcess] traceback.print_stack()

怀疑: start之后为啥马上就stop了.奇怪.

vim /usr/local/lib/python2.7/site-packages/celery/worker/__init__.py

到211行

def start(self):

"""Starts the workers main loop."""

try:

self.blueprint.start(self)

except WorkerTerminate:

self.terminate()

except Exception as exc:

logger.error('Unrecoverable error: %r', exc, exc_info=True)

self.stop()

except (KeyboardInterrupt, SystemExit):

self.stop()

为啥这地方使用了logger.error,我们没有看到日志呢,艹

剩下的就是想办法吧日志全打印出来,

通过代码跟踪需要配置2个日志名称celery.worker和celery.task

LOGGING = {

'version': 1,

'disable_existing_loggers': True,

'formatters': {

'direct': {

'format': '%(asctime)s - %(message)s'

},

'json': {

'format': '%(message)s'

},

},

'handlers': {

'celery.worker': {

'level': 'DEBUG',

'class': 'logging.FileHandler',

'filename': '/var/log/celery.worker.log',

'formatter': 'direct'

},

'celery.task': {

'level': 'DEBUG',

'class': 'logging.FileHandler',

'filename': '/var/log/celery.task.log',

'formatter': 'direct'

},

},

'loggers': {

'celery.worker': {

'handlers': ['celery.worker'],

'level': 'DEBUG'

},

'celery.task': {

'handlers': ['celery.task'],

'level': 'DEBUG'

},

}

}

配置上面,

然后启动celery,

[2017-01-16 13:06:36,550: ERROR/MainProcess] Unrecoverable error: WorkerLostError('Could not start worker processes',)

Traceback (most recent call last):

File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start

self.blueprint.start(self)

File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start

step.start(parent)

File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start

return self.obj.start()

File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 279, in start

blueprint.start(self)

File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start

step.start(parent)

File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 841, in start

c.loop(*c.loop_args())

File "/usr/local/lib/python2.7/site-packages/celery/worker/loops.py", line 48, in asynloop

raise WorkerLostError('Could not start worker processes')

WorkerLostError: Could not start worker processes

然后你会发现不同环境的这个错误是不一样的.

celery worker -c 2 -A project -l debug -P gevent

添加一个参数-P gevent能够正常启动了

查看一下-B参数的说明

-B, --beat            Also run the celery beat periodic task scheduler.

Please note that there must only be one instance of

this service