记录一次celery-beat进程CPU占比94%

最新推荐文章于 2022-08-14 21:29:24 发布

aaa～＃

最新推荐文章于 2022-08-14 21:29:24 发布

阅读量1.1k

点赞数

文章标签： python

本文链接：https://blog.csdn.net/haishan88996/article/details/110196218

版权

下面可以看到celery beat进程占用cpu很高

[root@iZbp1je77jss3x67qq75nuZ ~]# top -c
top - 15:55:29 up 37 days, 20:42,  3 users,  load average: 1.18, 1.64, 1.58
Tasks: 442 total,   2 running, 440 sleeping,   0 stopped,   0 zombie
%Cpu(s): 26.0 us,  4.1 sy,  0.0 ni, 67.1 id,  1.4 wa,  0.0 hi,  1.4 si,  0.0 st
KiB Mem : 32780992 total, 11452072 free, 15723096 used,  5605824 buff/cache
KiB Swap:  1023996 total,  1023996 free,        0 used. 16530292 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2183 root      20   0  650504  96312  13992 R  94.1  0.3 273:19.07 /usr/local/bin/python3.6 /usr/local/bin/celery -A hk_back_HQ beat --logfile celery_beat.log
17493 mysql     20   0 3029076 896032   9020 S  11.8  2.7 752:47.46 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mysqld.log --pid-file=/var/+
17907 root      20   0  639652  91632  14008 S   5.9  0.3   9:36.09 /usr/local/bin/python3.6 /usr/local/bin/celery -A hk_back_HQ worker --loglevel info --logfile celery_worker.log
26047 root      20   0  162552   2476   1612 R   5.9  0.0   0:00.02 top -c
    1 root      20   0  191132   4064   2620 S   0.0  0.0   7:29.94 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.12 [kthreadd]

解决问题过程:
1.查看进程堆栈信息，发现看不出来什么错（没解决）

[root@iZbp1je77jss3x67qq75nuZ ~]# pstack 2183
Thread 4 (Thread 0x7fc23ef90700 (LWP 2188)):
#0  0x00007fc2536cca35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc23f59e60b in blas_thread_server () from /usr/local/lib/python3.6/site-packages/numpy/core/../.libs/libopenblasp-r0-34a18dc3.3.7.so
#2  0x00007fc2536c8ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc252ce88dd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fc23e78f700 (LWP 2189)):
#0  0x00007fc2536cca35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc23f59e60b in blas_thread_server () from /usr/local/lib/python3.6/site-packages/numpy/core/../.libs/libopenblasp-r0-34a18dc3.3.7.so
#2  0x00007fc2536c8ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc252ce88dd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fc23df8e700 (LWP 2190)):
#0  0x00007fc2536cca35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc23f59e60b in blas_thread_server () from /usr/local/lib/python3.6/site-packages/numpy/core/../.libs/libopenblasp-r0-34a18dc3.3.7.so
#2  0x00007fc2536c8ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc252ce88dd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fc253aee740 (LWP 2183)):
#0  0x00007fc2536cfaeb in recv () from /lib64/libpthread.so.0
#1  0x00007fc247533d31 in sock_recv_impl (data=0x7fffe9dad240, s=0x7fc22d8647c8) at /home/Python-3.6.8/Modules/socketmodule.c:3070
#2  sock_call_ex (s=s@entry=0x7fc22d8647c8, data=data@entry=0x7fffe9dad240, timeout=-1000000000, err=0x0, connect=0, sock_func=0x7fc247531bf0 <sock_recv_impl>, writing=0) at /home/Python-3.6.8/Modules/socketmodule.c:900
#3  0x00007fc247533ea8 in sock_call (data=0x7fffe9dad240, func=0x7fc247531bf0 <sock_recv_impl>, writing=0, s=0x7fc22d8647c8) at /home/Python-3.6.8/Modules/socketmodule.c:952
#4  sock_recv_guts (flags=<optimized out>, len=<optimized out>, cbuf=<optimized out>, s=0x7fc22d8647c8) at /home/Python-3.6.8/Modules/socketmodule.c:3102
#5  sock_recv_into (s=0x7fc22d8647c8, args=<optimized out>, kwds=<optimized out>) at /home/Python-3.6.8/Modules/socketmodule.c:3197
#6  0x00000000004aa7d7 in _PyCFunction_FastCallDict (kwargs=0x0, nargs=1, args=0x7fc247533de0 <sock_recv_into>, func_obj=0x7fc22d7c0900) at Objects/methodobject.c:231
#7  _PyCFunction_FastCallKeywords (func=func@entry=0x7fc22d7c0900, stack=stack@entry=0x195ac70, nargs=1, kwnames=kwnames@entry=0x0) at Objects/methodobject.c:294
#8  0x0000000000540934 in call_function (pp_stack=pp_stack@entry=0x7fffe9dad470, oparg=<optimized out>, kwnames=kwnames@entry=0x0) at Python/ceval.c:4851
#9  0x00000000005424db in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3335
#10 0x000000000053f8e1 in PyEval_EvalFrameEx (throwflag=0, f=0x195aad8) at Python/ceval.c:754
#11 _PyFunction_FastCall (co=co@entry=0x7fc247776150, args=<optimized out>, args@entry=0x7fffe9dad670, nargs=nargs@entry=2, globals=globals@entry=0x7fc24776a288) at Python/ceval.c:4933
#12 0x0000000000549ef6 in _PyFunction_FastCallDict (func=func@entry=0x7fc247780f28, args=args@entry=0x7fffe9dad670, nargs=nargs@entry=2, kwargs=kwargs@entry=0x0) at Python/ceval.c:5035
#13 0x00000000004524af in _PyObject_FastCallDict (func=0x7fc247780f28, args=0x7fffe9dad670, nargs=2, kwargs=0x0) at Objects/abstract.c:2310
#14 0x00000000004525ab in _PyObject_Call_Prepend (func=0x7fc247780f28, obj=0x7fc233e2fa58, args=0x7fc22d794eb8, kwargs=0x0) at Objects/abstract.c:2373
#15 0x0000000000452362 in _PyObject_FastCallDict (func=0x7fc241cfeec8, args=0x7fffe9dad760, nargs=1, kwargs=0x0) at Objects/abstract.c:2331
#16 0x00000000004534e6 in PyObject_CallMethodObjArgs (callable=0x7fc241cfeec8, name=<optimized out>) at Objects/abstract.c:2759
#17 0x00000000005bc025 in _bufferedreader_raw_read (self=self@entry=0x7fc22d8525c8, start=<optimized out>, len=8192) at ./Modules/_io/bufferedio.c:1498
#18 0x00000000005bfb94 in _bufferedreader_fill_buffer (self=0x7fc22d8525c8) at ./Modules/_io/bufferedio.c:1530
#19 _bufferedreader_read_generic (n=<optimized out>, self=0x7fc22d8525c8) at ./Modules/_io/bufferedio.c:1716
#20 _io__Buffered_read_impl (n=<optimized out>, self=0x7fc22d8525c8) at ./Modules/_io/bufferedio.c:926
#21 _io__Buffered_read (self=0x7fc22d8525c8, args=<optimized out>) at ./Modules/_io/clinic/bufferedio.c.h:136

2.因为我的celery进程是通过supervisord来管理的，所以来看一下它的输出（没解决）

#tqxj-beat是自己在配置文件里面定义的名字

#查看celery-beat的输出
supervisorctl tail tqxj-beat stdout
#查看celery-beat的错误输出
supervisorctl tail tqxj-beat stderr

3.看一下celery的版本（解决）

[root@iZbp1je77jss3x67qq75nuZ ~]# pip list|grep celery
celery                3.1.25
celery-with-redis     3.0
django-celery         3.3.1

最后发现竟然是django-celery 版本问题，切换到django-celery==3.1.17版本解决

[root@iZbp1je77jss3x67qq75nuZ ~]# pip3 uninstall django-celery
Found existing installation: django-celery 3.3.1
Uninstalling django-celery-3.3.1:
  Would remove:
    /usr/local/lib/python3.6/site-packages/django_celery-3.3.1.dist-info/*
    /usr/local/lib/python3.6/site-packages/djcelery/*
Proceed (y/n)? y
  Successfully uninstalled django-celery-3.3.1
[root@iZbp1je77jss3x67qq75nuZ ~]# pip3 install django-celery==3.1.17
Looking in indexes: http://mirrors.cloud.aliyuncs.com/pypi/simple/
Collecting django-celery==3.1.17
  Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/56/cc/6b9731091887dcb1c76ef48521e580bb2b0f469e6f4934072aac75a478f3/django-celery-3.1.17.tar.gz (79 kB)
     |████████████████████████████████| 79 kB 10.2 MB/s
Requirement already satisfied: celery>=3.1.15 in /usr/local/lib/python3.6/site-packages (from django-celery==3.1.17) (3.1.25)
Requirement already satisfied: billiard<3.4,>=3.3.0.23 in /usr/local/lib/python3.6/site-packages (from celery>=3.1.15->django-celery==3.1.17) (3.3.0.23)
Requirement already satisfied: kombu<3.1,>=3.0.37 in /usr/local/lib/python3.6/site-packages (from celery>=3.1.15->django-celery==3.1.17) (3.0.37)
Requirement already satisfied: pytz>dev in /usr/local/lib/python3.6/site-packages (from celery>=3.1.15->django-celery==3.1.17) (2019.1)
Requirement already satisfied: amqp<2.0,>=1.4.9 in /usr/local/lib/python3.6/site-packages (from kombu<3.1,>=3.0.37->celery>=3.1.15->django-celery==3.1.17) (1.4.9)
Requirement already satisfied: anyjson>=0.3.3 in /usr/local/lib/python3.6/site-packages (from kombu<3.1,>=3.0.37->celery>=3.1.15->django-celery==3.1.17) (0.3.3)
Building wheels for collected packages: django-celery
  Building wheel for django-celery (setup.py) ... done
  Created wheel for django-celery: filename=django_celery-3.1.17-py3-none-any.whl size=67863 sha256=c4a534e4a99d79007701da1467d2ab9af4bd291c067efb588afcc0e9246bc0ca
  Stored in directory: /root/.cache/pip/wheels/f8/9e/2b/a864e9474bcaff43c71884a15c4e04c22a4c8346814a5cb126
Successfully built django-celery
Installing collected packages: django-celery
Successfully installed django-celery-3.1.17
[root@iZbp1je77jss3x67qq75nuZ ~]# pip3 list|grep celery
celery                3.1.25
celery-with-redis     3.0
django-celery         3.1.17

重启supervisord服务

[root@iZbp1je77jss3x67qq75nuZ hk_back_HQ]# systemctl restart supervisord

可以看到celery_beat的cpu占用率降下来了

[root@iZbp1je77jss3x67qq75nuZ hk_back_HQ]#top
Tasks: 449 total,  20 running, 429 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.5 us,  0.7 sy,  0.0 ni, 93.6 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 32780992 total, 11464244 free, 15689112 used,  5627636 buff/cache
KiB Swap:  1023996 total,  1023996 free,        0 used. 16563956 avail Mem
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
30420 root      20   0  650768  96560  13988 S   1.7  0.3   0:04.42 /usr/local/bin/python3.6 /usr/local/bin/celery -A hk_back_HQ beat --logfile celery_beat.log

通过将django-celery3.3.1版本切换到django-celery3.1.17解决

aaa～＃

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
1
评论
记录一次celery-beat进程CPU占比94%

下面可以看到celery beat进程占用cpu很高[root@iZbp1je77jss3x67qq75nuZ ~]# top -ctop - 15:55:29 up 37 days, 20:42, 3 users, load average: 1.18, 1.64, 1.58Tasks: 442 total, 2 running, 440 sleeping, 0 stopped, 0 zombie%Cpu(s): 26.0 us, 4.1 sy, 0.0 ni, 67.1 id
复制链接

扫一扫