nova-novncproxy出现zombie进程,且无法提供服务,日志文件也不输出

最新推荐文章于 2024-09-24 07:53:03 发布
最新推荐文章于 2024-09-24 07:53:03 发布
阅读量303 收藏
点赞数
文章标签： python
原文链接：http://www.cnblogs.com/oolo/p/nova-novncproxy-hang.html
版权

nova-novncproxy作用

nova-novncproxy提供vm的控制台访问方式,类似与物理机ipmi访问的方式,其优点在于,即使用户业务网络出现异常,用户也能够通过vnc访问其虚拟机. 因此nova-novncproxy是nova的核心模块.

遇到的问题描述

在使用nova-novncproxy的过程中(kilo 2015.1.3,websockify 0.6.0),我们遇到了一个棘手的问题:
每隔一两周, nova-novncproxy会出现很多zombie进程, 而且nova-novncproxy无法提供服务,于是看nova-novncproxy日志/var/log/nova/nova-novncproxy.log,发现最新的日志在好几天前. 但是只要一旦重启nova-novncproxy,立刻恢复
类似于:

 26327 ?        S      0:05  \_ /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf
 4765 ?        Z      0:00      \_ [nova-novncproxy] <defunct>
 4766 ?        Z      0:00      \_ [nova-novncproxy] <defunct>
 4767 ?        Z      0:00      \_ [nova-novncproxy] <defunct>
 4768 ?        Z      0:00      \_ [nova-novncproxy] <defunct>
 4769 ?        Z      0:00      \_ [nova-novncproxy] <defunct>

疑问:
1. 为什么有zombie进程?
2. nova-novncproxy卡在哪里?

分析

首先想知道的一个问题是,到底nova-novncproxy被阻塞在哪里?
于是想到了,可以用gdb attach到该进程上, 比如命令:gdb -p 26327, 具体的方法可以参考:http://www.cnblogs.com/dkblog/p/3806277.html
摘出核心的方法:
1）确保你的gdb版本>=7
2）安装python-debuginfo包（如：python-debuginfo- 2.6.6-29.el6_2.2.x86_64.rpm，这个版本号一定要跟你所用的python版本一致(可以rpm -qa|grep python查看你安装的python的详细版本号)。找包http://debuginfo.centos.org/6/x86_64/）
3）就可以用#gdb -p 进程号，进行调试了。

attach到进程之后, 输入bt, 打印进程栈,如下:

#0  0x00007fbd4bc703f3 in __select_nocancel () at /lib64/libc.so.6
#1  0x00007fbd4461e070 in time_sleep (secs=<optimized out>) at /usr/src/debug/Python-2.7.5/Modules/timemodule.c:948
#2  0x00007fbd4461e070 in time_sleep (self=<optimized out>, args=<optimized out>) at /usr/src/debug/Python-2.7.5/Modules/timemodule.c:206
#3  0x00007fbd4c949aa4 in PyEval_EvalFrameEx (oparg=<optimized out>, pp_stack=0x7ffe0f6a11e0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4098
#4  0x00007fbd4c949aa4 in PyEval_EvalFrameEx (f=f@entry=Frame 0x291c560, for file /usr/lib/python2.7/site-packages/eventlet/hubs/poll.py, line 82, in wait (self=<Hub(next_timers=[], clock=<built-in function time>, debug_exceptions=True, debug_blocking_resolution=1, modify=<built-in method modify of select.epoll object at remote 0x7fbd4cdbfc48>, running=True, debug_blocking=False, listeners={'read': {}, 'write': {}}, timers_canceled=0, greenlet=<greenlet.greenlet at remote 0x285a4b0>, closed=[], stopping=False, timers=[], poll=<select.epoll at remote 0x7fbd4cdbfc48>, secondaries={'read': {}, 'write': {}}, lclass=<type at remote 0x2922410>) at remote 0x28e1f90>, seconds=<float at remote 0x236f158>, readers={...}, writers={...}), throwflag=throwflag@entry=0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:2740
#5  0x00007fbd4c94b0bd in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=2, kws=0x291ebd0, kwcount=0, defs=0x28e1f68, defcount=1, closure=closure@entry=0x0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3330
#6  0x00007fbd4c94976f in PyEval_EvalFrameEx (nk=<optimized out>, na=2, n=2, pp_stack=0x7ffe0f6a13e0, func=<function at remote 0x28eac80>) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4194
#7  0x00007fbd4c94976f in PyEval_EvalFrameEx (oparg=<optimized out>, pp_stack=0x7ffe0f6a13e0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4119
#8  0x00007fbd4c94976f in PyEval_EvalFrameEx (f=f@entry=Frame 0x291ea20, for file /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py, line 346, in run (self=<Hub(next_timers=[], clock=<built-in function time>, debug_exceptions=True, debug_blocking_resolution=1, modify=<built-in method modify of select.epoll object at remote 0x7fbd4cdbfc48>, running=True, debug_blocking=False, listeners={'read': {}, 'write': {}}, timers_canceled=0, greenlet=<greenlet.greenlet at remote 0x285a4b0>, closed=[], stopping=False, timers=[], poll=<select.epoll at remote 0x7fbd4cdbfc48>, secondaries={'read': {}, 'write': {}}, lclass=<type at remote 0x2922410>) at remote 0x28e1f90>, a=(), kw={}, wakeup_when=None, sleep_time=<float at remote 0x236f158>), throwflag=throwflag@entry=0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:2740
#9  0x00007fbd4c94b0bd in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x28db928, argcount=1, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3330
#10 0x00007fbd4c8d7f68 in function_call (func=<function at remote 0x28ea0c8>, arg=(<Hub(next_timers=[], clock=<built-in function time>, debug_exceptions=True, debug_blocking_resolution=1, modify=<built-in method modify of select.epoll object at remote 0x7fbd4cdbfc48>, running=True, debug_blocking=False, listeners={'read': {}, 'write': {}}, timers_canceled=0, greenlet=<greenlet.greenlet at remote 0x285a4b0>, closed=[], stopping=False, timers=[], poll=<select.epoll at remote 0x7fbd4cdbfc48>, secondaries={'read': {}, 'write': {}}, lclass=<type at remote 0x2922410>) at remote 0x28e1f90>,), kw=0x0) at /usr/src/debug/Python-2.7.5/Objects/funcobject.c:526
#11 0x00007fbd4c8b30b3 in PyObject_Call (func=func@entry=<function at remote 0x28ea0c8>, arg=arg@entry=(<Hub(next_timers=[], clock=<built-in function time>, debug_exceptions=True, debug_blocking_resolution=1, modify=<built-in method modify of select.epoll object at remote 0x7fbd4cdbfc48>, running=True, debug_blocking=False, listeners={'read': {}, 'write': {}}, timers_canceled=0, greenlet=<greenlet.greenlet at remote 0x285a4b0>, closed=[], stopping=False, timers=[], poll=<select.epoll at remote 0x7fbd4cdbfc48>, secondaries={'read': {}, 'write': {}}, lclass=<type at remote 0x2922410>) at remote 0x28e1f90>,), kw=kw@entry=0x0) at /usr/src/debug/Python-2.7.5/Objects/abstract.c:2529
#12 0x00007fbd4c8c20a5 in instancemethod_call (func=<function at remote 0x28ea0c8>, arg=(<Hub(next_timers=[], clock=<built-in function time>, debug_exceptions=True, debug_blocking_resolution=1, modify=<built-in method modify of select.epoll object at remote 0x7fbd4cdbfc48>, running=True, debug_blocking=False, listeners={'read': {}, 'write': {}}, timers_canceled=0, greenlet=<greenlet.greenlet at remote 0x285a4b0>, closed=[], stopping=False, timers=[], poll=<select.epoll at remote 0x7fbd4cdbfc48>, secondaries={'read': {}, 'write': {}}, lclass=<type at remote 0x2922410>) at remote 0x28e1f90>,), kw=0x0) at /usr/src/debug/Python-2.7.5/Objects/classobject.c:2602
#13 0x00007fbd4c8b30b3 in PyObject_Call (func=func@entry=<instancemethod at remote 0x26d4eb0>, arg=arg@entry=(), kw=<optimized out>) at /usr/src/debug/Python-2.7.5/Objects/abstract.c:2529
#14 0x00007fbd4c944f07 in PyEval_CallObjectWithKeywords (func=<instancemethod at remote 0x26d4eb0>, arg=(), kw=<optimized out>) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3967
#15 0x00007fbd44825a9c in g_initialstub () at /usr/lib64/python2.7/site-packages/greenlet.so
#16 0x00007fbd448253e6 in g_switch () at /usr/lib64/python2.7/site-packages/greenlet.so
#17 0x0000000000000003 in  ()
#18 0x00000000028e5b08 in  ()
#19 0x00007fbd4c94b0bd in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x28e5b08, argcount=3, kws=kws@entry=0x7fbd4cdff068, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3330
#20 0x00007fbd4c8d805d in function_call (func=<function at remote 0x7fbd411126e0>, arg=(<KeywordArgumentAdapter(logger=<Logger(name='nova.console.websocketproxy', parent=<RootLogger(name='root', parent=None, handlers=[<WatchedFileHandler(stream=<file at remote 0x2867300>, level=0, lock=<_RLock(_Verbose__verbose=False, _RLock__owner=None, _RLock__block=<Semaphore(counter=0, _waiters=<collections.deque at remote 0x28dc670>) at remote 0x28db850>, _RLock__count=0) at remote 0x28db810>, encoding=None, dev=64769L, _name=None, ino=3221237281, baseFilename='/var/log/nova/nova-novncproxy.log', mode='a', filters=[], formatter=<ContextFormatter(project='nova', datefmt='%Y-%m-%d %H:%M:%S', version='unknown', _fmt='%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s %(funcName)s %(pathname)s:%(lineno)d', conf=<ConfigOpts(_groups={'remote_debug': <OptGroup(title='remote_debug options', _opts={'host': {'opt': <StrOpt(deprecated_for_removal=False, short=None, name='host', dest='host', required=False, _logged_deprecation=False, sample_default=None, deprecated_opts=[], position...(truncated), kw={}) at /usr/src/debug/Python-2.7.5/Objects/funcobject.c:526
#21 0x00007fbd4c8b30b3 in PyObject_Call (func=func@entry=<function at remote 0x7fbd411126e0>, arg=arg@entry=(<KeywordArgumentAdapter(logger=<Logger(name='nova.console.websocketproxy', parent=<RootLogger(name='root', parent=None, handlers=[<WatchedFileHandler(stream=<file at remote 0x2867300>, level=0, lock=<_RLock(_Verbose__verbose=False, _RLock__owner=None, _RLock__block=<Semaphore(counter=0, _waiters=<collections.deque at remote 0x28dc670>) at remote 0x28db850>, _RLock__count=0) at remote 0x28db810>, encoding=None, dev=64769L, _name=None, ino=3221237281, baseFilename='/var/log/nova/nova-novncproxy.log', mode='a', filters=[], formatter=<ContextFormatter(project='nova', datefmt='%Y-%m-%d %H:%M:%S', version='unknown', _fmt='%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s %(funcName)s %(pathname)s:%(lineno)d', conf=<ConfigOpts(_groups={'remote_debug': <OptGroup(title='remote_debug options', _opts={'host': {'opt': <StrOpt(deprecated_for_removal=False, short=None, name='host', dest='host', required=False, _logged_deprecation=False, sample_default=None, deprecated_opts=[], position...(truncated), kw=kw@entry={}) at /usr/src/debug/Python-2.7.5/Objects/abstract.c:2529
#22 0x00007fbd4c9462f7 in PyEval_EvalFrameEx (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7ffe0f6a1b90, func=<function at remote 0x7fbd411126e0>) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4411
*#23 0x00007fbd4c9462f7 in PyEval_EvalFrameEx (f=f@entry=Frame 0x292b920, for file /usr/lib/python2.7/site-packages/websockify/websocket.py, line 828, in vmsg (self=<NovaWebSocketProxy(wrap_times=[0, 0, 0], target_cfg=None, verbose=False, listen_host='0.0.0.0', tcp_keepintvl=None, ws_connection=False, tcp_keepidle=None, target_host=None, ssl_target=None, web='/usr/share/novnc', listen_port=6080, logger=<KeywordArgumentAdapter(logger=<Logger(name='nova.console.websocketproxy', parent=<RootLogger(name='root', parent=None, handlers=[<WatchedFileHandler(stream=<file at remote 0x2867300>, level=0, lock=<_RLock(_Verbose__verbose=False, _RLock__owner=None, _RLock__block=<Semaphore(counter=0, _waiters=<collections.deque at remote 0x28dc670>) at remote 0x28db850>, _RLock__count=0) at remote 0x28db810>, encoding=None, dev=64769L, _name=None, ino=3221237281, baseFilename='/var/log/nova/nova-novncproxy.log', mode='a', filters=[], formatter=<ContextFormatter(project='nova', datefmt='%Y-%m-%d %H:%M:%S', version='unknown', _fmt='%(asctime)s.%(msecs)03d %(process)d %(levelname)s %...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:2779*
#24 0x00007fbd4c94b0bd in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=2, kws=0x28fc260, kwcount=0, defs=0x0, defcount=0, closure=closure@entry=0x0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3330
#25 0x00007fbd4c94976f in PyEval_EvalFrameEx (nk=<optimized out>, na=2, n=2, pp_stack=0x7ffe0f6a1d90, func=<function at remote 0x180e848>) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4194
#26 0x00007fbd4c94976f in PyEval_EvalFrameEx (oparg=<optimized out>, pp_stack=0x7ffe0f6a1d90) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4119
*#27 0x00007fbd4c94976f in PyEval_EvalFrameEx (f=f@entry=Frame 0x28fc050, for file /usr/lib/python2.7/site-packages/websockify/websocket.py, line 996, in start_server (self=<NovaWebSocketProxy(wrap_times=[0, 0, 0], target_cfg=None, verbose=False, listen_host='0.0.0.0', tcp_keepintvl=None, ws_connection=False, tcp_keepidle=None, target_host=None, ssl_target=None, web='/usr/share/novnc', listen_port=6080, logger=<KeywordArgumentAdapter(logger=<Logger(name='nova.console.websocketproxy', parent=<RootLogger(name='root', parent=None, handlers=[<WatchedFileHandler(stream=<file at remote 0x2867300>, level=0, lock=<_RLock(_Verbose__verbose=False, _RLock__owner=None, _RLock__block=<Semaphore(counter=0, _waiters=<collections.deque at remote 0x28dc670>) at remote 0x28db850>, _RLock__count=0) at remote 0x28db810>, encoding=None, dev=64769L, _name=None, ino=3221237281, baseFilename='/var/log/nova/nova-novncproxy.log', mode='a', filters=[], formatter=<ContextFormatter(project='nova', datefmt='%Y-%m-%d %H:%M:%S', version='unknown', _fmt='%(asctime)s.%(msecs)03d %(process)d %(level...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:2740*
#28 0x00007fbd4c949860 in PyEval_EvalFrameEx (nk=<optimized out>, na=1, n=1, pp_stack=0x7ffe0f6a1ef0, func=<function at remote 0x180ecf8>) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4184
#29 0x00007fbd4c949860 in PyEval_EvalFrameEx (oparg=<optimized out>, pp_stack=0x7ffe0f6a1ef0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4119
#30 0x00007fbd4c949860 in PyEval_EvalFrameEx (f=f@entry=Frame 0x28af4f0, for file /usr/lib/python2.7/site-packages/nova/cmd/baseproxy.py, line 72, in proxy (host='0.0.0.0', port=6080), throwflag=throwflag@entry=0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:2740
#31 0x00007fbd4c94b0bd in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=0, kws=0x159c490, kwcount=2, defs=0x0, defcount=0, closure=closure@entry=0x0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3330
#32 0x00007fbd4c94976f in PyEval_EvalFrameEx (nk=<optimized out>, na=0, n=4, pp_stack=0x7ffe0f6a20f0, func=<function at remote 0x24e7c80>) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4194
#33 0x00007fbd4c94976f in PyEval_EvalFrameEx (oparg=<optimized out>, pp_stack=0x7ffe0f6a20f0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4119
#34 0x00007fbd4c94976f in PyEval_EvalFrameEx (f=f@entry=Frame 0x159c310, for file /usr/lib/python2.7/site-packages/nova/cmd/novncproxy.py, line 49, in main (), throwflag=throwflag@entry=0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:2740
#35 0x00007fbd4c949860 in PyEval_EvalFrameEx (nk=<optimized out>, na=0, n=0, pp_stack=0x7ffe0f6a2250, func=<function at remote 0x24ec1b8>) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4184
#36 0x00007fbd4c949860 in PyEval_EvalFrameEx (oparg=<optimized out>, pp_stack=0x7ffe0f6a2250) at /usr/src/debug/Python-2.7.5/Python/ceval.c:4119
#37 0x00007fbd4c949860 in PyEval_EvalFrameEx (f=f@entry=Frame 0x1352510, for file /usr/bin/nova-novncproxy, line 10, in <module> (), throwflag=throwflag@entry=0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:2740
#38 0x00007fbd4c94b0bd in PyEval_EvalCodeEx (co=co@entry=0x7fbd4cd828b0, globals=globals@entry={'__builtins__': <module at remote 0x7fbd4cdffb08>, '__file__': '/usr/bin/nova-novncproxy', '__package__': None, 'sys': <module at remote 0x7fbd4cdffbb0>, '__name__': '__main__', 'main': <function at remote 0x24ec1b8>, '__doc__': None}, locals=locals@entry={'__builtins__': <module at remote 0x7fbd4cdffb08>, '__file__': '/usr/bin/nova-novncproxy', '__package__': None, 'sys': <module at remote 0x7fbd4cdffbb0>, '__name__': '__main__', 'main': <function at remote 0x24ec1b8>, '__doc__': None}, args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=closure@entry=0x0) at /usr/src/debug/Python-2.7.5/Python/ceval.c:3330
#39 0x00007fbd4c94b1c2 in PyEval_EvalCode (co=co@entry=0x7fbd4cd828b0, globals=globals@entry={'__builtins__': <module at remote 0x7fbd4cdffb08>, '__file__': '/usr/bin/nova-novncproxy', '__package__': None, 'sys': <module at remote 0x7fbd4cdffbb0>, '__name__': '__main__', 'main': <function at remote 0x24ec1b8>, '__doc__': None}, locals=locals@entry={'__builtins__': <module at remote 0x7fbd4cdffb08>, '__file__': '/usr/bin/nova-novncproxy', '__package__': None, 'sys': <module at remote 0x7fbd4cdffbb0>, '__name__': '__main__', 'main': <function at remote 0x24ec1b8>, '__doc__': None}) at /usr/src/debug/Python-2.7.5/Python/ceval.c:689
#40 0x00007fbd4c9645ff in run_mod (mod=<optimized out>, filename=filename@entry=0x7ffe0f6a3f2c "/usr/bin/nova-novncproxy", globals=globals@entry={'__builtins__': <module at remote 0x7fbd4cdffb08>, '__file__': '/usr/bin/nova-novncproxy', '__package__': None, 'sys': <module at remote 0x7fbd4cdffbb0>, '__name__': '__main__', 'main': <function at remote 0x24ec1b8>, '__doc__': None}, locals=locals@entry={'__builtins__': <module at remote 0x7fbd4cdffb08>, '__file__': '/usr/bin/nova-novncproxy', '__package__': None, 'sys': <module at remote 0x7fbd4cdffbb0>, '__name__': '__main__', 'main': <function at remote 0x24ec1b8>, '__doc__': None}, flags=flags@entry=0x7ffe0f6a24b0, arena=arena@entry=0x1317f40) at /usr/src/debug/Python-2.7.5/Python/pythonrun.c:1374
#41 0x00007fbd4c9657be in PyRun_FileExFlags (fp=fp@entry=0x135fe20, filename=filename@entry=0x7ffe0f6a3f2c "/usr/bin/nova-novncproxy", start=start@entry=257, globals=globals@entry={'__builtins__': <module at remote 0x7fbd4cdffb08>, '__file__': '/usr/bin/nova-novncproxy', '__package__': None, 'sys': <module at remote 0x7fbd4cdffbb0>, '__name__': '__main__', 'main': <function at remote 0x24ec1b8>, '__doc__': None}, locals=locals@entry={'__builtins__': <module at remote 0x7fbd4cdffb08>, '__file__': '/usr/bin/nova-novncproxy', '__package__': None, 'sys': <module at remote 0x7fbd4cdffbb0>, '__name__': '__main__', 'main': <function at remote 0x24ec1b8>, '__doc__': None}, closeit=closeit@entry=1, flags=flags@entry=0x7ffe0f6a24b0) at /usr/src/debug/Python-2.7.5/Python/pythonrun.c:1360
#42 0x00007fbd4c966a49 in PyRun_SimpleFileExFlags (fp=fp@entry=0x135fe20, filename=filename@entry=0x7ffe0f6a3f2c "/usr/bin/nova-novncproxy", closeit=closeit@entry=1, flags=flags@entry=0x7ffe0f6a24b0) at /usr/src/debug/Python-2.7.5/Python/pythonrun.c:952
#43 0x00007fbd4c966f63 in PyRun_AnyFileExFlags (fp=fp@entry=0x135fe20, filename=filename@entry=0x7ffe0f6a3f2c "/usr/bin/nova-novncproxy", closeit=closeit@entry=1, flags=flags@entry=0x7ffe0f6a24b0) at /usr/src/debug/Python-2.7.5/Python/pythonrun.c:756
#44 0x00007fbd4c977b9f in Py_Main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/Python-2.7.5/Modules/main.c:640
#45 0x00007fbd4bba3b15 in __libc_start_main () at /lib64/libc.so.6
#46 0x0000000000400721 in _start ()

仔细分析该栈,发现卡在:
/usr/lib/python2.7/site-packages/websockify/websocket.py, line 996, in start_server
/usr/lib/python2.7/site-packages/websockify/websocket.py, line 828, in vmsg

大家可以分析下这个/usr/lib/python2.7/site-packages/websockify/websocket.py代码, 实际上start_server函数是一个标准的服务端程序, 建立一个socket,监听,有新的连接来了之后,就启动一个新进程去处理,主进程不停的轮询,当用户的连接断开后,子进程死亡,父进程收到sigchld,会根据消息处理函数self.multiprocessing_SIGCHLD来进行销毁.

通过上述的分析,可以解决我们的第一/二个疑问:

  1. nova-novncproxy卡在websockify库的websocket.py的写日志的函数:vmsg
  2. 因为主进程卡在vmsg函数中(写日志),因此,死亡的孩子进程无法被收尸,因此处于了defunct状态

卡在写日志vmsg的原因

这个问题,我没有能力去处理, 但是我们已经知道了,问题出在nova-novncproxy所依赖的websockify库中,那么靠google吧
搜索关键字: websockify hang
第一篇文章就是:
https://github.com/kanaka/noVNC/issues/556
顺着这个链接,找到了websockify的一个patch:
https://github.com/kanaka/websockify/pull/219/files
同时,确认了我们使用的nova-novncproxy是kilo版本的, 2015.1.3的,第一篇文章提到的nova-novncproxy的修改已经merge进去了
目前就按照该patch, 把websockify更新了下,同时知道vmsg有一定的问题,因此在nova-novncproxy的配置文件中,把debug,verbose置为false,这样会比较稳妥一些.

https://github.com/kanaka/websockify/pull/219
给出的原因是:
Openstack nova novnc-proxy services uses websockify to provide support
for nova vms using novnc proxy. At present, novnc hangs every couple of
weeks. It only resumes post restart of the novnc-proxy which is not
good. Hence, this code in websockify is updated to get rid of additional
signal calls to avoid novnc going in hang state even though process is
running. Basically, we are getting rid of existing msg and vmsg calls in
the websocket.py. This is kind of quick fix but we will need an
additional way of figuring out the logging to make it easy to trace in
case of any further failures in future.

意思是websocket.py中,msg与vmsg function是比较危险的,应该避免被调用到. 深层次的原因还需要深入的探究

暂时先跑着, 看看是否还有问题

转载于:https://www.cnblogs.com/oolo/p/nova-novncproxy-hang.html

weixin_30576827
关注
openstack-nova_novncproxy:OpenStack Nova novncproxy服务安装-Ansible角色
05-25
Nova Novncproxy OpenStack Nova novncproxy服务安装 在Ubuntu Precise（12.04）和Trusty（14.04）上进行了测试 要求 RabbitMQ服务器。 见下文。 对于RHEL / CentOS，需要RHOSP或RDO存储库。 角色变量 Nova novncproxy（由此角色设置） 名称 默认值 描述 笔记 my_ip {{ ansible_eth0.ipv4.address }} nova-novncproxy的管理IP vncserver_proxyclient_address {{ my_ip }} 用于连接到vnc代理的地址 vncserver_proxy_address {{ my_ip }} 代理客户端应连接到的地址 novncproxy_base_url "http://{{ vncserver_
openstack-nova-novncproxy-20.1.1-3.el7.noarch.rpm
12-28
官方离线安装包，测试可用。使用rpm -ivh [rpm完整包名] 进行安装
Reproxy：简单而强大的反向代理服务器
最新发布
gitblog_00819的博客
09-24 923
Reproxy：简单而强大的反向代理服务器 reproxy Simple edge server / reverse proxy 项目地址: https://gitcode.com/gh_mirrors/re/reproxy ...
nova vnc proxy基本原理
刘超的通俗云计算
05-23 2424
先上图 VNC Proxy的功能： 将公网(public network)和私网(private network)隔离VNC client运行在公网上，VNCServer运行在私网上，VNC Proxy作为中间的桥梁将二者连接起来VNC Proxy通过token对VNC Client进行验证VNC Proxy不仅仅使得私网的访问更加安全，而且将具体的VNC Server的实现分
nova分析（9）—— nova-novncproxy
weixin_33964094的博客
08-11 583
nova提供了novncproxy代理支持用户通过vnc来访问虚拟机，用户可以通过websocket、java客户端或者spicehtml5来访问。通过websket访问虚拟机的功能已经集成到horizon中，而通过java客户端则需要先安装相应的软件。为了方便用户访问虚拟机，nova通过有一个proxy来实现，proxy通常同nova-api一起部署。...
openstack重启openstack-nova-nonvcproxy解决VM无法访问问题
weixin_33726318的博客
02-12 516
今天我遇到一个问题是基于openstack平台VM无法打开novnc控制台，并且使用nova get-vnc-console $VM_UUID novnc获取URL，把URL粘贴到浏览器地址依然无法访问。 一开始我以为是Win7系统中的Firefox异常，然后重新安装了火狐。因为openstack 平台里面有bootstrap技术，火狐浏览器对openstack的平台支...
CS-GO-Zombie-Escape-Public:准备就绪的Zombie Escape安装文件可以复制到CS GO服务器
04-12
准备就绪的Zombie Escape Server基础安装程序可以复制到Linux或Windows CS GO服务器。 这样做是为了帮助新的服务器所有者开始使用ZE。 希望这会给我们整个社区带来更多的兴趣。 玩家和感兴趣的人越多，对每个人都越...
Arma-3-Ravage-Zombie-Gear:检查文件中的描述以了解如何进行设置
03-18
Arma-3-Ravage-Zombie-Gear:检查文件中的描述以了解如何进行设置
grunt-qunit-zombie:使用 Zombie 的 Grunt 的 QUnit 插件
06-29
此插件正在开发中，尚无法使用。 对不起！ 入门 这个插件需要 Grunt ~0.4.1 如果您以前没有使用过 ，请务必查看指南，因为它解释了如何创建以及安装和使用 Grunt 插件。 熟悉该过程后，您可以使用以下命令安装此...
Cohen-Game-Of-Chromes-Owning-The-Web-With-Zombie-Chrome-Extensio
10-25
Chrome浏览器作为全球最流行的浏览器之一，其扩展程序的广泛使用为用户提供了丰富的功能，同时也可能成为网络安全的薄弱环节。下面我们将深入解析这一主题。 首先，我们需要了解什么是Chrome扩展程序。Chrome扩展...
Responsibility-Request-Zombie
03-05
在IT行业中，"Responsibility-Request-Zombie"可能是指一种编程模式或问题，尤其是在使用C#编程语言时。这个术语可能源于对软件设计模式的一种比喻，尤其是与职责链模式（Responsibility Chain）和死锁（Deadlock）...
nova vnc proxy工作原理和配置方法
ztejiagn的专栏
04-17 5166
1. vnc proxy的实现原理 vnc 是nova提供的用来访问虚拟机的一项重要功能，用户可以通过websocket来访问，也可以通过java客户端来访问。通过websket访问虚拟机的功能已经集成到horizon中，而通过java客户端则需要先安装相应的软件。为了方便用户访问虚拟机，nova通过有一个proxy来实现，proxy通常放在一个所有人都可以访问的IP地址。 vnc访问
爬虫IP代理资源汇总
xiaobei16的博客
01-06 2258
通常我们要对某些网站或是app进行数据采集时，都需要解决采集软件爬虫和封IP的问题，采集软件暂且滤过，有多年网站、手机APP爬虫定制开发和数据批量采集服务经验的楚江数据团队，整理汇总爬虫IP代理服务器资源，以此来应对封IP的问题，供各位参考。         首先，代理服务器有什么用？         通过在web浏览器(chrome, firefox, ie, opera, safari a
springboot实现java代理IP池 Proxy Pool，提供可用率达到95%以上的代理IP
weixin_33912638的博客
10-07 622
一、背景 前段时间，写java爬虫来爬网易云音乐的评论。不料，爬了一段时间后ip被封禁了。由此，想到了使用ip代理，但是找了很多的ip代理网站，很少有可以用的代理ip。于是，抱着边学习的心态，自己开发了一个代理ip池。 二、相关技术及环境 技术：SpringBoot，SpringMVC, Hibernate, MySQL, Redis , Maven, Lombok, Boot...
【云计算学习笔记（十九）】之Nova服务架构与执行流程详解
开发小鸽
06-09 4276
文章目录本文章由公号【开发小鸽】发布！欢迎关注！！！一．Nova服务（一）Nova架构1.架构图2.组件介绍（1）API1) nova-api（2）Compute Core1) nova-scheduler2) nova-compute3) Hypervisor4) nova-comductor（3）Console Interface1) nova-console2) nova-consoleauth3) nova-cert（4）Database（5）Message Queue3.nova组件的物理部署方案
Nginx代理问题
administraor的博客
05-16 670
1. nginx代理问题 1.1 nginx反代到nginx **问题：**使用域名可以访问，使用IP报404 **解决：**在location段proxy_pass参数上面加上一行： proxy_set_header Host xxx.com; 1.2 nginx反代到项目 **问题：**location匹配到规则之后，无法从一个完整的url跳到另一个完整的url **解决：**在nginx配置文件的http段添加一行参数： underscores_in_headers on; 1.3. nginx反
OpenStack之Nova
weixin_42795092的博客
03-07 2053
使用OpenStack Compute来托管和管理云计算系统。OpenStack Compute是基础架构即服务（IaaS）系统的主要部分。主要模块在Python中实现1因为认证，与OpenStack 身份认证keystone 交互。2因为磁盘和服务器镜像， 与镜像服务Glance 交互。3因为提供图形界面，与仪表盘Dashboard交互。Nova是OpenStack云中的计算组织控制器。支持OpenStack云中实例（instances）生命周期的所有活动都由Nova处理。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值