亚信/青藤云监控服务导致进程写文件卡死或异常缓慢的问题排查

　　　　free

已于 2024-07-18 17:07:48 修改

阅读量447

点赞数

文章标签： linux

于 2022-09-28 17:40:27 首次发布

本文链接：https://blog.csdn.net/qq_42896627/article/details/127093608

版权

现场反馈我们的一个模块总是运行一段时间就突然不提供服务了。让现场同事采集了堆栈发我们，发现这个模块卡在了open64()，打开文件的这个接口处，模块写日志卡住导致业务无法继续往下进行了，我们查看了进程打开文件句柄数。和系统允许打开最大文件句柄数等等发现都是正常的。

最后没有办法去看了系统日志/var/log/message，搜索模块的名字，发现了
在这里插入图片描述
通过这个inode，去查看我们日志文件的inode对比是相同的

我们把系统日志的关键词复制，百度搜索。发现这是趋势科技的一款文件安全管控的产品。询问了客户，他们的服务器确实装了趋势（亚信）防病毒软件。让客户联系趋势科技的开发人员。把我们的相关目录和文件添加到白名单中，解决文件被hook住的问题。

可能是我们的模块写日志频率高被监控到了。这里还有个有趣的小现象。虽然卡住了但是我先后多次远程对这个模块使用gdb调试。每次调试完模块会继续运行，不再卡在写日志的地方，可能是内部内核事件没处理好，gdb attach一下触发了内核信号，打断了监控，让程序继续往下走了

以上是2022.6遇到的。下面遇到的是2023.11，本次非写文件卡死而是写文件缓慢，第一时间查看系统日志未发现异常

某天现场说web认证流程太慢了。认证流程的这些模块的数据走向都依赖于转发模块，所有的模块都链接转发模块，web想要调用指定的模块完成认证，通过转发模块发送到目的程序。web->转发模块->目的程序。所以web的同事发现认证的调用返回结果花费了较多时间，需要排查是否是转发模块的问题。我们让现场同事复现，在认证期间多次采集堆栈，以下为其中一次，多次采集的信息都很接近

(gdb) 
Thread 77 (Thread 0x7f2951bec700 (LWP 8340)):
#0  0x00007f29583e5e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f2958416704 in usleep () from /lib64/libc.so.6
#2  0x0000561485525881 in HThreadTimerEx::run() ()
#3  0x00005614855230c3 in startHook ()
#4  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 76 (Thread 0x7f29513eb700 (LWP 8341)):
#0  0x00007f295841f483 in epoll_wait () from /lib64/libc.so.6
#1  0x00005614855bc2e2 in SIIOEpollAcceptThread::run() ()
#2  0x00005614855230c3 in startHook ()
#3  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 75 (Thread 0x7f2950bea700 (LWP 8342)):
#0  0x00007f295841f483 in epoll_wait () from /lib64/libc.so.6
#1  0x00005614855afe75 in SIIOEpollRecvThread::run() ()
#2  0x0000561485523179 in startHook ()
#3  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 74 (Thread 0x7f294bfff700 (LWP 8343)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x00005614855cfdca in SISimpleClockEventManger::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 73 (Thread 0x7f294b7fe700 (LWP 8344)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x000056148562af4b in SICallMgrAsync::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 72 (Thread 0x7f294affd700 (LWP 8345)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x000056148562af4b in SICallMgrAsync::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 71 (Thread 0x7f294a7fc700 (LWP 8420)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 70 (Thread 0x7f2949ffb700 (LWP 8421)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 69 (Thread 0x7f29497fa700 (LWP 8422)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 68 (Thread 0x7f293bfff700 (LWP 8558)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x0000561485523179 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 67 (Thread 0x7f293b7fe700 (LWP 8559)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 66 (Thread 0x7f293a7fc700 (LWP 8799)):
#0  0x00007f29586fc75d in close () from /lib64/libpthread.so.0
#1  0x0000561485544915 in HFileBase::myclose() ()
#2  0x000056148554495d in HFileBase::closex() ()
#3  0x00005614855465e2 in HFileLog::write_file(std::vector<HStringForLog*, std::allocator<HStringForLog*> >&, bool) ()
#4  0x0000561485546e35 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#5  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#6  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#7  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#8  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855e19c4 in SIDoPacketThread::run() ()
#10 0x00005614855272a6 in HThreadReal::run() ()
#11 0x0000561485523179 in startHook ()
#12 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 65 (Thread 0x7f2939ffb700 (LWP 8800)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 64 (Thread 0x7f2938ff9700 (LWP 10446)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 63 (Thread 0x7f290bfff700 (LWP 10464)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x0000561485523179 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 62 (Thread 0x7f290b7fe700 (LWP 10682)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x00005614855230c3 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 61 (Thread 0x7f290affd700 (LWP 10683)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x0000561485523179 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 60 (Thread 0x7f290a7fc700 (LWP 10684)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x00005614855230c3 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 59 (Thread 0x7f2909ffb700 (LWP 10685)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 58 (Thread 0x7f29097fa700 (LWP 10686)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 57 (Thread 0x7f2908ff9700 (LWP 10687)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x00005614855230c3 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 56 (Thread 0x7f2903fff700 (LWP 10688)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x00005614855230c3 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 55 (Thread 0x7f29037fe700 (LWP 10689)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 54 (Thread 0x7f2902ffd700 (LWP 10690)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 53 (Thread 0x7f29027fc700 (LWP 10693)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 52 (Thread 0x7f2901ffb700 (LWP 10694)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 51 (Thread 0x7f29017fa700 (LWP 10695)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 50 (Thread 0x7f2900ff9700 (LWP 10696)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 49 (Thread 0x7f28dbfff700 (LWP 10697)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 48 (Thread 0x7f28d3fff700 (LWP 10698)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 47 (Thread 0x7f28db7fe700 (LWP 10699)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 46 (Thread 0x7f28daffd700 (LWP 10700)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 45 (Thread 0x7f28da7fc700 (LWP 10701)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 44 (Thread 0x7f28d9ffb700 (LWP 10702)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 43 (Thread 0x7f28d97fa700 (LWP 10703)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x7f28d8ff9700 (LWP 10704)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x7f28d37fe700 (LWP 10705)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7f28d2ffd700 (LWP 10706)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7f28d27fc700 (LWP 10707)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7f28d1ffb700 (LWP 10708)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7f28d17fa700 (LWP 10709)):
---Type <return> to continue, or q <return> to quit---#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7f28d0ff9700 (LWP 10710)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7f28cbfff700 (LWP 10711)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7f28cb7fe700 (LWP 13773)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
---Type <return> to continue, or q <return> to quit---#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7f2898ff9700 (LWP 3210)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7f2899ffb700 (LWP 14782)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7f28c8ff9700 (LWP 28479)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 30 (Thread 0x7f293affd700 (LWP 12964)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f28caffd700 (LWP 31623)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
---Type <return> to continue, or q <return> to quit---#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f284affd700 (LWP 31624)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f28ca7fc700 (LWP 1983)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f28c9ffb700 (LWP 19345)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---Thread 25 (Thread 0x7f289b7fe700 (LWP 6241)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f2831ffb700 (LWP 23557)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
---Type <return> to continue, or q <return> to quit---#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f2830ff9700 (LWP 14578)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f28317fa700 (LWP 14579)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f2948ff9700 (LWP 28320)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f280affd700 (LWP 28321)):
---Type <return> to continue, or q <return> to quit---#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f28327fc700 (LWP 28322)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f28c97fa700 (LWP 28784)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f29397fa700 (LWP 28785)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
---Type <return> to continue, or q <return> to quit---#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f289bfff700 (LWP 28786)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f289affd700 (LWP 30100)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f289a7fc700 (LWP 14999)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 13 (Thread 0x7f28997fa700 (LWP 15000)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f284bfff700 (LWP 16169)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
---Type <return> to continue, or q <return> to quit---#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f284a7fc700 (LWP 32258)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f284b7fe700 (LWP 20070)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f2849ffb700 (LWP 20071)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---Thread 8 (Thread 0x7f28497fa700 (LWP 20072)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f28337fe700 (LWP 14542)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
---Type <return> to continue, or q <return> to quit---#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f280bfff700 (LWP 21263)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f2832ffd700 (LWP 21264)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f2833fff700 (LWP 7189)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x0000561485523179 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f2848ff9700 (LWP 7190)):
---Type <return> to continue, or q <return> to quit---#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f280b7fe700 (LWP 7191)):
#0  0x00007f29586fbccf in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007f29586fbda7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007f29586fbe45 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x00005614856170f7 in HEvent::wait(int) ()
#4  0x0000561485527261 in HThreadReal::run() ()
#5  0x00005614855230c3 in startHook ()
#6  0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---#7  0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f29599ae740 (LWP 8338)):
#0  0x00007f29583e5e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f2958416704 in usleep () from /lib64/libc.so.6
#2  0x00005614854d782a in USStatus::run() ()
#3  0x00005614854c42ba in main ()
(gdb) quit
A debugging session is active.

	Inferior 1 [process 8338] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
Detaching from program: /opt/leagsoft/LeagView/Bin/UniCenter, process 8338

过滤后的信息发现多次采集堆栈均抓到写文件的接口

Thread 68 (Thread 0x7f293bfff700 (LWP 8558)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x0000561485523179 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 66 (Thread 0x7f293a7fc700 (LWP 8799)):
#0  0x00007f29586fc75d in close () from /lib64/libpthread.so.0
#1  0x0000561485544915 in HFileBase::myclose() ()
#2  0x000056148554495d in HFileBase::closex() ()
#3  0x00005614855465e2 in HFileLog::write_file(std::vector<HStringForLog*, std::allocator<HStringForLog*> >&, bool) ()
#4  0x0000561485546e35 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#5  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#6  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
//SIServerRecvBufferMgrV2::server_recv_call这个接口就是web调用目的程序，转发模块内部调用的接口
//由此可知是由于写日志慢导致的调用流程变慢 需要排查 1 io 2 磁盘挂载 3 是否安装杀毒或者文件监控相关软件
#7  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#8  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855e19c4 in SIDoPacketThread::run() ()
#10 0x00005614855272a6 in HThreadReal::run() ()
#11 0x0000561485523179 in startHook ()
#12 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 63 (Thread 0x7f290bfff700 (LWP 10464)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x0000561485523179 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 62 (Thread 0x7f290b7fe700 (LWP 10682)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x00005614855230c3 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 61 (Thread 0x7f290affd700 (LWP 10683)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x0000561485523179 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 60 (Thread 0x7f290a7fc700 (LWP 10684)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x00005614855230c3 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 57 (Thread 0x7f2908ff9700 (LWP 10687)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x00005614855230c3 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

Thread 56 (Thread 0x7f2903fff700 (LWP 10688)):
#0  0x00007f29586fc4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f29586f7de6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f29586f7cdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005614855046a1 in HMutex::Lock() ()
#4  0x000056148550474f in HAutoMutex::HAutoMutex(HMutex*) ()
#5  0x0000561485546e26 in HFileLog::log_real_pointer(HStringForLog&, bool) ()
#6  0x00005614855a69bc in SIServerRecvBufferMgrV2::deal_recv_async_callback(SICallNodeAsync&) ()
#7  0x000056148562a5e1 in SICallMgrAsync::notify(SICallInternal const&, void*) ()
#8  0x00005614855ab2ed in SIServerRecvBufferMgrV2::server_recv_call(SIAutoRefParam<SITCPSession> const&, SIClientAuthInfo&, SIRecvOnePacketInfo&, SISendPacket&) ()
#9  0x00005614855aa33e in SIServerRecvBufferMgrV2::recv_packet(SIAutoRefParam<SITCPSession> const&, SIRecvOnePacketInfo&, SISendPacket&) ()
#10 0x00005614855e19c4 in SIDoPacketThread::run() ()
#11 0x00005614855272a6 in HThreadReal::run() ()
#12 0x00005614855230c3 in startHook ()
#13 0x00007f29586f5dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f295841eead in clone () from /lib64/libc.so.6

后续排除了io，挂载磁盘的情况。怀疑是客户服务器是不是安装了杀毒或者文件监控相关软件，让现场和客户沟通，最终确认是titanagent这个进程的问题。最开始的认证流程缓慢花费的时间为1min，正常情况下是2s

在这里插入图片描述