PostgreSQL 无法kill(pg_terminate_backend, pg_cancel_backend)的情况分析 - 进程hang strace,pstack...

标签

PostgreSQL , pg_terminate_backend , pg_cancel_backend , hang , pstack , strace

---

背景

当PostgreSQL进程无法被cancel, terminate时，进程处于什么状态？为什么无法退出？

例子

1、无法被kill的进程

Type "help" for help.  
  
postgres=# select pg_cancel_backend(60827);  
 pg_cancel_backend   
-------------------  
 t  
(1 row)  
  
postgres=# select pg_terminate_backend(60827);  
 pg_terminate_backend   
----------------------  
 t  
(1 row)  
  
postgres=# select pg_terminate_backend(60827);  
 pg_terminate_backend   
----------------------  
 t  
(1 row)  

2、查看进程当时的STACK，卡在__epoll_wait_nocancel

$pstack 60827  
  
#0  0x00007f4bced78f13 in __epoll_wait_nocancel () from /lib64/libc.so.6  
#1  0x0000000000753c35 in WaitEventSetWait ()  
#2  0x000000000076d103 in ConditionVariableSleep ()  
#3  0x00000000004cc4e1 in _bt_parallel_seize ()  
#4  0x00000000004ce433 in ?? ()  
#5  0x00000000004ce72e in ?? ()  
#6  0x00000000004cf071 in _bt_first ()  
#7  0x00000000004ccc2d in btgettuple ()  
#8  0x00000000004c617a in index_getnext_tid ()  
#9  0x0000000000650f87 in ?? ()  
#10 0x000000000063efa1 in ExecScan ()  
#11 0x000000000063d7c7 in ?? ()  
#12 0x000000000064719e in ?? ()  
#13 0x000000000064903c in ?? ()  
#14 0x000000000063d7c7 in ?? ()  
#15 0x000000000064c0c1 in ?? ()  
#16 0x000000000063d7c7 in ?? ()  
#17 0x000000000064719e in ?? ()  
#18 0x000000000064903c in ?? ()  
#19 0x000000000063d7c7 in ?? ()  
#20 0x000000000063c4f0 in standard_ExecutorRun ()  
#21 0x00007f4bc4cd7288 in ?? () from pg_stat_statements.so  
#22 0x00007f4bc48cf87f in ?? () from auto_explain.so  
#23 0x000000000077ed0b in ?? ()  
#24 0x00000000007800d0 in PortalRun ()  
#25 0x000000000077dc88 in PostgresMain ()  
#26 0x000000000070782c in PostmasterMain ()  
#27 0x000000000067d060 in main ()  

3、查看进程的strace

$strace -e trace=all -T -tt -p 60827  
Process 60827 attached - interrupt to quit  
19:21:14.881369 epoll_wait(270,   
  
  
^C <unfinished ...>  
Process 60827 detached  

4、查看这个系统调用的描述，等待某个FD的IO

$man epoll_wait  
EPOLL_WAIT(2)              Linux Programmer’s Manual             EPOLL_WAIT(2)  
  
NAME  
       epoll_wait, epoll_pwait - wait for an I/O event on an epoll file descriptor  
  
SYNOPSIS  
       #include <sys/epoll.h>  
  
       int epoll_wait(int epfd, struct epoll_event *events,  
                      int maxeve