db2 SQL1034C The database is damaged. SQLSTATE=58031

背景：

维护的项目，大半夜的db2挂了，版本：V8.2，db2客服不支持服务了（PS：他们TM有多不待见低版本啊 擦！！！），竟然让我去谷歌怎么操作，我要在谷歌上找到了，我还找你800干嘛！

状态：

[db2inst@db2 ~]$ db2 connect to ddn
SQL1034C The database is damaged. All applications processing the database
have been stopped. SQLSTATE=58031
[db2inst@db2 ~]$
[db2inst@db2 ~]$ db2 restart db ddn
SQL1034C The database is damaged. All applications processing the database
have been stopped. SQLSTATE=58031

连接db2失败，SQL1034C，SQLSTATE=58031，dbstop提示active状态，关不了

[db2inst@db2 backup]$ db2stop
SQL1025N The database manager was not stopped because databases are still active.

使出杀手锏 db2stop force，然后。。。然后就没有然后了，一直在执行，20分钟之后还是在执行，ctrl+c也不管用了。。。果断kill掉！

正在绝望之际（我都想到了restore了），ps -ef|grep db2了一下

[db2inst@db2 backup]$ ps -ef|grep db2
root 4393 1 0 Nov11 ? 00:07:14 /opt/IBM/db2/V8.1/bin/db2fmcd 
root 31132 1 0 Nov11 ? 00:00:24 db2licd
root 7991 1 0 Dec02 ? 00:00:00 db2wdog 
db2inst 7997 7991 0 Dec02 ? 00:00:00 db2sysc 
root 7999 7997 0 Dec02 ? 00:06:13 db2ckpwd 
root 8001 7997 0 Dec02 ? 00:06:12 db2ckpwd 
root 8004 7997 0 Dec02 ? 00:06:13 db2ckpwd 
root 8007 7997 0 Dec02 ? 00:00:54 db2gds 
db2inst 8027 7997 0 Dec02 ? 00:06:32 db2ipccm 
db2inst 8033 7997 0 Dec02 ? 00:00:11 db2tcpcm 
db2inst 8034 7997 0 Dec02 ? 00:00:11 db2tcpcm 
db2inst 8242 7997 0 Dec02 ? 00:00:00 db2resync 
db2inst 8243 8007 0 Dec02 ? 00:00:00 db2srvlst 
db2inst 8245 7997 0 Dec02 ? 00:09:33 db2hmon ,0,0,0,1,0,0,0,1e014,2,0,1,9fe0,0x10db8000,0x10db8000,15fc000,daf0005,2,38b2800c
db2inst 8803 8034 0 Dec02 ? 00:05:47 db2agent (idle) 
db2inst 7661 8034 0 Dec02 ? 01:00:30 db2agent (idle) 
db2inst 23381 8027 0 Dec02 ? 00:58:57 db2agent (idle) 
db2inst 24551 8007 0 Dec02 ? 00:13:30 db2agent (idle) 
db2inst 24553 8007 0 Dec02 ? 00:43:35 db2agent (idle) 
db2inst 24554 8033 0 Dec02 ? 00:01:27 db2agent (idle) 
db2inst 29972 8034 0 Dec02 ? 00:57:41 db2agent (DDN) 
db2inst 29974 8034 0 Dec02 ? 00:43:32 db2agent (idle) 
db2inst 29975 8034 0 Dec02 ? 00:01:43 db2agent (idle) 
db2inst 29976 8034 0 Dec02 ? 00:59:13 db2agent (idle) 
db2inst 30002 8033 0 Dec02 ? 00:01:15 db2agent (idle) 
db2inst 30215 8027 0 Dec02 ? 00:59:10 db2agent (idle) 
db2inst 30216 8033 0 Dec02 ? 00:58:58 db2agent (idle) 
db2inst 30456 8027 0 Dec02 ? 00:58:15 db2agent (idle) 
db2inst 21007 8027 0 Dec02 ? 01:01:08 db2agent (idle) 
db2inst 19938 8027 0 Dec02 ? 00:01:41 db2agent (idle) 
db2fenc 12360 7997 0 Dec02 ? 00:00:00 db2fmp ,1,0,0,0,0,0,0,1e014,2,0,1,29fe0,0x10db8000,0x10db8000,15fc000,daf0005,2,1ae18032
db2inst 27792 8027 0 Dec02 ? 00:58:15 db2agent (idle) 
db2inst 7418 8027 0 Dec02 ? 00:56:58 db2agent (idle) 
db2inst 14286 8007 0 Dec02 ? 00:04:13 db2logts (DDN) 
db2inst 14287 8007 0 Dec02 ? 00:00:13 db2logmgr (DDN) 
db2inst 14288 8007 0 Dec02 ? 00:00:11 db2loggr (DDN) 
db2inst 14577 8007 0 Dec02 ? 00:24:32 db2loggw (DDN) 
db2inst 14578 8007 0 Dec02 ? 00:00:04 db2lfr (DDN) 
db2inst 14621 8007 0 Dec02 ? 00:00:01 db2dlock (DDN) 
db2inst 15092 8007 0 Dec02 ? 01:24:00 db2pfchr 
db2inst 15093 8007 0 Dec02 ? 00:09:28 db2pfchr 
db2inst 15094 8007 0 Dec02 ? 00:00:23 db2pfchr 
db2inst 15095 8007 0 Dec02 ? 00:00:06 db2pfchr 
db2inst 15096 8007 0 Dec02 ? 00:00:01 db2pfchr 
db2inst 15097 8007 0 Dec02 ? 00:00:00 db2pfchr 
db2inst 15098 8007 0 Dec02 ? 00:00:01 db2pclnr 
db2inst 15099 8007 0 Dec02 ? 00:00:01 db2pclnr 
db2inst 15100 8007 0 Dec02 ? 00:00:01 db2pclnr 
db2inst 15101 8007 0 Dec02 ? 00:00:01 db2pclnr 
db2inst 15103 8007 0 Dec02 ? 00:00:00 db2event (DB2DETAILDEADLOCK) 
db2inst 7967 8027 0 Dec06 ? 00:00:05 db2agent (idle) 
db2inst 28370 8033 0 Dec06 ? 00:00:14 db2agent (idle) 
db2inst 28371 8033 0 Dec06 ? 00:00:00 db2agent (idle) 
db2inst 28375 8034 0 Dec06 ? 00:00:00 db2agent (idle) 
db2inst 28376 8033 0 Dec06 ? 00:00:01 db2agent (idle) 
db2inst 28377 8034 0 Dec06 ? 00:00:00 db2agent (idle) 
db2inst 28378 8034 0 Dec06 ? 00:00:00 db2agent (idle) 
db2inst 28971 8027 0 Dec06 ? 00:00:13 db2agent (idle) 
db2inst 3723 8027 0 Dec10 ? 00:37:32 db2agent (instance) 
db2inst 20626 8034 0 Dec10 ? 00:00:00 db2agent (idle) 
db2inst 20900 8034 0 Dec10 ? 00:00:06 db2agent (idle) 
db2inst 24456 8033 0 Dec11 ? 00:00:18 db2agent (idle) 
db2inst 24459 8034 0 Dec11 ? 00:00:34 db2agent (idle) 
db2inst 16316 8027 0 Dec11 ? 00:00:12 db2agent (idle) 
db2inst 12288 8027 0 Dec12 ? 00:32:01 db2agent (idle) 
db2inst 17658 8027 0 Dec12 ? 00:31:56 db2agent (idle) 
db2inst 17219 8027 0 Dec12 ? 00:31:42 db2agent (idle) 
db2inst 17220 8033 0 Dec12 ? 00:32:10 db2agent (idle) 
db2inst 17221 8034 0 Dec12 ? 00:31:49 db2agent (idle) 
db2inst 21551 8034 0 Dec15 ? 00:00:01 db2agent (idle) 
db2inst 21552 8034 0 Dec15 ? 00:00:01 db2agent (idle) 
db2inst 21553 8033 0 Dec15 ? 00:00:01 db2agent (idle) 
db2inst 21554 8033 0 Dec15 ? 00:00:00 db2agent (idle) 
db2inst 21878 8027 0 Dec15 ? 00:00:01 db2agent (idle) 
db2inst 25209 8027 0 Dec16 ? 00:00:03 db2agent (idle) 
db2inst 25210 8027 0 Dec16 ? 00:00:26 db2agent (idle) 
db2inst 9376 8027 0 Dec16 ? 00:00:00 db2agent (idle) 
db2inst 9432 8027 0 Dec16 ? 00:00:22 db2agent (idle) 
db2inst 26772 8034 0 Dec16 ? 00:00:00 db2agent (idle) 
db2inst 26773 8033 0 Dec16 ? 00:00:00 db2agent (idle) 
db2inst 26774 8033 0 Dec16 ? 00:00:00 db2agent (idle) 
db2inst 26775 8033 0 Dec16 ? 00:00:00 db2agent (idle) 
db2inst 26776 8034 0 Dec16 ? 00:00:00 db2agent (idle) 
db2inst 26777 8034 0 Dec16 ? 00:00:00 db2agent (idle) 
db2inst 26778 8034 0 Dec16 ? 00:00:00 db2agent (idle) 
db2inst 26779 8034 0 Dec16 ? 00:00:00 db2agent (idle) 
db2inst 26780 8033 0 Dec16 ? 00:00:00 db2agent (idle) 
db2inst 26781 8034 0 Dec16 ? 00:15:37 db2agent (idle) 
db2inst 26782 8033 0 Dec16 ? 00:15:39 db2agent (idle) 
db2inst 23262 8027 0 Dec19 ? 00:17:16 db2agent (idle) 
db2inst 25730 8027 0 Dec19 ? 00:16:55 db2agent (idle) 
db2inst 17752 8027 0 Dec20 ? 00:00:04 db2agent (idle) 
db2inst 20781 8027 0 Dec20 ? 00:00:02 db2agent (idle) 
db2inst 22248 8027 0 Dec20 ? 00:00:03 db2agent (idle) 
db2inst 9924 8034 0 Dec23 ? 00:00:00 db2agent (idle) 
db2inst 10056 8033 0 Dec23 ? 00:00:00 db2agent (idle) 
db2inst 10073 8033 0 Dec23 ? 00:00:00 db2agent (idle) 
db2inst 10116 8034 0 Dec23 ? 00:00:11 db2agent (idle) 
db2inst 10124 8033 0 Dec23 ? 00:00:00 db2agent (idle) 
db2inst 10127 8033 0 Dec23 ? 00:00:08 db2agent (idle) 
db2inst 10132 8034 0 Dec23 ? 00:00:00 db2agent (idle) 
db2inst 10147 8027 0 Dec23 ? 00:00:03 db2agent (idle) 
db2inst 21278 8027 0 Dec23 ? 00:00:00 db2agent (idle) 
db2inst 24580 8027 0 Dec23 ? 00:00:09 db2agent (idle) 
db2inst 4780 8027 0 Dec23 ? 00:00:08 db2agent (idle) 
db2inst 15310 8027 0 Dec25 ? 00:00:24 db2agent (idle) 
db2inst 5176 8027 0 Dec26 ? 00:00:05 db2agent (idle) 
db2inst 8562 8027 1 Dec26 ? 00:56:14 db2agent (idle) 
db2inst 13939 8027 0 Dec26 ? 00:00:07 db2agent (idle) 
root 21800 13999 0 02:21 pts/0 00:00:00 su - db2inst
db2inst 21801 21800 0 02:21 pts/0 00:00:00 -bash
root 17435 7126 0 02:31 pts/1 00:00:00 su - db2inst
db2inst 17436 17435 0 02:31 pts/1 00:00:00 -bash
db2inst 28774 1 0 02:38 pts/1 00:00:00 /home/db2inst/sqllib/bin/db2bp 17436A506 5 A
root 15947 1 0 03:07 pts/2 00:00:00 /home/db2inst/sqllib/bin/db2bp 27905A0 5 A
db2inst 17407 1 0 03:07 pts/0 00:00:00 /home/db2inst/sqllib/bin/db2bp 21801A506 5 A
db2inst 4873 17436 0 03:32 pts/1 00:00:00 ps -ef
db2inst 4874 17436 0 03:32 pts/1 00:00:00 grep db2

突然发现了一条db2inst 15103 8007 0 Dec02 ? 00:00:00 db2event (DB2DETAILDEADLOCK)，我擦！难道定时任务执行删除把条给弄死了？！

kill -9 15103之后，世界瞬间安静了，幸福来的如此突然！！！

[db2inst@db2 backup]$ 
[db2inst@db2 backup]$ db2 connect to ddn
SQL1032N No start database manager command was issued. SQLSTATE=57019
[db2inst@db2 backup]$
[db2inst@db2 backup]$  
[db2inst@db2 backup]$ db2start
SQL1063N DB2START processing was successful.
[db2inst@db2 backup]$
[db2inst@db2 backup]$  
[db2inst@db2 backup]$ db2 connect to ddn
SQL1031N The database directory cannot be found on the indicated file system. 
SQLSTATE=58031
[db2inst@db2 backup]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 64G 48G 13G 79% /
/dev/sda1 190M 15M 167M 8% /boot
none 1013M 0 1013M 0% /dev/shm
[db2inst@db2 backup]$ 
[db2inst@db2 backup]$ 

竟然不报58031了，然后我就知道数据库双击切换到了备机，哈哈，从备机上连接数据库一切正常！

分享过程，与大家共勉！！