ORA-03113: end-of-file on communication channel

晚上有个项目经理问，客户机房断电后数据库无法打开，登录环境操作如下

SQL>  startup;
ORACLE instance started.

Total System Global Area 3.2212E+10 bytes
Fixed Size           30149216 bytes
Variable Size         2818572288 bytes
Database Buffers     2.9327E+10 bytes
Redo Buffers           36958208 bytes
ORA-03113: end-of-file on communication channel
Process ID: 17340
Session ID: 667 Serial number: 32855

SQL>  startup mount;
ORACLE instance started.

Total System Global Area 3.2212E+10 bytes
Fixed Size           30149216 bytes
Variable Size         2818572288 bytes
Database Buffers     2.9327E+10 bytes
Redo Buffers           36958208 bytes
ORA-03113: end-of-file on communication channel
Process ID: 17340
Session ID: 667 Serial number: 32855

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 3.2212E+10 bytes
Fixed Size           30149216 bytes
Variable Size         2818572288 bytes
Database Buffers     2.9327E+10 bytes
Redo Buffers           36958208 bytes
SQL> alter database mount;
alter database mount
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 22521
Session ID: 667 Serial number: 10125

到这里感觉是控制文件出问题，查看日志

.... (PID:21658): Redo network throttle feature is disabled at mount time
2023-05-22T19:23:18.596684+08:00
LGWR (ospid: 21625): terminating the instance
2023-05-22T19:23:18.597726+08:00
System state dump requested by (instance=1, osid=21625 (LGWR)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_diag_21602.trc
2023-05-22T19:23:18.812746+08:00
Dumping diagnostic data in directory=[cdmp_20230522192318], requested by (instance=1, osid=21625 (LGWR)), summary=[abn
ormal instance termination].
2023-05-22T19:23:19.904882+08:00
Instance terminated by LGWR, pid = 21625
2023-05-22T19:28:41.911980+08:00
Starting ORACLE instance (normal) (OS id: 22273)

查看trc文件，发现确实控制文件故障了

[root@db trace]# ll *21625*
-rw-r----- 1 oracle oinstall 595112 May 22 19:23 orcl_lgwr_21625.trc
-rw-r----- 1 oracle oinstall   7058 May 22 19:23 orcl_lgwr_21625.trm

[root@db trace]# strings orcl_lgwr_21625.trc |more
..........................
..........................
..........................
Oracle process number: 23
Unix process pid: 21625, image: oracle@db (LGWR)
*** 2023-05-22T19:21:33.552565+08:00
*** SESSION ID:(2186.11279) 2023-05-22T19:21:33.552584+08:00
*** CLIENT ID:() 2023-05-22T19:21:33.552590+08:00
*** SERVICE NAME:() 2023-05-22T19:21:33.552596+08:00
*** MODULE NAME:() 2023-05-22T19:21:33.552601+08:00
*** ACTION NAME:() 2023-05-22T19:21:33.552607+08:00
*** CLIENT DRIVER:() 2023-05-22T19:21:33.552612+08:00
Created 2 redo writer workers (2 groups of 1 each)
*** 2023-05-22T19:23:18.396667+08:00
Error: kccpb_sanity_check_2
Control file sequence number mismatch!
fhcsq: 348692 bhcsq: 348722 cfn 0 rpbn 326
..........................
..........................
..........................
 恢复过程就不要细说了，使用备份恢复控制文件或者新建控制文件

MOS文档：

Database Can Not Startup With Error "Control file sequence number mismatch!" (Doc ID 2411155.1)