继昨天发生san存储切换导致io等待异常高的问题后,晚上客户对测试环境的数据库进行了远程启动,因为库比较多,监控process都起来了。客户就发邮件通知测试组继续测试。
结果早上测试反馈有一个库怎么都等不了。
他们提供的日志如下:
sqlplus test3c/xxxx@testdb
SQL*Plus: Release 11.2.0.2.0 Production on Sat Jun 21 12:59:36 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
ERROR:
ORA-01033: ORACLE initialization or shutdown in progress
Process ID: 0
Session ID: 0 Serial number: 0
猛一看是数据库可能没起来,但是根据客户的邮件说库都起来了。
就开始排查,大周末的只能很艰难的连上vpn来看了。
通过客户端,问题可以复现,直接连入db服务器,通过dba账户来连接。
sqlplus jianrong/jianrong@testdb
SQL*Plus: Release 11.2.0.2.0 Production on Sat Jun 21 13:05:42 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
ERROR:
ORA-12154: TNS:could not resolve the connect identifier specified
通过sysdba来连接,貌似是连上了。
sqlplus / as sysdba
SQL*Plus: Release 11.2.0.2.0 Production on Sat Jun 21 13:07:06 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> show parameter back
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
background_core_dump string partial
查看对应的process的情况,process确实存在。而且是半夜1点多起的。
3071 18455 1 0 01:51 ? 00:00:00 ora_smon_TESTDB
查到这,还没有找到其他的线索,来看看alert日志怎么说吧。一查看直接晕倒。
。。。。
alter database open
Sat Jun 21 13:02:48 2014
Errors in file /opt/app/oracle/testdb/diag/rdbms/testdb/TESTDB/trace/TESTDB_dbw0_18440.trc:
ORA-01157: cannot identify/lock data file 3 - see DBWR trace file
ORA-01110: data file 3: '/u01/oracle/TESTDB/orarbs01/rbs/undotbs01.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/app/oracle/testdb/diag/rdbms/testdb/TESTDB/trace/TESTDB_ora_18358.trc:
ORA-01157: cannot identify/lock data file 3 - see DBWR trace file
ORA-01110: data file 3: '/u01/oracle/TESTDB/orarbs01/rbs/undotbs01.dbf'
ORA-1157 signalled during: alter database open...
原来库就没起来,被误导了。查看库的状态。
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
MOUNTED
剩下的事情就是从备份中拿到undo来做恢复了。给客户简单说了一下情况,马上问题就解决了,虚惊一场。
结果早上测试反馈有一个库怎么都等不了。
他们提供的日志如下:
sqlplus test3c/xxxx@testdb
SQL*Plus: Release 11.2.0.2.0 Production on Sat Jun 21 12:59:36 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
ERROR:
ORA-01033: ORACLE initialization or shutdown in progress
Process ID: 0
Session ID: 0 Serial number: 0
猛一看是数据库可能没起来,但是根据客户的邮件说库都起来了。
就开始排查,大周末的只能很艰难的连上vpn来看了。
通过客户端,问题可以复现,直接连入db服务器,通过dba账户来连接。
sqlplus jianrong/jianrong@testdb
SQL*Plus: Release 11.2.0.2.0 Production on Sat Jun 21 13:05:42 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
ERROR:
ORA-12154: TNS:could not resolve the connect identifier specified
通过sysdba来连接,貌似是连上了。
sqlplus / as sysdba
SQL*Plus: Release 11.2.0.2.0 Production on Sat Jun 21 13:07:06 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> show parameter back
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
background_core_dump string partial
。。。。。。
查看对应的process的情况,process确实存在。而且是半夜1点多起的。
3071 18455 1 0 01:51 ? 00:00:00 ora_smon_TESTDB
查到这,还没有找到其他的线索,来看看alert日志怎么说吧。一查看直接晕倒。
。。。。
alter database open
Sat Jun 21 13:02:48 2014
Errors in file /opt/app/oracle/testdb/diag/rdbms/testdb/TESTDB/trace/TESTDB_dbw0_18440.trc:
ORA-01157: cannot identify/lock data file 3 - see DBWR trace file
ORA-01110: data file 3: '/u01/oracle/TESTDB/orarbs01/rbs/undotbs01.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/app/oracle/testdb/diag/rdbms/testdb/TESTDB/trace/TESTDB_ora_18358.trc:
ORA-01157: cannot identify/lock data file 3 - see DBWR trace file
ORA-01110: data file 3: '/u01/oracle/TESTDB/orarbs01/rbs/undotbs01.dbf'
ORA-1157 signalled during: alter database open...
原来库就没起来,被误导了。查看库的状态。
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
MOUNTED
剩下的事情就是从备份中拿到undo来做恢复了。给客户简单说了一下情况,马上问题就解决了,虚惊一场。
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/23718752/viewspace-1190720/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/23718752/viewspace-1190720/