11.2.0.3 rac-单机的DG
由于数据库部署监控,部署时把DG端的tnsname.ora文件覆盖了,到时DG和主库RAC节点的通讯中断,然后就出现奇怪的问题;
DG节点:
alert.log:
ERROR: Failed to connect with connect string: (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0.3/
grid/bin/oracle)(ARGV0=oracle+ASM_pr00_popdg)(ENVS='ORACLE_HOME=/u01/app/11.2.0.3/grid,ORACLE_SID=+ASM')(ARGS
='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
NOTE: Deferred communication with ASM instance
Errors in file /u01/app/oracle/diag/rdbms/popdg/popdg/trace/popdg_pr00_8502.trc:
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
ORA-00308: cannot open archived log '+RECO_DM02/popdg/archivelog/2013_04_26/thread_2_seq_31174.2074.813766677
'
ORA-17503: ksfdopn:2 Failed to open file +RECO_DM02/popdg/archivelog/2013_04_26/thread_2_seq_31174.2074.81376
6677
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
NOTE: deferred map free for map id 946645589
Recovery interrupted!
grid/bin/oracle)(ARGV0=oracle+ASM_pr00_popdg)(ENVS='ORACLE_HOME=/u01/app/11.2.0.3/grid,ORACLE_SID=+ASM')(ARGS
='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
NOTE: Deferred communication with ASM instance
Errors in file /u01/app/oracle/diag/rdbms/popdg/popdg/trace/popdg_pr00_8502.trc:
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
ORA-00308: cannot open archived log '+RECO_DM02/popdg/archivelog/2013_04_26/thread_2_seq_31174.2074.813766677
'
ORA-17503: ksfdopn:2 Failed to open file +RECO_DM02/popdg/archivelog/2013_04_26/thread_2_seq_31174.2074.81376
6677
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
NOTE: deferred map free for map id 946645589
Recovery interrupted!
-------------------------------------------------------------------------------------------------------------
重新配置tnsnames.ora文件,保证DG和主库之间tnsping 正常通讯;
-------------------------------------------------------------------------------------------------------------
取消应用日志发现异常:
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
*
ERROR at line 1:
ORA-16136: Managed Standby Recovery not active
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
*
ERROR at line 1:
ORA-16136: Managed Standby Recovery not active
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ ONLY
--------------------
READ ONLY
很明确之前数据的恢复进程已经挂起了!
关闭并启动DG数据库
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 7.2421E+10 bytes
Fixed Size 2235808 bytes
Variable Size 5.1540E+10 bytes
Database Buffers 2.0670E+10 bytes
Redo Buffers 209866752 bytes
ORA-00205: error in identifying control file, check alert log for more info
Fixed Size 2235808 bytes
Variable Size 5.1540E+10 bytes
Database Buffers 2.0670E+10 bytes
Redo Buffers 209866752 bytes
ORA-00205: error in identifying control file, check alert log for more info
意外发生了,
检查DG的alert.log:
ERROR: Failed to connect with connect string: (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0.3/grid/bin/oracle)(ARGV0=oracle+ASM_asmb_popdg)(ENVS='ORACLE_HOME=/u01/app/11.2.0.3/grid,ORACLE_SID=+ASM')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+RECO_DM02/popdg/controlfile/standbyctl02.ctl.257.799339773'
ORA-17503: ksfdopn:2 Failed to open file +RECO_DM02/popdg/controlfile/standbyctl02.ctl.257.799339773
ORA-15001: diskgroup "RECO_DM02" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA_DM02/popdg/controlfile/standbyctl01.ctl.257.799339639'
ORA-17503: ksfdopn:2 Failed to open file +DATA_DM02/popdg/controlfile/standbyctl01.ctl.257.799339639
ORA-15001: diskgroup "DATA_DM02" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
ORA-205 signalled during: alter database mount...
Fri Apr 26 15:40:48 2013
Starting background process ASMB
Fri Apr 26 15:40:48 2013
ASMB started with pid=27, OS id=31912
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+RECO_DM02/popdg/controlfile/standbyctl02.ctl.257.799339773'
ORA-17503: ksfdopn:2 Failed to open file +RECO_DM02/popdg/controlfile/standbyctl02.ctl.257.799339773
ORA-15001: diskgroup "RECO_DM02" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA_DM02/popdg/controlfile/standbyctl01.ctl.257.799339639'
ORA-17503: ksfdopn:2 Failed to open file +DATA_DM02/popdg/controlfile/standbyctl01.ctl.257.799339639
ORA-15001: diskgroup "DATA_DM02" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
ORA-205 signalled during: alter database mount...
Fri Apr 26 15:40:48 2013
Starting background process ASMB
Fri Apr 26 15:40:48 2013
ASMB started with pid=27, OS id=31912
***********************************************************************
Fatal NI connect error 12547, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0.3/grid/bin/oracle)(ARGV0=oracle+ASM_asmb_popdg)(ENVS='ORACLE_HOME=/u01/app/11.2.0.3/grid,ORACLE_SID=+ASM')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser)(CONNECT_DATA=(CID=( PROGRAM=oracle@popdg)(HOST=popdg)(USER=oracle))))
(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0.3/grid/bin/oracle)(ARGV0=oracle+ASM_asmb_popdg)(ENVS='ORACLE_HOME=/u01/app/11.2.0.3/grid,ORACLE_SID=+ASM')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser)(CONNECT_DATA=(CID=( PROGRAM=oracle@popdg)(HOST=popdg)(USER=oracle))))
全部都是这种错误;
-------------------------------------------------------------------------------------------------------------
怀疑是DG节点的ASM盘组出现问题,检查DG节点的crs状态和磁盘组状态:
[grid@popdg ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....DM02.dg ora....up.type ONLINE ONLINE popdg
ora....ER.lsnr ora....er.type ONLINE ONLINE popdg
ora....DM02.dg ora....up.type ONLINE ONLINE popdg
ora.asm ora.asm.type ONLINE ONLINE popdg
ora.cssd ora.cssd.type ONLINE ONLINE popdg
ora.diskmon ora....on.type OFFLINE OFFLINE
ora.evmd ora.evm.type ONLINE ONLINE popdg
ora.ons ora.ons.type OFFLINE OFFLINE
Name Type Target State Host
------------------------------------------------------------
ora....DM02.dg ora....up.type ONLINE ONLINE popdg
ora....ER.lsnr ora....er.type ONLINE ONLINE popdg
ora....DM02.dg ora....up.type ONLINE ONLINE popdg
ora.asm ora.asm.type ONLINE ONLINE popdg
ora.cssd ora.cssd.type ONLINE ONLINE popdg
ora.diskmon ora....on.type OFFLINE OFFLINE
ora.evmd ora.evm.type ONLINE ONLINE popdg
ora.ons ora.ons.type OFFLINE OFFLINE
SQL> select name,state from v$asm_diskgroup;
NAME STATE
--------- ----------
DATA_DM02 MOUNTED
RECO_DM02 MOUNTED
--------- ----------
DATA_DM02 MOUNTED
RECO_DM02 MOUNTED
一切都正常!!!!!
------------------------------------------------------------------------------------------------------------
那么问题出现在哪儿,怀疑是ORACLE_HOME/bin和grid_home/bin下的oracel执行文件出现问题,于是检查两个bin下的oracle文件的执行权限:
[grid@node bin]$ls -l oracle
-rwxr-xr-x oracle
[oracle@node bin]$ ls -l oracle
-r-srws--x oracle
--------------------------------------------------------------------------------------------------------------
很明显grid_home/bin下的oracle执行文件的权限被修改了:
修改执行权限:
chmod 6751 oracle
然后启动数据库ok!!!
这问题相当的诡异,还没有研究明白为什么grid_home/bin下的oracle执行文件的权限会被修改?什么操作引起的权限修改?按道理说覆盖了tnsnames.ora文件,DG和主库的通讯中断,不会传输日志,不能进行恢复,为什么直接导致那么多问题出现,还有待研究!
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/14507784/viewspace-759401/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/14507784/viewspace-759401/