关闭RAC一个节点，另一个节点HANG…

关闭一个节点，另一个节点HANG住无法连接的问题

reconfiguration started (old inc 46, new inc 48)
List of instances:
 1 (myinst: 1)
 Global Resource Directory frozen
 * dead instance detected - domain 0 invalid = TRUE
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Wed Nov 20 10:02:58 2013
 LMS 5: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:02:58 2013
 LMS 4: 2 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:02:58 2013
 LMS 8: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:02:58 2013
 LMS 9: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:02:58 2013
 LMS 6: 16 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:02:58 2013
 LMS 2: 10 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:02:58 2013
 LMS 1: 14 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:02:58 2013
 LMS 0: 6 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:02:58 2013
 LMS 7: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:02:58 2013
 LMS 3: 14 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Post SMON to start 1st pass IR
Wed Nov 20 10:02:58 2013
Instance recovery: looking for dead threads
Wed Nov 20 10:02:58 2013
Starting background process CJQ0
Instance recovery: lock domain invalid but no dead threads
Wed Nov 20 10:02:58 2013
CJQ0 started with pid=133, OS id=12036
 Submitted all GCS remote-cache requests
 Post SMON to start 1st pass IR
 Fix write in gcs resources
Reconfiguration complete
Wed Nov 20 10:05:41 2013
Incremental checkpoint up to RBA [0x237.3e265.0], current log tail at RBA [0x237.429a7.0]
Wed Nov 20 10:08:00 2013
Reconfiguration started (old inc 48, new inc 50)
List of instances:
 1 2 (myinst: 1)
 Global Resource Directory frozen
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Wed Nov 20 10:08:01 2013
 LMS 7: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:08:01 2013
 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:08:01 2013
 LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:08:01 2013
 LMS 9: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:08:01 2013
 LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:08:01 2013
 LMS 6: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:08:01 2013
Wed Nov 20 10:08:01 2013
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:08:01 2013
Wed Nov 20 10:08:01 2013
 LMS 8: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 LMS 4: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 LMS 5: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
Wed Nov 20 10:13:16 2013
Stopping background process CJQ0
Wed Nov 20 10:21:43 2013
Reconfiguration started (old inc 50, new inc 52)
List of instances:
 1 (myinst: 1)
 Global Resource Directory frozen
 * dead instance detected - domain 0 invalid = TRUE
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Wed Nov 20 10:21:43 2013
Wed Nov 20 10:21:43 2013
 LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:21:43 2013
 LMS 6: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:21:43 2013
 LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:21:43 2013
 LMS 9: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 LMS 7: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:21:43 2013
 LMS 8: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:21:43 2013
 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:21:43 2013
 LMS 4: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:21:43 2013
 LMS 5: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Wed Nov 20 10:21:43 2013
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Post SMON to start 1st pass IR
Wed Nov 20 10:21:44 2013
Instance recovery: looking for dead threads
Wed Nov 20 10:21:44 2013
Starting background process CJQ0
Instance recovery: lock domain invalid but no dead threads
 Submitted all GCS remote-cache requests
 Post SMON to start 1st pass IR
 Fix write in gcs resources
Wed Nov 20 10:21:44 2013
CJQ0 started with pid=84, OS id=13552
Reconfiguration complete
Wed Nov 20 10:21:44 2013
minact-scn: Master considers inst:2 dead
Wed Nov 20 10:25:44 2013
Incremental checkpoint up to RBA [0x237.434b0.0], current log tail at RBA [0x237.446b1.0]
Wed Nov 20 10:27:40 2013

***********************************************************************

Fatal NI connect error 12547, connecting to:
 (LOCAL=NO)

  VERSION INFORMATION:
 TNS for Solaris: Version 11.2.0.3.0 - Production
 Oracle Bequeath NT Protocol Adapter for Solaris: Version 11.2.0.3.0 - Production
 TCP/IP NT Protocol Adapter for Solaris: Version 11.2.0.3.0 - Production
  Time: 20-NOV-2013 10:27:40
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12547
   
TNS-12547: TNS:lost contact
    ns secondary err code: 12560
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
opiodr aborting process unknown ospid (13697) as a result of ORA-609
Wed Nov 20 10:32:25 2013

***********************************************************************

Fatal NI connect error 12547, connecting to:
 (LOCAL=NO)

  VERSION INFORMATION:
 TNS for Solaris: Version 11.2.0.3.0 - Production
 Oracle Bequeath NT Protocol Adapter for Solaris: Version 11.2.0.3.0 - Production
 TCP/IP NT Protocol Adapter for Solaris: Version 11.2.0.3.0 - Production
  Time: 20-NOV-2013 10:32:25
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12547
   
TNS-12547: TNS:lost contact
    ns secondary err code: 12560
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
opiodr aborting process unknown ospid (14099) as a result of ORA-609
Wed Nov 20 10:32:33 2013
Errors in file /u01/app/db/diag/rdbms/test3/TEST31/trace/TEST31_qmnc_24715.trc  (incident=260633):
ORA-00445: background process "q003" did not start after 120 seconds
Incident details in: /u01/app/db/diag/rdbms/test3/TEST31/incident/incdir_260633/TEST31_qmnc_24715_i260633.trc
Wed Nov 20 10:32:39 2013
Dumping diagnostic data in directory=[cdmp_20131120103239], requested by (instance=1, osid=24715 (QMNC)), summary=[incident=260633].
Wed Nov 20 10:41:19 2013

 

 

 

 

参考文档：

 ***Checked for relevance on 13-Jun-2012***

Error:  ORA 445 
Text:   background process did not start after seconds
-------------------------------------------------------------------------------
Cause:  The specified process did not start.
Action: Check and, if necessary, correct problems indicated by one or more of
        the following:
        the size of the SGA
        the operating system-specific initialization parameters
        accompanying messages
        the background trace file
        the executable image is not in the right location with the correct
        protections
        See also your operating system-specific Oracle documentation.