2010-03-24 16:21:36.865: [ OCRMSG][1485781312]prom_rpc: CLSC recv failure..ret code 7
2010-03-24 16:21:36.865: [ OCRMSG][1485781312]prom_rpc: possible OCR retry scenario
2010-03-24 16:21:36.865: [ OCRMAS][1215129920]th_master:13: I AM THE NEW OCR MASTER at incar 2. Node Number 2
2010-03-24 16:21:36.865: [ OCRMSG][1487882560]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.
2010-03-24 16:21:36.865: [ OCRMSG][1521453376]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.
2010-03-24 16:21:36.865: [ OCRMSG][1523554624]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.
2010-03-24 16:21:36.865: [ OCRMSG][1525655872]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.
2010-03-24 16:21:36.866: [ OCRMSG][1527757120]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.
2010-03-24 16:21:37.206: [ CRSCOMM][1529858368]0CLEANUP: Searching for connections to failed node racdb5
2010-03-24 16:21:37.206: [ CRSEVT][1529858368]0Processing member leave for racdb5, incarnation: 3
2010-03-24 16:21:37.207: [ CRSD][1529858368]0SM: recovery in process: 8
2010-03-24 16:21:37.207: [ CRSEVT][1529858368]0Do failover for: racdb5
2010-03-24 16:21:37.985: [ OCRRAW][1215129920]proprioo: for disk 0 (/dev/raw/raw11), id match (1), my id set (395007438,1028247821) total id sets (1), 1st set (395007438,1028247821), 2nd set (0,0) my votes (2), total votes (2)
2010-03-24 16:21:38.002: [ OCRMAS][1215129920]th_master: Deleted ver keys from cache (master)
2010-03-24 16:21:39.672: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.705: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.730: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.754: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.783: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.807: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.832: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.866: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.891: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:40.261: [ CRSRES][1521453376]0startRunnable: setting CLI values
2010-03-24 16:21:40.529: [ CRSRES][1527757120]0startRunnable: setting CLI values
2010-03-24 16:21:40.544: [ CRSRES][1487882560]0startRunnable: setting CLI values
2010-03-24 16:21:40.739: [ CRSRES][1521453376]0Attempting to start `ora.racdb5.vip` on member `racdb6`
2010-03-24 16:21:40.839: [ CRSRES][1487882560]0Attempting to start `ora.racweb.db` on member `racdb6`
2010-03-24 16:21:40.905: [ CRSRES][1527757120]0Attempting to start `ora.racweb.racwebdb.cs` on member `racdb6`
2010-03-24 16:21:42.658: [ CRSRES][1527757120]0Start of `ora.racweb.racwebdb.cs` on member `racdb6` succeeded.
2010-03-24 16:21:42.664: [ CRSRES][1487882560]0Start of `ora.racweb.db` on member `racdb6` succeeded.
2010-03-24 16:21:45.886: [ CRSRES][1521453376]0Start of `ora.racdb5.vip` on member `racdb6` succeeded.
2010-03-24 16:21:45.931: [ CRSEVT][1529858368]0Post recovery done evmd event for: racdb5
2010-03-24 16:21:45.931: [ CRSD][1529858368]0SM: recoveryDone: 0
2010-03-24 16:21:45.932: [ CRSEVT][1529858368]0Processing RecoveryDone
2010-03-24 16:22:08.968: [ CRSRES][1529858368]0StopResource: setting CLI values
2010-03-24 16:39:20.607: [ OCRSRV][1529858368]s_update_remote_cache_int: FAILED TO RCV ACK FROM node 1 retcode 7
2010-03-24 16:39:20.729: [ CRSCOMM][1485781312]0CLEANUP: Searching for connections to failed node racdb5
2010-03-24 16:39:20.729: [ CRSEVT][1485781312]0Processing member leave for racdb5, incarnation: 5
2010-03-24 16:39:20.730: [ CRSD][1485781312]0SM: recovery in process: 8
2010-03-24 16:39:20.730: [ CRSEVT][1485781312]0Do failover for: racdb5
2010-03-24 16:39:21.751: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.772: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.786: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.801: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.817: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.831: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.846: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.968: [ CRSRES][1529858368]0startRunnable: setting CLI values
2010-03-24 16:39:21.979: [ CRSRES][1529858368]0Attempting to start `ora.racdb5.vip` on member `racdb6`
2010-03-24 16:39:25.268: [ CRSRES][1529858368]0Start of `ora.racdb5.vip` on member `racdb6` succeeded.
2010-03-24 16:39:25.289: [ CRSEVT][1485781312]0Post recovery done evmd event for: racdb5
2010-03-24 16:39:25.289: [ CRSD][1485781312]0SM: recoveryDone: 0
2010-03-24 16:39:25.290: [ CRSEVT][1485781312]0Processing RecoveryDone
2010-03-24 16:39:52.033: [ CRSRES][1485781312]0StopResource: setting CLI values
2010-03-24 16:21:36.865: [ OCRMSG][1485781312]prom_rpc: possible OCR retry scenario
2010-03-24 16:21:36.865: [ OCRMAS][1215129920]th_master:13: I AM THE NEW OCR MASTER at incar 2. Node Number 2
2010-03-24 16:21:36.865: [ OCRMSG][1487882560]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.
2010-03-24 16:21:36.865: [ OCRMSG][1521453376]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.
2010-03-24 16:21:36.865: [ OCRMSG][1523554624]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.
2010-03-24 16:21:36.865: [ OCRMSG][1525655872]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.
2010-03-24 16:21:36.866: [ OCRMSG][1527757120]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.
2010-03-24 16:21:37.206: [ CRSCOMM][1529858368]0CLEANUP: Searching for connections to failed node racdb5
2010-03-24 16:21:37.206: [ CRSEVT][1529858368]0Processing member leave for racdb5, incarnation: 3
2010-03-24 16:21:37.207: [ CRSD][1529858368]0SM: recovery in process: 8
2010-03-24 16:21:37.207: [ CRSEVT][1529858368]0Do failover for: racdb5
2010-03-24 16:21:37.985: [ OCRRAW][1215129920]proprioo: for disk 0 (/dev/raw/raw11), id match (1), my id set (395007438,1028247821) total id sets (1), 1st set (395007438,1028247821), 2nd set (0,0) my votes (2), total votes (2)
2010-03-24 16:21:38.002: [ OCRMAS][1215129920]th_master: Deleted ver keys from cache (master)
2010-03-24 16:21:39.672: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.705: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.730: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.754: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.783: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.807: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.832: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.866: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:39.891: [ CRSRES][1529858368]0 startup = 0
2010-03-24 16:21:40.261: [ CRSRES][1521453376]0startRunnable: setting CLI values
2010-03-24 16:21:40.529: [ CRSRES][1527757120]0startRunnable: setting CLI values
2010-03-24 16:21:40.544: [ CRSRES][1487882560]0startRunnable: setting CLI values
2010-03-24 16:21:40.739: [ CRSRES][1521453376]0Attempting to start `ora.racdb5.vip` on member `racdb6`
2010-03-24 16:21:40.839: [ CRSRES][1487882560]0Attempting to start `ora.racweb.db` on member `racdb6`
2010-03-24 16:21:40.905: [ CRSRES][1527757120]0Attempting to start `ora.racweb.racwebdb.cs` on member `racdb6`
2010-03-24 16:21:42.658: [ CRSRES][1527757120]0Start of `ora.racweb.racwebdb.cs` on member `racdb6` succeeded.
2010-03-24 16:21:42.664: [ CRSRES][1487882560]0Start of `ora.racweb.db` on member `racdb6` succeeded.
2010-03-24 16:21:45.886: [ CRSRES][1521453376]0Start of `ora.racdb5.vip` on member `racdb6` succeeded.
2010-03-24 16:21:45.931: [ CRSEVT][1529858368]0Post recovery done evmd event for: racdb5
2010-03-24 16:21:45.931: [ CRSD][1529858368]0SM: recoveryDone: 0
2010-03-24 16:21:45.932: [ CRSEVT][1529858368]0Processing RecoveryDone
2010-03-24 16:22:08.968: [ CRSRES][1529858368]0StopResource: setting CLI values
2010-03-24 16:39:20.607: [ OCRSRV][1529858368]s_update_remote_cache_int: FAILED TO RCV ACK FROM node 1 retcode 7
2010-03-24 16:39:20.729: [ CRSCOMM][1485781312]0CLEANUP: Searching for connections to failed node racdb5
2010-03-24 16:39:20.729: [ CRSEVT][1485781312]0Processing member leave for racdb5, incarnation: 5
2010-03-24 16:39:20.730: [ CRSD][1485781312]0SM: recovery in process: 8
2010-03-24 16:39:20.730: [ CRSEVT][1485781312]0Do failover for: racdb5
2010-03-24 16:39:21.751: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.772: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.786: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.801: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.817: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.831: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.846: [ CRSRES][1485781312]0 startup = 0
2010-03-24 16:39:21.968: [ CRSRES][1529858368]0startRunnable: setting CLI values
2010-03-24 16:39:21.979: [ CRSRES][1529858368]0Attempting to start `ora.racdb5.vip` on member `racdb6`
2010-03-24 16:39:25.268: [ CRSRES][1529858368]0Start of `ora.racdb5.vip` on member `racdb6` succeeded.
2010-03-24 16:39:25.289: [ CRSEVT][1485781312]0Post recovery done evmd event for: racdb5
2010-03-24 16:39:25.289: [ CRSD][1485781312]0SM: recoveryDone: 0
2010-03-24 16:39:25.290: [ CRSEVT][1485781312]0Processing RecoveryDone
2010-03-24 16:39:52.033: [ CRSRES][1485781312]0StopResource: setting CLI values
本文转自 jxwpx 51CTO博客,原文链接:http://blog.51cto.com/jxwpx/287564,如需转载请自行联系原作者