今天维护人员告知有个9i的rac实例无法正常启动,一直处于mount状态,也没有报错信息,一直hang在lmon registered with NM - instance id 2 (internal mem no 1)这里不动。
Sat Sep 10 14:00:33 2011
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2 Version 1.0
PMON started with pid=2
DIAG started with pid=3
LMON started with pid=4
LMD0 started with pid=5
LMS0 started with pid=6
LMS1 started with pid=7
DBW0 started with pid=8
LGWR started with pid=9
CKPT started with pid=10
SMON started with pid=11
RECO started with pid=12
CJQ0 started with pid=13
Sat Sep 10 14:00:36 2011
ARCH: STARTING ARCH PROCESSES
ARC0 started with pid=14
ARC0: Archival started
ARC1 started with pid=15
ARC1: Archival started
Sat Sep 10 14:00:37 2011
ARCH: STARTING ARCH PROCESSES COMPLETE
Sat Sep 10 14:00:37 2011
ARC1: Thread not mounted
Sat Sep 10 14:00:37 2011
ARC0: Thread not mounted
Sat Sep 10 14:00:51 2011
alter database mount
Sat Sep 10 14:00:51 2011
lmon registered with NM - instance id 2 (internal mem no 1)
开始怀疑是HA有问题,导致无法读取共享资源hang住,查看两边的pv属性,发现都是正常的concurrent状态,重启了HA后故障依旧。
后通过查看集群状态发现有两个心跳ip状态是down,两个节点无相ping果然无法ping通,通过询问维护人员知道因换过电源,初步怀疑是换电源的时候把网线给碰松或者碰掉了导致,去机房查看状态发现果然两块网卡指示灯都不亮,接实所有网线后状态指示灯呈正常绿色闪烁状态,再次启动数据库后正常启动。
clstat - HACMP Cluster Status Monitor
-------------------------------------
Cluster: kkdb_cluster (1149099876)
Sat Sep 10 16:49:01 BEIST 2011
State: UP Nodes: 2
SubState: STABLE
Node: kkdb1 State: UP
Interface: kkdb1_boot1 (1) Address: 192.168.0.21
State: UP
Interface: kkdb1_boot2 (1) Address: 192.168.1.21
State: UP
Interface: kkdb1_hb (0) Address: 192.168.3.21
State: DOWN
Interface: kkdb1_svc (1) Address: 10.10.13.62
State: UP
Interface: kkdb1_rac (2) Address: 192.168.2.21
State: DOWN
Resource Group: kkdb1_rg State: On line
Resource Group: kkdb_share_rg State: On line
Node: kkdb2 State: UP
Interface: kkdb2_boot1 (1) Address: 192.168.0.23
State: UP
Interface: kkdb2_boot2 (1) Address: 192.168.1.23
State: UP
Interface: kkdb2_hb (0) Address: 192.168.3.23
State: DOWN
Interface: kkdb2_svc (1) Address: 10.10.13.64
State: UP
Interface: kkdb2_rac (2) Address: 192.168.2.23
State: DOWN
Resource Group: kkdb2_rg State: On line
Resource Group: kkdb_share_rg State: On line