打不死的小强-记一次RAC下ONS资源为UNKNOWN的处理过程-CSDN博客

1.背景描述
客户反映，有一AIX环境（OS:5.3),2节点RAC，其中GSD和OSN资源在其中的第2节点的state列始终显示unknown,但是不影响连接使用。
CRS和ORACLE 库版本:11.1.0.6；数据库类型为:待上线的测试库
node2$crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....01.lsnr application    ONLINE    ONLINE    ahdx...ra01
ora....a01.gsd application    ONLINE    ONLINE    ahdx...ra01
ora....a01.ons application    ONLINE    ONLINE    ahdx...ra01
ora....a01.vip application    ONLINE    ONLINE    ahdx...ra01
ora....02.lsnr application    ONLINE    ONLINE    ahdx...ra02
ora....a02.gsd application    ONLINE    UNKNOWN   ahdx...ra02
ora....a02.ons application    ONLINE    UNKNOWN   ahdx...ra02
ora....a02.vip application    ONLINE    ONLINE    ahdx...ra02
ora.gdomain.db application    ONLINE    UNKNOWN   ahdx...ra02
ora....n1.inst application    ONLINE    ONLINE    ahdx...ra01
ora....n2.inst application    ONLINE    ONLINE    ahdx...ra02

如上数据，除了我们提到的gsd,ons 状态为UNKNOWN外，db的state状态也为UNKNOWN,这个资源可通过停止和启动变为online.

2.处理过程
2.1通过单独停止资源的方法
通常这种UNKNOWN状态的资源，通过单独停止和启动后，状态都能变为ONLINE，这也是网络上的多数观点，
经过这种方法尝试，发现状态还是UNKNOWN

2.2重新停止和启动apps方式
这种停止里面包含的资源有gsd,ons,vip
node2$srvctl stop nodeapps -n NODE1-ORA02
再启动
node2$srvctl start nodeapps -n NODE1-ORA02
查看状态：
./crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....01.lsnr application    ONLINE    ONLINE    ahdx...ra01
ora....a01.gsd application    ONLINE    ONLINE    ahdx...ra01
ora....a01.ons application    ONLINE    ONLINE    ahdx...ra01
ora....a01.vip application    ONLINE    ONLINE    ahdx...ra01
ora....02.lsnr application    ONLINE    ONLINE    ahdx...ra02
ora....a02.gsd application    ONLINE    UNKNOWN   ahdx...ra02
ora....a02.ons application    ONLINE    UNKNOWN   ahdx...ra02
ora....a02.vip application    ONLINE    ONLINE    ahdx...ra02
ora.gdomain.db application    ONLINE    ONLINE    ahdx...ra01
ora....n1.inst application    ONLINE    ONLINE    ahdx...ra01
ora....n2.inst application    ONLINE    ONLINE    ahdx...ra02

如上，停止和重新启动nodeapps后，问题依旧

2.3比对节点1和节点2的racgwrap的文件是否有不同
鉴于节点1的资源都正常，涉及到和ons资源有关的文件racgwrap，看看此两个节点的文件是否有差异
将节点2上的/opt/oracle/crs/bin/racgwrap 文件拷贝到node1 上的/tmp下
使用diff命令比对　节点上的 /opt/oracle/crs/bin/racgwrap 和/tmp目录下节点2的racgwrap文件比较
结构发现两个文件是相同的

2.4 比对两个节点上ons资源的属性
[root@node1# ./crs_getperm ora.node1-ora01.ons
Name: ora.node1-ora01.ons
owner:oracle:rwx,pgrp:oinstall:r-x,other::r--,

[root@ NODE1-ORA01 bin]# ./crs_stat -p ora.node1-ora01.ons
NAME=ora.node1-ora01.ons
TYPE=application
ACTION_SCRIPT=/opt/oracle/crs/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1
CHECK_INTERVAL=60
DESCRIPTION=CRS application for ONS on node
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS= NODE1-ora01
OPTIONAL_RESOURCES=
PLACEMENT=restricted
REQUIRED_RESOURCES=
RESTART_ATTEMPTS=3
SCRIPT_TIMEOUT=600
START_TIMEOUT=600
STOP_TIMEOUT=600
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba
USR_ORA_DEBUG=5
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=

-------
[root@ NODE2-ORA02 ~]# cd /opt/oracle/crs/bin
[root@ NODE2-ORA02 bin]# ./crs_getperm ora.node1-ora02.ons
Name: ora.node1-ora02.ons
owner:oracle:rwx,pgrp:oinstall:r-x,other::r--,

[root@ NODE2-ORA02 bin]# ./crs_stat -p ora.node1-ora02.ons
NAME=ora.node1-ora02.ons
TYPE=application
ACTION_SCRIPT=/opt/oracle/crs/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1
CHECK_INTERVAL=60
DESCRIPTION=CRS application for ONS on node
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS= NODE1-ora02
OPTIONAL_RESOURCES=
PLACEMENT=restricted
REQUIRED_RESOURCES=
RESTART_ATTEMPTS=3
SCRIPT_TIMEOUT=600
START_TIMEOUT=600
STOP_TIMEOUT=600
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba
USR_ORA_DEBUG=5
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=

经过比对，没有明显差异。

2.5尝试改变ons的远程端口，看是否有效果
ONS 资源在启动后会占用一个local端口和一个远程端口，本地端口用于本机的通信，远程端口用于和远程节点进行通信
尝试修改远程端口后，看看结果。

ons资源本地的端口配置文件在$CRS_HOME/opmn/conf 下：
more ons.config

localport=6150
useocr=on
allowgroup=true
usesharedinstall=true

localport :指定ons启动时需要的本地port,默认为6150

远程端口可通过命令onsctl start 查看
./onsctl start
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
{node = NODE1-ORA02, port = 6251}
Setting remote port from OCR repository to 2200
Adding remote host NODE1-ORA02:6251
onscfg[1]
{node = NODE1-ORA01, port = 6251}
Adding remote host NODE1-ORA01:6251

Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
{node = NODE1-ORA02, port = 6251}
Setting remote port from OCR repository to 2200
Adding remote host NODE1-ORA02:2200
onscfg[1]
{node = NODE1-ORA01, port = 6251}
Adding remote host NODE1-ORA01:6251
onsctl: ons started

如上显示，两个节点的remote port都是6251,下面将Node2的remote port 改为2200

先停止node2 上的nodeapps
srvctl stop nodeapps -n node2

清除ocr资料库中的6251端口，改为2200port
[oracle@ NODE1-ORA02 bin]$ ./racgons remove_config NODE1-ORA02:6150
racgons: Existing key value on NODE1-ORA02 = 6150.
racgons: NODE1-ORA02:6150 removed from OCR.

增加2200por
[oracle@ NODE1-ORA02 bin]$ ./racgons add_config NODE1-ORA02:2200

启动服务
srvctl start nodeapps -n node2

验证是否修改，发现已经改为2200了
./onsctl start
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
   {node = NODE1-ORA02, port = 2200}
Setting remote port from OCR repository to 2200
Adding remote host NODE1-ORA02:2200
onscfg[1]
   {node = NODE1-ORA01, port = 6251}
Adding remote host NODE1-ORA01:6251
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
   {node = NODE1-ORA02, port = 2200}
Setting remote port from OCR repository to 2200
Adding remote host NODE1-ORA02:2200
onscfg[1]
   {node = NODE1-ORA01, port = 6251}
Adding remote host NODE1-ORA01:6251
onsctl: ons started

检查crs 状态

node2$crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....01.lsnr application    ONLINE    ONLINE    ahdx...ra01
ora....a01.gsd application    ONLINE    ONLINE    ahdx...ra01
ora....a01.ons application    ONLINE    ONLINE    ahdx...ra01
ora....a01.vip application    ONLINE    ONLINE    ahdx...ra01
ora....02.lsnr application    ONLINE    ONLINE    ahdx...ra02
ora....a02.gsd application    ONLINE    UNKNOWN   ahdx...ra02
ora....a02.ons application    ONLINE    UNKNOWN   ahdx...ra02
ora....a02.vip application    ONLINE    ONLINE    ahdx...ra02
ora.gdomain.db application    ONLINE    ONLINE    ahdx...ra02
ora....n1.inst application    ONLINE    ONLINE    ahdx...ra01
ora....n2.inst application    ONLINE    ONLINE    ahdx...ra02

发现状态依旧。

2.6 手工启动ons服务

./onsctl start
./onsctl start
Number of onsconfiguration retrieved, numcfg = 3
onscfg[0]
   {node = AHDX-RZLC01-ORA02, port = 2200}
Adding remote host AHDX-RZLC01-ORA02:2200
onscfg[1]
   {node = AHDX-RZLC01-ORA02, port = 6150}
Adding remote host AHDX-RZLC01-ORA02:6150
onscfg[2]
   {node = AHDX-RZLC01-ORA01, port = 6251}
Adding remote host AHDX-RZLC01-ORA01:6251
Number of onsconfiguration retrieved, numcfg = 3
onscfg[0]
   {node = AHDX-RZLC01-ORA02, port = 2200}
Adding remote host AHDX-RZLC01-ORA02:2200
onscfg[1]
   {node = AHDX-RZLC01-ORA02, port = 6150}
Adding remote host AHDX-RZLC01-ORA02:6150
onscfg[2]
   {node = AHDX-RZLC01-ORA01, port = 6251}
Adding remote host AHDX-RZLC01-ORA01:6251
onsctl: ons started

上述提示，表示使用onsctl start 重启此服务后，ons可以正常启动了
显示状态

发现状态依旧。

2.7 ons -a ping 查看Ons 进程是否已经启动
ons -a ping
Number of onsconfiguration retrieved, numcfg = 3
onscfg[0]
   {node = AHDX-RZLC01-ORA02, port = 2800}
No remoteport in local config file and multiple ports defined for this host!
Can not determine value for remote port, exiting.
[oracle@AHDX-RZLC01-ORA02 bin]$ ./ onsctl start
-bash: ./: is a directory
[oracle@AHDX-RZLC01-ORA02 bin]$ ./onsctl start
Number of onsconfiguration retrieved, numcfg = 3
onscfg[0]
   {node = AHDX-RZLC01-ORA02, port = 2800}
Adding remote host AHDX-RZLC01-ORA02:2800
onscfg[1]
   {node = AHDX-RZLC01-ORA02, port = 6150}
Adding remote host AHDX-RZLC01-ORA02:6150
onscfg[2]
   {node = AHDX-RZLC01-ORA01, port = 6251}
Adding remote host AHDX-RZLC01-ORA01:6251
onsctl: ons is already running

2.8 ons debug
./onsctl debug
Number of onsconfiguration retrieved, numcfg = 3
onscfg[0]
   {node = AHDX-RZLC01-ORA02, port = 2800}
Adding remote host AHDX-RZLC01-ORA02:2800
onscfg[1]
   {node = AHDX-RZLC01-ORA02, port = 6150}
Adding remote host AHDX-RZLC01-ORA02:6150
onscfg[2]
   {node = AHDX-RZLC01-ORA01, port = 6251}
Adding remote host AHDX-RZLC01-ORA01:6251
HTTP/1.1 200 OK
Content-Length: 1432
Content-Type: text/html
Response:

======== NS ========

Listeners:

NAME    BIND ADDRESS   PORT   FLAGS   SOCKET
------- --------------- ----- -------- ------
Local   127.000.000.001 6150 00000142      6
Remote 192.168.001.041 2800 00000101      7
Request     No listener

Server connections:

    ID           IP        PORT    FLAGS    SENDQ     WORKER   BUSY SUBS
---------- --------------- ----- -------- ---------- -------- ------ -----
         1 192.168.001.041 6150 00104205          0               1     0
         2 192.168.001.040 6251 00010005          0               1     0

Client connections:

    ID           IP        PORT    FLAGS    SENDQ     WORKER   BUSY SUBS
---------- --------------- ----- -------- ---------- -------- ------ -----
         3 127.000.000.001 6150 0001001a          0               1     0
         4 127.000.000.001 6150 0001001a          0               1     1

Pending connections:

ID IP PORT FLAGS SENDQ WORKER BUSY SUBS
---------- --------------- ----- -------- ---------- -------- ------ -----
0 127.000.000.001 6150 00020812 0 1 0

Worker Ticket: 1/1, Idle: 183

THREAD FLAGS
-------- --------
41c47940 00000012
42648940 00000012
43049940 00000012

Resources:

Notifications:
Received: 0, in Receive Q: 0, Processed: 0, in Process Q: 0

Pools:
Message: 24/25 (1), Link: 25/25 (1), Subscription: 24/25 (1)

上述结果可以看出本地Port 和远程port 的配置情况

3.尽管尝试了上述各种能想到的方法，但是问题仍旧没有解决，但是将后台的ons服务已经启动。
鉴于此数据库版本为11.0.6.0，建议客户将数据库先升级到11.0.7.0,此问题可能解决。

--The end of beijing

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/388619/viewspace-750606/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/388619/viewspace-750606/