(Doc ID 1633123.1)
Applies to:
Oracle Database - Enterprise Edition - Version 11.2.0.3 and laterInformation in this document applies to any platform.
Symptoms
The ONS resource shows state OFFLINE for one or mode nodes and refuses to start.
The crsctl command output shows the following (in the example on a 3 node cluster where node 2 has the problem):
ora.ons
ONLINE ONLINE Node1
ONLINE OFFLINE Node2
ONLINE ONLINE Node3
Attempting start of ONS fails with:
CRS-2672: Attempting to start 'ora.ons' on 'Node2'
CRS-5016: Process "/u01/app/grid/11.2.0/gi_1/opmn/bin/onsctli" spawned by agent "/u01/app/grid/11.2.0/gi_1/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/grid/11.2.0/gi_1/log/hkgx1unx002-ora/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-2674: Start of 'ora.ons' on 'Node2' failed
CRS-4000: Command Start failed, or completed with errors.
The node alert log (<GI home>/log/<node>/alert<node>.log) only shows:
ONS log file (<GI home>/opmn/logs/ons.log.<node>) shows:
[2014-03-06T18:02:05+08:00] [ons] [ERROR:1] [17] [ons-listener] 127.0.0.1,6100: BIND (Address already in use)
Netstat output shows ONS port number 6100 already in use:
tcp4 0 0 127.0.0.1.6100 *.* LISTEN
Cause
Another process is using (listening on) local ONS port# 6100.
Solution
Ensure the port number selected for ONS is free and not being used or selected by other software.
ONS port number information can be found in the Oracle Documentation
To determine if a particular port is in use tools like lsof, fuser and netstat can be used, below are some examples:
Unix generic:
ons 23415 oragrid 5u IPv4 117974 TCP 127.0.0.1:6100 (LISTEN)
# netstat -an|egrep '6100.*LISTEN'
tcp 0 0 127.0.0.1:6100 0.0.0.0:* LISTEN
Linux only:
# fuser -n tcp 61006100/tcp: 23415
# netstat -tlnp|grep 6100
tcp 0 0 127.0.0.1:6100 0.0.0.0:* LISTEN 23415/ons