Start Of ONS Resource Fails With CRS-2672, CRS-5016, CRS-2674 and CRS-4000

(Doc ID 1633123.1)

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.

Symptoms

The ONS resource shows state OFFLINE for one or mode nodes and refuses to start.

The crsctl command output shows the following (in the example on a 3 node cluster where  node 2 has the problem):

$ crsctl stat res ora.ons -t
ora.ons
             ONLINE  ONLINE       Node1
             ONLINE  OFFLINE      Node2
             ONLINE  ONLINE       Node3

 
Attempting start of ONS fails with:

$ onsctl start
CRS-2672: Attempting to start 'ora.ons' on 'Node2'
CRS-5016: Process "/u01/app/grid/11.2.0/gi_1/opmn/bin/onsctli" spawned by agent "/u01/app/grid/11.2.0/gi_1/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/grid/11.2.0/gi_1/log/hkgx1unx002-ora/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-2674: Start of 'ora.ons' on 'Node2' failed
CRS-4000: Command Start failed, or completed with errors.

 
The node alert log (<GI home>/log/<node>/alert<node>.log) only shows:

[crsd(5636096)]CRS-2765:Resource 'ora.ons' has failed on server 'Node2'.

ONS log file (<GI home>/opmn/logs/ons.log.<node>) shows:

[2014-03-06T18:02:05+08:00] [ons] [NOTIFICATION:1] [104] [ons-internal] ONS server initiated
[2014-03-06T18:02:05+08:00] [ons] [ERROR:1] [17] [ons-listener] 127.0.0.1,6100: BIND (Address already in use)

Netstat output shows ONS port number 6100 already in use:

$ netstat -an|grep 6100

tcp4       0      0  127.0.0.1.6100         *.*                    LISTEN

 

Cause

Another process is using (listening on)  local ONS port#  6100.
 

Solution

Ensure the port number selected for ONS is free and not being used or selected by other software.

ONS port number information can be found in the Oracle Documentation

To determine if a particular port is in use tools like lsof, fuser and netstat can be used, below are some examples:

Unix generic:

# lsof -Pni|egrep '6100.*LISTEN'
ons       23415 oragrid    5u  IPv4 117974       TCP 127.0.0.1:6100 (LISTEN)

# netstat -an|egrep '6100.*LISTEN'
tcp        0      0 127.0.0.1:6100              0.0.0.0:*                   LISTEN 

 Linux only:

# fuser -n tcp 6100
6100/tcp:            23415

# netstat -tlnp|grep 6100
tcp        0      0 127.0.0.1:6100              0.0.0.0:*                   LISTEN      23415/ons