Port 3938 is already used during configuring dbcontrol in RAC

Some days before, I helped my clients replace nodes in RAC DB. The replacement procedure was finished successfully, but I met a failure during the web dbcontrol reconfiguration.
The following is the main output info, you could see that the process was terminated because the port 3938 is already used.

[@more@]

Some days before, I helped my clients replace nodes in RAC DB. The replacement procedure was finished successfully, but I met a failure during the web dbcontrol reconfiguration.
The following is the main output info, you could see that the process was terminated because the port 3938 is already used.

[oracle@rac1 dbs]$ emca -config dbcontrol db -cluster

STARTED EMCA at Jan 28, 2011 7:24:02 PM
EM Configuration Assistant, Version 10.2.0.1.0 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.

Enter the following information:
Database unique name: RACDB
Database Control is already configured for the database RACDB
You have chosen to configure Database Control for managing the database RACDB
This will remove the existing configuration and the default settings and perform a fresh configuration
Do you wish to continue? [yes(Y)/no(N)]: Y
Listener port number: 1521
Cluster name: crs
Password for SYSMAN user: Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
ASM ORACLE_HOME [ /oracle/db_1 ]:
ASM port [ 1521 ]:
ASM user role [ SYSDBA ]:
ASM username [ SYS ]:
ASM user password: s
-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /oracle/db_1

Database instance hostname ................ rac1.domainname.com
Listener port number ................ 1521
Cluster name ................ crs
Database unique name ................ RACDB
Email address for notifications ...............
Outgoing Mail (SMTP) server for notifications ...............
ASM ORACLE_HOME ................ /oracle/db_1
ASM port ................ 1521
ASM user role ................ SYSDBA
ASM username ................ SYS

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Jan 28, 2011 7:25:07 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /oracle/db_1/cfgtoollogs/emca/RACDB/emca_2011-01-28_07-24-02-PM.log.
Jan 28, 2011 7:25:09 PM oracle.sysman.emcp.util.PortManager isPortInUse
WARNING: Specified port 5540 is already in use.
Jan 28, 2011 7:25:09 PM oracle.sysman.emcp.util.PortManager isPortInUse
WARNING: Specified port 5520 is already in use.
Jan 28, 2011 7:25:09 PM oracle.sysman.emcp.util.PortManager isPortInUse
WARNING: Specified port 1158 is already in use.
Jan 28, 2011 7:25:09 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Jan 28, 2011 7:25:14 PM oracle.sysman.emcp.EMDBCConfig instantiateOC4JConfigFiles
INFO: Propagating /oracle/db_1/oc4j/j2ee/OC4J_DBConsole_rac1_RACDB1 to remote nodes ...
Jan 28, 2011 7:25:15 PM oracle.sysman.emcp.EMDBCConfig instantiateOC4JConfigFiles
INFO: Propagating /oracle/db_1/oc4j/j2ee/OC4J_DBConsole_racdb2_RACDB2 to remote nodes ...
Jan 28, 2011 7:25:15 PM oracle.sysman.emcp.EMDBCConfig instantiateOC4JConfigFiles
INFO: Propagating /oracle/db_1/oc4j/j2ee/OC4J_DBConsole_racdb3_RACDB3 to remote nodes ...
Jan 28, 2011 7:25:16 PM oracle.sysman.emcp.EMAgentConfig deployStateDirs
INFO: Propagating /oracle/db_1/rac1_RACDB1 to remote nodes ...
Jan 28, 2011 7:25:18 PM oracle.sysman.emcp.EMAgentConfig deployStateDirs
INFO: Propagating /oracle/db_1/racdb2_RACDB2 to remote nodes ...
Jan 28, 2011 7:25:19 PM oracle.sysman.emcp.EMAgentConfig deployStateDirs
INFO: Propagating /oracle/db_1/racdb3_RACDB3 to remote nodes ...
Jan 28, 2011 7:25:19 PM oracle.sysman.emcp.EMConfig perform
SEVERE: Cannot start Database Control. The following ports are already in use: [EM agent port:3938].
Refer to the log file at /oracle/db_1/cfgtoollogs/emca/RACDB/emca_2011-01-28_07-24-02-PM.log for more details.
Could not complete the configuration. Refer to the log file at /oracle/db_1/cfgtoollogs/emca/RACDB/emca_2011-01-28_07-24-02-PM.log for more details.
[oracle@rac1 dbs]$ netstat -anp |grep 3938
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)

So I used the netstat command to find out which node was using this port, you could see that it's the second node.
[oracle@rac1 dbs]$
[oracle@rac1 dbs]$ exit
exit
[root@rac1 ~]# netstat -anp |grep 3938
[root@rac1 ~]# netstat -tln |grep 3938
[root@rac1 ~]# su - oracle
[oracle@rac1 ~]$ exit

[root@racdb3 ~]# netstat -tln |grep 3938
[root@racdb3 ~]#

[oracle@racdb2 ~]$ netstat -tln |grep 3938
tcp 0 0 0.0.0.0:3938 0.0.0.0:* LISTEN

Then continued to find out which OS process was using this port and found that it's dbcontrol agent on the second nodes.Just to kill it by the following command.
[oracle@racdb2 ~]$ lsof -i :3938
-bash: lsof: command not found
[oracle@racdb2 ~]$ exit
logout
[root@racdb2 ~]# lsof -i :3938
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
emagent 16316 oracle 5u IPv4 33109 TCP *:dbcontrol_agent (LISTEN)
[root@racdb2 ~]# kill -9 16316
[root@racdb2 ~]#

After killing the specific process, it's good to configure web control for this RAC DB.

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/38542/viewspace-1045380/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/38542/viewspace-1045380/