ISSUE
"One Click Extend Cluster Database" was used to add TWO additional nodes to an
existing cluster with an existing database named ORASID
Procedure completed and created two instances called:
bash-3.00$ crs_stat | grep orasid
NAME=ora.orasid.ORASID3.inst
NAME=ora.orasid.ORASID4.inst
bash-3.00$
NOTE the database name is in lower case "orasid" -- it should have been upper
case "ORASID" (this is caused by a bug in EM Grid Control).
Attempt to remove these instances results in the following error message:
bash-3.00$ srvctl remove instance -d orasid -i ORASID3
PRKO-2007 : Invalid instance name: ORASID3
bash-3.00$ srvctl remove instance -d orasid -i ORASID4
PRKO-2007 : Invalid instance name: ORASID4
SOLUTION
Use crs_unregister to remove erroneous instances with lowercase database name:
bash-3.00$ crs_stat | grep orasid
NAME=ora.orasid.ORASID3.inst
NAME=ora.orasid.ORASID4.inst
bash-3.00$ crs_unregister ora.orasid.ORASID3.inst
bash-3.00$ crs_unregister ora.orasid.ORASID4.inst
bash-3.00$ crs_stat | grep orasid
Then if you already removed the database (srvctl remove database -d ORASID
)-- re-register it with correct instance names:
bash-3.00$ srvctl add database -d ORASID \
> -o /u01/app/oracle/product/10.2.0/db_1 \
> -p +SYS/ORASID/spfileORASID.ora
bash-3.00$ srvctl add instance -d ORASID -i ORASID1 -n qarac1
bash-3.00$ srvctl add instance -d ORASID -i ORASID2 -n qarac2
bash-3.00$ srvctl add instance -d ORASID -i ORASID3 -n qarac4
bash-3.00$ srvctl add instance -d ORASID -i ORASID4 -n qarac3
bash-3.00$ srvctl modify instance -d ORASID -i ORASID1 -s +ASM1
bash-3.00$ srvctl modify instance -d ORASID -i ORASID2 -s +ASM2
bash-3.00$ srvctl modify instance -d ORASID -i ORASID3 -s +ASM3
bash-3.00$ srvctl modify instance -d ORASID -i ORASID4 -s +ASM4
bash-3.00$ srvctl start database -d ORASID
bash-3.00$ srvctl status database -d ORASID
Instance ORASID1 is running on node qarac1
Instance ORASID2 is running on node qarac2
Instance ORASID3 is running on node qarac4
Instance ORASID4 is running on node qarac3
bash-3.00$ srvctl config database -d ORASID -a
qarac1 ORASID1 /u01/app/oracle/product/10.2.0/db_1
qarac2 ORASID2 /u01/app/oracle/product/10.2.0/db_1
qarac4 ORASID3 /u01/app/oracle/product/10.2.0/db_1
qarac3 ORASID4 /u01/app/oracle/product/10.2.0/db_1
DB_NAME: null
ORACLE_HOME: /u01/app/oracle/product/10.2.0/db_1
SPFILE: +SYS/ORASID/spfileORASID.ora
DOMAIN: null
DB_ROLE: null
START_OPTIONS: null
POLICY: AUTOMATIC
ENABLE FLAG: DB ENABLED
bash-3.00$
bash-3.00$ crs_stat -t
Name Type Target State
Host
------------------------------------------------------------
ora....D1.inst application ONLINE ONLINE qarac1
ora....D2.inst application ONLINE ONLINE qarac2
ora....D3.inst application ONLINE ONLINE qarac4
ora....D4.inst application ONLINE ONLINE qarac3
ora.ORASID.db application ONLINE ONLINE qarac1
ora....SM1.asm application ONLINE ONLINE qarac1
ora....C1.lsnr application ONLINE ONLINE qarac1
ora.qarac1.gsd application ONLINE ONLINE qarac1
ora.qarac1.ons application ONLINE ONLINE qarac1
ora.qarac1.vip application ONLINE ONLINE qarac1
ora....SM2.asm application ONLINE ONLINE qarac2
ora....C2.lsnr application ONLINE ONLINE qarac2
ora.qarac2.gsd application ONLINE ONLINE qarac2
ora.qarac2.ons application ONLINE ONLINE qarac2
ora.qarac2.vip application ONLINE ONLINE qarac2
ora....SM4.asm application ONLINE ONLINE qarac3
ora....C3.lsnr application ONLINE ONLINE qarac3
ora.qarac3.gsd application ONLINE ONLINE qarac3
ora.qarac3.ons application ONLINE ONLINE qarac3
ora.qarac3.vip application ONLINE ONLINE qarac3
ora....SM3.asm application ONLINE ONLINE qarac4
ora....C4.lsnr application ONLINE ONLINE qarac4
ora.qarac4.gsd application ONLINE ONLINE qarac4
ora.qarac4.ons application ONLINE ONLINE qarac4
ora.qarac4.vip application ONLINE ONLINE qarac4
bash-3.00$
It's also very easy to reproduce this issue with a simple test case:
## 1) NOTE the database name is upper case ORASID
##
bash-3.00$ srvctl config database
ORASID
## 2) create a bogus instance "ORASID5" but
## register it under database "orasid" in lower-case
##
bash-3.00$ srvctl add instance -d orasid -i ORASID5 -n qarac4
## Q1: why is this being allowed?
## 3) attempt to remove "ORASID5" using the same
## syntax that was created with the lower case
## database name "orasid"
##
bash-3.00$ srvctl remove instance -d orasid -i ORASID5
Remove instance ORASID5 from the database orasid? (y/[n]) y
CRS-0210: Could not find resource ora.ORASID.ORASID5.inst.
## Q2: why here ORASID is being converted to upper case?
## 4) verify the real name for this instance in CRS
##
bash-3.00$ crs_stat | grep ORASID5
NAME=ora.orasid.ORASID5.inst
## 5) the only solution now is to use crs_unregister
##
bash-3.00$ crs_unregister ora.orasid.ORASID5.inst
bash-3.00$
## BUT you are not done yet -- check this out:
##
## NOTE: ORASID5 is still there -- listed under the ORASID database!
##
bash-3.00$ srvctl config database -d ORASID -a
qarac1 ORASID1 /u01/app/oracle/product/10.2.0/db_1
qarac2 ORASID2 /u01/app/oracle/product/10.2.0/db_1
qarac4 ORASID3 /u01/app/oracle/product/10.2.0/db_1
qarac3 ORASID4 /u01/app/oracle/product/10.2.0/db_1
qarac4 ORASID5 /u01/app/oracle/product/10.2.0/db_1
DB_NAME: null
ORACLE_HOME: /u01/app/oracle/product/10.2.0/db_1
SPFILE: +SYS/ORASID/spfileORASID.ora
DOMAIN: null
DB_ROLE: null
START_OPTIONS: null
POLICY: AUTOMATIC
ENABLE FLAG: DB ENABLED
## If you attempt to remove it you get "CRS-0210"
##
bash-3.00$ srvctl remove instance -d ORASID -i ORASID5
Remove instance ORASID5 from the database ORASID? (y/[n]) y
CRS-0210: Could not find resource ora.ORASID.ORASID5.inst.
## you can't even stop the database at this point
##
bash-3.00$ srvctl stop database -d ORASID
CRS-0210: Could not find resource ora.ORASID.ORASID5.inst.
## Solution? Surprisingly it's to add ORASID5 with the
## right database name! This is really messed up!
##
bash-3.00$ srvctl add instance -d ORASID -i ORASID5 -n qarac4
## After ORASID5 is added with the right database name
## CRS will allow you to remove it
##
bash-3.00$ srvctl remove instance -d ORASID -i ORASID5
Remove instance ORASID5 from the database ORASID? (y/[n]) y
bash-3.00$
## And finally the database/instances are in synch
##
bash-3.00$ srvctl config database -d ORASID -a
qarac1 ORASID1 /u01/app/oracle/product/10.2.0/db_1
qarac2 ORASID2 /u01/app/oracle/product/10.2.0/db_1
qarac4 ORASID3 /u01/app/oracle/product/10.2.0/db_1
qarac3 ORASID4 /u01/app/oracle/product/10.2.0/db_1
DB_NAME: null
ORACLE_HOME: /u01/app/oracle/product/10.2.0/db_1
SPFILE: +SYS/ORASID/spfileORASID.ora
DOMAIN: null
DB_ROLE: null
START_OPTIONS: null
POLICY: AUTOMATIC
ENABLE FLAG: DB ENABLED
bash-3.00$
## stop/start/status works fine now ...
##
bash-3.00$ srvctl stop database -d ORASID
bash-3.00$ srvctl start database -d ORASID
bash-3.00$ srvctl status database -d ORASID
Instance ORASID1 is running on node qarac1
Instance ORASID2 is running on node qarac2
Instance ORASID3 is running on node qarac4
Instance ORASID4 is running on node qarac3
bash-3.00$