PRKP-1001 : Error starting instance …. followed by CRS-0215

Well, I am sure this is going to be the last post for 2009 as somehow I am running out of time to fix more things and write about them here.

The problem:

I was asked to create a POC (Proof Of Concept) RAC database on one of our QA clusters as we may decide to share the cluster for two different businesses.

The specification was quite simple, 3 node RAC running on Linux, ASM (using ASMLib) and Oracle 10.2.0.4.1.

The creation of the ASM disks was quite simple, as well as the creation of the database from a full RMAN backup using the “duplicate target” command.

Once all components were registered on the cluster, I went for the start of the database:

srvctl start database -d …

I had the following error:

PRKP-1001 : Error starting instance iiiiiii on node nnnnnna
CRS-0215: Could not start resource ‘ora.dddddd.iiiiii1.inst’.

After having a look on the logfiles and in google, the answers did not look really good. Some people suggested to stop the crs (impossible for us as it was shared with QA and the POC database and could not justify the downtime). Other people suggested a patch to fix a bug (Bug 4337645), which I thought it was unlikely as it was running before without problems.

The Solution:

Well, it was quite easy now that I know the answer. The steps I followed were:

1.- Try to start the database using sqlplus:

sqlplus ‘/as sysdba’

SQL*Plus: Release 10.2.0.4.0 – Production on Tue Dec 15 10:52:39 2009

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup nomount
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file ‘+DATA_POC/dddddd/spfileddddd.ora’
ORA-17503: ksfdopn:2 Failed to open file +DATA_POC/dddddd/spfileddddd.ora
ORA-15077: could not locate ASM instance serving a required diskgroup
SQL> exit

2.- check the disk on ASMLib (as ROOT):

/etc/init.d/oracleasm listdisks

/etc/init.d/oracleasm querydisk

3.- I knew the ASM instance was up and running as the QA database was working. So the problem had to be on the new diskgroups. The following sql statement confirmed my suspicions, the diskgroups were not mounted:

SQL> select name, state from v$asm_diskgroup;

SQL> select name, state from v$asm_diskgroup;

NAME                           STATE
—————————— —————–
CTRLLOG                        MOUNTED
DATA                             MOUNTED
FRA                                MOUNTED
DATA_POC                      DISMOUNTED
FRA_POC                         DISMOUNTED
LOGCTL_POC                   DISMOUNTED

3.- Mounting the diskgroups (on all 3 ASM instances) was fairly simple:

SQL> alter diskgroup data_poc mount;

Diskgroup altered.

SQL>  select name, state from v$asm_diskgroup;

NAME                           STATE
—————————— —————–
CTRLLOG                        MOUNTED
DATA                             MOUNTED
FRA                                MOUNTED
DATA_POC                      MOUNTED
FRA_POC                         DISMOUNTED
LOGCTL_POC                   DISMOUNTED

and the same for fra_poc and logctl_poc.

once all 3 diskgroups were mounted on all 3 nodes, I try to start the database again using srvctl.

Couple of things:

1.- ASMLib have to be execute on all servers

2.- ASM Diskgroups have to be mounted and all ASM instances.

3.- When moving from a single database to a RAC database, remember to set the cluster_database and any other RAC related parameters.