[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
2.Full backup database (honcho)
[oracle@honcho /]$ rman target /
Recovery Manager: Release 10.2.0.1.0 - Production on Mon Nov 7 23:29:05 2011
Copyright (c) 1982, 2005, Oracle. All rights reserved.
connected to target database: HONCHO (DBID=3640055948)
RMAN> run
2> {
3> allocate channel c1 type disk;
4> backup full database format='/u01/backup/honcho_full_%U_%T' tag='honcho_full';
5> sql 'alter system archive log current';
6> backup archivelog all tag='arc_bak' format='/u01/backup/arch_%U_%T' delete input;
7> backup current controlfile tag='bak_ctlfile' format='/u01/backup/ctl_file_%U_%T';
8> backup spfile tag='spfile' format='/u01/backup/honcho_spfile_%U_%T';
9> release channel c1;
10> }
View backup information:
RMAN> list backup summary;
List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- ------------------- ------- ------- ---------- ---
1 B F A DISK 2011-11-07 23:31:12 1 1 NO HONCHO_FULL
2 B F A DISK 2011-11-07 23:31:15 1 1 NO TAG20111107T233114
3 B A A DISK 2011-11-07 23:31:35 1 1 NO ARC_BAK
4 B F A DISK 2011-11-07 23:31:43 1 1 NO BAK_CTLFILE
5 B F A DISK 2011-11-07 23:32:45 1 1 NO SPFILE
6 B F A DISK 2011-11-07 23:32:46 1 1 NO TAG20111107T233246
3.Copy backup file to RAC server :
[oracle@honcho backup]$ scp * 192.168.1.131:/u01/backup
The authenticity of host '192.168.1.131 (192.168.1.131)' can't be established.
RSA key fingerprint is 97:f1:d1:ff:9c:ac:fb:95:70:98:4d:11:64:ef:11:b7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.131' (RSA) to the list of known hosts.
oracle@192.168.1.131's password:
arch_0amr3hmd_1_1_20111107 100% 100MB 12.5MB/s 00:08
ctl_file_0bmr3hmv_1_1_20111107 100% 6944KB 6.8MB/s 00:01
honcho_full_08mr3hj3_1_1_20111107 100% 594MB 8.3MB/s 01:12
honcho_spfile_0cmr3hot_1_1_20111107 100% 96KB 96.0KB/s 00:00
4.create spfile from pfile( spfile should be stored in shared device)
[oracle@rac1 ~]$ export ORACLE_SID=honcho1
[oracle@rac1 ~]$ rman target /
Recovery Manager: Release 10.2.0.1.0 - Production on Mon Nov 7 23:43:05 2011
Copyright (c) 1982, 2005, Oracle. All rights reserved.
connected to target database (not started)
RMAN> startup nomount
startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/10.0.2.1/db_1/dbs/inithoncho1.ora'
Starting Oracle instance without parameter file for retrival of spfile
Oracle instance started
Total System Global Area 159383552 bytes
Fixed Size 1218268 bytes
Variable Size 58722596 bytes
Database Buffers 96468992 bytes
Redo Buffers 2973696 bytes
RMAN> restore spfile to pfile '/u01/app/oracle/product/10.0.2.1/db_1/dbs/inithoncho1.ora'
2> from '/u01/backup/honcho_spfile_0cmr3hot_1_1_20111107';
Starting restore at 07-NOV-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=33 devtype=DISK
channel ORA_DISK_1: autobackup found: /u01/backup/honcho_spfile_0cmr3hot_1_1_20111107
channel ORA_DISK_1: SPFILE restore from autobackup complete
Finished restore at 07-NOV-11
5.Check the parameters of the recovery files:
[oracle@rac1 backup]$ cd /u01/app/oracle/product/10.0.2.1/db_1/dbs/
[oracle@rac1 dbs]$ ls
ab_+ASM1.dat hc_devdb1.dat init+ASM1.ora inithoncho1.ora orapw+ASM1
hc_+ASM1.dat hc_honcho1.dat initdw.ora init.ora
[oracle@rac1 dbs]$ cat inithoncho1.ora
honcho.__db_cache_size=331350016
honcho.__java_pool_size=4194304
honcho.__large_pool_size=4194304
honcho.__shared_pool_size=100663296
honcho.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/honcho/adump'
*.background_dump_dest='/u01/app/oracle/admin/honcho/bdump'
*.compatible='10.2.0.1.0'
*.control_files='/u01/app/oracle/oradata/honcho/control01.ctl','/u01/app/oracle/oradata/honcho/control02.ctl','/u01/app/oracle/oradata/honcho/control03.ctl'
*.core_dump_dest='/u01/app/oracle/admin/honcho/cdump'
*.db_block_size=8192
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_name='honcho'
*.db_recovery_file_dest='/u01/app/oracle/flash_recovery_area'
*.db_recovery_file_dest_size=2147483648
*.dispatchers='(PROTOCOL=TCP) (SERVICE=honchoXDB)'
*.job_queue_processes=10
*.open_cursors=300
*.pga_aggregate_target=147849216
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=444596224
*.undo_management='AUTO'
*.undo_tablespace='UNDOTBS1'
*.user_dump_dest='/u01/app/oracle/admin/honcho/udump'
There is a single instance of parameters, we need to bring this into the RAC parameters.
SQL> select state,redundancy,total_mb,free_mb,name,failgroup from v$asm_disk;
STATE REDUNDA TOTAL_MB FREE_MB NAME FAILGROUP
---------- ------- ---------- ---------- --------------- --------------------
NORMAL UNKNOWN 2047 0
NORMAL UNKNOWN 2047 0
NORMAL UNKNOWN 2047 1642 RECOVERY_0000 RECOVERY_0000
NORMAL UNKNOWN 3067 2973 DATA_0001 DATA_0001
NORMAL UNKNOWN 3067 2973 DATA_0000 DATA_0000
SQL> select group_number,name,state,type,total_mb,free_mb,unbalanced from v$asm_diskgroup;
GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB U
------------ --------------- ---------- ------ ---------- ---------- -
1 DATA MOUNTED NORMAL 6134 5946 N
2 RECOVERY MOUNTED EXTERN 2047 1642 N
6.According to the parameter file creation corresponding director:
[oracle@rac1 admin]$ mkdir -p /u01/app/oracle/admin/honcho/adump
[oracle@rac1 admin]$ mkdir -p /u01/app/oracle/admin/honcho/bdump
[oracle@rac1 admin]$ mkdir -p /u01/app/oracle/admin/honcho/cdump
[oracle@rac1 admin]$ mkdir -p /u01/app/oracle/flash_recovery_area
[oracle@rac1 admin]$ mkdir -p /u01/app/oracle/admin/honcho/udump
ASMCMD> ls
DATA/
RECOVERY/
ASMCMD> cd data
ASMCMD> ls
ASMCMD> pwd
+data
ASMCMD> mkdir HONCHO
ASMCMD> mkdir CONTROLFILE DATAFILE TEMPFILE ONLINELOG PARAMETERFILE
ASMCMD> ls
CONTROLFILE/
DATAFILE/
ONLINELOG/
PARAMETERFILE/
TEMPFILE/
ASMCMD> cd ..
ASMCMD> ls
HONCHO/
ASMCMD> cd ..
ASMCMD> ls
DATA/
RECOVERY/
ASMCMD> cd recovery
ASMCMD> mkdir HONCHO
ASMCMD> ls
HONCHO/
ASMCMD> cd honcho
ASMCMD> mkdir CONTROLFILE
ASMCMD> ls
CONTROLFILE/
ASMCMD> mkdir ARCHIVELOG
ASMCMD> ls
ARCHIVELOG/
CONTROLFILE/
7.Edit initialization parameter file, amended parameters as follows:
honcho.__db_cache_size=331350016
honcho.__java_pool_size=4194304
honcho.__large_pool_size=4194304
honcho.__shared_pool_size=100663296
honcho.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/honcho/adump'
*.background_dump_dest='/u01/app/oracle/admin/honcho/bdump'
*.compatible='10.2.0.1.0'
*.core_dump_dest='/u01/app/oracle/admin/honcho/cdump'
*.db_block_size=8192
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_recovery_file_dest_size=2147483648
*.dispatchers='(PROTOCOL=TCP) (SERVICE=honchoXDB)'
*.job_queue_processes=10
*.open_cursors=300
*.pga_aggregate_target=147849216
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=444596224
*.undo_management='AUTO'
*.user_dump_dest='/u01/app/oracle/admin/honcho/udump'
*.cluster_database_instances=2
*.cluster_database=true
*.control_files='+DATA/honcho/controlfile/control01.ctl','+RECOVERY/honcho/controlfile/control02.ctl'
*.db_create_file_dest='+DATA'
*.db_name='honcho'
*.db_recovery_file_dest='+RECOVERY'
honcho1.instance_name='honcho1'
honcho2.instance_name='honcho2'
honcho1.instance_number=1
honcho2.instance_number=2
*.log_archive_dest_1='LOCATION=+RECOVERY/honcho/archivelog'
honcho1.thread=1
honcho2.thread=2
honcho1.undo_tablespace='UNDOTBS1'
honcho2.undo_tablespace='UNDOTBS2'
8.Create spfile
SQL> create spfile='+DATA/HONCHO/PARAMETERFILE/spfilehoncho.ora' from pfile='/u01/app/oracle/product/10.0.2.1/db_1/dbs/inithoncho1.ora';
File created.
SQL> shutdown immediate
ORA-01507: database not mounted
ORACLE instance shut down.
9.In all nodes, modify pfile , and it will point to the spfile at share equipment
[oracle@rac1 ~]$ echo "SPFILE='+DATA/HONCHO/PARAMETERFILE/spfilehoncho.ora'" > /u01/app/oracle/product/10.0.2.1/db_1/dbs/inithoncho1.ora
[oracle@rac2 dbs]$ touch inithoncho2.ora
[oracle@rac2 dbs]$ echo "SPFILE='+DATA/HONCHO/PARAMETERFILE/spfilehoncho.ora'" > /u01/app/oracle/product/10.0.2.1/db_1/dbs/inithoncho2.ora
[oracle@rac2 dbs]$ cat inithoncho2.ora
SPFILE='+DATA/HONCHO/PARAMETERFILE/spfilehoncho.ora'
10.In all nodes, create password file.
[oracle@rac1 ~]$ cd $ORACLE_HOME/dbs
[oracle@rac1 dbs]$ orapwd file=orapwhoncho1 password=oracle entries=20
[oracle@rac2 dbs]$ orapwd file=orapwhoncho2 password=oracle entries=20
[oracle@rac1 ~]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.1.0 - Production on Tue Nov 8 00:36:26 2011
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup nomount
ORACLE instance started.
Total System Global Area 444596224 bytes
Fixed Size 1219904 bytes
Variable Size 125829824 bytes
Database Buffers 314572800 bytes
Redo Buffers 2973696 bytes
SQL> show parameter spfile
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +DATA/honcho/parameterfile/spf
ilehoncho.ora
11.Recovery controlfile from backupset.
[oracle@rac1 backup]$ rman target /
Recovery Manager: Release 10.2.0.1.0 - Production on Tue Nov 8 00:37:43 2011
Copyright (c) 1982, 2005, Oracle. All rights reserved.
connected to target database: honcho (not mounted)
RMAN> restore controlfile from '/u01/backup/ctl_file_0bmr3hmv_1_1_20111107';
Starting restore at 08-NOV-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=150 instance=honcho1 devtype=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:17
output filename=+DATA/honcho/controlfile/control01.ctl
output filename=+RECOVERY/honcho/controlfile/control02.ctl
Finished restore at 08-NOV-11
12.start the database in mount stage
RMAN> alter database mount;
database mounted
released channel: ORA_DISK_1
13.Rename the datafile,redo logs to new shared device
RMAN> run {
2> set newname for datafile 1 to '+DATA/honcho/DATAFILE/system01.dbf';
3> set newname for datafile 2 to '+DATA/honcho/DATAFILE/undotbs01.dbf';
4> set newname for datafile 3 to '+DATA/honcho/DATAFILE/sysaux01.dbf';
5> set newname for datafile 4 to '+DATA/honcho/DATAFILE/users01.dbf';
6> set newname for datafile 5 to '+DATA/honcho/DATAFILE/example01.dbf';
7> set newname for datafile 6 to '+DATA/honcho/DATAFILE/honcho01.dbf';
8> set newname for tempfile 1 to '+DATA/honcho/TEMPFILE/temp01.dbf';
9> restore database;
10> switch datafile all;
11> switch tempfile all;
12> }
ASMCMD> ls -l
Type Redund Striped Time Sys Name
DATAFILE MIRROR COARSE NOV 08 01:00:00 Y EXAMPLE.269.766630817
DATAFILE MIRROR COARSE NOV 08 01:00:00 Y HONCHO.266.766630819
DATAFILE MIRROR COARSE NOV 08 01:00:00 Y SYSAUX.263.766631113
DATAFILE MIRROR COARSE NOV 08 01:00:00 Y SYSTEM.270.766631113
DATAFILE MIRROR COARSE NOV 08 01:00:00 Y UNDOTBS1.265.766630821
DATAFILE MIRROR COARSE NOV 08 01:00:00 Y USERS.262.766630823
N example01.dbf => +DATA/HONCHO/DATAFILE/EXAMPLE.269.766630817
N honcho01.dbf => +DATA/HONCHO/DATAFILE/HONCHO.266.766630819
N sysaux01.dbf => +DATA/HONCHO/DATAFILE/SYSAUX.263.766631113
N system01.dbf => +DATA/HONCHO/DATAFILE/SYSTEM.270.766631113
N undotbs01.dbf => +DATA/HONCHO/DATAFILE/UNDOTBS1.265.766630821
N users01.dbf => +DATA/HONCHO/DATAFILE/USERS.262.766630823
ASMCMD>
RMAN> recover database;
Starting recover at 08-NOV-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=145 instance=honcho1 devtype=DISK
starting media recovery
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=6
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=7
channel ORA_DISK_1: reading from backup piece /u01/backup/arch_0amr3hmd_1_1_20111107
channel ORA_DISK_1: restored backup piece 1
piece handle=/u01/backup/arch_0amr3hmd_1_1_20111107 tag=ARC_BAK
channel ORA_DISK_1: restore complete, elapsed time: 00:00:16
archive log filename=+RECOVERY/honcho/archivelog/1_6_765821516.dbf thread=1 sequence=6
archive log filename=+RECOVERY/honcho/archivelog/1_7_765821516.dbf thread=1 sequence=7
unable to find archive log
archive log thread=1 sequence=8
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 11/08/2011 10:17:27
RMAN-06054: media recovery requesting unknown log: thread 1 seq 8 lowscn 520688
RMAN> recover database until sequence 7;
Starting recover at 08-NOV-11
using channel ORA_DISK_1
starting media recovery
archive log thread 1 sequence 6 is already on disk as file +RECOVERY/honcho/archivelog/1_6_765821516.dbf
archive log filename=+RECOVERY/honcho/archivelog/1_6_765821516.dbf thread=1 sequence=6
media recovery complete, elapsed time: 00:00:03
Finished recover at 08-NOV-11
SQL> select * from v$logfile;
rows will be truncated
GROUP# STATUS TYPE MEMBER
---------- ------- ------- -----------------------------------------------------
3 ONLINE /u01/app/oracle/oradata/honcho/redo03.log
2 ONLINE /u01/app/oracle/oradata/honcho/redo02.log
1 ONLINE /u01/app/oracle/oradata/honcho/redo01.log
SQL> alter database rename file '/u01/app/oracle/oradata/honcho/redo03.log' to '+DATA/HONCHO/ONLINELOG/redo03.log';
Database altered.
SQL> alter database rename file '/u01/app/oracle/oradata/honcho/redo02.log' to '+DATA/HONCHO/ONLINELOG/redo02.log';
Database altered.
SQL> alter database rename file '/u01/app/oracle/oradata/honcho/redo01.log' to '+DATA/HONCHO/ONLINELOG/redo01.log';
Database altered.
SQL> select * from v$logfile;
rows will be truncated
GROUP# STATUS TYPE MEMBER
---------- ------- ------- -----------------------------------------------------
3 ONLINE +DATA/honcho/onlinelog/redo03.log
2 ONLINE +DATA/honcho/onlinelog/redo02.log
1 ONLINE +DATA/honcho/onlinelog/redo01.log
14.Open the database with resetlogs
SQL> alter database open resetlogs;
SQL> select open_mode from v$database;
OPEN_MODE
----------
READ WRITE
SQL> alter system set cluster_database=true scope=spfile;
System altered.
SQL> alter system set cluster_database_instances=2 scope=spfile;
System altered.
SQL> alter system set instance_number=1 scope=spfile sid='honcho1';
System altered.
SQL> alter system set instance_number=2 scope=spfile sid='honcho2';
System altered.
SQL> alter system set thread=1 scope=spfile sid='honcho1';
System altered.
SQL> alter system set thread=2 scope=spfile sid='honcho2';
System altered.
15.create the second (or more) instance undo tablespace from existing instance
SQL> create undo tablespace UNDOTBS2 datafile '+DATA/HONCHO/datafile/undotbs02.dbf' size 50m;
Tablespace created.
SQL> alter system set undo_tablespace='UNDOTBS2' scope=spfile sid='honcho2';
System altered.
16.Add second instance redo logs (or more when multiple instances will be started)
SQL> alter database add logfile thread 2 group 4 '+DATA' size 50m;
Database altered.
SQL> alter database add logfile thread 2 group 5 '+DATA' size 50m;
Database altered.
SQL> alter database add logfile thread 2 group 6 '+DATA' size 50m;
Database altered.
SQL> alter database enable thread 2;
Database altered.
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 444596224 bytes
Fixed Size 1219904 bytes
Variable Size 138412736 bytes
Database Buffers 301989888 bytes
Redo Buffers 2973696 bytes
Database mounted.
Database opened.
SQL> show parameter cluster
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_database boolean TRUE
cluster_database_instances integer 2
cluster_interconnects string
[oracle@rac2 honcho]$ cd ~
[oracle@rac2 ~]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.1.0 - Production on Tue Nov 8 14:00:15 2011
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 444596224 bytes
Fixed Size 1219904 bytes
Variable Size 125829824 bytes
Database Buffers 314572800 bytes
Redo Buffers 2973696 bytes
Database mounted.
Database opened.
SQL> select instance_number,instance_name,host_name from gv$instance;
INSTANCE_NUMBER INSTANCE_NAME HOST_NAME
--------------- ---------------- ----------------------------------------------------------------
2 honcho2 rac2
1 honcho1 rac1
17.Open your database (i.e. alter database open;) and run $ORACLE_HOME/rdbms/admin/catclust.sql to create cluster database specific views within the existing
instance
SQL> @?/rdbms/admin/catclust.sql
Create temporary tablespace
SQL> select name from v$tempfile;
NAME
-------------------------------------------------------------
+DATA/honcho/tempfile/temp01.dbf
SQL> alter tablespace temp add tempfile '+DATA/honcho/tempfile/temp02.dbf' size 100M;
Tablespace altered.
SQL> select name from v$tempfile;
NAME
------------------------------------------------------------
+DATA/honcho/tempfile/temp01.dbf
+DATA/honcho/tempfile/temp02.dbf
SQL> alter database tempfile '+DATA/honcho/tempfile/temp01.dbf' offline;
Database altered.
SQL> alter database tempfile '+DATA/honcho/tempfile/temp01.dbf' drop including datafiles;
Database altered.
SQL> select name from v$tempfile;
NAME
------------------------------------------------------------
+DATA/honcho/tempfile/temp02.dbf
[oracle@rac1 honcho]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
18.Use netca configuration monitor
[oracle@rac1 honcho]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
19.After configuring the listener,you have to add the database in cluster as below
[oracle@rac1 ~]$ srvctl add database -d honcho -o $ORACLE_HOME -p +DATA/HONCHO/PARAMETERFILE/spfilehoncho.ora
[oracle@rac1 ~]$ srvctl add instance -d honcho -i honcho1 -n rac1
[oracle@rac1 ~]$ srvctl add instance -d honcho -i honcho2 -n rac2
20.In case ASM is used, add the rdbms instance / asm dependency
[oracle@rac1 ~]$ srvctl modify instance -d honcho -i honcho1 -s +ASM1
[oracle@rac1 ~]$ srvctl modify instance -d honcho -i honcho2 -s +ASM2
[oracle@rac1 ~]$ srvctl start database -d honcho
[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.honcho.db application ONLINE ONLINE rac1
ora....o1.inst application ONLINE ONLINE rac1
ora....o2.inst application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
OK.Game over !