oracle rac节点挂掉,存储挂掉 重建oracle rac



9.24 存储挂掉   oracle 主节点重建   oracle 版本11gr2

crs ocr 资源还在不打算重新安装节点。重建后从备端重新同步数据

SQL> select group_number,name,FAILGROUP,REDUNDANCY from v$asm_disk;

GROUP_NUMBER NAME       FAILGROUP      REDUNDA

------------ ------------------------------ ------------------------------ -------

0           UNKNOWN

0           UNKNOWN

0           UNKNOWN

3 CRSVOL1       CRSVOL1      UNKNOWN

重新创建磁盘组

SQL> create diskgroup ARCHVOL1  external redundancy disk ''/dev/oracleasm/disks/ARCHVOL1';

create diskgroup ARCHVOL1  external redundancy disk ''/dev/oracleasm/disks/ARCHVOL1'

*

ERROR at line 1:

ORA-15106: missing or invalid operating system disk locator string

SQL> create diskgroup ARCHVOL1  external redundancy disk '/dev/oracleasm/disks/ARCHVOL1';

create diskgroup ARCHVOL1  external redundancy disk '/dev/oracleasm/disks/ARCHVOL1'

*

ERROR at line 1:

ORA-15018: diskgroup cannot be created

ORA-15031: disk specification '/dev/oracleasm/disks/ARCHVOL1' matches no disks

ORA-15014: path '/dev/oracleasm/disks/ARCHVOL1' is not in the discovery set

SQL> show parameter asm_diskstring;

NAME         TYPE  VALUE

------------------------------------ ----------- ------------------------------

asm_diskstring        string

设置asm_diskstring

SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*';

alter system set asm_diskstring='/dev/oracleasm/disks/*'

*

ERROR at line 1:

ORA-02097: parameter cannot be modified because specified value is invalid

ORA-15014: path 'ORCL:CRSVOL1' is not in the discovery set

从报错信息中看出 当前处于正常MOUNTED状态的磁盘组

并不是/dev/oracleasm/disk

SQL>  create diskgroup ARCHVOL1  external redundancy disk 'ORCL:ARCHVOL1';

Diskgroup created.

创建成功

SQL> create diskgroup DATAVOL1 external redundancy disk 'ORCL:DATAVOL1';

Diskgroup created.

SQL> create diskgroup VOL1 external redundancy disk 'ORCL:VOL1';

create diskgroup VOL1 external redundancy disk 'ORCL:VOL1'

*

ERROR at line 1:

ORA-15018: diskgroup cannot be created

ORA-15033: disk 'ORCL:VOL1' belongs to diskgroup "DATADG"

创建磁盘组不对

删除 新创建的磁盘组

SQL> drop diskgroup ARCHVOL1;

Diskgroup dropped.

SQL>  drop diskgroup DATAVOL1;

Diskgroup dropped.

SQL> select group_number,name,FAILGROUP,REDUNDANCY from v$asm_disk;

GROUP_NUMBER NAME       FAILGROUP      REDUNDA

------------ ------------------------------ ------------------------------ -------

0           UNKNOWN

0           UNKNOWN

0           UNKNOWN

3 CRSVOL1       CRSVOL1      UNKNOWN

SQL>

应当创建如下两个磁盘组

ARCHDG/

DATADG/

SQL>  create diskgroup ARCHDG  external redundancy disk 'ORCL:ARCHVOL1';

Diskgroup created.

SQL>  create diskgroup DATADG external redundancy disk 'ORCL:ARCHVOL1,ORCL:VOL1';

SQL>  create diskgroup DATADG external redundancy disk 'ORCL:ARCHVOL1,ORCL:VOL1';

create diskgroup DATADG external redundancy disk 'ORCL:ARCHVOL1,ORCL:VOL1'

*

ERROR at line 1:

ORA-15018: diskgroup cannot be created

ORA-15031: disk specification 'ORCL:ARCHVOL1,ORCL:VOL1' matches no disks

ORA-15186: ASMLIB error function = [kfk_dscv_lib_init:disc],  error = [4],  mesg = [Invalid

discovery string]

SQL>  create diskgroup DATADG external redundancy disk 'ORCL:DATAVOL1','ORCL:VOL1';

SQL>  create diskgroup DATADG external redundancy disk 'ORCL:DATAVOL1','ORCL:VOL1';

create diskgroup DATADG external redundancy disk 'ORCL:DATAVOL1','ORCL:VOL1'

*

ERROR at line 1:

ORA-15018: diskgroup cannot be created

ORA-15033: disk 'ORCL:VOL1' belongs to diskgroup "DATADG"

SQL> drop diskgroup DATADG;

drop diskgroup DATADG

*

ERROR at line 1:

ORA-15039: diskgroup not dropped

ORA-15001: diskgroup "DATADG" does not exist or is not mounted

SQL> drop diskgroup DATADG;

drop diskgroup DATADG

*

ERROR at line 1:

ORA-15039: diskgroup not dropped

ORA-15001: diskgroup "DATADG" does not exist or is not mounted

必须把ORCL:VOL1  盘头信息擦除

1028  /etc/init.d/oracleasm deletedisk   VOL1

1029  /etc/init.d/oracleasm createdisk VOL1  /dev/sde1

1030  /etc/init.d/oracleasm scandisks;

1031  /etc/init.d/oracleasm listdisks;

SQL> create diskgroup DATADG external redundancy disk 'ORCL:DATAVOL1','ORCL:VOL1';

Diskgroup created.

创建成功

用原来备份的pfile 启动

用pfile 启动

SQL> startup nomount pfile='/u01/app/oracle/product/11.2.0/dbhome_1/dbs/pfile'

ORACLE instance started.

Total System Global Area 5010685952 bytes

Fixed Size      2261848 bytes

Variable Size   2835352744 bytes

Database Buffers  2164260864 bytes

Redo Buffers      8810496 bytes

SQL>

用rman 重新同步数据

rman target sys/*****@racdg  auxiliary sys/***@racdb

duplicate target database for standby from active database;

发现  RMAN-03009: failure of backup command on ORA_DISK_1 channel at 09/26/2015 16:11:56

ORA-17628: Oracle error 19505 returned by remote Oracle server

continuing other job steps, job failed will not be re-run

channel ORA_DISK_1: starting datafile copy

input datafile file number=00170 name=+DATADG/racdg/datafile/sae.436.888841821

RMAN-03009: failure of backup command on ORA_DISK_1 channel at 09/26/2015 16:11:58

ORA-17628: Oracle error 19505 returned by remote Oracle server

continuing other job steps, job failed will not be re-run

channel ORA_DISK_1: starting datafile copy

好像是空间不足

再次执行  of Duplicate Db command at 09/26/2015 19:36:30

RMAN-05501: aborting duplication of target database

RMAN-06136: ORACLE error from auxiliary database: ORA-00200: control file could not be created

ORA-00202: control file: '+datadg'

ORA-17502: ksfdcre:4 Failed to create file +datadg

ORA-15041: diskgroup "DATADG" space exhausted

删除数据文件

修改racdb1 的  vim /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames.ora

修改成

RACDG =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.124)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = racdg)

(UR=A)

)

)

把racdg2  上的tnsnames.ora 修改成

重新开始复制

racdb

ASMCMD> lsdg

State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N         512   4096  1048576    307199   306632                0          306632              0             N  ARCHDG/

MOUNTED  EXTERN  N         512   4096  1048576     10239     9759                0            9759              0             Y  DATA/

MOUNTED  EXTERN  N         512   4096  1048576   2098514  2098178                0         2098178              0             N  DATADG/

ASMCMD>

racdg

State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N         512   4096  1048576    307196   201520                0          201520              0             N  ARCHDG/

MOUNTED  EXTERN  N         512   4096  1048576     10236     9840                0            9840              0             Y  DATA/

MOUNTED  EXTERN  N         512   4096  1048576   3460299  1308874                0         1308874              0             N  DATADG/

发现datadg 大小相差很大

怀疑  创建的diskgroup 有问题

发现asmlib 里racdb 的 datavol 明显有问题

重新对相应的磁盘分区

发现大小正常

重新同步

racdb2

datafile 177 switched to datafile copy

input datafile copy RECID=584 STAMP=891493391 file name=+DATADG/racdb/datafile/sae.410.891492627

Finished Duplicate Db at 27-SEP-2015 05:03:35

alter system set INSTANCE_NUMBER=1 scope=spfile sid='racdb1';

alter system set INSTANCE_NUMBER=2 scope=spfile sid='racdb2';

alter system set INSTANCE_NUMBER=3 scope=spfile sid='racdb3';

alter system set INSTANCE_NUMBER=4 scope=spfile sid='racdb4';

alter system set thread=1 scope=spfile sid='racdb1';

alter system set thread=2 scope=spfile sid='racdb2';

alter system set thread=3 scope=spfile sid='racdb3';

alter system set thread=4 scope=spfile sid='racdb4';

alter system set undo_tablespace='UNDOTBS1' sid='racdb1';

alter system set undo_tablespace='UNDOTBS2' sid='racdb2';

alter system set undo_tablespace='UNDOTBS3' sid='racdb3';

alter system set undo_tablespace='UNDOTBS4' sid='racdb4';

alter system set core_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb1/cdump' scope=spfile sid='racdb1';

alter system set core_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb2/cdump' scope=spfile sid='racdb2';

alter system set core_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb3/cdump' scope=spfile sid='racdb3';

alter system set core_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb4/cdump' scope=spfile sid='racdb4';

alter system set user_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb1/trace' scope=spfile sid='racdb1';

alter system set user_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb2/trace' scope=spfile sid='racdb2';

alter system set user_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb3/trace' scope=spfile sid='racdb3';

alter system set user_dump_dest='/u01/app/oracle/diag/rdbms/racdb/racdb4/trace' scope=spfile sid='racdb4';

alter system set local_listener=’(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.3.129)(PORT=11521))))’ scope=both;