Standby 库, 根盘暴满,虚惊一场 [zt]

最新推荐文章于 2024-09-27 11:30:45 发布

cuizhu0832

最新推荐文章于 2024-09-27 11:30:45 发布

阅读量139

点赞数

文章标签：数据库系统架构人工智能

突然收到备库上以下错误，吓了一大跳,hehe
根盘空间有3多个G呀，怎么会突然暴满?
哎，别想了，先解决问题要紧

Jul 28 16:20:13 hpdb2 vmunix: vxfs: NOTICE: msgcnt 1895 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block ext
ent)
Jul 28 16:20:28 hpdb2 vmunix: vxfs: NOTICE: msgcnt 1897 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block ext
ent)
Jul 28 16:20:56 hpdb2 vmunix: vxfs: NOTICE: msgcnt 1899 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block ext
ent)
Jul 28 16:21:15 hpdb2 vmunix: vxfs: NOTICE: msgcnt 1901 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block ext
ent)
Jul 28 16:21:30 hpdb2 vmunix: vxfs: NOTICE: msgcnt 1903 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block ext
ent)

bdf确认了一下，根盘使用100%了.汗一个。

root@hpdb2:/jbindx02/oraindx/indx#bdf
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol3    3145728 3145728       0 100% /
/dev/vg00/lvol1     505392 129008 325840   28% /stand
/dev/vg00/lvol8    5242880 1534784 3679208   29% /var
/dev/vg00/lvol7    5242880 2020824 3196896   39% /usr
/dev/vg00/u01      14680064 8635816 5998560   59% /u01
/dev/vg00/lvol6    2097152 1050976 1038128   50% /tmp
/dev/vg00/lvol5    5996544 3900936 2079272   65% /opt
/dev/vg00/lvol4     524288   83336 437552   16% /home
/dev/vgdgehrdata/lvolvgdgehrdata
                   819200000 435723320 380480784   53% /dgehrdata
/dev/vgdgehrarch/lvolvgdgehrarch
                   256000000 165329936 89961952   65% /dgehrarch
/dev/vgdba01/lvoldba01
                   1331232768 8767432 1312133584    1% /dba01
/dev/vgdba02/lvoldba02
                   1331232768   61856 1320771152    0% /dba02
/dev/vgdgjbarch/lvolvgdgjbarch
                   363528192 112804792 248764768   31% /dgjbarch
/dev/vgdgjbdata/lvolvgdgjbdata
                   1873936384 1122876504 745192272   60% /dgjbdata

糊乱找了一通，终于发现以下目录下多出了两个oracle的数据文件.

root@hpdb2:/jbindx02/oraindx/indx#ll
total 5596240
-rw-r----- 1 oracle dba 2097160192 Jul 28 09:24 rsm_idx80.dbf
-rw-r----- 1 oracle dba 771751936 Jul 28 09:24 rsm_idx81.dbf

嘿嘿,发现了就赶快删除吧,别急哦,不能删除? 为什么呢,请听我慢慢道来.(这里有点绕)

先说明介绍一下系统架构.

这个出问题的主机(简称Q主机),这个主机既是其他三台主机的HA(MC/SG),又是这台三台主机上数据的standby(Oracle/DG).

上面各位看到的那个/jbindx02/oraindx/indx 目录,其实是给HA/Clustering用的，一个没有mount任何存储的目录。

而dataguard其实是用得另外一套目录，由于在一台主库上有一个存放索引的目录/jbindx02/oraindx/indx 一直没有存放数据文件进去，所以就忘记了加在 db_file_name_convert中，今天加表空间时，看到其他目录快满了，就向这个目录放数据文件了，所以就造成了以上问题。

问题找到了，想办法解决备库问题吧.

解决过程如下:

1.先查看一下alert.log

WARNING: File being created with same name as in Primary
Existing file may be overwritten
Mon Jul 28 09:24:10 2008
File #546 added to control file as 'UNNAMED00546'.
Originally created as:
'/jbindx02/oraindx/indx/rsm_idx81.dbf'
Recovery was unable to create the file as:
'/jbindx02/oraindx/indx/rsm_idx81.dbf'
Errors with log /dgjbarch/dgjb_1_27376_638807858.log
MRP0: Background Media Recovery terminated with error 19502
Mon Jul 28 09:24:10 2008
Errors in file /u01/oracle/admin/hpjob/bdump/dghpjob_mrp0_21025.trc:
ORA-19502: write error on file "/jbindx02/oraindx/indx/rsm_idx81.dbf", blockno 93696 (blocksize=8192)
ORA-27072: File I/O error
HP-UX Error: 2: No such file or directory
Additional information: 4
Additional information: 93696
Additional information: 540672
Mon Jul 28 09:24:51 2008
Some recovered datafiles maybe left media fuzzy
Media recovery may continue but open resetlogs may fail
Mon Jul 28 09:24:51 2008
Errors in file /u01/oracle/admin/hpjob/bdump/dghpjob_mrp0_21025.trc:
ORA-19502: write error on file "/jbindx02/oraindx/indx/rsm_idx81.dbf", blockno 93696 (blocksize=8192)
ORA-27072: File I/O error
HP-UX Error: 2: No such file or directory
Additional information: 4
Additional information: 93696
Additional information: 540672
Mon Jul 28 09:24:51 2008
MRP0: Background Media Recovery process shutdown (dghpjob)

2.先停止恢复
alter database recover managed standby database cancel ;

3.把需要转用的目录加进去
ALTER SYSTEM SET db_file_name_convert='xxx,xxx' scope=both;

4.关闭数据库
shtudown immediate;

5.移动数据文件
mv rsm_idx80.dbf /xxxx
mv rsm_idx81.dbf /xxxx

6.启动数据库到mount
ALTER SYSTEM SET standby_file_management='MANUAL' SCOPE=BOTH;
alter database rename file '/jbindx02/oraindx/indx/rsm_idx80.dbf' to '/dgjbdata/indx/rsm_idx80.dbf' ;
---成功

alter database rename file '/jbindx02/oraindx/indx/rsm_idx80.dbf' to '/dgjbdata/indx/rsm_idx80.dbf' ;

---失败(错误信息如下)

ORA-01111: name for data file 546 is unknown - rename to correct file
ORA-01110: data file 546: '/u01/oracle/product/10.2.0.3/dbs/UNNAMED00546'
ORA-01157: cannot identify/lock data file 546 - see DBWR trace file
ORA-01111: name for data file 546 is unknown - rename to correct file
ORA-01110: data file 546: '/u01/oracle/product/10.2.0.3/dbs/UNNAMED00546'

其实从上面看这个文件的大小和alert.log中的信息，已经知道了，这个文件没有创建成功。

7.对这个损坏的文件进行恢复

SQL> select name from v$datafile where file# = 546;

NAME
--------------------------------------------------------------------------------
/u01/oracle/product/10.2.0.3/dbs/UNNAMED00546

SQL> alter database create datafile '/u01/oracle/product/10.2.0.3/dbs/UNNAMED00546' as '/dgjbdata/indx/rsm_idx81.dbf';

Database altered.

SQL> select name from v$datafile where file# = 546;

NAME
--------------------------------------------------------------------------------
/dgjbdata/indx/rsm_idx81.dbf

-------过来了^_^

SQL> alter system set standby_file_management=AUTO scope=both;

SQL> alter database create datafile '/u01/oracle/product/10.2.0.3/dbs/UNNAMED00546' as '/dgjbdata/indx/rsm_idx81.dbf';

Database altered.