ASM diskgroup在线迁移记录

Oracle ASM磁盘组使用新存储替换旧存储方案

存储信息确认及配置

1.1、ASM磁盘组及旧存储磁盘信息说明

磁盘组名称冗余方式磁盘WWID及大小ASM磁盘路径
CRSNORMAL36000c29130bc55d46b0093070ebc8221 2G 36000c29130bc55d46b0093070ebc8222 2G 36000c29130bc55d46b0093070ebc8223 2G 36000c29130bc55d46b0093070ebc8224 2G/dev/asm-ocr1 /dev/asm-ocr1 /dev/asm-ocr3 /dev/asm-ocr4
DATAEXTERNAL36000c29130bc55d46b0093070ebc8225 500G 36000c29130bc55d46b0093070ebc8226 500G 36000c29130bc55d46b0093070ebc8227 500G 36000c29130bc55d46b0093070ebc8228 500G/dev/asm-data1 /dev/asm-data2 /dev/asm-data3 /dev/asm-data4

1.2、新存储磁盘信息说明

磁盘WWID及大小ASM路径新增到的磁盘组
36000c123456c55d46b0093070ebc8221 2G 36000c123456c55d46b0093070ebc8222 2G 36000c123456c55d46b0093070ebc8223 2G 36000c123456c55d46b0093070ebc8224 2G/dev/asm-ocr5 /dev/asm-ocr6 /dev/asm-ocr7 /dev/asm-ocr8CRS
36000c123456c55d46b0093070ebc8225 500G 36000c123456c55d46b0093070ebc8226 500G 36000c123456c55d46b0093070ebc8227 500G 36000c123456c55d46b0093070ebc8228 500G/dev/asm-data5 /dev/asm-data6 /dev/asm-data7 /dev/asm-data8DATA

1.3、确认新存储的磁盘wwid

-- 扫盘(如果存储已经分配存储盘了,但是数据库服务器上不显示)
-- 如果确定知道是host多少,指定host进行扫描,不确定的话可以全部扫描一遍
#echo "- - -">/sys/class/scsi_host/host0/scan
#echo "- - -">/sys/class/scsi_host/host1/scan
#echo "- - -">/sys/class/scsi_host/host2/scan
#echo "- - -">/sys/class/scsi_host/host3/scan
...
#echo "- - -">/sys/class/scsi_host/hostX/scan

或者使用脚本
#sh /usr/bin/rescan-scsi-bus.sh
-- 确定wwid
#lsscsi -si  --安装了lsscsi-0.27-6.el7.x86_64或其他版本的rpm包
#multipath -ll --使用了multipath作为多路径软件
#ls -l /dev/disk/by-id

2、修改multipath添加新增的磁盘

#cd /etc/ && cp multipath.conf multipath.conf.bak20220801
#vi /etc/multipath.conf
multipath {
      wwid 36000c123456c55d46b0093070ebc8221 
      alias ocr5 
      path_grouping_policy multibus
}
multipath {
      wwid 36000c123456c55d46b0093070ebc8222
      alias ocr6 
      path_grouping_policy multibus
}
multipath {
      wwid 36000c123456c55d46b0093070ebc8223
      alias ocr7
      path_grouping_policy multibus
}
multipath {
      wwid 36000c123456c55d46b0093070ebc8224
      alias ocr8
      path_grouping_policy multibus
}
multipath {
      wwid 36000c123456c55d46b0093070ebc8225
      alias data5 
      path_grouping_policy multibus
}
multipath {
      wwid 36000c123456c55d46b0093070ebc8226
      alias data6
      path_grouping_policy multibus
}
multipath {
      wwid 36000c123456c55d46b0093070ebc8227
      alias data7
      path_grouping_policy multibus
}
multipath {
      wwid 36000c123456c55d46b0093070ebc8228
      alias data8
      path_grouping_policy multibus
}
-- reload multipath
#multipath -r

3、修改udev添加新增的磁盘

#vi /etc/udev/rules.d/99-oracle-asmdevices.rules
ENV{DM_NAME}=="ocr5", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="asm-ocr5"
ENV{DM_NAME}=="ocr6", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="asm-ocr6"
ENV{DM_NAME}=="ocr7", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="asm-ocr7"
ENV{DM_NAME}=="ocr8", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="asm-ocr8"
ENV{DM_NAME}=="data5", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="asm-data5"
ENV{DM_NAME}=="data6", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="asm-data6"
ENV{DM_NAME}=="data7", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="asm-data7"
ENV{DM_NAME}=="data8", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="asm-data8"

-- 触发权限生效
#udevadm control --reload-rules
#udevadm trigger --sysname-match=dm-15
#udevadm trigger --sysname-match=dm-16
#udevadm trigger --sysname-match=dm-17
#udevadm trigger --sysname-match=dm-18
#udevadm trigger --sysname-match=dm-19
#udevadm trigger --sysname-match=dm-20
#udevadm trigger --sysname-match=dm-21
#udevadm trigger --sysname-match=dm-22

-- 检查属组及权限
#ll /dev/mapper/*
#ll /dev/dm-*|grep grid

4、确认磁盘可用性

#su - grid
$asmcmd lsdsk --candidate
$sqlplus / as sysasm<<eof
set linesize 200 pages 1000;
col name format a16;
col path format a30;
col HEADER_STATUS format a16;
select name,path,HEADER_STATUS,TOTAL_MB
from v$asm_disk;
eof

5、备份ocr和votedisk

#/u01/app/11.2.0.4/grid/bin/ocrconfig -export /backup/20220801/ocr_20220801.exp
dd if=/dev/asm-ocr1 of=/backup/20220801/votedisk_20220801_ocr1.bak
dd if=/dev/asm-ocr2 of=/backup/20220801/votedisk_20220801_ocr2.bak
dd if=/dev/asm-ocr3 of=/backup/20220801/votedisk_20220801_ocr3.bak
dd if=/dev/asm-ocr4 of=/backup/20220801/votedisk_20220801_ocr4.bak

6、CRS磁盘组替换新存储

#su - grid
$sqlplus / as sysasm<<eof
alter diskgroup CRS add disk
'/dev/asm-ocr5',
'/dev/asm-ocr6',
'/dev/asm-ocr7',
'/dev/asm-ocr8'
drop disk
OCR_0000,
OCR_0001,
OCR_0002,
OCR_00003
rebalance power 8;
eof

--查看rebalance进度
#su - grid
$sqlplus / as sysasm<<eof
select * from v$asm_operation;
eof

7、DATA磁盘组替换新存储

#su - grid
$sqlplus / as sysasm<<eof
alter diskgroup DATA add disk
'/dev/asm-data5',
'/dev/asm-data6',
'/dev/asm-data7',
'/dev/asm-data8'
drop disk
DATA_0000,
DATA_0001,
DATA_0002,
DATA_0003
rebalance power 8;
eof

--查看rebalance进度
#su - grid
$sqlplus / as sysasm<<eof
select * from v$asm_operation;
eof

8、查看磁盘组磁盘状态

– 旧存储磁盘的状态为FORMER,新存储磁盘的状态为MEMBER

$ sqlplus / as sysasm<<eof
set linesize 200 pages 1000;
col name format a16;
col path format a30;
col HEADER_STATUS format a16;
select name,path,HEADER_STATUS,TOTAL_MB
from v\$asm_disk;
eof

9、检查集群状态及数据库状态

#su - grid
$crsctl stat res -t

#su - oracle
$sqlplus / as sysdba<<eof
select instance_name,status from gv\$instance;
select name,open_mode from gv\$database;
eof

-----------------oracle 磁盘组丢失,模拟failgroup磁盘丢失处理

磁盘组:datadg

磁盘:/dev/asm-diskb,/dev/asm-diskc

冗余级别:normal,一个failgroup只有一个磁盘

测试过程:

1:mount磁盘组启动数据库写入测试数据。

2:关闭虚拟机拔掉一块硬盘/dev/asm-diskb

3:拔掉硬盘之后看磁盘组是否能还能工作并查询数据。

下面开始

首先进行第一步插入测试数据

create table testtab(id int);

insert into testtab values(1);

insert into testtab values(2);

commit;

select * from testtab;

然后进行第二步发现磁盘组datadg不能mount,查看日志如下

SQL> ALTER DISKGROUP DATADG MOUNT NOTE: cache registered group DATADG number=1

incarn=0xd22857aa

NOTE: cache began mount (first) of group DATADG number=1

incarn=0xd22857aa

Mon Jul 14 12:38:57 2014

ASM Health Checker found 1 new failures

NOTE: Assigning number (1,1) to disk (/dev/asm-diskc)

NOTE: GMON heartbeating for grp 1

GMON querying group 1 at 17 for pid 17, osid 1386

Mon Jul 14 12:39:03 2014

NOTE: Assigning number (1,0) to disk ()

GMON querying group 1 at 18 for pid 17, osid 1386

NOTE: cache dismounting (clean) group 1/0xD22857AA

(DATADG) NOTE: messaging CKPT to quiesce pins Unix process pid: 1386,

image: oracle@asmstudy (TNS

V1-V3)

NOTE: dbwr not being msg'd to dismount

NOTE: lgwr not being msg'd to dismount

NOTE: cache dismounted group 1/0xD22857AA

(DATADG) NOTE: cache ending mount (fail) of group DATADG number=1

incarn=0xd22857aa

NOTE: cache deleting context for group DATADG

1/0xd22857aa

GMON dismounting group 1 at 19 for pid 17, osid 1386

NOTE: Disk in mode 0x7f marked for

de-assignment

NOTE: Disk DATADG_0001 in mode 0x7f marked for

de-assignment

ERROR: diskgroup DATADG was not mounted

WARNING: Disk Group DATADG containing spfile for this instance is

not mounted

ORA-15032: not all alterations performed

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk "0" is missing from group number

"1" ERROR: ALTER DISKGROUP DATADG MOUNT Mon Jul 14 12:39:03 2014

ASM Health Checker found 1 new failures

意思很明白了,就是少了一个disk,所以mount失败了。

但是datadg是normal冗余的,另外一个磁盘还可以用,所以我们强制mount

[grid@asmstudy

~]$ sqlplus / as sysasm

SQL*Plus: Release

11.2.0.4.0 Production on Mon Jul 14 12:43:33 2014

Copyright (c)

1982, 2013, Oracle. All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit

Production

With the Automatic Storage Management option

SQL> alter

diskgroup datadg mount force;

Diskgroup

altered.

SQL> select name,path,failgroup,mount_status from v$asm_disk

where group_number=1;

NAME PATH FAILGROUP MOUNT_STATUS

------------------------------ --------------------

-------------------- --------------------

_DROPPED_0000_DATADG DATADG_0000  MISSING

DATADG_0001 /dev/asm-diskc DATADG_0001  CACHED

发现上面有块盘是丢失状态,我们启动数据库然后查询数据,而数据还在,因为normal级别的冗余有2个数据备份,

一个failgroup存放primary copy,另一个failgroup存放备用copy。

现在我们将磁盘还原回去。

关闭虚拟机,然后将磁盘加回去,然后启动,之前删除的是 /dev/asm-diskb

所以拔掉磁盘可用不做操作,自动消失的。

SQL> alter

diskgroup datadg add disk '/dev/asm-diskb' name diskb rebalance

power 5 wait;

alter diskgroup datadg add disk '/dev/asm-diskb' name diskb

rebalance power 5 wait

*

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15033: disk '/dev/asm-diskb' belongs to diskgroup "DATADG"

报这个错误时因为磁盘设备头文件还存储了之前磁盘组的信息,因此需要清除头部的信息。

[root@asmstudy

~]# dd if=/dev/zero of=/dev/asm-diskb bs=1M count=200

200+0 records in

200+0 records out

209715200 bytes (210 MB) copied, 5.86341 s, 35.8 MB/s

然后再次执行上面的命令增加磁盘到磁盘组datadg中,并等待其平衡数据到新的磁盘中 具体可以查询视图v$asm_operation

SQL> alter

diskgroup datadg add disk '/dev/asm-diskb' name diskb rebalance

power 5 wait;

Diskgroup altered.

然后就可以查询v$asm_disk视图,恢复原样了。

直接替换存储表决磁盘的ASM

 1.新加三块共享盘设置权限(记得两个节点都做处理)
[root@rac1 ~]# raw /dev/raw/raw6 /dev/sdg
/dev/raw/raw6:  bound to major 8, minor 96
[root@rac1 ~]# raw /dev/raw/raw7 /dev/sdh
/dev/raw/raw7:  bound to major 8, minor 112
[root@rac1 ~]# raw /dev/raw/raw8 /dev/sdi
/dev/raw/raw8:  bound to major 8, minor 128
[root@rac1 ~]#
[root@rac1 ~]# chown grid:asmadmin /dev/raw/raw6
[root@rac1 ~]# chown grid:asmadmin /dev/raw/raw7
[root@rac1 ~]# chown grid:asmadmin /dev/raw/raw8
[root@rac1 ~]# 
[root@rac1 ~]# chmod 660 /dev/raw/raw6
[root@rac1 ~]# chmod 660 /dev/raw/raw7
[root@rac1 ~]# chmod 660 /dev/raw/raw8
2.创建一个磁盘组,包含一个投票故障组
[grid@rac1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Thu Nov 26 13:16:44 2015
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL>    create diskgroup ocr_data normal redundancy
  2       failgroup fg1 disk '/dev/raw/raw6'
  3       failgroup fg2 disk '/dev/raw/raw7'
  4       quorum failgroup fg3 disk '/dev/raw/raw8'
  5       attribute 'compatible.asm'='11.2.0.0.0';
Diskgroup created.
 
SQL> col name for a10
SQL> select v1.name, v2.failgroup, v2.failgroup_type, v2.total_mb, v2.free_mb from                                  
  2    v$asm_diskgroup v1, v$asm_disk v2
  3      where v1.group_number = v2.group_number
  4        and v1.name = upper('ocr_data');
 
NAME       FAILGROUP                      FAILGRO   TOTAL_MB    FREE_MB
---------- ------------------------------ ------- ---------- ----------
OCR_DATA   FG3                            QUORUM        5120       5118
OCR_DATA   FG2                            REGULAR       5120       5067
OCR_DATA   FG1                            REGULAR       5120       5067
3.将表决磁盘文件迁移到新创建的磁盘组:
[root@rac1 bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
1. ONLINE   924682b9e7fb4fa5bf93e8a2a12a916f (/dev/raw/raw1) [OCR]
2. ONLINE   39d0f0108a734f32bf1aee2abf464db3 (/dev/raw/raw2) [OCR]
3. ONLINE   3e457cf73bba4f06bf92ca64e672f796 (/dev/raw/raw3) [OCR]
Located 3 voting disk(s).
[root@rac1 bin]# ./crsctl replace votedisk +OCR_DATA
Successful addition of voting disk 74692688b1184fcebf04f871a1fa0f31.
Successful addition of voting disk 1bcea48766164f79bf20752c8d54781d.
Successful addition of voting disk 4ed3c378eea04f1cbfd1d0021ebced2c.
Successful deletion of voting disk 924682b9e7fb4fa5bf93e8a2a12a916f.
Successful deletion of voting disk 39d0f0108a734f32bf1aee2abf464db3.
Successful deletion of voting disk 3e457cf73bba4f06bf92ca64e672f796.
Successfully replaced voting disk group with +OCR_DATA.
CRS-4266: Voting file(s) successfully replaced
4.查看磁盘组
SQL> col name for a10
SQL> select v1.name, v2.failgroup, v2.failgroup_type, v2.total_mb, v2.free_mb from
  2    v$asm_diskgroup v1, v$asm_disk v2
  3      where v1.group_number = v2.group_number
  4        and v1.name = upper('ocr_data');
 
NAME       FAILGROUP                      FAILGRO   TOTAL_MB    FREE_MB
---------- ------------------------------ ------- ---------- ----------
OCR_DATA   FG2                            REGULAR       5120       5035
OCR_DATA   FG1                            REGULAR       5120       5035
OCR_DATA   FG3                            QUORUM        5120       5086
[root@rac1 bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
1. ONLINE   74692688b1184fcebf04f871a1fa0f31 (/dev/raw/raw8) [OCR_DATA]
2. ONLINE   1bcea48766164f79bf20752c8d54781d (/dev/raw/raw6) [OCR_DATA]
3. ONLINE   4ed3c378eea04f1cbfd1d0021ebced2c (/dev/raw/raw7) [OCR_DATA]
Located 3 voting disk(s).
 

1、存储层面先划分好LUN映射到每个节点

note:注意配置好多路径软件,每个主流存储都有自家的多路径软件

2、绑定ASM磁盘

note:asm标记的方法很多,linux建议使用udev绑定的方式,直接使用块设备

查看磁盘wwpn号

14f504e46494c455263364179366a2d766d5a782d49376866
[root@19c-h1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdi
14f504e46494c45524c47426b63702d474d48592d53685741
[root@19c-h1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdj
14f504e46494c4552394e535369362d6d486c692d50783334
[root@19c-h1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdk
153dfsfss494c4552394e535369362d6d486c692d54832335

note:linux6的scsi_id位置为/sbin/scsi_id

编写rule文件,增加新的磁盘

vi /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="14f504e46494c455263364179366a2d766d5a782d49376866", SYMLINK+="asmdisk/vote01", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="14f504e46494c45524c47426b63702d474d48592d53685741", SYMLINK+="asmdisk/vote02", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="14f504e46494c4552394e535369362d6d486c692d50783334", SYMLINK+="asmdisk/vote03", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="153dfsfss494c4552394e535369362d6d486c692d54832335", SYMLINK+="asmdisk/data04", OWNER="grid", GROUP="asmadmin", MODE="0660"

重启udev

systemctl restart systemd-udev-trigger.service

3、更换仲裁盘

note:集群的关键,利用failgroup特性进行更换,通常我们的RAC仲裁盘都normal冗余的有3块仲裁盘,且OCR文件和ASM参数文件也放在一起,12c以上多了口令文件

直接在盘级替换,不涉及上面步骤的迁移oracle文件。

检查OCR

ocrcheck

检查vote disk

crsctl query css votedisk

查看failgroup状态
note:这里查出仲裁盘对应的failgroup,下面我们需要对应添加新的asm磁盘进来

set line 300
col path for a30
select NAME,PATH,FAILGROUP,STATE from  v$asm_disk where GROUP_NUMBER=3;

添加新的ASM磁盘

alter diskgroup vote add 
failgroup VOTE_0000 disk '/dev/asmdisk/vote01'
failgroup VOTE_0001 disk '/dev/asmdisk/vote02'
failgroup VOTE_0002 disk '/dev/asmdisk/vote03'
rebalance power 6;

note:注意以上3个failgroup就是前面我们查出来的,一一对应

查看平衡操作

select * from v$asm_operation;

查看failgroup状态
note:下面可以看到每个failgroup对应2个asm磁盘

set line 300
col path for a30
select NAME,PATH,FAILGROUP,STATE from  v$asm_disk where GROUP_NUMBER=3;

删除旧的磁盘
note:等待平衡完毕,我们可以删除旧的asm磁盘,完成更换操作

alter diskgroup vote drop disk VOTE_0000,VOTE_0001,VOTE_0002 rebalance power 6;

再次等待平衡完成

select * from v$asm_operation;

最后查看failgroup状态

set line 300
col path for a30
select NAME,PATH,FAILGROUP,STATE from  v$asm_disk where GROUP_NUMBER=3;

最后检查仲裁盘
note:此时发现vote disk为新的磁盘

crsctl query css votedisk

4、更换常规数据磁盘组

note:这里我们就可以利用ASM很好的特性rebalance进行更换

往磁盘组添加磁盘
note:注意平衡等级可以根据业务情况适当调整,建议的下班时间进行操作

alter diskgroup DATA add disk 
'/dev/asmdisk/asm_data04'
rebalance power 6;

查看平衡操作

select * from v$asm_operation;

查看磁盘

set line 300
select b.name group_name, a.name, a.path from v$asm_disk a , v$asm_diskgroup b where a.group_number=b.group_number order by group_name;

删除磁盘
note:等待平衡完毕,可以删除旧的磁盘已完成磁盘更换

alter diskgroup DATA drop disk DATA2
rebalance power 6;

最后等待平衡完成

select * from v$asm_operation;

oracleasm

--------------------------------------------------------------
目的:一套11.2.0.4 双节点的RAC当前使用的是阵列里的FC(Fibre channel)盘,运行一段时间后发现数据库比较空闲,所以打算将其迁移到同阵列的SATA盘上,以将FC盘预留给对IO要求更高的应用。
迁移方法:由ASM实现存储管理的数据库,使用ASM diskgroup在线迁移方式是再好不过的选择
挑战:难度虽然不大,但前一次做diskgroup在线迁移还是3年以前的事情,这次特意从头到尾将整个步骤整理如下,供参考

//
// 第一部分:新盘准备
//
1、shajx1主机上当前使用的dg及盘
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576    743632   350106                0          350106              0             N  AJXDG1/
MOUNTED  EXTERN  N         512   4096  1048576    743632   321451                0          321451              0             N  AJXDG2/
MOUNTED  EXTERN  N         512   4096  1048576      7500     7134                0            7134              0             N  SYSDG1/
MOUNTED  EXTERN  N         512   4096  1048576      7500     7103                0            7103              0             Y  SYSDG2/
MOUNTED  EXTERN  N         512   4096  1048576      7500     7403                0            7403              0             N  SYSDG3/
ASMCMD> lsdsk -G sysdg1
Path
/dev/rhdiskpower44
/dev/rhdiskpower45
ASMCMD> lsdsk -G sysdg2
Path
/dev/rhdiskpower46
/dev/rhdiskpower47
ASMCMD> lsdsk -G sysdg3
Path
/dev/rhdiskpower48
/dev/rhdiskpower49


2、shajx1主机上认到的新盘及大小如下:
/dev/rhdiskpower58     13FD 09E:0 05A:C8  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower59     1401 09E:0 06A:C7  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower60     1405 09E:0 05A:CA  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower61     1409 09E:0 06A:C9  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower62     140D 09E:0 05A:CC  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower63     1411 09E:0 06A:CB  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower64     1415 09E:0 06C:D0  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower65     1419 09E:0 05C:D1  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower66     141D 09E:0 06C:D2  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower67     1421 09E:0 05C:D3  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower68     1425 09E:0 06C:D4  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower69     1429 09E:0 06A:CD  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower70     142D 09E:0 05D:D6  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower71     1431 09E:0 06D:D5  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower72     1435 09E:0 05D:D8  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower73     1439 09E:0 06D:D7  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower74     143D 09E:0 05D:DA  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower75     1441 09E:0 06D:D9  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower76     1445 09E:0 05D:DC  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower77     1449 09E:0 06D:DB  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower78     144D 09E:0 06B:C0  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower79     1451 09E:0 05B:C1  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower80     1455 09E:0 06B:C2  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower81     1459 09E:0 05B:C3  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower82     145D 09E:0 06B:C4  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower83     1461 09E:0 06A:D6  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower84     1465 09E:0 05A:D5  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower85     1469 09E:0 06A:D8  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower86     146D 09E:0 05A:D7  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower87     1471 09E:0 06A:DA  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower88     1475 09E:0 05A:D9  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower89     1479 09E:0 06A:DC  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower90     147D 09E:0 05A:DB  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower91     1481 09E:0 05C:C0  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower92     1485 09E:0 06C:C1  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower93     1489 09E:0 05C:C2  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower94     148D 09E:0 06C:C3  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower95     1491 09E:0 05C:C4  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower96     1495 09E:0 06D:C6  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower97     1499 09E:0 05D:C5  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower98     149D 09E:0 06D:C8  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower99     14A1 09E:0 05D:C7  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower100    14A5 09E:0 06D:CA  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower101    14A9 09E:0 05D:C9  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower102    40DF 09E:0 10A:D18 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower103    40E0 09E:0 10C:D18 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower104    40E1 09E:0 09B:D18 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower105    40E2 09E:0 09D:D18 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower106    40E3 09E:0 09A:D19 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower107    40E4 09E:0 09C:D19 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower108    40DD 09E:0 09B:C1B 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower109    40DE 09E:0 10D:C1A 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower110    40D5 09E:0 08B:D1D 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower111    40D6 09E:0 08D:D1D 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower112    40D7 09E:0 10A:C1B 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower113    40D8 09E:0 09C:C1A 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower114    40D9 09E:0 10B:C1A 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower115    40DA 09E:0 09D:C19 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower116    40DB 09E:0 09A:C1C 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower117    40DC 09E:0 10C:C1B 2-Way Mir     N/Grp'd      RW     938


其中hdiskpower58~89用于数据文件,hdiskpower90~hdiskpower101保留用于archivelog文件系统,hdiskpower102~hdiskpower117用于新建OCR、vote盘所在DG


3、修改hdiskpower58~hdiskpower89、hdiskpower102~hdiskpower117的磁盘属性及属主,属性为crw-rw----,属主为grid.asmadmin,两个节点都要修改
shajx1、shajx2都要运行:
i=58
while [ $i -lt 90 ]
do
chown grid.asmadmin /dev/rhdiskpower$i
((i=i+1))
done


i=102
while [ $i -lt 118 ]
do
chown grid.asmadmin /dev/rhdiskpower$i
((i=i+1))
done


4、登陆ASM实例确认新加的盘都能认到,两个节点都要执行
sqlplus '/as sysasm'
col path format a40
set linesize 160 pagesize 200
select mount_status,header_status,mode_status,os_mb,state,path from v$asm_disk where header_status='CANDIDATE' order by path;
MOUNT_S HEADER_STATU MODE_ST      OS_MB STATE    PATH
------- ------------ ------- ---------- -------- ----------------------------------------
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower0
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower1
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower10
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower102
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower103
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower104
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower105
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower106
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower107
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower108
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower109
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower11
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower110
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower111
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower112
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower113
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower114
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower115
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower116
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower117
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower2
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower3
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower4
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower5
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower59
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower6
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower60
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower61
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower62
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower63
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower64
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower65
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower66
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower67
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower68
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower69
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower7
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower70
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower71
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower72
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower73
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower74
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower75
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower76
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower77
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower78
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower79
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower8
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower80
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower81
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower82
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower83
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower84
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower85
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower86
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower87
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower88
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower89
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower9


5、挑首尾两块盘作读测试及kfed测试,使用oracle或者grid用户执行:
dd if=/dev/rhdiskpower58 of=/dev/null count=100 bs=8192
100+0 records in.
100+0 records out.


dd if=/dev/rhdiskpower89 of=/dev/null count=100 bs=8192
100+0 records in.
100+0 records out.


kfed read /dev/rhdiskpower58
kfed read /dev/rhdiskpower89


kfed的输出如下:
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000


MOS里关于KFBTYP_INVALID的描述:If the disk is not a FORMER disk, then kfed on new disks will return:    kfbh.type: 0 ; 0x002: KFBTYP_INVALID and that is expected


6、修改hdiskpowerX的reserve_policy属性,两节点都要执行
i=58
while [ $i -lt 90 ]
do
chdev -l hdiskpower${i} -a reserve_policy=no_reserve
(( i=i+1 ))
done


i=102
while [ $i -lt 118 ]
do
chdev -l hdiskpower${i} -a reserve_policy=no_reserve
(( i=i+1 ))
done


***验证reserve_policy是否已经成功
i=58
while [ $i -lt 90 ]
do
echo "hdiskpower${i} " `lsattr -El hdiskpower${i} |grep reserve_policy | awk '{print $2}'`
(( i=i+1 ))
done


i=102
while [ $i -lt 118 ]
do
echo "hdiskpower${i} " `lsattr -El hdiskpower${i} |grep reserve_policy | awk '{print $2}'`
(( i=i+1 ))
done


7、选一块新的盘在上面创建一个Diskgroup
create diskgroup dgnew1 external redundancy disk '/dev/rhdiskpower58';


SQL> SELECT NAME,TOTAL_MB,usable_file_mb,state FROM V$ASM_DISKGROUP where name='DGNEW1';


NAME                             TOTAL_MB USABLE_FILE_MB STATE
------------------------------ ---------- -------------- -----------
DGNEW1                              46477          46427 MOUNTED                <---损耗主要体现在Allocation Table、PST、Disk Directory等各种directory


diskgroup里所有的metadata见下:


PST
Allocation Table
Free Space Table
Disk Directory  (file number 2)
File Directory   (file number 1)
ACD  (file number 3)
Alias Directory  (file number 6)
Attribute Directory (file number 9)
User and Group Directory
Used Space Directory (file number 8)
How many partners
Continuing Operations Directory  (file number 4)
Template Directory (file number 5)
Volume Directory
staleness_directory_and_registry


8、检查crs里新建diskgroup的状态
grid@qc740701a:/home/grid>crsctl status res ora.DGNEW1.dg -t
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DGNEW1.dg
               OFFLINE OFFLINE      qc740701a        <---第二个节点上需要mount
               ONLINE  ONLINE       qc740702a     


因为是RAC环境,在第二个节点上也把这个diskgroup mount起来
su - grid
srvctl start diskgroup -g dgnew1 -n qc740701a


crsctl status res ora.DGNEW1.dg -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DGNEW1.dg
               ONLINE  ONLINE       qc740701a
               ONLINE  ONLINE       qc740702a


9、如果在第8步里,两个节点都能mount起DG,那么将这个diskgroup从N-1个节点都dismount,只留最后一个节点用于drop
qc740702a: dismount
srvctl stop diskgroup -g dgnew1 -n qc740702a


qc740701a: drop
drop diskgroup dgnew1


10、至此测试完毕,新认的盘已可以使用


//
// 第二部分:迁移ASM spfile, voting disk,ocr所在的DG至SATA盘
//
1、ASM spfile, voting disk 存放在SYSDG1、SYSDG2中,SYSDG3里暂时没有文件
ASMCMD> spget
+SYSDG1/shajx-cluster/asmparameterfile/registry.253.854489087    <---spfile
ASMCMD> quit


grid@qc740702a:/home/grid>ocrcheck -config   <---ocr file
Oracle Cluster Registry configuration is :
         Device/File Name         :    +sysdg1
         Device/File Name         :    +sysdg2
         
grid@qc740702a:/home/grid>crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
1. ONLINE   46c6fdafabb74fa1bf2987f4d9a846b5 (/dev/rhdiskpower46) [SYSDG2]   <---voting disk
Located 1 voting disk(s).


2、将hdiskpower102~hdiskpower117添加到SYSDG1、SYSDG2、SYSDG3
SYSDG1:
alter diskgroup sysdg1 add disk '/dev/rhdiskpower102','/dev/rhdiskpower103','/dev/rhdiskpower104','/dev/rhdiskpower105' drop disk SYSDG1_0000,SYSDG1_0001 rebalance power 500;


SYSDG2:
alter diskgroup sysdg2 add disk '/dev/rhdiskpower106','/dev/rhdiskpower107','/dev/rhdiskpower108','/dev/rhdiskpower109' drop disk SYSDG2_0000,SYSDG2_0001 rebalance power 500;


SYSDG3:
alter diskgroup sysdg3 add disk '/dev/rhdiskpower110','/dev/rhdiskpower111','/dev/rhdiskpower112','/dev/rhdiskpower113' drop disk SYSDG3_0000,SYSDG3_0001 rebalance power 500;


3、将hdiskpower58~hdiskpower73添加到AJXDG1
AJXDG1: 通过shell脚本获取SQL脚本
i=58
while [ $i -lt 74 ]
do
echo "alter diskgroup AJXDG1 add disk '/dev/rhdiskpower${i}' rebalance power 0;";
(( i=i+1 ))
done


得到如下脚本直接运行
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower58' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower59' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower60' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower61' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower62' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower63' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower64' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower65' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower66' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower67' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower68' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower69' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower70' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower71' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower72' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower73' rebalance power 0;


执行如下SQL
select 'alter diskgroup '||d.name||' drop disk '||s.name ||' rebalance power 0;' from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG1');


得到drop old disk的cmd
alter diskgroup AJXDG1 drop disk AJXDG1_0000 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0001 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0002 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0003 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0004 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0005 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0006 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0007 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0008 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0009 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0010 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0011 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0012 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0013 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0014 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0015 rebalance power 0;


  ---3.1 观察rebalance前的磁盘状态
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG1');


NAME                           STATE      TOTAL_MB    FREE_MB PATH                           CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG1_0000                    DROPPING      46477      21880 /dev/rhdiskpower12             02-AUG-14
AJXDG1_0001                    DROPPING      46477      21885 /dev/rhdiskpower13             02-AUG-14
AJXDG1_0002                    DROPPING      46477      21883 /dev/rhdiskpower14             02-AUG-14
AJXDG1_0003                    DROPPING      46477      21878 /dev/rhdiskpower15             02-AUG-14
AJXDG1_0004                    DROPPING      46477      21883 /dev/rhdiskpower16             02-AUG-14
AJXDG1_0005                    DROPPING      46477      21874 /dev/rhdiskpower17             02-AUG-14
AJXDG1_0006                    DROPPING      46477      21886 /dev/rhdiskpower18             02-AUG-14
AJXDG1_0007                    DROPPING      46477      21879 /dev/rhdiskpower19             02-AUG-14
AJXDG1_0008                    DROPPING      46477      21879 /dev/rhdiskpower20             02-AUG-14
AJXDG1_0009                    DROPPING      46477      21885 /dev/rhdiskpower21             02-AUG-14
AJXDG1_0010                    DROPPING      46477      21887 /dev/rhdiskpower22             02-AUG-14
AJXDG1_0011                    DROPPING      46477      21884 /dev/rhdiskpower23             02-AUG-14
AJXDG1_0012                    DROPPING      46477      21881 /dev/rhdiskpower24             02-AUG-14
AJXDG1_0013                    DROPPING      46477      21880 /dev/rhdiskpower25             02-AUG-14
AJXDG1_0014                    DROPPING      46477      21885 /dev/rhdiskpower26             02-AUG-14
AJXDG1_0015                    DROPPING      46477      21877 /dev/rhdiskpower27             02-AUG-14
AJXDG1_0017                    NORMAL        46477      46475 /dev/rhdiskpower59             21-APR-16
AJXDG1_0018                    NORMAL        46477      46475 /dev/rhdiskpower60             21-APR-16
AJXDG1_0019                    NORMAL        46477      46475 /dev/rhdiskpower61             21-APR-16
AJXDG1_0020                    NORMAL        46477      46475 /dev/rhdiskpower62             21-APR-16
AJXDG1_0021                    NORMAL        46477      46475 /dev/rhdiskpower63             21-APR-16
AJXDG1_0022                    NORMAL        46477      46475 /dev/rhdiskpower64             21-APR-16
AJXDG1_0023                    NORMAL        46477      46475 /dev/rhdiskpower65             21-APR-16
AJXDG1_0024                    NORMAL        46477      46475 /dev/rhdiskpower66             21-APR-16
AJXDG1_0025                    NORMAL        46477      46475 /dev/rhdiskpower67             21-APR-16
AJXDG1_0026                    NORMAL        46477      46475 /dev/rhdiskpower68             21-APR-16
AJXDG1_0027                    NORMAL        46477      46475 /dev/rhdiskpower69             21-APR-16
AJXDG1_0028                    NORMAL        46477      46475 /dev/rhdiskpower70             21-APR-16
AJXDG1_0029                    NORMAL        46477      46475 /dev/rhdiskpower71             21-APR-16
AJXDG1_0030                    NORMAL        46477      46475 /dev/rhdiskpower72             21-APR-16
AJXDG1_0031                    NORMAL        46477      46475 /dev/rhdiskpower73             21-APR-16
AJXDG1_0016                    NORMAL        46477      46475 /dev/rhdiskpower58             21-APR-16


---3.2 发起rebalance,查看rebalance进度
alter diskgroup ajxdg1 rebalance power 500;


select * from v$asm_operation;
GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           1 REBAL RUN         500        500      46040     393494      15207          22    <---est_rate=15207表示每分钟正在被move的AU个数,AU_SIZE=1M时表示每分钟15GB/分钟,折合250MB/s

***调高优先级,观察同步速度是否加快

alter diskgroup ajxdg1 rebalance power 999;

select * from v$asm_operation;


GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           1 REBAL RUN         999        999      13462     236076      15482          14     <---改变了优先级,相当于将原来优先级为500的rebalance先停止然后以新的优先级发起所以EST_WORK值比一开始时小,刨掉了已经完成同步的AU


select * from v$asm_operation;


GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           1 REBAL RUN         999        999      13822     236076      15469          14


看出调整为999之后速率并未明显上升,可能是因为IO请求已经出现了排队
powermt display every=1
Symmetrix logical device count=118
==============================================================================
----- Host Bus Adapters ---------  ------ I/O Paths -----  ------ Stats ------
###  HW Path                       Summary   Total   Dead  IO/Sec Q-IOs Errors
==============================================================================
   0 fscsi0                        optimal     118      0     117     1      0   <---Q-IOs!=0
   1 fscsi2                        optimal     118      0     132     2      0
   2 fscsi4                        optimal     118      0     127     2      0
   3 fscsi6                        optimal     118      0     124     1      0


---3.3 rebalance完成后的结果检查
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG1');


NAME                           STATE      TOTAL_MB    FREE_MB PATH                           CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG1_0017                    NORMAL        46477      21882 /dev/rhdiskpower59             21-APR-16
AJXDG1_0018                    NORMAL        46477      21882 /dev/rhdiskpower60             21-APR-16
AJXDG1_0019                    NORMAL        46477      21882 /dev/rhdiskpower61             21-APR-16
AJXDG1_0020                    NORMAL        46477      21881 /dev/rhdiskpower62             21-APR-16
AJXDG1_0021                    NORMAL        46477      21881 /dev/rhdiskpower63             21-APR-16
AJXDG1_0022                    NORMAL        46477      21882 /dev/rhdiskpower64             21-APR-16
AJXDG1_0023                    NORMAL        46477      21882 /dev/rhdiskpower65             21-APR-16
AJXDG1_0024                    NORMAL        46477      21881 /dev/rhdiskpower66             21-APR-16
AJXDG1_0025                    NORMAL        46477      21882 /dev/rhdiskpower67             21-APR-16
AJXDG1_0026                    NORMAL        46477      21882 /dev/rhdiskpower68             21-APR-16
AJXDG1_0027                    NORMAL        46477      21882 /dev/rhdiskpower69             21-APR-16
AJXDG1_0028                    NORMAL        46477      21882 /dev/rhdiskpower70             21-APR-16
AJXDG1_0029                    NORMAL        46477      21882 /dev/rhdiskpower71             21-APR-16
AJXDG1_0030                    NORMAL        46477      21881 /dev/rhdiskpower72             21-APR-16
AJXDG1_0031                    NORMAL        46477      21880 /dev/rhdiskpower73             21-APR-16
AJXDG1_0016                    NORMAL        46477      21882 /dev/rhdiskpower58             21-APR-16


4、将hdiskpower74~hdiskpower89添加到AJXDG2
AJXDG1: 通过shell脚本获取SQL脚本
i=74
while [ $i -lt 90 ]
do
echo "alter diskgroup AJXDG2 add disk '/dev/rhdiskpower${i}' rebalance power 0;";
(( i=i+1 ))
done


得到如下添加新盘的命令直接运行:
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower74' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower75' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower76' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower77' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower78' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower79' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower80' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower81' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower82' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower83' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower84' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower85' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower86' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower87' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower88' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower89' rebalance power 0;


执行如下SQL:
select 'alter diskgroup '||d.name||' drop disk '||s.name ||' rebalance power 0;' from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG2');


得到drop old disk的command
alter diskgroup AJXDG2 drop disk AJXDG2_0000 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0001 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0002 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0003 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0004 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0005 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0006 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0007 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0008 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0009 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0010 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0011 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0012 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0013 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0014 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0015 rebalance power 0;


  ---4.1 观察rebalance前的磁盘状态
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG2');
NAME                           STATE      TOTAL_MB    FREE_MB PATH                           CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG2_0000                    DROPPING      46477      20092 /dev/rhdiskpower28             02-AUG-14
AJXDG2_0001                    DROPPING      46477      20091 /dev/rhdiskpower29             02-AUG-14
AJXDG2_0002                    DROPPING      46477      20088 /dev/rhdiskpower30             02-AUG-14
AJXDG2_0003                    DROPPING      46477      20090 /dev/rhdiskpower31             02-AUG-14
AJXDG2_0004                    DROPPING      46477      20092 /dev/rhdiskpower32             02-AUG-14
AJXDG2_0005                    DROPPING      46477      20097 /dev/rhdiskpower33             02-AUG-14
AJXDG2_0006                    DROPPING      46477      20092 /dev/rhdiskpower34             02-AUG-14
AJXDG2_0007                    DROPPING      46477      20089 /dev/rhdiskpower35             02-AUG-14
AJXDG2_0008                    DROPPING      46477      20092 /dev/rhdiskpower36             02-AUG-14
AJXDG2_0009                    DROPPING      46477      20089 /dev/rhdiskpower37             02-AUG-14
AJXDG2_0010                    DROPPING      46477      20089 /dev/rhdiskpower38             02-AUG-14
AJXDG2_0011                    DROPPING      46477      20089 /dev/rhdiskpower39             02-AUG-14
AJXDG2_0012                    DROPPING      46477      20091 /dev/rhdiskpower40             02-AUG-14
AJXDG2_0013                    DROPPING      46477      20094 /dev/rhdiskpower41             02-AUG-14
AJXDG2_0014                    DROPPING      46477      20089 /dev/rhdiskpower42             02-AUG-14
AJXDG2_0015                    DROPPING      46477      20087 /dev/rhdiskpower43             02-AUG-14
AJXDG2_0016                    NORMAL        46477      46475 /dev/rhdiskpower74             22-APR-16
AJXDG2_0028                    NORMAL        46477      46475 /dev/rhdiskpower86             22-APR-16
AJXDG2_0029                    NORMAL        46477      46475 /dev/rhdiskpower87             22-APR-16
AJXDG2_0030                    NORMAL        46477      46475 /dev/rhdiskpower88             22-APR-16
AJXDG2_0031                    NORMAL        46477      46475 /dev/rhdiskpower89             22-APR-16
AJXDG2_0024                    NORMAL        46477      46475 /dev/rhdiskpower82             22-APR-16
AJXDG2_0025                    NORMAL        46477      46475 /dev/rhdiskpower83             22-APR-16
AJXDG2_0026                    NORMAL        46477      46475 /dev/rhdiskpower84             22-APR-16
AJXDG2_0027                    NORMAL        46477      46475 /dev/rhdiskpower85             22-APR-16
AJXDG2_0021                    NORMAL        46477      46475 /dev/rhdiskpower79             22-APR-16
AJXDG2_0022                    NORMAL        46477      46475 /dev/rhdiskpower80             22-APR-16
AJXDG2_0023                    NORMAL        46477      46475 /dev/rhdiskpower81             22-APR-16
AJXDG2_0017                    NORMAL        46477      46475 /dev/rhdiskpower75             22-APR-16
AJXDG2_0018                    NORMAL        46477      46475 /dev/rhdiskpower76             22-APR-16
AJXDG2_0019                    NORMAL        46477      46475 /dev/rhdiskpower77             22-APR-16
AJXDG2_0020                    NORMAL        46477      46475 /dev/rhdiskpower78             22-APR-16


  ---4.2 发起rebalance,查看rebalance进度
alter diskgroup ajxdg2 rebalance power 1011;


SQL> select * from v$asm_operation


GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           2 REBAL RUN        1011       1011       7834     422149      14768          28


  ---4.3 rebalance完成后的结果检查
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG2') order by s.name;
NAME                           STATE      TOTAL_MB    FREE_MB PATH                           CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG2_0016                    NORMAL        46477      20090 /dev/rhdiskpower74             22-APR-16
AJXDG2_0017                    NORMAL        46477      20091 /dev/rhdiskpower75             22-APR-16
AJXDG2_0018                    NORMAL        46477      20091 /dev/rhdiskpower76             22-APR-16
AJXDG2_0019                    NORMAL        46477      20091 /dev/rhdiskpower77             22-APR-16
AJXDG2_0020                    NORMAL        46477      20091 /dev/rhdiskpower78             22-APR-16
AJXDG2_0021                    NORMAL        46477      20090 /dev/rhdiskpower79             22-APR-16
AJXDG2_0022                    NORMAL        46477      20091 /dev/rhdiskpower80             22-APR-16
AJXDG2_0023                    NORMAL        46477      20091 /dev/rhdiskpower81             22-APR-16
AJXDG2_0024                    NORMAL        46477      20090 /dev/rhdiskpower82             22-APR-16
AJXDG2_0025                    NORMAL        46477      20091 /dev/rhdiskpower83             22-APR-16
AJXDG2_0026                    NORMAL        46477      20091 /dev/rhdiskpower84             22-APR-16
AJXDG2_0027                    NORMAL        46477      20091 /dev/rhdiskpower85             22-APR-16
AJXDG2_0028                    NORMAL        46477      20090 /dev/rhdiskpower86             22-APR-16
AJXDG2_0029                    NORMAL        46477      20090 /dev/rhdiskpower87             22-APR-16
AJXDG2_0030                    NORMAL        46477      20091 /dev/rhdiskpower88             22-APR-16
AJXDG2_0031                    NORMAL        46477      20091 /dev/rhdiskpower89             22-APR-16


5、(可选),将原先的Votedisk从external redundancy的DG迁移到normal redundancy的DG
grid@qc740702a:/home/grid>crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a48d1185b7be4f17bf06f6dd01e84d88 (/dev/rhdiskpower106) [SYSDG2]    <---原来voting disk只有一块盘
Located 1 voting disk(s).


***创建normal redundancy的diskgroup
create diskgroup sysdg3 normal redundancy failgroup vdfg1 disk '/dev/rhdiskpower110','/dev/rhdiskpower111' failgroup vdfg2 disk '/dev/rhdiskpower112','/dev/rhdiskpower113' failgroup vdfg3 disk '/dev/rhdiskpower114','/dev/rhdiskpower115' attribute 'compatible.asm'='11.2.0.3','compatible.rdbms'='11.2.0.0.0' ;


***迁移到sysdg3
grid@qc740702a:/home/grid>crsctl replace votedisk +sysdg3
Successful addition of voting disk 72d81026f64d4f86bf7a7ec0845201db.
Successful addition of voting disk 2fa55993ec494fa8bf839661e2c43e89.
Successful addition of voting disk 1127e47a60084f8ebf6f2292143710c0.
Successful deletion of voting disk a48d1185b7be4f17bf06f6dd01e84d88.
Successfully replaced voting disk group with +sysdg3.
CRS-4266: Voting file(s) successfully replaced


grid@qc740702a:/home/grid>crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   72d81026f64d4f86bf7a7ec0845201db (/dev/rhdiskpower110) [SYSDG3] <---现在voting disk有三块了
 2. ONLINE   2fa55993ec494fa8bf839661e2c43e89 (/dev/rhdiskpower112) [SYSDG3]
 3. ONLINE   1127e47a60084f8ebf6f2292143710c0 (/dev/rhdiskpower114) [SYSDG3]
Located 3 voting disk(s).

 

  • 0
    点赞
  • 4
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值