l 随着业务数据的快速增长，我们经常需要对数据库空间进行扩容，以满足空间快速增长的需要。 目前数据库使用存储做为存储介质，数据库数据文件、控制文件、在线重做日志均存储在存储上。
AIX操作系统增加存储能够做到在线认盘。Linux操作系统目前的操作是增加存储lun后，重启服务器来认盘。
l 为了进一步提高业务可用性，减少停机时间。探索在linux平台上也实现增加存储后能在线认盘。
本次测试基于redhat 5.4、redhat 6.2两个操作系统版本，测试在有压力的情况下增加存储lun。
二、 多路径软件安装
操作系统为redhat6.2时，多路径软件版本：hdlm7.2
操作系统为redhat5.4时，多路径软件版本：hdlm6.3
两个版本的安装步骤一样，以hdlm6.3为例：
1）解压缩hdlm6.3_linux.zip
unziphdlm6.3_linux.zip
2）把所有文件的权限改777
chmod -R777 *
3）把license拷贝到/var/tmp
cphdlm_license /var/tmp
4）安装
./installhdlm
5）重启服务器
三、 在线增加存储lun
Redhat 6.2 和redhat 5.4操作系统版本下，步骤稍有不同。Redhat 6.2环境下可自动认盘，redhat 5.4环境下需要手动认盘。下面分别描述具体步骤：
1. redhat 6.2操作系统下，在线增加存储lun
操作系统：Red HatEnterprise Linux Server release 6.2 (Santiago) x86_64
存储： Hitachi storage HUS150
在测试过程中，运行IO测试工具orion：
#./orion_linux_x86-64 -run oltp -testname test-num_disks 4
1.1先用管理软件把新增的lun分配到相应的host group.
测试环境hitachi管理软件：http://192.168.6.38:23015/StorageNavigatorModular/jsp/Title.jsp
选择存储array name: HUS150_93012124
Lun: 0600 当前链路0
操作host group: TESTZHF_0A、TESTZHF_1A、TESTZHF_0B、TESTZHF_1B
Edit host group—available volumes, 把lun 0600 分别加到4个host group, 因为有四条链路。
现在每个host group有5个lun: 0104，0200，0300，0404，0600
1.2.linux上，动态探测新设备的SCSI/FC bus
操作系统版本redhat 6 , QLogic HBA驱动
# echo 1 > /sys/class/fc_host/host1/issue_lip
# echo 1 > /sys/class/fc_host/host2/issue_lip
# echo "- - -" >/sys/class/scsi_host/host1/scan
# echo "- - -" >/sys/class/scsi_host/host2/scan
注：/sys/class/scsi_host/host1/scan
Host后的数值n表示: host port number
1.3．检查操作系统是否认到盘
#cat/proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 32 Lun: 00
Vendor:DP Model: BACKPLANE Rev: 1.09
Type: Enclosure ANSI SCSI revision: 05
Host: scsi0 Channel: 02 Id: 00 Lun: 00
Vendor:DELL Model: PERC H700 Rev: 2.10
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 00 Lun: 01
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 00 Lun: 02
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 00 Lun: 03
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 01 Lun: 00
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 01 Lun: 01
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 01 Lun: 02
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 01 Lun: 03
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 01
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 02
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 03
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 01 Lun: 00
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 01 Lun: 01
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 01 Lun: 02
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2 Channel: 00 Id: 01 Lun: 03
Vendor:HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1Channel: 00 Id: 00 Lun: 04
Vendor: HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi1Channel: 00 Id: 01 Lun: 04
Vendor: HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2Channel: 00 Id: 00 Lun: 04
Vendor: HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi2Channel: 00 Id: 01 Lun: 04
Vendor: HITACHI Model: DF600F Rev: 0000
Type: Direct-Access ANSI SCSI revision: 04
lun编码为04的即为新加的盘，已认到
1.4.更新HDLM配置信息
#/sbin/dlmcfgmgr -r
KAPL10339-I This operation will change theconfiguration of HDLM devices. Do you want to continue? [y/n]: y
KAPL10341-I The HDLM device configurations havebeen changed.
KAPL10302-I /sbin/dlmcfgmgr completed normally.
1.5.确认结果
#/sbin/dlmcfgmgr -v
HDevName Management Device Host Channel Target Lun
/dev/sddlmaa configured /dev/sdb 1 0 0 0
/dev/sdf 1 0 1 0
/dev/sdj 2 0 0 0
/dev/sdn 2 0 1 0
/dev/sddlmab configured /dev/sdc 1 0 0 1
/dev/sdg 1 0 1 1
/dev/sdk 2 0 0 1
/dev/sdo 2 0 1 1
/dev/sddlmac configured /dev/sdd 1 0 0 2
/dev/sdh 1 0 1 2
/dev/sdl 2 0 0 2
/dev/sdp 2 0 1 2
/dev/sddlmad configured /dev/sde 1 0 0 3
/dev/sdi 1 0 1 3
/dev/sdm 2 0 0 3
/dev/sdq 2 0 1 3
/dev/sddlmae configured /dev/sdr 1 0 0 4
/dev/sds 1 0 1 4
/dev/sdt 2 0 0 4
/dev/sdu 2 0 1 4
KAPL10302-I /sbin/dlmcfgmgr completed normally.
--/dev/sddlmae为新加的盘.
如果/dev/sddlmae 状态为unconfigured，表示未被HDLM多路径软件管理。
执行以下命令让HDLM管理/dev/sddlmae
#/sbin/dlmcfgmgr -i /dev/sddlmae
再次确认：
#/sbin/dlmcfgmgr -v
查看多路径信息，已能识别设备： /dev/sddlmae，并且4条链路都是online.
#/opt/DynamicLinkManager/bin/dlnkmgr view -lu
Product : DF600F
SerialNumber : 93012124
LUs : 5
iLU HDevName Device PathIDStatus
0104 sddlmaa /dev/sdb 000000 Online
/dev/sdf 000004 Online
/dev/sdj 000008 Online
/dev/sdn 000009 Online
0200 sddlmab /dev/sdc 000001 Online
/dev/sdg 000005 Online
/dev/sdk 000010 Online
/dev/sdo 000011 Online
0300 sddlmac /dev/sdd 000002 Online
/dev/sdh 000006 Online
/dev/sdl 000012 Online
/dev/sdp 000013 Online
0404 sddlmad /dev/sde 000003 Online
/dev/sdi 000007 Online
/dev/sdm 000014 Online
/dev/sdq 000015 Online
0600 sddlmae /dev/sdr 000016 Online
/dev/sds 000017 Online
/dev/sdt 000018 Online
/dev/sdu 000019 Online
KAPL01001-I The HDLM command completed normally.Operation name = view, completion time = 2014/01/10 15:34:02
Orion测试结果显示，现有存储lun无明显异常。测试结果见：《redhat 5.4在线认盘_orion测试结果》
2． Redhat 5.4操作系统下，在线增加存储lun
此处以生产历史库的操作为例。
操作系统版本：Red HatEnterprise Linux Server release 5.4 (Tikanga) x86_64
存储：Hitachi storage AMS2300
2.1先用管理软件把新增的lun分配到相应的host group.
新增lun id: 2000
2.2 手动添加SCSI设备
# cd/proc/scsi/qla2xxx/
# ls
0 1
使用以下命令：
# echo"scsi-qlascan" > /proc/scsi/qla2xxx/0
# echo"scsi-qlascan" > /proc/scsi/qla2xxx/1
查看：Id:Lun部分
# cat/proc/scsi/qla2xxx/0
( 0: 7):Total reqs 0, Pending reqs 0, flags 0x0*, Dflags 0x0, 0:0:81 00
( 1: 7):Total reqs 0, Pending reqs 0, flags 0x0*, Dflags 0x0, 0:0:82 00
# cat/proc/scsi/qla2xxx/1
( 0: 7):Total reqs 0, Pending reqs 0, flags 0x0*, Dflags 0x0, 1:0:81 00
( 1: 7):Total reqs 0, Pending reqs 0, flags 0x0*, Dflags 0x0, 1:0:82 00
手动加入:
# echo"scsi add-single-device 0 0 0 7" > /proc/scsi/scsi
# echo"scsi add-single-device 0 0 1 7" > /proc/scsi/scsi
# echo"scsi add-single-device 1 0 0 7" > /proc/scsi/scsi
# echo"scsi add-single-device 1 0 1 7" > /proc/scsi/scsi
验证:
cat/proc/scsi/scsi
2.3更新HDLM配置信息
#/sbin/dlmcfgmgr -r
KAPL10339-IThis operation will change the configuration of HDLM devices. Do you want tocontinue? [y/n]: y
KAPL10341-IThe HDLM device configurations have been changed.
KAPL10302-I/sbin/dlmcfgmgr completed normally.
2.4 确认结果
#/sbin/dlmcfgmgr -v
/dev/sddlmah configured /dev/sdad 0 0 0 7
/dev/sdae 0 0 1 7
/dev/sdaf 1 0 0 7
/dev/sdag 1 0 1 7
验证发现已认到盘
四、 在线认到lun后，重启服务器看路径是否变化
此处以生产历史库的操作为例。新增加的lun id: 2000 设备名称:/dev/sddlmah
测试过程概述：先在新增的lun上创建文件系统并挂载，验证是否能正常读写。
重启服务器，看存储链路是否不变，能否正常mount设备，并正常读写。
1．重启前路径状态
[10:12:13 root(grid)@histdb ~]# /sbin/dlmcfgmgr -v
HDevName Management Device Host Channel Target Lun
/dev/sddlmaa configured /dev/sdc 0 0 0 2
/dev/sdj 0 0 1 2
/dev/sdq 1 0 0 2
/dev/sdx 1 0 1 2
/dev/sddlmab configured /dev/sdo 1 0 0 0
/dev/sdv 1 0 1 0
/dev/sdh 0 0 1 0
/dev/sda 0 0 0 0
/dev/sddlmac configured /dev/sdp 1 0 0 1
/dev/sdw 1 0 1 1
/dev/sdi 0 0 1 1
/dev/sdb 0 0 0 1
/dev/sddlmad configured /dev/sdd 0 0 0 3
/dev/sdk 0 0 1 3
/dev/sdr 1 0 0 3
/dev/sdy 1 0 1 3
/dev/sddlmae configured /dev/sde 0 0 0 4
/dev/sdl 0 0 1 4
/dev/sds 1 0 0 4
/dev/sdz 1 0 1 4
/dev/sddlmaf configured /dev/sdf 0 0 0 5
/dev/sdm 0 0 1 5
/dev/sdt 1 0 0 5
/dev/sdaa 1 0 1 5
/dev/sddlmag configured /dev/sdg 0 0 0 6
/dev/sdn 0 0 1 6
/dev/sdu 1 0 0 6
/dev/sdab 1 0 1 6
/dev/sddlmah configured /dev/sdad 0 0 0 7
/dev/sdae 0 0 1 7
/dev/sdaf 1 0 0 7
/dev/sdag 1 0 1 7
[09:38:15 root(grid)@histdb bin]# ./dlnkmgr view-lu
Product : AMS
SerialNumber : 85003997
LUs : 8
iLU HDevName Device PathIDStatus
0021 sddlmab /dev/sda 000000 Online
/dev/sdh 000006 Online
/dev/sdo 000012 Online
/dev/sdv 000013 Online
0022 sddlmac /dev/sdb 000001 Online
/dev/sdi 000007 Online
/dev/sdp 000014 Online
/dev/sdw 000015 Online
0023 sddlmaa /dev/sdc 000002 Online
/dev/sdj 000008 Online
/dev/sdq 000016 Online
/dev/sdx 000017 Online
0024 sddlmad /dev/sdd 000003 Online
/dev/sdk 000009 Online
/dev/sdr 000018 Online
/dev/sdy 000019 Online
0025 sddlmae /dev/sde 000004 Online
/dev/sdl 000010 Online
/dev/sds 000020 Online
/dev/sdz 000021 Online
0026 sddlmaf /dev/sdf 000005 Online
/dev/sdm 000011 Online
/dev/sdt 000022 Online
/dev/sdaa 000023 Online
0027 sddlmag /dev/sdg 000024 Online
/dev/sdn 000025 Online
/dev/sdu 000026 Online
/dev/sdab 000027 Online
2000 sddlmah /dev/sdad 000028 Online
/dev/sdae 000029 Online
/dev/sdaf 000030 Online
/dev/sdag 000031Online
2．/dev/sddlmah划分区,整个lun划一个分区
# fdisk /dev/sddlmah
Device contains neither a valid DOS partitiontable, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remainin memory only,
until you decide to write them. After that, ofcourse, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4will be corrected by w(rite)
Command (m for help): p
Disk /dev/sddlmah: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1011, default 1): 1
Last cylinder or +size or +sizeM or +sizeK(1-1011, default 1011):
Using default value 1011
Command (m for help): p
Disk /dev/sddlmah: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Device Boot Start End Blocks Id System
/dev/sddlmah1 1 1011 1048376+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
3． 创建文件系统并挂载
mkfs.ext3 /dev/sddlmah1
创建目录： mkdir /data1
在/etc/fstab中加入:
/dev/sddlmah1 /data1 ext3 defaults 1 0
挂载目录:
mount /data1 //此处可以确认/etc/fstab中是否正确
4.测试能否正常读写
cd /data1
vi test.txt
hello~
online test for read write on /dev/sddlmah.
cat test.txt
5.重启
1）关闭数据库
$jdb
$sqlplus /”as sysdba”
alter system archive log current;
alter system checkpoint;
shutdown immediate;
2）关闭ASM
$jgrid
$sqlplus /”as sysdba”
shutdown immediate;
3）关闭listener
$jdb
$lsnrctl status
$lsnrctl stop
4) check
$jdb
$lsnrctl status
$ps -ef|grep ora_
$ps -ef|grep asm
5)重启服务器
#sync;sync;sync;
#reboot
6.服务器重启后启动数据库
查看crs资源是否启动：./crsctl stat res –t
启动listener:
$jdb
$lsnrctl start
启动asm:
$srvctl start asm
启动数据库：
$sqlplus /”as sysdba”
Startup;
7.检查新加lun的盘符
# /sbin/dlmcfgmgr -v
#/opt/DynamicLinkManager/bin/dlnkmgr view -lu
盘的 Host ID (host port number)、 Channel number (bus number)、Target ID、 Lun (host LUnumber)
不变
8.验证重启后能否正常读写
cd /data1
vi testaf.txt
hello~
read write test on /dev/sddlmah after reboot.
9测试完成,umount /data1
#umount /data1
#vi /etc/fstab
删除这行:/dev/sddlmah1 /data1 ext3 defaults 1 0
五、 在线删除lun
经测试redhat 5.4与redhat 6.2环境下的操作步骤相似，这里以redhat 6.2环境下为例展示具体操作步骤。删除lun过程中，数据库运行压力测试。收集每秒IOSTAT信息
1) 停止访问需要删除的LUN
2) 用管理软件从相应的host group中去除lun,即删除链路
3) 删除scsi设备
从hdlm管理中去除/dev/sddlmae
#dlmcfgmgr -o /dev/sddlmae
KAPL10339-I This operation will change theconfiguration of HDLM devices. Do you want to continue? [y/n]: y
KAPL10341-I The HDLM device configurations havebeen changed.
KAPL10302-I /sbin/dlmcfgmgr completed normally.
去除多路径：
#echo "scsi remove-single-device 1 0 04" > /proc/scsi/scsi
#echo "scsi remove-single-device 2 0 04" > /proc/scsi/scsi
#echo "scsi remove-single-device 2 0 14" > /proc/scsi/scsi
#echo "scsi remove-single-device 1 0 14" > /proc/scsi/scsi
说明：以上的数值a,b,c,d 分别代表
a: Host ID (host port number)
b: Channel number (bus number)
c: Target ID
d: Lun (host LU number)
以上值可以在/sbin/dlmcfgmgr -v命令看到
#dlmcfgmgr -i /dev/sddlmae
KAPL10339-I This operation will change theconfiguration of HDLM devices. Do you want to continue? [y/n]: y
KAPL10341-I The HDLM device configurations havebeen changed.
KAPL10302-I /sbin/dlmcfgmgr completed normally.
4）执行HDLM配置定义，使变更生效
#/sbin/dlmcfgmgr -r
KAPL10339-I This operation will change theconfiguration of HDLM devices. Do you want to continue? [y/n]: y
KAPL10302-I /sbin/dlmcfgmgr completed normally.
5）确认操作,被删除的设备前有“-”
[13:38:26 root(grid)@testzhf zhf] #/sbin/dlmcfgmgr-v
HDevName Management Device Host Channel Target Lun
/dev/sddlmaa configured /dev/sdb 1 0 0 0
/dev/sdq 1 0 1 0
/dev/sdg 2 0 0 0
/dev/sdl 2 0 1 0
/dev/sddlmab configured /dev/sdc 1 0 0 1
/dev/sdr 1 0 1 1
/dev/sdh 2 0 0 1
/dev/sdm 2 0 1 1
/dev/sddlmac configured /dev/sdd 1 0 0 2
/dev/sds 1 0 1 2
/dev/sdi 2 0 0 2
/dev/sdn 2 0 1 2
/dev/sddlmad configured /dev/sde 1 0 0 3
/dev/sdt 1 0 1 3
/dev/sdj 2 0 0 3
/dev/sdo 2 0 1 3
/dev/sddlmae configured - 1 0 0 4
- 1 0 1 4
- 2 0 0 4
- 2 0 1 4
KAPL10302-I /sbin/dlmcfgmgr completed normally.
6) 从HDLM中删除路径
针对被删除lun执行以下命令，从HDLM中删除路径
#/sbin/dlmcfgmgr -u /dev/sddlmae
KAPL10339-I This operation will change theconfiguration of HDLM devices. Do you want to continue? [y/n]: y
KAPL10341-I The HDLM device configurations havebeen changed.
KAPL10302-I /sbin/dlmcfgmgr completed normally.
7) 确认lun已被删除
#/sbin/dlmcfgmgr -v
HDevName Management Device Host Channel Target Lun
/dev/sddlmaa configured /dev/sdb 1 0 0 0
/dev/sdq 1 0 1 0
/dev/sdg 2 0 0 0
/dev/sdl 2 0 1 0
/dev/sddlmab configured /dev/sdc 1 0 0 1
/dev/sdr 1 0 1 1
/dev/sdh 2 0 0 1
/dev/sdm 2 0 1 1
/dev/sddlmac configured /dev/sdd 1 0 0 2
/dev/sds 1 0 1 2
/dev/sdi 2 0 0 2
/dev/sdn 2 0 1 2
/dev/sddlmad configured /dev/sde 1 0 0 3
/dev/sdt 1 0 1 3
/dev/sdj 2 0 0 3
/dev/sdo 2 0 1 3
KAPL10302-I /sbin/dlmcfgmgr completed normally.
在执行以上操作时，数据库不停止，仍运行压力测试，收集iostat数据
iostat -xmnztd 1 >/home/oracle/zhf/storage_scan_test/iostat_0108.txt
iostat数据分析结果:iostat 数据连续，IO无明显异常
iostat 结果详见：iostat_0108_在线删lun_redhat6.2.txt
915
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
