md3600i存储服务器连接 iscsi+multipath配置
存储器:dell ms3600i
管理IP:192.168.128.101
RAID0 iSCSI ip1:192.168.10.11
RAID0 iSCSI ip2:192.168.10.12
RAID1 iSCSI ip1:192.168.10.13
RAID1 iSCSI ip2:192.168.10.14
操作系统:centos6.3
/etc/hosts
192.168.10.101 rac1
192.168.10.111 rac1-vip
192.168.20.101 rac1-private
192.168.10.102 rac2
192.168.20.102 rac2-private
192.168.10.112 rac2-vip
#scan ip
192.168.10.100 rac-scan
存储器IP端口无法ping ,找到原因:此端口设置了一个ipv4: [Y]以太网优先级
以致无法ping
******************************************************************************
iscsi 命令:
1.发现iscsi存储: iscsiadm -m discovery -t st -p ISCSI_IP
2.查看iscsi发现记录 iscsiadm -m node
3.删除iscsi发现记录 iscsiadm -m node -o delete -T LUN_NAME -p ISCSI_IP
4.登录iscsi存储 iscsiadm -m node -T LUN_NAME -p ISCSI_IP -l
5.登出iscsi存储 iscsiadm -m node -T LUN_NAME -p ISCSI_IP -u
---------------------------------------------------------------------
安装步骤:
1.检查是否安装iscsi-initiator-utils
[root@localhost ~]# rpm -qa iscsi-initiator-utils
iscsi-initiator-utils-6.2.0.872-41.el6.x86_64
2.添加连接
[root@localhost ~]# iscsiadm -m discovery -t sendtargets -p 192.168.10.11:3260
注:此命令会把所有端口都保存到可以访问的线路列表中。系统在重启时,会从此列表中,各个线路进行测试检查。如果不能访问,会出
错误提示:
Mar 20 13:20:22 rac2 kernel: sd 8:0:0:3: SCSI error: return code = 0x08000002
Mar 20 13:20:22 rac2 kernel: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
Mar 20 13:20:22 rac2 kernel: sdu: Current: sense key: Illegal Request
Mar 20 13:20:22 rac2 kernel: <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
192.168.10.11:3260,1 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
192.168.10.12:3260,1 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
192.168.10.13:3260,2 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
192.168.10.14:3260,2 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
#删除用不到的控制器2的两个接口
[root@rac2 ~]# iscsiadm -m node -o delete -T iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871 -p 192.168.10.13:3260
[root@rac2 ~]# iscsiadm -m node -o delete -T iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871 -p 192.168.10.14:3260
#用控制器1的两个接口的IP登录
[root@localhost ~]# iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871 -p 192.168.10.11:3260 --login
[root@localhost ~]# iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871 -p 192.168.10.12:3260 --login
[root@localhost ~]# iscsiadm -m session #访问路径,这只是说明,这个列表中的路径,已在访问成功中。
tcp: [1] 192.168.10.13:3260,2 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
tcp: [2] 192.168.10.11:3260,1 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
*******************************************************************
Linux系统用device-mapper multipathing driver管理
http://www.aixchina.net/home/space.php?uid=2947&do=blog&id=31276
REDHAT 官方说明
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/DM_Multipath/config_file_defaults.html
1、multipath.conf文件的配置
# This is a basic configuration file with some examples, for device mapper
# multipath.
# For a complete list of the default configuration values, see
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
# For a list of configuration options with descriptions, see
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated
#注:因各版本的配置不太一样,千万不能使用网络上看到的别人的配置文件拷贝过来就用,有些参数在各版本中不支持。
#会出错,但错误信息又不显示。
blacklist {
devnode "^sda"
}
defaults {
user_friendly_names yes
path_grouping_policy multibus multibus #默认的路径组策略
failback immediate#故障恢复的模式
no_path_retry fail#在disable queue之前系统尝试使用失效路径的次数的数值
#加上后面3个参数后,无法初始化多路,可能是此版本不支持
#getuid_callout "/sbin/scsi_id -g -u -s /block/%n" #获得唯一设备号使用的默认程序
#prio_callout "/sbin/acs_prio_alua %d" #获取有限级数值使用的默认程序
#path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法
}
multipaths {
multipath {
wwid 3690b11c0002be80e00000ce3514380db
alias asm-crs01
}
multipath {
wwid 3690b11c0002be80e00000ce5514380f8
alias asm-data01
}
multipath {
wwid 3690b11c0002be80e00000ce75143810c
alias asm-data02
}
multipath {
wwid 3690b11c0002be80e00000ce95143811c
alias asm-recovery01
}
multipath {
wwid 3690b11c0002be7310000099851438137
alias asm-recovery02
}
}
# vi /etc/multipath.conf
#/usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
----------------------------------------------------------------------------------
4、常用的一些multipath工具
清空已有的multipath记录
multipath -F
扫描磁盘,系统中会出现链路聚合之后的dm设备,同时也会在/dev/mapper/、/dev/mpath/目录下生成相应的设备。
multipath -v2
查看multipath拓扑结构
multipath -ll
--------------------------------------------------------------------
[root@rac1-private mapper]# multipath -ll
Mar 06 15:13:50 | DM multipath kernel driver not loaded
Mar 06 15:13:50 | DM multipath kernel driver not loaded
如果模块没有加载成功请使用下列命初始化DM,或重启系统
[root@rac1-private mapper]# modprobe dm-multipath
[root@rac1-private mapper]# modprobe dm-round-robin
[root@rac1-private mapper]# service multipathd start
正在启动守护进程multipathd:
[root@rac1-private mapper]# multipath –v2
[root@rac1-private mapper]# multipath -ll
modprobe dm-multipath
modprobe dm-round-robin
service multipathd start
multipath –v2
初始化完了之后再通过multipath -ll命令查看是否加载成功
[root@rac2 ~]# multipath -ll
mpathr (3690b11c0002be80e000005285136a6f9) dm-2 DELL,MD36xxi
size=2.0T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 2:0:0:1 sdf 8:80 active ready running
|- 5:0:0:1 sdi 8:128 active ready running
|- 3:0:0:1 sdg 8:96 failed faulty running
`- 4:0:0:1 sdh 8:112 failed faulty running
mpathq (3690b11c0002be731000004b25136a708) dm-1 DELL,MD36xxi
size=2.0T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:0:2 sdl 8:176 active ready running
|- 4:0:0:2 sdk 8:160 active ready running
|- 2:0:0:2 sdj 8:144 failed faulty running
`- 5:0:0:2 sdm 8:192 failed faulty running
mpathp (3690b11c0002be731000004b05136a6d9) dm-0 DELL,MD36xxi
size=4.1G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:0:0 sdc 8:32 active ready running
|- 4:0:0:0 sdd 8:48 active ready running
|- 2:0:0:0 sdb 8:16 failed faulty running
`- 5:0:0:0 sde 8:64 failed faulty running
# 看到只有两条线路是正常的。后来拨插了网线,看到两张控制卡,各工作一条网线。
测试多路的速度及负载均衡
dd if=/dev/zero if=/dev/mapper/mpatha bs=1M count=200
iostat
multipath磁盘的基本操作
要对多路径软件生成的磁盘进行操作直接操作/dev/mapper/目录下的磁盘就行.
在对多路径软件生成的磁盘进行分区之前最好运行一下pvcreate命令
(原因:http://book.51cto.com/art/201012/240911.htm):
# pvcreate /dev/mapper/mpath0
# fdisk /dev/mapper/mpath0
[root@rac2 ~]# /etc/init.d/oracleasm createdisk CRS1 /dev/mapper/asmvol1
Marking disk "CRS1" as an ASM disk: [ OK ]
存储器:dell ms3600i
管理IP:192.168.128.101
RAID0 iSCSI ip1:192.168.10.11
RAID0 iSCSI ip2:192.168.10.12
RAID1 iSCSI ip1:192.168.10.13
RAID1 iSCSI ip2:192.168.10.14
操作系统:centos6.3
/etc/hosts
192.168.10.101 rac1
192.168.10.111 rac1-vip
192.168.20.101 rac1-private
192.168.10.102 rac2
192.168.20.102 rac2-private
192.168.10.112 rac2-vip
#scan ip
192.168.10.100 rac-scan
存储器IP端口无法ping ,找到原因:此端口设置了一个ipv4: [Y]以太网优先级
以致无法ping
******************************************************************************
iscsi 命令:
1.发现iscsi存储: iscsiadm -m discovery -t st -p ISCSI_IP
2.查看iscsi发现记录 iscsiadm -m node
3.删除iscsi发现记录 iscsiadm -m node -o delete -T LUN_NAME -p ISCSI_IP
4.登录iscsi存储 iscsiadm -m node -T LUN_NAME -p ISCSI_IP -l
5.登出iscsi存储 iscsiadm -m node -T LUN_NAME -p ISCSI_IP -u
---------------------------------------------------------------------
安装步骤:
1.检查是否安装iscsi-initiator-utils
[root@localhost ~]# rpm -qa iscsi-initiator-utils
iscsi-initiator-utils-6.2.0.872-41.el6.x86_64
2.添加连接
[root@localhost ~]# iscsiadm -m discovery -t sendtargets -p 192.168.10.11:3260
注:此命令会把所有端口都保存到可以访问的线路列表中。系统在重启时,会从此列表中,各个线路进行测试检查。如果不能访问,会出
错误提示:
Mar 20 13:20:22 rac2 kernel: sd 8:0:0:3: SCSI error: return code = 0x08000002
Mar 20 13:20:22 rac2 kernel: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
Mar 20 13:20:22 rac2 kernel: sdu: Current: sense key: Illegal Request
Mar 20 13:20:22 rac2 kernel: <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
192.168.10.11:3260,1 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
192.168.10.12:3260,1 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
192.168.10.13:3260,2 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
192.168.10.14:3260,2 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
#删除用不到的控制器2的两个接口
[root@rac2 ~]# iscsiadm -m node -o delete -T iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871 -p 192.168.10.13:3260
[root@rac2 ~]# iscsiadm -m node -o delete -T iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871 -p 192.168.10.14:3260
#用控制器1的两个接口的IP登录
[root@localhost ~]# iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871 -p 192.168.10.11:3260 --login
[root@localhost ~]# iscsiadm -m node -T iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871 -p 192.168.10.12:3260 --login
[root@localhost ~]# iscsiadm -m session #访问路径,这只是说明,这个列表中的路径,已在访问成功中。
tcp: [1] 192.168.10.13:3260,2 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
tcp: [2] 192.168.10.11:3260,1 iqn.1984-05.com.dell:powervault.md3600i.690b11c0002be7310000000051197871
*******************************************************************
Linux系统用device-mapper multipathing driver管理
http://www.aixchina.net/home/space.php?uid=2947&do=blog&id=31276
REDHAT 官方说明
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/DM_Multipath/config_file_defaults.html
1、multipath.conf文件的配置
# This is a basic configuration file with some examples, for device mapper
# multipath.
# For a complete list of the default configuration values, see
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
# For a list of configuration options with descriptions, see
# /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated
#注:因各版本的配置不太一样,千万不能使用网络上看到的别人的配置文件拷贝过来就用,有些参数在各版本中不支持。
#会出错,但错误信息又不显示。
blacklist {
devnode "^sda"
}
defaults {
user_friendly_names yes
path_grouping_policy multibus multibus #默认的路径组策略
failback immediate#故障恢复的模式
no_path_retry fail#在disable queue之前系统尝试使用失效路径的次数的数值
#加上后面3个参数后,无法初始化多路,可能是此版本不支持
#getuid_callout "/sbin/scsi_id -g -u -s /block/%n" #获得唯一设备号使用的默认程序
#prio_callout "/sbin/acs_prio_alua %d" #获取有限级数值使用的默认程序
#path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法
}
multipaths {
multipath {
wwid 3690b11c0002be80e00000ce3514380db
alias asm-crs01
}
multipath {
wwid 3690b11c0002be80e00000ce5514380f8
alias asm-data01
}
multipath {
wwid 3690b11c0002be80e00000ce75143810c
alias asm-data02
}
multipath {
wwid 3690b11c0002be80e00000ce95143811c
alias asm-recovery01
}
multipath {
wwid 3690b11c0002be7310000099851438137
alias asm-recovery02
}
}
# vi /etc/multipath.conf
#/usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
----------------------------------------------------------------------------------
4、常用的一些multipath工具
清空已有的multipath记录
multipath -F
扫描磁盘,系统中会出现链路聚合之后的dm设备,同时也会在/dev/mapper/、/dev/mpath/目录下生成相应的设备。
multipath -v2
查看multipath拓扑结构
multipath -ll
--------------------------------------------------------------------
[root@rac1-private mapper]# multipath -ll
Mar 06 15:13:50 | DM multipath kernel driver not loaded
Mar 06 15:13:50 | DM multipath kernel driver not loaded
如果模块没有加载成功请使用下列命初始化DM,或重启系统
[root@rac1-private mapper]# modprobe dm-multipath
[root@rac1-private mapper]# modprobe dm-round-robin
[root@rac1-private mapper]# service multipathd start
正在启动守护进程multipathd:
[root@rac1-private mapper]# multipath –v2
[root@rac1-private mapper]# multipath -ll
modprobe dm-multipath
modprobe dm-round-robin
service multipathd start
multipath –v2
初始化完了之后再通过multipath -ll命令查看是否加载成功
[root@rac2 ~]# multipath -ll
mpathr (3690b11c0002be80e000005285136a6f9) dm-2 DELL,MD36xxi
size=2.0T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 2:0:0:1 sdf 8:80 active ready running
|- 5:0:0:1 sdi 8:128 active ready running
|- 3:0:0:1 sdg 8:96 failed faulty running
`- 4:0:0:1 sdh 8:112 failed faulty running
mpathq (3690b11c0002be731000004b25136a708) dm-1 DELL,MD36xxi
size=2.0T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:0:2 sdl 8:176 active ready running
|- 4:0:0:2 sdk 8:160 active ready running
|- 2:0:0:2 sdj 8:144 failed faulty running
`- 5:0:0:2 sdm 8:192 failed faulty running
mpathp (3690b11c0002be731000004b05136a6d9) dm-0 DELL,MD36xxi
size=4.1G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:0:0 sdc 8:32 active ready running
|- 4:0:0:0 sdd 8:48 active ready running
|- 2:0:0:0 sdb 8:16 failed faulty running
`- 5:0:0:0 sde 8:64 failed faulty running
# 看到只有两条线路是正常的。后来拨插了网线,看到两张控制卡,各工作一条网线。
测试多路的速度及负载均衡
dd if=/dev/zero if=/dev/mapper/mpatha bs=1M count=200
iostat
multipath磁盘的基本操作
要对多路径软件生成的磁盘进行操作直接操作/dev/mapper/目录下的磁盘就行.
在对多路径软件生成的磁盘进行分区之前最好运行一下pvcreate命令
(原因:http://book.51cto.com/art/201012/240911.htm):
# pvcreate /dev/mapper/mpath0
# fdisk /dev/mapper/mpath0
[root@rac2 ~]# /etc/init.d/oracleasm createdisk CRS1 /dev/mapper/asmvol1
Marking disk "CRS1" as an ASM disk: [ OK ]