linux rdac管理,Linux® RDAC Multipath Drivers

這篇文章適用於 IBM Storage DS3000, DS4000 以及 DS5000 (感謝我同事 David 提供的文章)

在 Linux 底下有一個服務為 multipathd ,開啟後就可以讓 multipath 的功能啟動. 設定方式請參考 http://benjr.tw/159

不過這 multipath 的裝置非原始的 sdx 而是變成 /dev/mapper/mpathxx 裝置.雖然裝置就如同一般的硬碟裝置,並且具備 MPIO (multipath I/O: 主要是為了確保連線不會因為單一的網路斷線而失去連線) 的功能.但感覺上???

如果你的儲存裝置為 IBM System Storage DS3000, DS4000 或是 DS5000 就可以使用 IBM 所提供的 multipath 驅動程式 RDAC (Redundant Disk Array Controller),這時就不需要透過 RHEL 預設的 Multipath 功能,但不是所有的 HBA 都支援 RDAC.

最簡單的 Multipath I/O 連接方式如下,不管是 Server 端還是 Storage 端都一定是 Redundant (備源) 的接法,這是最基礎的接法,真實應用可能會更為複雜.

目前測試環境為

Server + NIC Port x2

OS : RHEL 6.3

Storage : IBM DS3524 (iSCSI Storage)

不過不是所有的 HBA 都支援 RDAC,在 RDAC 的 Readme.txt 支援列表如下

FCoE CNA

PCI-E :QLE8042

PCI-E :LP21000, LP21002

PCI-E :BR-1020

iSCSI Qlogic

PCI-X :Qlogic 4050C, Qlogic 4052C

PCI-E :Qlogic 4060C, Qlogic 4062C

Fibre Channel Qlogic

PCI-E : QLE2560, QLE2562 (8Gb)

PCI-E : QLE2460, QLE2462 (4Gb)

PCI-E : QLE236x (2Gb)

PCI-X : QLA2460, QLA2462 (4Gb)

PCI-X : QLA2310F, QLA2340, QLA2342 (2Gb)

PCI-X : QLA2000 (1Gb)

SAS LSI Logic

PCI-X : LSISAS3800-X, LSI SAS3442X-R, LSI SAS3442E-R

PCI-X : 3801X (SAS1 3G)

PCI-E : 3801E (SAS1 3G)

PCI-E : 92XX-XXX (SAS2 6G)

Fibre Channel LSI Logic:

PCI-X : LSI7204XP-LC, LSI7404XP-LC, LSI7104XP, (4Gb)

PCI-X : LSI7202XP, LSI7402XP (2Gb)

LSI7104EP, LSI7204EP

PCI and PCI-X: LSIFC929X (2Gb)

Fibre Channel IBM Branded Emulex

PCI-X : IBM FC 5758, 5759 (4Gb)

PCI-X : IBM FC 5716, (2Gb)

PCI : IBM FC 6228 (1Gb) , 6239 (2Gb)

Fibre Channel Emulex

PCI-E : LPe12000, LPe12002 (8Gb)

PCI-E : LPe11000, LPe11002, LPe1150 (4Gb)

PCI-E : LP10000ExDC (2Gb)

PCI-X : LP11000, LP11000DC, LP11002, LP1150 (4Gb)

PCI-X : LP10000, LP10000DC, LP1050, LP9802, LP9802DC, LP982 (2Gb)

PCI : LP9002DC, LP952L, LP9002L, LP9000 (2Gb)

Fibre Channel Brocade

PCI-E : Brocade 815, Brocade 825 (8Gb)

Fibre Channel ATTO

PCI-E : Celerity FC-84EN, Celerity FC-82EN, Celerity FC-81EN (8Gb)

Infiniband HCA Mellanox

PCI-E : InfiniHost III Lx MHGS18-XTC, InfiniHost Ex MHGA28-XTC, ConnectX MHGH28-XTC, ConnectX MHQH29

Infiniband HCA Voltaire

PCI-E : HCA 400EX-D (MHGA28-XTC), HCA 410EX-D (MHGS18-XTC), HCA 500EX-D (MHGH28-XTC), 600Ex2

Infiniband HCA QLogic

PCI-E : 7104-HCA-128LPX-DDR (MHGA28-1TC), 7104-HCA-LPX1P-DDR (MHGS18-XTC), 7104-HCA-LPX2P-DDR (MHGA28-XTC)

步驟:

接 2 條網路線到 DS3524的 iscsi port,並先設定好DS3524, 設定方式雷同 IBM DS3400 主要是把 Lun 指派給 iscsi initiator

LinuxR RDAC Multipath Drivers 驅動程式安裝方式很簡單, 因為需要 make 所以要安裝 kernel-devel kernel-header make,接著

下載 RDAC,解開 source packages,並手動建立驅動程式,使用新建立好的 initrd 更改 menu.lst 選單,重新開機.

[root@benjr Desktop]# yum install kernel-devel kernel-header

[root@benjr Desktop]# tar zxvf rdac-LINUX-09.03.0C05.0642-source.tar.gz

[root@benjr Desktop]# cd Linuxrdac-09.03.0C05.0642

[root@benjr Desktop]# make clean

[root@benjr Desktop]# make

[root@benjr Desktop]# make install

在 GRUB 選單中,從新建立的選項開機.

IP 設完後,用 #iscsiadm command,將建立2個 iscsi 的連線,使用 #fdisk -l 只會看到1個 Lun.

建立第一條connection

[root@benjr Desktop]# iscsiadm -m node -T iqn.xxxxx -p 192.168.132.101 -l

建立第二條 connection.

[root@benjr Desktop]# iscsiadm -m node -T iqn.xxxxx -p 192.168.133.101 -l

看到 Lun 之後,把它格式化+掛載,試著存取它.

移除其中一條網路線,會停頓一下,之後就正常.如果你拔除網路線之後,跳出 Input/Output error 的訊息,代表失敗.

系統的一些訊息可證明 MPIO 有功能的.

[root@benjr Desktop]# dmesg

igb: eth0 NIC Link is Down

connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4303063833, last ping 4303068833, now 4303073833

connection1:0: detected conn error (1011)

session1: session recovery timed out after 144 secs

一邊存取同時拔線會頓一下,需要約 2分多鐘 的時間才會走另一條路徑

94 [RAIDarray.mpp]DS3524:0:0:0 Selection Retry count exhausted

7 [RAIDarray.mpp]DS3524:0:0 Path Failed

495 [RAIDarray.mpp]DS3524:0:0:0 Cmnd failed-retry on a new path. vcmnd SN 1228 pdev H7:C0:T0:L0 0×00/0×00/0×00 0x000f0000 mpp_status:6

我們還可以觀察一下在 /proc/mpp/DS3524 下

[root@benjr DS3524]# cat virtualLun0

Linux MPP driver. Version:09.03.0C05.0638 Build:Tue Apr 17 15:31:54 CDT 2012

Lun WWN:60080e50001b70880000073e500df5a0

Virtual Scsi Address: host_no:8 channel:0 target:0 Lun:0

Queue Depth = 30

I/O Statistics:

Number of IOs:489

Longest trip of all I/Os:148

Shortest trip of all I/Os:0

Number of occurrences of path failover events:2

Number of occurrences of controller failover events:0

The longest Controller Failover Time:0

The shortest Controller Failover Time:0

total size:491

Linux MPP driver. Version:09.03.0C05.0638 Build:Tue Apr 17 15:31:54 CDT 2012

Lun WWN:60080e50001b70880000073e500df5a0

Physical HBA driver: iscsi_tcp

Device Scsi Address: host_no:9 channel:0 target:0 Lun:0

Queue Depth = 32

I/O Statistics:

Number of IOs:101

Longest trip of all I/Os:0

Shortest trip of all I/Os:0

Number of occurences of IO failed events:18

Device state: [6] OPTIMAL

Device state: [7] FAILED

Device state: [8] FAILED_NEED_CHECK

Device state: [9] FAILED_CHECKING

Device state: [0] FAILED

Device state: [1] FAILED_NEED_CHECK

Device state: [2] FAILED_CHECKING

Device state: [3] FAILED

Device state: [4] FAILED_NEED_CHECK

Device state: [5] FAILED_CHECKING

Path state:[0] OPTIMAL

Path state:[1] OPTIMAL

Path state:[2] OPTIMAL_NEED_CHECK

Path state:[3] OPTIMAL_CHECKING

Path state:[4] OPTIMAL

Path state:[5] OPTIMAL_NEED_CHECK

Path state:[6] OPTIMAL_CHECKING

Path state:[7] OPTIMAL

Path state:[8] OPTIMAL_NEED_CHECK

Path state:[9] OPTIMAL_CHECKING

Controller Failed? 0

Outstanding IOs on this device:

total size:1121

把線路插回去會看到 Recovery 的訊息

Jul 27 17:01:58 localhost avahi-daemon[2652]: Registering new address record for 192.168.132.102 on eth0.IPv4.

Jul 27 17:01:58 localhost iscsid: connect to 192.168.132.101:3260 failed (Connection refused)

Jul 27 17:02:02 localhost iscsid: connection1:0 is operational after recovery (1358 attempts)

沒有解決問題,試試搜尋本站其他內容