Redhat iscsi共享存储+ gfs安装配置

感谢伟哥的文档跟帮助,伟哥的blog://http://ylw6006.blog.51cto.com 此人大牛也... 

环境:

系统:redhat 5.4
服务端:10.0.0.52
节点1:10.0.0.53
节点2:10.0.0.54

Redhat iscsi共享存储安装:

服务器端： 

首先需要安装scsi-target-utils工具包，然后将tgtd服务设置成开机自动启动，然后划出一个LVM做共享盘阵，LVM的名称为mydata

[root@localhost ~]# yum install -y scsi-target-utils.x86_64
[root@localhost ~]# service tgtd start
Starting SCSI target daemon:                               [  OK  ]
[root@localhost ~]# chkconfig tgtd on
[root@localhost ~]# lvs
  LV       VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  Lvmydata vg01 -wi-ao 310.41G                                      
  Lvroot   vg01 -wi-ao  50.00G                                      
  Lvusr    vg01 -wi-ao  50.00G

每次重启tgtd服务的时候，之前使用tgtdadmin绑定的target和logicalunit都会失效，因而写了个脚本用于简化操作

IQN命名规范：iqn.date. reverse.domain.name:optional name，例如：iqn.2011-12-15.com.hsf.data:shareddisk
这里设置允许所有的IP进行进行挂载，取消target绑定的时候需要先取消logicalunit然后取消target   

[root@localhost ~]# vi /etc/init.d/tgtdrules   
#!/bin/sh  
# chkconfig: - 59 85  
# Source function library.  
 
. /etc/rc.d/init.d/functions  
 
start() {  
        echo -e "Starting Tgtdrules Server:\n"  
 
        # Target  
        tgtadm  --lld iscsi --op new --mode target --tid 1 -T iqn.2011-12-15.com.hsf.data:shareddisk  
 
        # Lun  
        tgtadm  --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/vg01/Lvmydata  
 
        # Init  
        #tgtadm  --lld iscsi --op bind --mode target --tid 1 -I 10.0.0.53  
        #tgtadm  --lld iscsi --op bind --mode target --tid 1 -I 10.0.0.54  
        tgtadm  --lld iscsi --op bind --mode target --tid 1 -I ALL  
}  
 
stop() {  
        echo -e "Stopping Tgtdrules Server:\n"  
 
        # Lun  
        tgtadm  --lld iscsi --op delete --mode logicalunit --tid 1 --lun 1   
 
        # Target  
        tgtadm  --lld iscsi --op delete --mode target --tid 1   
 
        # Init  
        #tgtadm  --lld iscsi --op unbind --mode target --tid 1 -I 10.0.0.53  
        #tgtadm  --lld iscsi --op unbind --mode target --tid 1 -I 10.0.0.54  
        tgtadm  --lld iscsi --op unbind --mode target --tid 1 -I ALL  
}  
 
status() {  
        tgtadm --lld iscsi -o show -m target
}  
case "$1" in  
  start)
  start
  ;;  
 
  stop)  
  stop  
  ;;  
 
  status)  
  status  
  ;;  
 
  *)  
  echo {1}quot;Usage: tgtdrules {start|stop|status}"  
  ;;  
 
esac  
exit 0
[root@localhost ~]# chmod  +x /etc/init.d/tgtdrules

服务器端测试： 

[root@localhost ~]# service tgtd status 
tgtd (pid 29246 29245) is running...
[root@localhost ~]# netstat -ntpl| grep :3260
tcp        0      0 0.0.0.0:3260                0.0.0.0:*                   LISTEN      29245/tgtd          
tcp        0      0 :::3260                     :::*                        LISTEN      29245/tgtd
[root@localhost ~]# service tgtdrules start 
[root@localhost ~]# service tgtdrules status 
Target 1: iqn.2011-12-15.com.hsf.data:shareddisk
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: deadbeaf1:0
            SCSI SN: beaf10
            Size: 0 MB
            Online: Yes
            Removable media: No
            Backing store: No backing store
        LUN: 1
            Type: disk
            SCSI ID: deadbeaf1:1
            SCSI SN: beaf11
            Size: 333296 MB
            Online: Yes
            Removable media: No
            Backing store: /dev/vg01/Lvmydata
    Account information:
    ACL information:
        ALL

节点安装:

安装iscsi-initiator-utils软件包，设置iscsi进程开机自动启动 

[root@localhost ~]# yum install -y iscsi-initiator-utils.x86_64
[root@localhost ~]# service iscsi start
iscsid is stopped
Turning off network shutdown. Starting iSCSI daemon:       [  OK  ]
                                                           [  OK  ]
Setting up iSCSI targets: iscsiadm: No records found!
                                                           [  OK  ]
[root@localhost ~]# chkconfig iscsi on
[root@localhost ~]# service iscsi status
iscsid (pid  29527) is running...

自动挂载iscsi脚本内容如下，挂载和卸载之前先发现，探测一次服务器端的共享是否正常 

[root@localhost ~]# vi /etc/init.d/iscsiadmrules    
#!/bin/bash  
# chkconfig: - 20 85  
# Source function library.  
 
. /etc/rc.d/init.d/functions  
 
start() {  
echo -e "Starting Iscsiadmrules Server:\n"  
iscsiadm --mode discovery --type sendtargets --portal 10.0.0.52  
iscsiadm --mode node --targetname iqn.2011-12-15.com.hsf.data:shareddisk --portal 10.0.0.52:3260 --login  
}  
 
stop() {  
echo -e "Stopping Iscsiadmrules Server:\n"  
iscsiadm --mode discovery --type sendtargets --portal 10.0.0.52 
iscsiadm --mode node --targetname iqn.2011-12-15.com.hsf.data:shareddisk --portal 10.0.0.52:3260 --logout  
}  
 
case "$1" in  
 start)  
 start  
 ;;  
 
 stop)  
 stop  
 ;;  
esac  
exit 0  
[root@localhost ~]# chmod  +x /etc/rc.d/init.d/iscsiadmrules

挂载:  

[root@localhost ~]# service iscsiadmrules start 
Starting Iscsiadmrules Server:
10.0.0.52:3260,1 iqn.2011-12-15.com.hsf.data:shareddisk
Logging in to [iface: default, target: iqn.2011-12-15.com.hsf.data:shareddisk, portal: 10.0.0.52,3260]
Login to [iface: default, target: iqn.2011-12-15.com.hsf.data:shareddisk, portal: 10.0.0.52,3260]: successful
[root@localhost ~]# service iscsi status
iscsid (pid  29527) is running...
[root@localhost ~]# fdisk -l
Disk /dev/sda: 449.4 GB, 449495171072 bytes
255 heads, 63 sectors/track, 54648 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          25      200781   83  Linux
/dev/sda2              26        1069     8385930   82  Linux swap / Solaris
/dev/sda3            1070       54648   430373317+  8e  Linux LVM
Disk /dev/sdb: 333.2 GB, 333296173056 bytes
255 heads, 63 sectors/track, 40520 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table

共享存储以/sdb形式加载进来了.

到服务端看一下.

[root@localhost ~]# service tgtdrules status 
Target 1: iqn.2011-12-15.com.hsf.data:shareddisk
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 1
            Initiator: iqn.1994-05.com.redhat:424056be2e26
            Connection: 0
                IP Address: 10.0.0.53
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: deadbeaf1:0
            SCSI SN: beaf10
            Size: 0 MB
            Online: Yes
            Removable media: No
            Backing store: No backing store
        LUN: 1
            Type: disk
            SCSI ID: deadbeaf1:1
            SCSI SN: beaf11
            Size: 333296 MB
            Online: Yes
            Removable media: No
            Backing store: /dev/vg01/Lvmydata
    Account information:
    ACL information:
        ALL

可以看到10.0.0.53已经挂载了.

在另外一台客户端也进行同步的操作再去服务端查看下:

[root@localhost ~]# service tgtdrules status 
Target 1: iqn.2011-12-15.com.hsf.data:shareddisk
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 1
            Initiator: iqn.1994-05.com.redhat:424056be2e26
            Connection: 0
                IP Address: 10.0.0.53
        I_T nexus: 2
            Initiator: iqn.1994-05.com.redhat:ba96a2c6310
            Connection: 0
                IP Address: 10.0.0.54
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: deadbeaf1:0
            SCSI SN: beaf10
            Size: 0 MB
            Online: Yes
            Removable media: No
            Backing store: No backing store
        LUN: 1
            Type: disk
            SCSI ID: deadbeaf1:1
            SCSI SN: beaf11
            Size: 333296 MB
            Online: Yes
            Removable media: No
            Backing store: /dev/vg01/Lvmydata
    Account information:
    ACL information:
        ALL

GFS安装配置

先配置下yum源

[root@localhost ~]# cat /etc/yum.repos.d/base.repo 
[base]
name=RHEL 5.4 Server
baseurl=ftp://10.0.0.23/pub/Server/
gpgcheck=0
[VT]
name=RHEL 5.4 VT
baseurl=ftp://10.0.0.23/pub/VT/
gpgcheck=0
[Cluster]
name=EHRL 5.4 Cluster
baseurl=ftp://10.0.0.23/pub/Cluster
gpgcheck=0
[ClusterStorage]
name=EHRL 5.4 ClusterStorage
baseurl=ftp://10.0.0.23/pub/ClusterStorage
gpgcheck=0
[root@localhost ~]#  yum -y groupinstall "Cluster Storage"  "Clustering"

创建配置文件,启动相关进程，两个节点做同样的配置 

[root@localhost ~]# vi /etc/cluster/cluster.conf
<?xml version="1.0" ?>
<cluster config_version="2" name="file_gfs">
        <fence_daemon post_fail_delay="0" post_join_delay="3"/>
        <clusternodes>
                <clusternode name="10.0.0.53" nodeid="1" votes="1">
                        <fence/>
                </clusternode>
                <clusternode name="10.0.0.54" nodeid="2" votes="1">
                        <fence/>
                </clusternode>
        </clusternodes>
        <cman expected_votes="1" two_node="1"/>
        <fencedevices/>
        <rm>
                <failoverdomains/>
                <resources/>
        </rm>
</cluster>
[root@localhost ~]# lvmconf --enable-cluster
[root@localhost ~]# service rgmanager start
Starting Cluster Service Manager:                          [  OK  ]
[root@localhost ~]# chkconfig rgmanager on
[root@localhost ~]# service ricci start
Starting system message bus:                               [  OK  ]
Starting oddjobd:                                          [  OK  ]
generating SSL certificates...  done
Starting ricci:                                            [  OK  ]
[root@localhost ~]#  chkconfig ricci on
[root@localhost ~]# service cman start
Starting cluster: 
   Loading modules... done
   Mounting configfs... done
   Starting ccsd... done
   Starting cman... done
   Starting daemons... done
   Starting fencing... done
                                                           [  OK  ]
[root@localhost ~]# chkconfig cman on
[root@localhost ~]# service clvmd start
Starting clvmd:                                            [  OK  ]
Activating VGs:   3 logical volume(s) in volume group "vg01" now active
                                                           [  OK  ]
[root@localhost ~]# chkconfig clvmd on
[root@localhost ~]#  clustat 
Cluster Status for file_gfs @ Thu Dec 15 13:53:40 2011
Member Status: Quorate
 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 10.0.0.53                                                           1 Online, Local
 10.0.0.54                                                           2 Offline

在共享存储上划分LVM,在一个节点上操作即可 

[root@localhost ~]# pvcreate /dev/sdb1 
  Physical volume "/dev/sdb1" successfully created
[root@localhost ~]# vgcreate file_gfs /dev/sdb1 
  Clustered volume group "file_gfs" successfully created
[root@localhost ~]# vgdisplay file_gfs
  --- Volume group ---
  VG Name               file_gfs
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  Clustered             yes
  Shared                no
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               310.40 GB
  PE Size               4.00 MB
  Total PE              79462
  Alloc PE / Size       0 / 0   
  Free  PE / Size       79462 / 310.40 GB
  VG UUID               XebmJz-qHFl-Pe5U-gvEL-y1Ml-HbRV-j3GvSd
[root@localhost mnt]# lvcreate -n gfs -l 79462 file_gfs
  Logical volume "gfs" created

如果报以下的错误:

[root@localhost ~]# lvcreate -n gfs1 -l 79462 file_gfs
  Error locking on node 10.0.0.54: Volume group for uuid not found: XebmJzqHFlPe5UgvELy1MlHbRVj3GvSdig4REYsoFSRsUE7byErDdOYcJwu3DTct
  Aborting. Failed to activate new LV to wipe the start of it.

可以先重启下两台节点的clvmd.

[root@localhost ~]# service clvmd restart
Deactivating VG file_gfs:   0 logical volume(s) in volume group "file_gfs" now active
                                                           [  OK  ]
Stopping clvm:                                             [  OK  ]
Starting clvmd:                                            [  OK  ]
Activating VGs:   0 logical volume(s) in volume group "file_gfs" now active
  3 logical volume(s) in volume group "vg01" now active
                                                           [  OK  ]

如果还是不行.就在两台节点上vgs查看下.是不是都能看到这个vg.

[root@localhost mnt]# vgs
  VG       #PV #LV #SN Attr   VSize   VFree
  file_gfs   1   1   0 wz--nc 310.40G    0 
  vg01       1   3   0 wz--n- 410.41G    0 
[root@localhost ~]# service clvmd restart
Deactivating VG file_gfs:   0 logical volume(s) in volume group "file_gfs" now active
                                                           [  OK  ]
Stopping clvm:                                             [  OK  ]
Starting clvmd:                                            [  OK  ]
Activating VGs:   1 logical volume(s) in volume group "file_gfs" now active
  3 logical volume(s) in volume group "vg01" now active
                                                           [  OK  ]

格式化lvm卷 

[root@localhost ~]# gfs_mkfs -h
Usage:
gfs_mkfs [options] <device>
Options:
  -b <bytes>       Filesystem block size
  -D               Enable debugging code
  -h               Print this help, then exit
  -J <MB>          Size of journals
  -j <num>         Number of journals
  -O               Don't ask for confirmation
  -p <name>        Name of the locking protocol
  -q               Don't print anything
  -r <MB>          Resource Group Size
  -s <blocks>      Journal segment size
  -t <name>        Name of the lock table
  -V               Print program version information, then exit
[root@localhost ~]# gfs_mkfs -p lock_dlm  -t file_gfs:gfs -j 2 /dev/file_gfs/gfs   
This will destroy any data on /dev/file_gfs/gfs.
Are you sure you want to proceed? [y/n] y
Device:                    /dev/file_gfs/gfs
Blocksize:                 4096
Filesystem Size:           81297320
Journals:                  2
Resource Groups:           1242
Locking Protocol:          lock_dlm
Lock Table:                file_gfs:gfs
Syncing...
All Done

在两个节点上分别挂载，并测试写入数据 

[root@localhost ~]# mount -t gfs /dev/file_gfs/gfs /data/
[root@localhost data]# dd if=/dev/zero of=aa bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 1.25589 seconds, 835 MB/s