VMware Workstation 共享磁盘和Linux的UDEV使用
因为要测试Oracle RAC的安装,需要使用共享磁盘。 一般情况下做与oracle相关的试验使用Oracle VM VirtualBox会比较好。不过由于个人比较偏爱VMware的虚拟工具,所以尝试用VMware Workstation做测试平台。但是也经历了一些困难。在这里做个总结,便于以后查阅。
实体机环境:Windows Server 2008 R2 Enterprise Edition, 64-bit , Service Pack 1
虚拟机环境:VMware Workstation 11.0.0
测试环境:CentOS 6.5_64bit,OracleLinux-R6-U3-x86_64 (测试过这两个系统)
参考资料:
关于udev与asmlib 以及Multipath的问题,提问前先看这个
http://www.askmaclean.com/archives/question-about-udev-asmlib.html
使用UDEV在Oracle linux 6上安装Oracle 11g RAC(11.2.0.3)
http://blog.csdn.NET/staricqxyz/article/details/8447495
使用udev管理asmdisk执行/sbin/scsi_id不显示UUID
http://blog.itpub.net/19602/viewspace-1172958/
使用udev管理asmdisk常用的命令
http://blog.itpub.Net/19602/viewspace-1173071/
Oracle_rac在vmware上面安装时,udev磁盘没有scsi_id返回值
http://www.68idc.cn/help/jiabenmake/qita/20141224149948.html
Vmware Workstation虚拟机如何设置共享磁盘
http://longyue4585.blog.163.com/blog/static/3817153220126645339463/
How to use udev for Oracle ASM in Oracle Linux 6
http://www.dbform.com/html/2011/1708.html
VMwarelinux虚拟机无法获取uuid
http://www.bitscn.com/os/other/563854.html
Using the Virtual Machine UUID
http://pubs.vmware.com/workstation-9/index.jsp?topic=%2Fcom.vmware.ws.using.doc%2FGUID-533B2C4F-7BD5-41EB-8392-2B9FE687AE50.html
查看VMWARE UUID
http://doorgods.blog.163.com/blog/static/7854785720133100591281/
更改或保留已移动虚拟机的 UUID (2040913)
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2040913
oracle_使用udev绑定磁盘方法
http://blog.sina.com.cn/s/blog_8317516b0101agp5.html
用UDEV服务解决RAC ASM存储设备名
http://blog.csdn.net/evils798/article/details/7616432
linux下获取scsi上的磁盘设备的序列号
http://blog.csdn.net/wwyyxx26/article/details/7739070
Linux下查看UUID方法
http://os.chinaunix.net/a2010/0312/1061/000001061871.shtml
Oracle RAC (11.2.0.4版本) For AIX 6.1安装手册
http://blog.csdn.net/alangmei/article/details/18310381
总结:
VMware 下面使用共享磁盘后有两种方法可以做UDEV。
scsi0:1.present = "TRUE"
scsi0:1.fileName = "D:\VMwareWorkstation\CentOS_Oracle11g\CentOS_Oracle11g_asmdisk01.vmdk"
scsi0:1.redo = ""
scsi0:1.mode = "independent-persistent"
scsi0:1.writeThrough="TRUE"
scsi0:1.sharedBus="virtual"
scsi0:1.deviceType = "disk"
//第二个磁盘的设置
scsi0:2.present = "TRUE"
scsi0:2.fileName = "D:\VMwareWorkstation\CentOS_Oracle11g\CentOS_Oracle11g_asmdisk02.vmdk"
scsi0:2.redo = ""
scsi0:2.mode = "independent-persistent"
scsi0:2.writeThrough="TRUE"
scsi0:2.sharedBus="virtual"
scsi0:2.deviceType = "disk"
//第三个磁盘的设置
scsi0:3.present = "TRUE"
scsi0:3.fileName = "D:\VMwareWorkstation\CentOS_Oracle11g\CentOS_Oracle11g_asmdisk03.vmdk"
scsi0:3.redo = ""
scsi0:3.mode = "independent-persistent"
scsi0:3.writeThrough="TRUE"
scsi0:3.sharedBus="virtual"
scsi0:3.deviceType = "disk"
//这一段没看懂,不过照抄就行
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"
disk.locking = "FALSE"
scsi1.virtualDev = "lsilogic"
disk.enableUUID = "TRUE"
Disk /dev/sda: 64.4 GB, 64424509440 bytes
255 heads, 63 sectors/track, 7832 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 38 305203+ 83 Linux
/dev/sda2 39 7318 58476600 83 Linux
/dev/sda3 7319 7832 4128705 82 Linux swap / Solaris
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sdc: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdd: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdd doesn't contain a valid partition table
[root@localhost ~]# rpm -qa|grep udev
udev-095-14.21.el5
do
echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\", RESULT==\"`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
done
//将文件拷贝到节点2(虚拟系统2)
scp /etc/udev/rules.d/99-oracle-asmdevices.rules node2:/etc/udev/rules.d/
[root@server2 ~]# /sbin/udevcontrol reload_rules
[root@server2 ~]# /sbin/start_udev
Starting udev: [ OK ]
[root@server2 ~]# ls -l /dev/asm-disk*
brw-rw---- 1 grid asmadmin 8, 17 Sep 30 17:49 /dev/asm-diskb
brw-rw---- 1 grid asmadmin 8, 33 Sep 30 17:52 /dev/asm-diskc
brw-rw---- 1 grid asmadmin 8, 49 Sep 30 17:49 /dev/asm-diskd
/dev/sda1: UUID="ec3f5c5d-634c-43d4-b21a-def742702519" TYPE="ext4"
/dev/sda2: UUID="kfSFtZ-7GAT-IgBn-S63P-2ebp-S7nz-KBazer" TYPE="LVM2_member"
/dev/sdd: TYPE="oracleasm"
/dev/sdc: TYPE="oracleasm"
/dev/sdb: TYPE="oracleasm"
/dev/mapper/VolGroup-lv_root: UUID="65bb5403-072c-4138-b480-847bec086b19" TYPE="ext4"
/dev/mapper/VolGroup-lv_swap: UUID="95bc6aed-e5b2-4e5e-8e07-1cc91120e63d" TYPE="swap"
dm-name-VolGroup-lv_root
dm-name-VolGroup-lv_swap
dm-uuid-LVM-CwWwocr3fxAFTFMb9sdfuse8MPYXdVex5rHCNc6WGuaWwFoCubYd6g0RAve6cczA
dm-uuid-LVM-CwWwocr3fxAFTFMb9sdfuse8MPYXdVexOLWGlOt9ZmIXcDUDoeYazVI9vqLS2qvz
scsi-36000c2949053c54c1a238a917b4bcbf2
scsi-36000c294e125c50c4a72627b5e47573a
scsi-36000c2958faf5c7b43bee1e668b0bb2f
scsi-36000c298df1f4f89724d320de9127853
scsi-36000c29cca1ea663013f8c339467da8d
scsi-36000c29cca1ea663013f8c339467da8d-part1
scsi-36000c29cca1ea663013f8c339467da8d-part2
wwn-0x6000c2949053c54c1a238a917b4bcbf2
wwn-0x6000c294e125c50c4a72627b5e47573a
wwn-0x6000c2958faf5c7b43bee1e668b0bb2f
wwn-0x6000c298df1f4f89724d320de9127853
wwn-0x6000c29cca1ea663013f8c339467da8d
wwn-0x6000c29cca1ea663013f8c339467da8d-part1
wwn-0x6000c29cca1ea663013f8c339467da8d-part2
36000c2958faf5c7b43bee1e668b0bb2f
[root@server1 by-id]# /sbin/scsi_id -g /dev/sdb
36000c2958faf5c7b43bee1e668b0bb2f
[root@server1 by-id]# /sbin/scsi_id -g /dev/sdc
36000c298df1f4f89724d320de9127853
[root@server1 by-id]# /sbin/scsi_id -g /dev/sde
36000c2949053c54c1a238a917b4bcbf2
由于使用的是vmware虚拟机,默认磁盘是没有scsi_id返回值的,那不得不考虑使用别的方法
第一种是从VMWARE层面去解决,用文本编辑器修改vmx文件,在vmx文件中任意位置(通常在最后)添加如下行:
disk.EnableUUID = "TRUE"
这种方法有坏处,就是每次修改完磁盘或者网卡信息以后,参数又会变成默认的false,每次都得重新修改,这不符合实际的生产环境要求。
另外一种方法就是,我们的udev采用别的取值方法来绑定磁盘,比如
root@udev ~]# udevinfo -a -p /sys/block/sdc/sdc1
Udevinfo starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.
looking at device '/block/sdc/sdc1':
KERNEL=="sdb1"
SUBSYSTEM=="block"
SYSFS{dev}=="8:33"
SYSFS{partition}=="1"
SYSFS{start}=="63"
SYSFS{size}=="20964762"
我们用上面标红的四个值来取代scsi_id,所以我的规则文件可以写成下面这样
取以下四个值写入文件/etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sdb1"
SUBSYSTEM=="block"
SYSFS{size}=="20964762"
SYSFS{start}=="63"
KERNEL=="sdb1", SUBSYSTEM=="block", SYSFS{size}=="20964762", SYSFS{start}=="63", NAME="asm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sdc1", SUBSYSTEM=="block", SYSFS{size}=="208782", SYSFS{start}=="63", NAME="asm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sdd1", SUBSYSTEM=="block", SYSFS{size}=="20964762", SYSFS{start}=="63", NAME="asm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660"
#end
主题思路就是通过udevinfo这个命令去找其他唯一的值来代替
#重新载入规则,启动udev服务,或者重启服务器即可
[root@server2 ~]# /sbin/udevcontrol reload_rules
[root@server2 ~]# /sbin/start_udev
Starting udev: [ OK ]
[root@server2 ~]# ls -l /dev/asm-disk*
brw-rw---- 1 grid asmadmin 8, 17 Sep 30 17:49 /dev/asm-diskb
brw-rw---- 1 grid asmadmin 8, 33 Sep 30 17:52 /dev/asm-diskc
brw-rw---- 1 grid asmadmin 8, 49 Sep 30 17:49 /dev/asm-diskd
#udev是否启动了我们直接在os上查找和udev相关的进程就可以了
ps -ef|grep udev
scsi_id --whitelisted --replace-whitespace --device=/dev/sdb
scsi_id只支持相对路径,直接使用绝对路径会因为找不到设备属性文件而无返回值。
直接执行/sbin/scsi_id -g -u -s /block/sdb后,返回WWID