Configuring raw devices (singlepath) for Oracle RHEL5/OEL5 [ID 465001.1]

Configuring raw devices (singlepath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5 [ID 465001.1]
		修改时间 16-JUL-2010     类型 HOWTO     状态 PUBLISHED

In this Document
  Goal
  Solution
     Support for Raw Devices
     A Bit About Udev and Device Name Persistency
     Configuring raw devices (singlepath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5
     Assumptions
     1. Configure SCSI_ID to Return Unique Device Identifiers
     1a. Whitelist all SCSI devices
     1b. List all SCSI (Clusterware) devices
     1c. Obtain Clusterware device unique SCSI identifiers
     2. Configure Udev for Persistent Naming of Clusterware Devices
     2a. Create udev naming rule file /etc/udev/rules.d/55-oracle-naming.rules
     2b. Add udev naming rules to /etc/udev/rules.d/55-oracle-naming.rules file
     2c. Test udev rules
     2d. Restart udev service
     3. Binding Raw Devices
     3a. Check existence of udev raw mapping rule file /etc/udev/rules.d/60-raw.rules
     3b. Add udev raw binding rules to /etc/udev/rules.d/61-oracleraw.rules file
     3c. Create udev raw permissions file /etc/udev/rules.d/65-raw-permissions.rules
     3d. Add udev raw permission rules to /etc/udev/rules.d/65-raw-permissions.rules file
     3e. (Re)test udev rules
     3f. Restart udev service
     4. Verify Files/Devices Created
     5. Complete Oracle RAC Installation
  References

---

Applies to:

Linux OS - Version: 5.0 to 5.0 - Release: OEL5 to OEL5
Linux x86
Linux x86-64
Linux Itanium
Linux Kernel - Version: 5.0 to 5.0

Goal

This article is intended for Oracle on Linux Database and System Administrators, particularly those intending to install (or migrate to) Oracle 10g Release 2 (10.2.0) Real Application Cluster (RAC) on Red Hat/Oracle Enterprise Linux 5 (EL5). The article is solely intended to describe the configuration of raw devices against singlepath devices on EL5 in preparation for RAC Clusterware usage.

Examples were taken from a working system of the following configuration:

• Enterprise Linux 5 (GA) - 2.6.18-8.el5
• Oracle Clusterware 10g Release 2 (10.2.0)
• Shared storage for Clusterware files served via iSCSI via NetApp filer
  

This document differs to  Note.564580.1 that describes how to configure raw devices against multipath devices. This Note describes how to configure raw devices against singlepath devices only.

Solution

Support for Raw Devices

In versions prior to EL5, applications such as Oracle, could access unstructured data on block devices by binding to them via character devices, such as /dev/raw/raw1, using the  raw(8) command. Persistent device assignments could be configured using the  /etc/sysconfig/rawdevices file in conjunction with the  rawdevices service.

Support for raw devices was initially deprecated in the Linux 2.6 kernel (EL5 < U4) in favour of directio ( O_DIRECT) access, however was later undeprecated from EL5 U4 ( initscripts-8.45.30-2).

For details of the deprecation and undeprecation of support for rawio, refer to Linux kernel/version documentation including:

• /usr/share/doc/kernel-doc-2.6.18/Documentation/feature-removal-schedule.txt
• Enterprise Linux 4/5 Release notes

Both the  /etc/sysconfig/rawdevices file (EL4) and  /etc/udev/rules.d/60-raw.rules file (EL5) similarly discuss deprecation of raw.

OCFS2, Oracle's Cluster Filesystem version 2 ( http://oss.oracle.com/projects/ocfs2), is an extent based, POSIX-compliant file system that provides for shared,  O_DIRECT file access. For certified ports and distributions, Oracle extends free support of OCFS2 users with an Oracle database license for use in storing Oracle datafiles, redologs, archivelogs, control files, voting disk (CRS), cluster registry (OCR), etc. along with shared Oracle Home.

A Bit About Udev and Device Name Persistency

Unlike  devlabel in the 2.4 kernel,  udev(8) (the 2.6 kernel device file naming scheme) dynamically creates device file names at boot time. This can, however, give rise to the possibility that device file names may change - a device that may have once been named  /dev/sdd say, may be named /dev/sdf, say, after reboot. Without specific configuration, if  udev is left to dynamically name devices, the potential exists for devices referred to, or inadvertently accessed by, their arbitrary kernel-assigned name (e.g. Oracle Clusterware files; Cluster Registry, Voting disks, etc.) to be unavailable or become corrupt.

Should you still wish to configure Oracle Clusterware files on raw devices, consider yourself cautioned and read on.

Configuring raw devices (singlepath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5

The following procedure outlines the steps necessary to configure udev for both a.) persistent device naming and b.) binding of raw devices in preparation for installing Oracle Clusterware 10g Release 2 (10.2.0). The procedure can also be used to configure raw devices under EL4, in which case minor  udev syntax modification is necessary, as well as cesation of  rawdevices service. Unless otherwise stated, all steps should be performed on each cluster node and as a privileged user. For the sake of brevity, the procedure describes raw device creation for only one OCR and Voting device i.e. no redundancy.

Assumptions

The following procedure assumes the following to have occured for all Clusterware devices:

• devices have been created on supported shared storage
• devices have been partitioned
• devices have been appropriately sized according to Oracle 10g Release 2 (10.2.0) RAC documentation

If intending to implement multipathing of Clusterware devices, refer to  Note.564580.1 that specifically describes configuration of raw devices against multipathed devices.

1. Configure SCSI_ID to Return Unique Device Identifiers

1a. Whitelist all SCSI devices

Before being able to configure udev to explicitly name devices,  scsi_id(8) should first be configured to return their device identifiers.

Modify the  /etc/scsi_id.config file - add or replace the ' option=-b' parameter/value pair (if exists) with ' option=-g', for example:

# cd /etc
# cp scsi_id.config scsi_id.config.orig
# grep -v ^# /etc/scsi_id.config
vendor="ATA",options=-p 0x80
options=-g

1b. List all SCSI (Clusterware) devices

Clusterware devices must be visible and accessible to all cluster nodes.
Typically, cluster node operating systems need to be updated in order to see newly provisioned (or modified) devices on shared storage i.e. use ' /sbin/partprobe ', ' /sbin/sfdisk -r ', etc., or simply reboot. Resolve any issues preventing cluster nodes from correctly seeing or accessing Clusterware devices before proceeding.

Run the  fdisk(8) command to ensure Clusterware devices are visible, for example:

# /sbin/fdisk -l /dev/sdb1 /dev/sde1

Disk /dev/sdb1: 261 MB, 261890048 bytes
9 heads, 56 sectors/track, 1014 cylinders
Units = cylinders of 504 * 512 = 258048 bytes

Disk /dev/sdb1 doesn't contain a valid partition table

Disk /dev/sde1: 52 MB, 52403200 bytes
2 heads, 50 sectors/track, 1023 cylinders
Units = cylinders of 100 * 512 = 51200 bytes

Disk /dev/sde1 doesn't contain a valid partition table

Note: at this point, cluster nodes may refer to the would-be Clusterware devices by different device file names - this is expected.

1c. Obtain Clusterware device unique SCSI identifiers

Run the scsi_id(8) command against Clusterware devices from one cluster node to obtain their unique device identifiers. When running the scsi_idcommand with the '-s' argument, the device path and name passed should be that relative to sysfs directory /sys/ i.e. /block/ when referring to /sys/block/. Record the unique SCSI identifiers of Clusterware devices - these are required later (Step 2b.), for example:

# /sbin/scsi_id -g -s /block/sdb/sdb1
360a98000686f6959684a453333524174

# /sbin/scsi_id -g -s /block/sde/sde1
360a98000686f6959684a453333524179

Note: Irrespective of which cluster node the  scsi_id(8) command is run from, the value returned for a given device should always be the same.

2. Configure Udev for Persistent Naming of Clusterware Devices

The purpose of this step is to provide persistent and meaningful user-defined Clusterware device file names. This step is provided to help ensure correct use of the intended Clusterware devices, that could otherwise be confused if solely relying on kernel-assigned device file names, especially when many raw devices happen to be used. Note, this step is recommended though may be skipped if persistent device naming can be guaranteed via other means, such as vendor-specific storage configuration.

Udev(8) sequentially executes directives (rules) defined in rule files ( /etc/udev/rules.d/*.rules). Rule files are executed in lexical order i.e. starting with the rule file with the lowest alphanumeric value. In other words, the order of rule file execution will be that returned from running command ` /bin/ls /etc/udev/rules.d/*.rules `.

Consider creating custom rule files for any rules you create - this will not only serve to protect default rules/rule files, but also ensure custom rules are not lost in the event of package upgrade.

When creating custom rule files, the following naming format should be observed;  [<number>]-[<name>][.rules]. Any custom rule files, such as those described below, should be created with values greater than 50 (i.e. 51 - 99 ...) to ensure their execution only after all default rules are executed.

Example listing of default EL5 udev rule files:

# ls -l /etc/udev/rules.d/
total 180
-rw-r--r-- 1 root root   515 Jun  7  2007 05-udev-early.rules
-rw-r--r-- 1 root root   920 Jun  5  2007 40-multipath.rules
-rw-r--r-- 1 root root 14344 Jun  7  2007 50-udev.rules
-rw-r--r-- 1 root root   471 Jun  7  2007 51-hotplug.rules
-rw-r--r-- 1 root root 58016 Jun  7  2007 60-libsane.rules
-rw-r--r-- 1 root root   143 Apr 11  2006 60-net.rules
-rw-r--r-- 1 root root  1088 Jun  7  2007 60-pcmcia.rules
-rw-r--r-- 1 root root   452 Jun  7  2007 60-raw.rules
-rw-r--r-- 1 root root   143 Jun  6  2007 60-wacom.rules
-rw-r--r-- 1 root root   114 Jun  5  2007 90-alsa.rules
-rw-r--r-- 1 root root    82 Jun  6  2007 90-hal.rules
-rw-r--r-- 1 root root   107 Jun  7  2007 95-pam-console.rules
-rw-r--r-- 1 root root  2319 Jun  5  2007 bluetooth.rules
-rw-r--r-- 1 root root   590 Jun 13  2007 xen-backend.rules

# rpm -qf /etc/udev/rules.d/60-raw.rules
util-linux-2.13-0.44.el5

# rpm -qf /etc/udev/rules.d/50-udev.rules
udev-095-14.5.el5

Note: the existence of particular rule files depend on the package set that is installed.

2a. Create udev naming rule file /etc/udev/rules.d/55-oracle-naming.rules

Create a custom  udev device naming rule file,  /etc/udev/rules.d/55-oracle-naming.rules say, for example:

# touch /etc/udev/rules.d/55-oracle-naming.rules

2b. Add udev naming rules to /etc/udev/rules.d/55-oracle-naming.rules file

Having earlier identified the unique SCSI identifiers of your Clusterware devices (Step 1c.), add the device matching rules to the  /etc/udev/rules.d/55-oracle-naming.rules file, for example:

# cat /etc/udev/rules.d/55-oracle-naming.rules
# Configure persistent, user-defined Oracle Clusterware device file names
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a453333524174", NAME="ocr1", WNER="root", GROUP="oinstall", MODE="0640"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a453333524179", NAME="vote1", WNER="oracle", GROUP="oinstall", MODE="0640"

For each rule, if all specified keys ( KERNEL, BUS, PROGRAM, RESULT) are matched, the rule is applied and the specified assignments ( NAME, OWNER, GROUP, MODE) are made and applied to the device. If, however, one or more keys are unmatched, the rule is completely ignored and the default (arbitrary) kernel-assigned device file name is assigned to devices.

Note: In the example rules above, Clusterware devices are created with group  oinstall, a loose requirement of running the Cluster verification utility ( cluvfy) when installing the Oracle Clusterware.

2c. Test udev rules

Run the  udevtest(8) command to test the devices to be configured by your  udev rules. The  udevtest program expects the relative  sysfs block device ( (/sys)/block/) to be passed, for example:

# udevtest /block/sdb/sdb1
main: looking at device '/block/sdb/sdb1' from subsystem 'block'
udev_rules_get_name: add symlink 'disk/by-id/scsi-360a98000686f6959684a453333524174-part1'
udev_rules_get_name: add symlink 'disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-part1'
udev_node_mknod: preserve file '/dev/.tmp-8-17', because it has correct dev_t
run_program: '/lib/udev/vol_id --export /dev/.tmp-8-17'
run_program: '/lib/udev/vol_id' returned with status 4
run_program: '/sbin/scsi_id'
run_program: '/sbin/scsi_id' (stdout) '360a98000686f6959684a453333524174'
run_program: '/sbin/scsi_id' returned with status 0
udev_rules_get_name: rule applied, 'sdb1' becomes 'ocr1'
udev_device_event: device '/block/sdb/sdb1' validate currently present symlinks
udev_node_add: creating device node '/dev/ocr1', major = '8', minor = '17', mode = '0640', uid = '0', gid = '500'
udev_node_add: creating symlink '/dev/disk/by-id/scsi-360a98000686f6959684a453333524174-part1' to '../../ocr1'
udev_node_add: creating symlink '/dev/disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-part1' to '../../ocr1'
main: run: 'socket:/org/kernel/udev/monitor'
main: run: '/lib/udev/udev_run_devd'
main: run: 'socket:/org/freedesktop/hal/udev_event'
main: run: '/sbin/pam_console_apply /dev/ocr1 /dev/disk/by-id/scsi-360a98000686f6959684a453333524174-part1 /dev/disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-part1'

In the output above, note that OCR device  /dev/sdb1 is to be 'renamed' as  /dev/ocr1.

2d. Restart udev service

Having verified your  udev naming rules, run the  start_udev(8) command to restart the  udev service. This will apply (implement) the newly defined/modified rules, for example:

# start_udev

# ls -l /dev | grep -e 'ocr1\|vote1'
brw-r----- 1 root   oinstall 8, 17 Oct 29 15:31 ocr1
brw-rw---- 1 oracle oinstall 8, 65 Oct 29 15:31 vote1

3. Binding Raw Devices

As stated earlier, support for raw devices was initially deprecated in EL5 (GA-U3), but later undeprecated from EL5 U4. Notwithstanding, regardless of version, raw devices, including the setting of their permissions can all be configured via  udev as described below.

3a. Check existence of udev raw mapping rule file /etc/udev/rules.d/60-raw.rules

On EL5,  udev raw mapping rule file  /etc/udev/rules.d/60-raw.rules should exist by default as part of the  util-linux package, for example:

# ls /etc/udev/rules.d/60-raw.rules 
/etc/udev/rules.d/60-raw.rules

Create a custom udev raw mapping rule file,  /etc/udev/rules.d/61-oracleraw.rules say, for example:

# touch /etc/udev/rules.d/61-oracleraw.rules

3b. Add udev raw binding rules to /etc/udev/rules.d/61-oracleraw.rules file

Add the  udev raw binding rules to the  /etc/udev/rules.d/61-oracleraw.rules file, for example:

# cat /etc/udev/rules.d/61-oracleraw.rules
# Raw bind to Oracle Clusterware devices
ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a453333524174", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a453333524179", RUN+="/bin/raw /dev/raw/raw2 %N" 

If migrating to EL5 (GA-U3) from an earlier Linux version that relied on /etc/sysconfig/rawdevices file for persistent raw device binding, the following script. can be used as a basis for generating your EL5 udev rules. Note, the script. does not factor device name persistency.

#!/bin/bash
grep -v "^ *#" /etc/sysconfig/rawdevices | grep -v "^$" | while read dev major minor ; do
if [ -z "$minor" ]; then
echo "ACTION==\"add\", KERNEL==\"${major##/dev/}\", RUN+=\"/bin/raw $dev %N\""
else
echo "ACTION==\"add\", ENV{MAJOR}==\"$major\", ENV{MINOR}==\"$minor\", RUN+=\"/bin/raw $dev %M %m\""
fi
done

3c. Create udev raw permissions file /etc/udev/rules.d/65-raw-permissions.rules

The ' RUN+=' directive, as specified in udev raw binding rule file (Step 3b) is always the very last directive to execute in a rule file, therefore a separate file is used to permission resultant raw devices.

Create a udev raw permissions file,  /etc/udev/rules.d/65-raw-permissions.rules say, for example:

# touch /etc/udev/rules.d/65-raw-permissions.rules

3d. Add udev raw permission rules to /etc/udev/rules.d/65-raw-permissions.rules file

To set the ownership of raw devices, add  udev raw permission rules to the   /etc/udev/rules.d/65-raw-permissions.rules file, for example:

# cat /etc/udev/rules.d/65-raw-permissions.rules
# Set permissions of raw bindings to Oracle Clusterware devices
KERNEL=="raw1", WNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", WNER="oracle", GROUP="oinstall", MODE="640"

3e. (Re)test udev rules

Run the  udevtest(8) command again to test the creation and proper permissioning of raw bindings to Clusterware devices, for example:

# udevtest /block/sdb/sdb1
main: looking at device '/block/sdb/sdb1' from subsystem 'block'
udev_rules_get_name: add symlink 'disk/by-id/scsi-360a98000686f6959684a453333524174-part1'
udev_rules_get_name: add symlink 'disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-part1'
udev_node_mknod: preserve file '/dev/.tmp-8-17', because it has correct dev_t
run_program: '/lib/udev/vol_id --export /dev/.tmp-8-17'
run_program: '/lib/udev/vol_id' returned with status 4
run_program: '/sbin/scsi_id'
run_program: '/sbin/scsi_id' (stdout) '360a98000686f6959684a453333524174'
run_program: '/sbin/scsi_id' returned with status 0
udev_rules_get_name: rule applied, 'sdb1' becomes 'ocr1'
udev_device_event: device '/block/sdb/sdb1' validate currently present symlinks
udev_node_add: creating device node '/dev/ocr1', major = '8', minor = '17', mode = '0640', uid = '0', gid = '500'
udev_node_add: creating symlink '/dev/disk/by-id/scsi-360a98000686f6959684a453333524174-part1' to '../../ocr1'
udev_node_add: creating symlink '/dev/disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-part1' to '../../ocr1'
main: run: 'socket:/org/kernel/udev/monitor'
main: run: '/lib/udev/udev_run_devd'
main: run: 'socket:/org/freedesktop/hal/udev_event'
main: run: '/sbin/pam_console_apply /dev/ocr1 /dev/disk/by-id/scsi-360a98000686f6959684a453333524174-part1 /dev/disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-part1'
main: run: '/bin/raw /dev/raw/raw1 /dev/.tmp-8-17'

3f. Restart udev service

Having verified your  udev naming and raw binding rules above, re-run the  start_udev(8) command to restart the  udev service. This will apply (implement) the newly defined/modified rules, for example:

# start_udev

4. Verify Files/Devices Created

Upon completion of the procedure, you should now have correctly permissioned, cluster-consistent and persistent Clusterware device names and raw bindings similar to the following:

# fdisk -l /dev/ocr1 /dev/vote1

Disk /dev/ocr1: 261 MB, 261890048 bytes
9 heads, 56 sectors/track, 1014 cylinders
Units = cylinders of 504 * 512 = 258048 bytes

Disk /dev/ocr1 doesn't contain a valid partition table

Disk /dev/vote1: 52 MB, 52403200 bytes
2 heads, 50 sectors/track, 1023 cylinders
Units = cylinders of 100 * 512 = 51200 bytes

Disk /dev/vote1 doesn't contain a valid partition table

# ls -l /dev | grep -ie 'ocr\|vote'
brw-r----- 1   root dba  8, 17 Oct 29 15:31 ocr1
brw-rw---- 1 oracle dba  8, 65 Oct 29 15:31 vote1

# udevinfo -q all -n /dev/ocr1
P: /block/sdb/sdb1
N: ocr1
S: disk/by-id/scsi-360a98000686f6959684a453333524174-part1
S: disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-part1
E: ID_VENDOR=NETAPP
E: ID_MODEL=LUN
E: ID_REVISION=0.2
E: ID_SERIAL=360a98000686f6959684a453333524174
E: ID_TYPE=disk
E: ID_BUS=scsi
E: ID_PATH=ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085

# udevinfo -q all -n /dev/vote1
P: /block/sde/sde1
N: vote1
S: disk/by-id/scsi-360a98000686f6959684a453333524179-part1
S: disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-part1
E: ID_VENDOR=NETAPP
E: ID_MODEL=LUN
E: ID_REVISION=0.2
E: ID_SERIAL=360a98000686f6959684a453333524179
E: ID_TYPE=disk
E: ID_BUS=scsi
E: ID_PATH=ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085

# raw -qa
/dev/raw/raw1: bound to major 8, minor 17
/dev/raw/raw2: bound to major 8, minor 65

# ls -l /dev/raw/raw*
crw-r----- 1 root   oinstall 162, 11 Oct 30 12:54 /dev/raw/raw1
crw-r----- 1 oracle oinstall 162, 21 Oct 30 14:26 /dev/raw/raw2

5. Complete Oracle RAC Installation

At this stage, assuming the proper creation of raw device files and bindings to Oracle Clusterware devices on all cluster nodes, proceed with installation of Oracle RAC as documented. When prompted for OCR/Voting disk devices, ensure to specify the appropriate raw device names to the Oracle Universal Installer (OUI) rather than block device names.

References

NOTE:357492.1 - Linux 2.6 Kernel Deprecation of Raw Devices
NOTE:564580.1 - Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5
NOTE:605828.1 - Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0) on RHEL5/OEL5
man raw(8)
man scsi_id(8)
man udev(7)
man udevinfo(8)
man udevtest(8)
http://oss.oracle.com/projects/ocfs2/

相关内容 产品 Oracle Linux and Virtualization > Oracle Linux > Operating System > Linux OS 关键字 OCR; MAPPING RULE; OCFS2; RAW DEVICE; CLUSTERWARE 错误 TS-8

返回页首

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/9410636/viewspace-710022/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/9410636/viewspace-710022/