How to Dynamically Add and Remove SCSI Devices on Linux (文档 ID 603868.1)-CSDN博客

In this Document

APPLIES TO:

Linux OS - Version 2.4 to 2.6
Linux x86
Linux x86-64
Linux Kernel - Version: 2.4 to 2.6
***Checked for relevance on 29-Feb-2012***
***Checked for relevance on 03-Sep-2013***
***Checked for relevance on 19-Jun-2015***
***Checked for relevance on 21-Nov-2016***
***Checked for relevance on 15-Jan-2018***

GOAL

This article is primarily intended for Linux System and Storage Administrators.
It aims to list and describe the various, low-level methods available to identify, add and remove SCSI devices from a Linux system.

Examples used are based on Oracle Linux Server 5 GA (2.6.18-8.el5).

SOLUTION

1. SCSI Device Addressing

A four-part addressing scheme is used to define the location of SCSI devices within a system. The attributes include:

<H>ost: Instance of hostadapter to which device is attached
us: SCSI Bus or Channel on the hostadapter
<T>arget: SCSI Id assigned to an individual device
<L>un: Logical unit number on the device

Each attribute, <H> <T> <L>, refers to a part of the device location, similar to how number, street, suburb and state all form an address.

References to device addresses are readily visible from system logs and various command output, though the availability of certain commands or utilities depends on the distribution and operating system version used. The lsscsi(8) utility, for example, is natively available on Oracle Linux 5.

The following denotes the use of several native commands to describe a relatively simple SCSI system.

   # dmesg 
  
 ... 
  
 SCSI subsystem initialized 
  
 libata version 2.21 loaded. 
  
 ata_piix 0000:00:07.1: version 2.12 
  
 scsi0 : ata_piix 
  
 scsi1 : ata_piix 
  
 ata1: PATA max UDMA/33 cmd 0x000101f0 ctl 0x000103f6 bmdma 0x0001ffa0 irq 14 
  
 ata2: PATA max UDMA/33 cmd 0x00010170 ctl 0x00010376 bmdma 0x0001ffa8 irq 15 
  
 ata1.00: ATA-5: IC35L040AVVA07-0, VA2OA51A, max UDMA/100 
  
 ata1.00: 78165360 sectors, multi 8: LBA  
  
 ata1.01: ATA-5: WDC WD400BB-32CLB0, 05.04E05, max UDMA/100 
  
 ata1.01: 78165360 sectors, multi 8: LBA  
  
 ata1.00: configured for UDMA/33 
  
 ata1.01: configured for UDMA/33 
  
 ata2.00: ATA-5: QUANTUM FIREBALLP AS60.0, A1Y.1500, max UDMA/100 
  
 ata2.00: 117266688 sectors, multi 8: LBA  
  
 ata2.01: ATAPI: JLMS DVD-ROM XJ-HD166, DD05, max UDMA/33 
  
 ata2.00: configured for UDMA/33 
  
 ata2.01: configured for UDMA/33 
  
 scsi 0:0:0:0: Direct-Access ATA IC35L040AVVA07-0 VA2O PQ: 0 ANSI: 5 
  
 sd 0:0:0:0: [sda] 78165360 512-byte hardware sectors (40021 MB) 
  
 sd 0:0:0:0: [sda] Write Protect is off 
  
 sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 
  
 sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
  
 sd 0:0:0:0: [sda] 78165360 512-byte hardware sectors (40021 MB) 
  
 sd 0:0:0:0: [sda] Write Protect is off 
  
 sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 
  
 sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
  
 sda: sda1 sda2 sda3 
  
 sd 0:0:0:0: [sda] Attached SCSI disk 
  
 scsi 0:0:1:0: Direct-Access ATA WDC WD400BB-32CL 05.0 PQ: 0 ANSI: 5 
  
 sd 0:0:1:0: [sdb] 78165360 512-byte hardware sectors (40021 MB) 
  
 sd 0:0:1:0: [sdb] Write Protect is off 
  
 sd 0:0:1:0: [sdb] Mode Sense: 00 3a 00 00 
  
 sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
  
 sd 0:0:1:0: [sdb] 78165360 512-byte hardware sectors (40021 MB) 
  
 sd 0:0:1:0: [sdb] Write Protect is off 
  
 sd 0:0:1:0: [sdb] Mode Sense: 00 3a 00 00 
  
 sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
  
 sdb: sdb1 sdb2 
  
 sd 0:0:1:0: [sdb] Attached SCSI disk 
  
 scsi 1:0:0:0: Direct-Access ATA QUANTUM FIREBALL A1Y. PQ: 0 ANSI: 5 
  
 sd 1:0:0:0: [sdc] 117266688 512-byte hardware sectors (60041 MB) 
  
 sd 1:0:0:0: [sdc] Write Protect is off 
  
 sd 1:0:0:0: [sdc] Mode Sense: 00 3a 00 00 
  
 sd 1:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
  
 sd 1:0:0:0: [sdc] 117266688 512-byte hardware sectors (60041 MB) 
  
 sd 1:0:0:0: [sdc] Write Protect is off 
  
 sd 1:0:0:0: [sdc] Mode Sense: 00 3a 00 00 
  
 sd 1:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
  
 sdc: sdc1 
  
 sd 1:0:0:0: [sdc] Attached SCSI disk 
  
 scsi 1:0:1:0: CD-ROM JLMS DVD-ROM XJ-HD166 DD05 PQ: 0 ANSI: 5 
  
 ... 
  
 # lspci | grep -i ide 
  
 00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01) 
  
 # lsscsi 
  
 [0:0:0:0]    disk    ATA      IC35L040AVVA07-0 VA2O  /dev/sda 
  
 [0:0:1:0]    disk    ATA      WDC WD400BB-32CL 05.0  /dev/sdb 
  
 [1:0:0:0]    disk    ATA      QUANTUM FIREBALL A1Y.  /dev/sdc 
  
 [1:0:1:0]    cd/dvd  JLMS     DVD-ROM XJ-HD166 DD05  /dev/sr0 
  
 # cat /proc/scsi/scsi 
  
 Attached devices: 
  
 Host: scsi0 Channel: 00 Id: 00 Lun: 00 
  
 Vendor: ATA Model: IC35L040AVVA07-0 Rev: VA2O 
  
 Type: Direct-Access ANSI SCSI revision: 05 
  
 Host: scsi0 Channel: 00 Id: 01 Lun: 00 
  
 Vendor: ATA Model: WDC WD400BB-32CL Rev: 05.0 
  
 Type: Direct-Access ANSI SCSI revision: 05 
  
 Host: scsi1 Channel: 00 Id: 00 Lun: 00 
  
 Vendor: ATA Model: QUANTUM FIREBALL Rev: A1Y. 
  
 Type: Direct-Access ANSI SCSI revision: 05 
  
 Host: scsi1 Channel: 00 Id: 01 Lun: 00 
  
 Vendor: JLMS Model: DVD-ROM XJ-HD166 Rev: DD05 
  
 Type: CD-ROM ANSI SCSI revision: 05 
  
 # grep host /etc/modprobe.conf 
  
 alias scsi_hostadapter ata_piix 
  
 # ls -ld /sys/class/scsi_host/host*/ 
  
 drwxr-xr-x 2 root root 0 2008-08-06 17:25 /sys/class/scsi_host/host0/ 
  
 drwxr-xr-x 2 root root 0 2008-08-06 17:25 /sys/class/scsi_host/host1/ 
  
 # cat /sys/class/scsi_host/host[0-1]/proc_name 
  
 ata_piix 
  
 ata_piix

Above, it's evident that four devices are attached to two hostadapters, both of type ata_piix i.e. host0: sda sdb, host1: sdc sr0.

Note too, there is sufficient overlap between various command output that device addresses and naming can be easily identified and mapped.

2. SCSI Device Naming

The name assigned to a SCSI device is completely independent to it's SCSI address. In fact, taking the linux 2.6 kernel as an example, the device naming system used, udev(8), dynamically allocates device names upon each boot.

During system initialisation, hardware is scanned and devices are named according to their discovery order. This means, however, that the same device may not always be assigned the same name. This may have implications on some systems, especially those that solely rely on device names such as file /etc/fstab for mounting filesystems.

For such cases, explicit udev(8) configuration may be required to guarantee cross-reboot persistency of device naming. Alternatively, other methods, such as mount-by-label or mount-by-UUID (where supported), may be employed to ensure only intended devices are selected and used, regardless of their arbitrary device name. udev(8), though mentioned here, is not described in any detail - it's relevance, however, becomes more apparent later. Refer to the udev(8) and mount(8) man pages and references below for more information.

3. Scanning, Adding and Removing Devices

From time to time, it may be necessary to add, remove, replace or even reorganise SCSI devices in a system. Broadly speaking, there are two main approaches to how this can be achieved - offline and online.

The method one might choose to add/remove devices usually depends on several factors, such as:

distribution, operating system and version
hardware type, driver version/capability
system availability requirements
storage availability requirements
availability of backup/redundant systems
familiarity of system, I/O stack and storage
acceptance of associated risk

Following are several methods of device addition and removal. Regardless of which method you use, always ensure to perform thorough testing before use within a production environment.

3.1 System reboot

Adding and removing devices whilst a system is shutdown is considered the simplest and safest method. Clean filesystem unmount and ordered shutdown of all of components/layers/drivers involved in the I/O path avoids the potential risks associated with online (dynamic) device removal. On boot, ordered driver load allows newly added/removed devices to be correctly discovered and identified. This method, obviously, necessitates total system unavailability.

3.2 Reload hostadapter driver

The installation (loading) of common host bus adapter (HBA) driver modules initiates a scan (or rescan) of the associated device, resulting in the (re)discovery of any newly added/removed devices. Whilst performed with the system online, to be able to reload the hostadapter module, it must first be unloaded. This, therefore, means that any (all) filesystem, volume or array on devices associated with the driver must first be offlined (in the case of arrays or logical volumes) and/or unmounted. In fact, depending on the complexity of the I/O stack involved, this method is likely to also require the shutdown of related storage services and the unloading of other related or dependent modules e.g. multipathing.

3.3 procfs /proc/scsi/scsi (2.4 kernel)

Linux provides the ability to dynamically interact with the running kernel via the /proc (procfs) filesystem. Dynamically adding or removing devices can be accomplished via the /proc/scsi/scsiinterface i.e.:

To remove a specific device:

   # echo "scsi remove-single-device <H> <B> <T> <L>" > /proc/scsi/scsi 
 

where <H> <T> <L> refers to Host, Bus, Target and Lun

To add a specific device:

   # echo "scsi add-single-device <H> <B> <T> <L>" > /proc/scsi/scsi 
 

where <H> <T> <L> refers to Host, Bus, Target and Lun

This method allows for all filesytems, volumes and arrays, except those immediately involved in specific device removal, to remain online and mounted. However, this method carries a high element of risk. Strong knowledge of the system and all storage layers/devices involved in the I/O path is required. Removal of an incorrect device or premature removal of an intended device may result in volume corruption or, in the presence of Clusterware (e.g. Oracle Cluster File System 2 (OCFS2) or Real Application Clusters (RAC)), could result in node fence (eviction).

Use this method as required when using a 2.4 kernel. The 2.6 kernel provides an improved sysfs interface (/sys, described below) for managing devices.

The following example illustrates the removal, then addition of a SCSI device:

   # lsscsi 
  
 [0:0:0:0]    disk    ATA      IC35L040AVVA07-0 VA2O  /dev/sda 
  
 [0:0:1:0]    disk    ATA      WDC WD400BB-32CL 05.0  /dev/sdb 
  
 [1:0:0:0]    disk    ATA      QUANTUM FIREBALL A1Y.   
  /dev/sdc 
  
 [1:0:1:0]    cd/dvd  JLMS     DVD-ROM XJ-HD166 DD05  /dev/sr0 
  
 # echo "scsi remove-single-device 1 0 0 0" > /proc/scsi/scsi 
  
 # lsscsi 
  
 [0:0:0:0]    disk    ATA      IC35L040AVVA07-0 VA2O  /dev/sda 
  
 [0:0:1:0]    disk    ATA      WDC WD400BB-32CL 05.0  /dev/sdb 
  
 [1:0:1:0]    cd/dvd  JLMS     DVD-ROM XJ-HD166 DD05  /dev/sr0 
  
 # echo "scsi add-single-device 1 0 0 0" > /proc/scsi/scsi 
  
 # lsscsi 
  
 [0:0:0:0]    disk    ATA      IC35L040AVVA07-0 VA2O  /dev/sda 
  
 [0:0:1:0]    disk    ATA      WDC WD400BB-32CL 05.0  /dev/sdb 
  
 [1:0:0:0]    disk    ATA      QUANTUM FIREBALL A1Y.   
  /dev/sdd 
  
 [1:0:1:0]    cd/dvd  JLMS     DVD-ROM XJ-HD166 DD05  /dev/sr0

Note that the removed device, originally known as /dev/sdc, was added back to the system, but as /dev/sdd. This highlights the potential risk associated of solely relying on arbitrary kernel-assigned device file names.

3.4 sysfs /sys/class/scsi_host/ (2.6 kernel)

The 2.6 kernel provides the /sys (sysfs) interface for interacting and managing system devices. In the case of SCSI devices, the /sys/class/scsi_host/ interface can be used to dynamically rescan a hostadapter, as well as add or remove specific devices.

To rescan a hostadapter:

   # echo '- - -' > /sys/class/scsi_host/host<H>/scan 
 

where <H> refers to the hostadapter or instance of hostadapter where multiple adapters of the same type exist on the system

To remove a specific device:

   # echo 1 > /sys/class/scsi_host/host<H>/device/target<H>:<B>:<T>/<H>:<B>:<T>:<L>/delete 
 

where <H> <T> <L> refers to Host, Bus, Target and Lun

To add a specific device:

   # echo "<B> <T> <L>" > /sys/class/scsi_host/host<H>/scan 
 

where <H> <T> <L> refers to Host, Bus, Target and Lun

Like the /proc/scsi/scsi interface, the /sys/class/scsi_host/ interface similarly allows for all filesytems, volumes and arrays, except those immediately involved in specific device removal, to remain online and mounted. Again, this method carries a high level of risk and requires a strong knowledge of the system and all storage layers/devices involved in the I/O path.

Where Fiber Channel (FC) Host Bus Adapters (HBA) are used, separate procfs and/or sysfs entries are created in various locations depending on Operating System and driver type and version used. In such cases, FC HBA driver level re-scan should precede SCSI Bus rescan. For example, for QLogic (qla2xxx):

Enterprise Linux 4:

   # echo "scsi-qlascan" >> /proc/scsi/qla2xxx/<H> 
  
 # echo "scsi add-single-device <H> <B> <T> <L>" > /proc/scsi/scsi

Enterprise Linux 5:

   # echo 1 > /sys/class/fc_host/host<H>/issue_lip 
  
 # echo '- - -' > /sys/class/scsi_host/host<H>/scan

The following example illustrates (non-FC HBA) SCSI hostadapter rescan, then removal and addition of a device:

   # echo '- - -' > /sys/class/scsi_host/host1/scan 
  
 # lsscsi 
  
 [0:0:0:0]    disk    ATA      IC35L040AVVA07-0 VA2O  /dev/sda 
  
 [0:0:1:0]    disk    ATA      WDC WD400BB-32CL 05.0  /dev/sdb 
  
  [1:0:0:0]    disk    ATA      QUANTUM FIREBALL A1Y.  /dev/sdc 
  
 [1:0:1:0]    cd/dvd  JLMS     DVD-ROM XJ-HD166 DD05  /dev/sr0 
  
 [root@toxic ~]# echo 1 > /sys/class/scsi_host/host1/device/target1:0:0/1:0:0:0/delete 
  
 # lsscsi 
  
 [0:0:0:0]    disk    ATA      IC35L040AVVA07-0 VA2O  /dev/sda 
  
 [0:0:1:0]    disk    ATA      WDC WD400BB-32CL 05.0  /dev/sdb 
  
 [1:0:1:0]    cd/dvd  JLMS     DVD-ROM XJ-HD166 DD05  /dev/sr0 
  
 [root@toxic /]# echo '0 0 0' > /sys/class/scsi_host/host1/scan 
  
 # lsscsi 
  
 [0:0:0:0]    disk    ATA      IC35L040AVVA07-0 VA2O  /dev/sda 
  
 [0:0:1:0]    disk    ATA      WDC WD400BB-32CL 05.0  /dev/sdb 
  
  [1:0:0:0]    disk    ATA      QUANTUM FIREBALL A1Y.  /dev/sdd 
  
 [1:0:1:0]    cd/dvd  JLMS     DVD-ROM XJ-HD166 DD05  /dev/sr0

Once again, note that the name of the re-added device (/dev/sdd) differs to it's original name (/dev/sdc).

The entries and names beneath /sys/class/scsi_host/ may vary depending on the operating system, kernel version and type of devices in use on the system. For example, if using iSCSI devices, additional directory entry session exists:

   # lsscsi 
  
 [0:0:0:0]    disk    ATA      WDC WD1600JS-75N 10.0  /dev/sda 
  
 [2:0:0:0]    disk    IET      VIRTUAL-DISK     0     /dev/sdb 
  
 [3:0:0:0]    disk    IET      VIRTUAL-DISK     0     /dev/sdc 
  
 [7:0:0:0]    disk    IET      VIRTUAL-DISK     0     /dev/sdd 
  
 [10:0:0:0]   disk    IET      VIRTUAL-DISK     0     /dev/sde 
  
 # ls -ld /sys/class/scsi_host/host*/device/session*/target*/[0-9]* 
  
 drwxr-xr-x 3 root root 0 Aug 7 22:35   /sys/class/scsi_host/host10/device/session8/target10:0:0/10:0:0:0 
  
 drwxr-xr-x 3 root root 0 Aug 7 22:35   /sys/class/scsi_host/host2/device/session0/target2:0:0/2:0:0:0 
  
 drwxr-xr-x 3 root root 0 Aug 7 22:35   /sys/class/scsi_host/host3/device/session1/target3:0:0/3:0:0:0 
  
 drwxr-xr-x 3 root root 0 Aug 7 22:35   /sys/class/scsi_host/host7/device/session5/target7:0:0/7:0:0:0

3.5 Other Native and Vendor Supplied Scripts

Some hostadapter/driver vendors supply their own scripts that can be used to scan, add and remove devices. These scripts may be used in favour of manual device interaction via native kernel interfaces. The scripts should, however, be thoroughly tested before their reliance in Production. Naturally, where provided by a third-party vendor, any issues arising from their use should be referred to the originating supplier.

Examples of vendor supplied scripts include:

scsi-rescan/rescan-scsi-bus.sh (Linux sg3_utils package)
ql-dynamic-tgt-lun-disc.sh (QLogic)
qlun_disc.sh (QLogic)
lun_scan (Emulex)
hp_rescan (HP)
proprietary and custom others

Contact your hostadapter vendor for latest available and recommended scripts.

4. Removing a Multipath Device by Example

Let's look at a more complex, yet typical scenario involving multipathing (device-mapper-multipath).
In this instance, explicitly white-listed iSCSI-served partitioned target devices (LUNs) are multipathed on the initiator using user-defined names; ocr1, voting1, etc.

   # dmsetup ls | sort 
  
 ocr1    (253, 5) 
  
 ocr1p1  (253, 9) 
  
 ocr2    (253, 6) 
  
 ocr2p1  (253, 10) 
  
 ocr3    (253, 7) 
  
 ocr3p1  (253, 11) 
  
 voting1 (253, 0) 
  
 voting1p1       (253, 3) 
  
 voting2 (253, 1) 
  
 voting2p1       (253, 4) 
  
 voting3 (253, 2) 
  
 voting3p1       (253, 8) 
  
 # multipath -ll 
  
 ocr3 (149455400000000000000000001000000ca0200000d000000) dm-7 IET,VIRTUAL-DISK 
  
 [size=980M][features=0][hwhandler=0] 
  
 \_ round-robin 0 [prio=0][active] 
  
  \_ 1:0:0:10 sdn 8:208 [active][ready] 
  
 \_ round-robin 0 [prio=0][enabled] 
  
  \_ 1:0:0:11 sdo 8:224 [active][ready] 
  
 ocr2 (149455400000000000000000001000000ed0200000d000000) dm-6 IET,VIRTUAL-DISK 
  
 [size=980M][features=0][hwhandler=0] 
  
 \_ round-robin 0 [prio=0][active] 
  
  \_ 1:0:0:8 sdl 8:176 [active][ready] 
  
 \_ round-robin 0 [prio=0][enabled] 
  
  \_ 1:0:0:9 sdm 8:192 [active][ready] 
  
 ocr1 (149455400000000000000000001000000e80200000d000000) dm-5 IET,VIRTUAL-DISK 
  
 [size=980M][features=0][hwhandler=0] 
  
 \_ round-robin 0 [prio=0][active] 
  
  \_ 1:0:0:6 sdj 8:144 [active][ready] 
  
 \_ round-robin 0 [prio=0][enabled] 
  
  \_ 1:0:0:7 sdk 8:160 [active][ready] 
  
 voting3 (149455400000000000000000001000000e30200000d000000) dm-2 IET,VIRTUAL-DISK 
  
 [size=965M][features=0][hwhandler=0] 
  
 \_ round-robin 0 [prio=0][active] 
  
  \_ 1:0:0:4 sdh 8:112 [active][ready] 
  
 \_ round-robin 0 [prio=0][enabled] 
  
  \_ 1:0:0:5 sdi 8:128 [active][ready] 
  
 voting2 (149455400000000000000000001000000de0200000d000000) dm-1 IET,VIRTUAL-DISK 
  
 [size=965M][features=0][hwhandler=0] 
  
 \_ round-robin 0 [prio=0][active] 
  
  \_ 1:0:0:2 sdf 8:80 [active][ready] 
  
 \_ round-robin 0 [prio=0][enabled] 
  
  \_ 1:0:0:3 sdg 8:96 [active][ready] 
  
 voting1 (149455400000000000000000001000000d90200000d000000) dm-0 IET,VIRTUAL-DISK 
  
 [size=965M][features=0][hwhandler=0] 
  
 \_ round-robin 0 [prio=0][active] 
  
  \_ 1:0:0:1 sde 8:64 [active][ready] 
  
 \_ round-robin 0 [prio=0][enabled] 
  
  \_ 1:0:0:0 sdd 8:48 [active][ready] 
  
 # cat /proc/partitions 
  
 major minor #blocks name 
  
    8     0   6291456 sda 
  
    8     1   5735173 sda1 
  
    8     2    554242 sda2 
  
    8    16   2097152 sdb 
  
    8    17   2096451 sdb1 
  
    8    32   2097152 sdc 
  
    8    33   2096451 sdc1 
  
    8    48    987966 sdd 
  
    8    49    987681 sdd1 
  
    8    64    987966 sde 
  
    8    65    987681 sde1 
  
    8    80    987966 sdf 
  
    8    81    987681 sdf1 
  
    8    96    987966 sdg 
  
    8    97    987681 sdg1 
  
    8   112    987966 sdh 
  
    8   113    987681 sdh1 
  
    8   128    987966 sdi 
  
    8   129    987681 sdi1 
  
    8   144   1004031 sdj 
  
    8   145   1003873 sdj1 
  
    8   160   1004031 sdk 
  
    8   161   1003873 sdk1 
  
    8   176   1004031 sdl 
  
    8   177   1003873 sdl1 
  
    8   192   1004031 sdm 
  
    8   193   1003873 sdm1 
  
    8   208   1004031 sdn 
  
    8   209   1003873 sdn1 
  
    8   224   1004031 sdo 
  
    8   225   1003873 sdo1 
  
  253     0    987966 dm-0 
  
  253     1    987966 dm-1 
  
  253     2    987966 dm-2 
  
  253     3    987681 dm-3 
  
  253     4    987681 dm-4 
  
  253     5   1004031 dm-5 
  
  253     6   1004031 dm-6 
  
  253     7   1004031 dm-7 
  
  253     8    987681 dm-8 
  
  253     9   1003873 dm-9 
  
  253    10   1003873 dm-10 
  
  253    11   1003873 dm-11

Below, unused multipath device ocr3 is dynamically removed from the system, as are its associated underlying devices. The devices were verified to no longer be in use or required by any program or service before their removal.

Note, whether the /sbin/multipath or /sbin/dmsetup command is used, the result is the same. However, when using the dmsetup command to remove partitioned multipathed devices, the multipath aliases for all of its partitions must first be removed before the multipath alias of the device itself can be removed, otherwise the command fails with a 'device or resource busy error' message. When using the multipath command to remove partitioned devices, removal of the multipath alias of the device will automatically remove all multipath aliases for all of its partitions.

   # multipath -ll ocr3 
  
  ocr3 (149455400000000000000000001000000ca0200000d000000)  
  dm-7 IET,VIRTUAL-DISK 
  
 [size=980M][features=0][hwhandler=0] 
  
 \_ round-robin 0 [prio=0][active] 
  
  \_ 1:0:0:10  
  sdn 8:208 [active][ready] 
  
 \_ round-robin 0 [prio=0][enabled] 
  
  \_ 1:0:0:11  
  sdo 8:224 [active][ready] 
  
 # dmsetup ls | sort 
  
 ocr1    (253, 5) 
  
 ocr1p1  (253, 9) 
  
 ocr2    (253, 6) 
  
 ocr2p1  (253, 10) 
  
  ocr3    (253, 7)
 ocr3p1  (253, 11) 
  
 voting1 (253, 0) 
  
 voting1p1       (253, 3) 
  
 voting2 (253, 1) 
  
 voting2p1       (253, 4) 
  
 voting3 (253, 2) 
  
 voting3p1       (253, 8) 
  
 # cat /proc/partitions | grep -e 'sdo\|sdn\|dm-7\|dm-11' 
  
    8   208    1004031 sdn 
  
    8   209    1003873 sdn1 
  
    8   224    1004031 sdo 
  
    8   225    1003873 sdo1 
  
  253     7    1004031 dm-7 
  
  253    11    1003873 dm-11 
  
 # multipath -f ocr3 
  
  OR 
  
 # dmsetup remove ocr3p1 
  
 # dmsetup remove ocr3 
  
 # multipath -ll ocr3 
  
 # 
  
 # dmsetup ls | sort 
  
 ocr1    (253, 5) 
  
 ocr1p1  (253, 9) 
  
 ocr2    (253, 6) 
  
 ocr2p1  (253, 10) 
  
 voting1 (253, 0) 
  
 voting1p1       (253, 3) 
  
 voting2 (253, 1) 
  
 voting2p1       (253, 4) 
  
 voting3 (253, 2) 
  
 voting3p1       (253, 8) 
  
 # cat /proc/partitions | grep -e 'sdo\|sdn\|dm-7\|dm-11' 
  
    8   208    1004031 sdn 
  
    8   209    1003873 sdn1 
  
    8   224    1004031 sdo 
  
    8   225    1003873 sdo1 
  
 # echo 1 > /sys/class/scsi_host/host1/device/session0/target1:0:0/1:0:0:10/delete 
  
 # echo 1 > /sys/class/scsi_host/host1/device/session0/target1:0:0/1:0:0:11/delete 
  
 # cat /proc/partitions | grep -e 'sdo\|sdn\|dm-7\|dm-11' 
  
 #

Note that the device-mapper devices previously associated with multipath device ocr3 (/dev/dm-7, /dev/dm-11) are removed, however the underlying device paths (/dev/sdn, /dev/sdo) remain until explicitly removed from the system.

REFERENCES

NOTE:357472.1 - Configuring device-mapper for CRS/ASM
NOTE:414897.1 - Linux: How To Setup UDEV Rules For RAC OCR And Voting Devices On SLES10, RHEL5, OL5
NOTE:456239.1 - Understanding Device-mapper in Linux 2.6 Kernel
NOTE:465001.1 - Configuring raw devices (singlepath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OL5
NOTE:564580.1 - Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OL5
NOTE:605828.1 - Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0, 11.2.0) on OL5/RHEL5