oracle asm service can not find disks which are configured available when restart server

Environment:

OS: Linux ocloud 2.6.18-164.el5 #1 SMP Thu Sep 3 04:15:13 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

ASM_INSTANCE: 

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for Linux: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

Declare:

This is a single db server with grid infrastructure. 

After once restarted the pc server, the asm instance started successfully but could not find the disks.

[root@ocloud ~]# crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    OFFLINE
ora....ER.lsnr ora....er.type ONLINE    ONLINE    ocloud
ora.asm        ora.asm.type   ONLINE    ONLINE    ocloud
ora.cssd       ora.cssd.type  ONLINE    ONLINE    ocloud
ora.diskmon    ora....on.type OFFLINE   OFFLINE
ora.evmd       ora.evm.type   ONLINE    ONLINE    ocloud
ora.key.db     ora....se.type ONLINE    OFFLINE
ora.ons        ora.ons.type   OFFLINE   OFFLINE


[root@ocloud ~]# service oracleasm listdisks
[root@ocloud ~]# service oracleasm scandisks
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@ocloud ~]# service oracleasm listdisks
[root@ocloud ~]#


/var/log/oracleasm
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
oracleasm-read-label: Unable to open device "/dev/dm-0": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-0": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-0": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-0": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-1": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-1": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-1": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-1": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-2": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-2": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-2": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-2": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-3": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-3": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-3": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-3": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-4": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-4": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-4": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-4": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-5": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-5": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-5": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-5": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-6": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-6": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-6": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-6": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-7": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-7": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-7": No such file or directory
oracleasm-read-label: Unable to open device "/dev/dm-7": No such file or directory

Check the /sys/block  as follows:

[root@ocloud ~]# ls /sys/block
dm-0  dm-2  dm-4  dm-6  fd0  md0   ram1   ram11  ram13  ram15  ram3  ram5  ram7  ram9  sdb
dm-1  dm-3  dm-5  dm-7  hdc  ram0  ram10  ram12  ram14  ram2   ram4  ram6  ram8  sda




[root@ocloud ~]# udevinfo -a -p /sys/block/dm-0


Udevinfo starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.


  looking at device '/block/dm-0':
    KERNEL=="dm-0"
    SUBSYSTEM=="block"
    SYSFS{stat}=="   93931        0  2582466   706162   206406        0  1644040   410876        0   278463  1117329"
    SYSFS{size}=="189005824"
    SYSFS{removable}=="0"
    SYSFS{range}=="1"
    SYSFS{dev}=="253:0”

Then I scanned the /etc/udev/rules.d/90-dm.rules

eg:

#KERNEL=="dm-[0-9]*", ACTION=="add", OPTIONS+="ignore_device"

And I edited the file as follow:

#KERNEL=="dm-[0-9]*", ACTION=="add", OPTIONS+="ignore_device"
KERNEL=="dm-0", SUBSYSTEM=="block", SYSFS{size}=="189005824", SYSFS{dev}=="253:0"
KERNEL=="dm-1", SUBSYSTEM=="block", SYSFS{size}=="20447232", SYSFS{dev}=="253:1"
KERNEL=="dm-2", SUBSYSTEM=="block", SYSFS{size}=="4194304", SYSFS{dev}=="253:2"
KERNEL=="dm-3", SUBSYSTEM=="block", SYSFS{size}=="4194304", SYSFS{dev}=="253:3"
KERNEL=="dm-4", SUBSYSTEM=="block", SYSFS{size}=="10485760", SYSFS{dev}=="253:4"
KERNEL=="dm-5", SUBSYSTEM=="block", SYSFS{size}=="10485760", SYSFS{dev}=="253:5"
KERNEL=="dm-6", SUBSYSTEM=="block", SYSFS{size}=="10485760", SYSFS{dev}=="253:6"
KERNEL=="dm-7", SUBSYSTEM=="block", SYSFS{size}=="10485760", SYSFS{dev}=="253:7”

Then do that

[root@ocloud rules.d]# start_udev 


[root@ocloud rules.d]# service oracleasm scandisks
Scanning the system for Oracle ASMLib disks:               [  OK  ]

[root@ocloud rules.d]# service oracleasm listdisks
VOL1
VOL2
VOL3
VOL4
VOL5
VOL6

[grid@ocloud ~]$ crsctl stop resource -all
CRS-2500: Cannot stop resource 'ora.diskmon' as it is not running
CRS-2500: Cannot stop resource 'ora.ons' as it is not running
CRS-2673: Attempting to stop 'ora.evmd' on 'ocloud'
CRS-2673: Attempting to stop 'ora.asm' on 'ocloud'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'ocloud'
CRS-2677: Stop of 'ora.evmd' on 'ocloud' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'ocloud' succeeded
CRS-2677: Stop of 'ora.asm' on 'ocloud' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'ocloud'
CRS-2677: Stop of 'ora.cssd' on 'ocloud' succeeded
CRS-4000: Command Stop failed, or completed with errors.
[grid@ocloud ~]$ crsctl start resource -all
CRS-2501: Resource 'ora.ons' is disabled
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'ocloud'
CRS-2672: Attempting to start 'ora.cssd' on 'ocloud'
CRS-2672: Attempting to start 'ora.diskmon' on 'ocloud'
CRS-2672: Attempting to start 'ora.evmd' on 'ocloud'
CRS-2676: Start of 'ora.diskmon' on 'ocloud' succeeded
CRS-2676: Start of 'ora.evmd' on 'ocloud' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'ocloud' succeeded
CRS-2676: Start of 'ora.cssd' on 'ocloud' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'ocloud'
CRS-2676: Start of 'ora.asm' on 'ocloud' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'ocloud'
CRS-2676: Start of 'ora.DATA.dg' on 'ocloud' succeeded
CRS-2672: Attempting to start 'ora.key.db' on 'ocloud'
CRS-2676: Start of 'ora.key.db' on 'ocloud' succeeded
CRS-4000: Command Start failed, or completed with errors.
[grid@ocloud ~]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    ocloud
ora....ER.lsnr ora....er.type ONLINE    ONLINE    ocloud
ora.asm        ora.asm.type   ONLINE    ONLINE    ocloud
ora.cssd       ora.cssd.type  ONLINE    ONLINE    ocloud
ora.diskmon    ora....on.type OFFLINE   OFFLINE
ora.evmd       ora.evm.type   ONLINE    ONLINE    ocloud
ora.key.db     ora....se.type ONLINE    ONLINE    ocloud
ora.ons        ora.ons.type   OFFLINE   OFFLINE
[grid@ocloud ~]$

The ASM instance start successfully!