All nodes in Oracle RAC cluster reboot when SAN controller fails on Red Hat Enterprise Linux 5.10

最新推荐文章于 2020-01-26 11:31:57 发布

victoruu

最新推荐文章于 2020-01-26 11:31:57 发布

阅读量603

点赞数

分类专栏： Linux KB

Linux KB 专栏收录该内容

35 篇文章 0 订阅

订阅专栏

Environment

Red Hat Enterprise Linux 5.10 with kernel-2.6.18-371.6.1.el5 or later.
Oracle RAC 10G environment with two or more nodes
- Oracle RAC's voting disk timeout is 200 seconds
Device-mapper-multipath
SAN environment suffering controller failure (for example, director board failure)
- Similar symptoms may be seen if a failure on the fabric occurs and RSCN's are not sent to alert targets of the change.
Raw devices used for some Oracle devices

Issue

A SAN event resulted in all eight nodes in a RHEL 5.10 Oracle 10g cluster rebooting simultaneously. The director board failure caused one of two paths per LUN to fail.
- The kernel logs from right before the reboot showed that there were many SCSI errors/timeouts and ended with "SysRq : Resetting" before the systems starting booting.
- Why didn't the multipath daemon didn't take advantage of the working paths (or rather disable the failed paths)?

Resolution

Adjust SCSI timeout and eh_deadline to ensure that lost I/O's are aborted before Oracle RAC can timeout. You must be using kernel-2.6.18-371.6.1.el5 or later to configure eh_deadline

Some example settings:
- If we set the scsi_timeout to 30 seconds and eh_deadline to 90 seconds, we'd get (scsi timeout = 30) + (eh_deadline = 90) = 120 seconds plus time to reissue the I/O.
- No need to adjust eh_timeout because eh_deadline=90 provides plenty of time for the commands to complete.

We can use udev to configure SCSI timeout and eh_deadline at boot time. Modify /etc/udev/rules.d/50-udev.rules:

Modify the following line to set the SCSI timeout to 30 seconds:

    # vim /etc/udev/rules.d/50-udev.rules
        ACTION=="add", SUBSYSTEM=="scsi" , SYSFS{type}=="0|7|14", \
                RUN+="/bin/sh -c 'echo 30 > /sys$$DEVPATH/timeout'"

The following will set eh_deadline to 90 seconds for all hostadapaters on the system. Add the following to /etc/rc.d/rc.local:
/etc/rc.d/rc.local:
Raw
```
for i in $(ls /sys/class/fc_host/); do echo 90 > /sys/class/scsi_host/$i/eh_deadline; done
```

Root Cause

When the SAN controller failed, no error was received by the host and outstanding I/O's were lost in the fabric. New I/O's could NOT be sent until the SCSI Error Handler (scsi_eh) kicked in and cleaned up the missing I/O's.
Even though device-mapper-multipath was in-use, the path-checker I/O's were also lost in the fabric so dm-multipath could not detect the failure until the SCSI Error Handler aborted the I/O's.
Unfortunately, the SCSI error handler took more than 200 seconds to abort the I/O's and recover. As a result, Oracle RAC rebooted the cluster nodes because they were unable to update their voting disks.

Diagnostic Steps

The following symptoms, in conjunction with the Environmental variables, are thought to represent this issue:

First sign of the SAN controller failure may look like the following:

The Cluster is performing correctly, sending I/O's down both multipath paths when suddenly the director fails. I/O commands that are "in-flight" start their SCSI timeout countdown shortly before the failure (by default, 60 seconds), and new I/O's generated continue to be sent down the failed path, starting their SCSI timeouts. Note that device-mapper-multipath is also sending I/O's to check on the path status, and its I/O's are also "hung" so it can't detect that the path has failed... until after the error handler kicks in.

Fast forward 60 seconds (default scsi timeout) from the failure of the director, the first commands suffer their SCSI timeout [1] and will be added to the list of commands to pass to the error handler. From the following log messages, it appears that it took 58 more seconds for all outstanding commands to time-out [2]. Once all the commands have timed out, then the error handler will wake up [2] and begin aborting the commands [3]:

Jun 24 19:35:37 node1 kernel: sd 5:0:2:0: Done: TIMEOUT   [1]
Jun 24 19:35:37 node1 kernel:         0 sd 5:0:2:0: 
Jun 24 19:35:37 node1 kernel:         command: Test Unit Ready: 00 00 00 00 00 00
<...>
Jun 24 19:36:35 node1 kernel: sd 5:0:2:7: Done: TIMEOUT
Jun 24 19:36:35 node1 kernel:         0 sd 5:0:2:7:
Jun 24 19:36:35 node1 kernel:         command: Read(10): 28 00 00 f0 ce 70 00 00 10 00
Jun 24 19:36:35 node1 kernel: scsi_eh_5: waking up 0/200/200                         [2]
Jun 24 19:36:35 node1 kernel: Total of 200 commands on 14 devices require eh work
Jun 24 19:36:36 node1 kernel: qla2xxx 0000:06:00.0: scsi(5:2:0): Abort command issued -- 1 1afec22f 2002.  [3]
= approximately 58 seconds between timeout and error handler waking up.

I/O's could have been hung for 120 seconds at this stage and they are not unstuck yet. Now the error handler starts up and it begins by trying to abort the outstanding commands (it sends an abort message to the fabric to tell the storage not to complete the I/O). The error handler will stop any I/O's going to that hostadapter so no more I/O's can be sent to host5. In this event, the SAN director is down so the Abort command attempts timeout (another 60 seconds), and then we cascade to the next level of SCSI error-handler recovery. From the beginning of the abort messages until we finally reset hostadapter5, it takes another 2 minute 26 seconds (146 seconds - 19:36:36 to 19:39:01):

Jun 24 19:36:35 node1 kernel: scsi_eh_5: waking up 0/200/200                         
Jun 24 19:36:35 node1 kernel: Total of 200 commands on 14 devices require eh work
Jun 24 19:36:36 node1 kernel: qla2xxx 0000:06:00.0: scsi(5:2:0): Abort command issued -- 1 1afec22f 2002.
<... several Abort commands omitted for clarity ...>
Jun 24 19:37:16 node1 kernel: qla2xxx 0000:06:00.0: scsi(5:2:29): Abort command issued -- 1 1afebcc1 2002.
<... 60 second delay here because the Abort command times out like the initial request ...>
Jun 24 19:38:18 node1 kernel: qla2xxx 0000:06:00.0: scsi(5:2:29): DEVICE RESET ISSUED.   <-- there are several of these.  Omitted for clarity
Jun 24 19:38:18 node1 kernel: qla2xxx 0000:06:00.0: scsi(5:2:0): LOOP RESET ISSUED.
Jun 24 19:38:28 node1 kernel: qla2xxx 0000:06:00.0: scsi(5:2:0): ADAPTER RESET ISSUED.
Jun 24 19:39:01 node1 kernel: qla2xxx 0000:06:00.0: qla2xxx_eh_host_reset: reset succeeded

At this point, the I/O's that were outstanding have been aborted but there hasn't been a write to disk for quite some time:
- For paths on the faulty controller (target#2 on hostadapter5), no updates have made it through for 4m24secs (264 seconds - 19:34:37 - 19:39:01).
- For all other paths on hostadapter5 (target 0,1,3), no updates have been made since the error handler started at 19:36:35 - 2m26secs (146 seconds)
Now that the in-flight path checker commands have failed, the dm-multipath devices begin to fail, but the commands can't be retried on the other multipath path because they were aborted/reset above. New I/O's can be directed down the other path but the previously in-flight commands have been lost forever:
Raw
```
Jun 24 19:39:11 node1 multipathd: sdjg: tur checker reports path is down 
Jun 24 19:39:11 node1 multipathd: sdjk: tur checker reports path is down 
<..some omitted..>
Jun 24 19:39:12 node1 multipathd: sdnk: tur checker reports path is down 
```

So at this point, any voting disk I/O's on the failed director were stalled for 264 seconds and Oracle would have timed out after 200 seconds. It appears that this would take out 2 of the voting devices, which matches the log message:

The message from Oracle CSSD indicates that 2 voting disks are unavailable:

Jun 24 19:39:28 node1 oraAlert_CRS: [cssd(12132)]CRS-1606:CSSD Insufficient voting
   files available [1 of 3]. Details in /oracle/crs/product/10.2.0/log/node1/cssd/ocssd.log.

Voting disks 1 and 3 have paths on host5. Multipath using round-robin so eventually these paths would have been used:

360000970000192604738533038333430 dm-184 EMC,SYMMETRIX          <-- voting1
[size=1.5G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 8:0:1:20 sdhn 133:208 [active][ready]
 \_ 5:0:2:20 sdmh 69:400  [active][ready]                    <-- path on host5
360000970000192604738533038333530 dm-239 EMC,SYMMETRIX          <-- voting2
[size=1.5G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 6:0:2:20 sdml 69:464  [active][ready]
 \_ 7:0:2:20 sdmt 70:336  [active][ready]
360000970000192604738533038333630 dm-178 EMC,SYMMETRIX          <-- voting3
[size=1.5G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 5:0:1:22 sdhg 133:96  [active][ready]                    <-- path on host5
 \_ 8:0:2:22 sdmm 69:480  [active][ready]

For 200 seconds to timeout before the Oracle RAC cssd message, the last voting disk update was 19:36:08, which is after the first timeout message but before the last timeout message.

Shortly after, several I/O-bound processes report hung-task (including Oracle ones..):

Jun 24 19:39:04 node1 kernel: INFO: task pdflush:5944 blocked for more than 120 seconds.
<...>
Jun 24 19:39:04 node1 kernel: INFO: task kjournald:11175 blocked for more than 120 seconds.
<...>
Jun 24 19:39:04 node1 kernel: INFO: task multipathd:16765 blocked for more than 120 seconds.
<...>
Jun 24 19:39:04 node1 kernel: INFO: task snmpd:8059 blocked for more than 120 seconds.
<...>
Jun 24 19:39:04 node1 kernel: INFO: task crsd.bin:5167 blocked for more than 120 seconds.
<...>
Jun 24 19:39:04 node1 kernel: INFO: task ocssd.bin:14751 blocked for more than 120 seconds.
Jun 24 19:39:04 node1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 24 19:39:04 node1 kernel: ocssd.bin     D ffffffff801582dc     0 14751  10349         17037 14750 (NOTLB)
Jun 24 19:39:04 node1 kernel:  ffff81045e0c5b98 0000000000000086 0000002083ba8b88 0000000000000001
Jun 24 19:39:04 node1 kernel:  0000003300000000 0000000000000001 ffff81101ea58080 ffff8110583b6080
Jun 24 19:39:04 node1 kernel:  00025de92553fdb5 00000000000108f0 ffff81101ea58268 0000003300000092
Jun 24 19:39:04 node1 kernel: Call Trace:
Jun 24 19:39:04 node1 kernel:  [<ffffffff88036e65>] :jbd:log_wait_commit+0xa3/0xf5
Jun 24 19:39:04 node1 kernel:  [<ffffffff800a3f62>] autoremove_wake_function+0x0/0x2e
Jun 24 19:39:04 node1 kernel:  [<ffffffff880317a1>] :jbd:journal_stop+0x1d3/0x203
Jun 24 19:39:04 node1 kernel:  [<ffffffff8002ff4c>] __writeback_single_inode+0x1dd/0x31c
Jun 24 19:39:04 node1 kernel:  [<ffffffff880317c5>] :jbd:journal_stop+0x1f7/0x203
Jun 24 19:39:04 node1 kernel:  [<ffffffff800f8ca1>] write_inode_now+0x77/0xbf
Jun 24 19:39:04 node1 kernel:  [<ffffffff8002f57f>] generic_osync_inode+0x96/0xd9
Jun 24 19:39:04 node1 kernel:  [<ffffffff800ca604>] sync_page_range+0x62/0xa0
Jun 24 19:39:04 node1 kernel:  [<ffffffff80021e51>] generic_file_aio_write+0xa8/0xc3
Jun 24 19:39:04 node1 kernel:  [<ffffffff8804c372>] :ext3:ext3_file_write+0x16/0x91
Jun 24 19:39:04 node1 kernel:  [<ffffffff800183ce>] do_sync_write+0xc7/0x104
Jun 24 19:39:04 node1 kernel:  [<ffffffff800a3f62>] autoremove_wake_function+0x0/0x2e
Jun 24 19:39:04 node1 kernel:  [<ffffffff80016b4b>] vfs_write+0xce/0x174
Jun 24 19:39:04 node1 kernel:  [<ffffffff80017414>] sys_write+0x45/0x6e
Jun 24 19:39:04 node1 kernel:  [<ffffffff8005d29e>] tracesys+0xd5/0xdf
Jun 24 19:39:04 node1 kernel: 
Jun 24 19:39:04 node1 kernel: INFO: task ocssd.bin:17037 blocked for more than 120 seconds.
Jun 24 19:39:04 node1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 24 19:39:04 node1 kernel: ocssd.bin     D ffff81101fef2040     0 17037  10349         17050 14751 (NOTLB)
Jun 24 19:39:04 node1 kernel:  ffff81147f349b98 0000000000000086 ffff811019e0fa00 ffff811019e0faf0
Jun 24 19:39:04 node1 kernel:  0000000000000000 0000000000000001 ffff81201be65040 ffff81101fef2040
Jun 24 19:39:04 node1 kernel:  00025de87d81e0fd 0000000000005f84 ffff81201be65228 0000003600000000
Jun 24 19:39:04 node1 kernel: Call Trace:
Jun 24 19:39:04 node1 kernel:  [<ffffffff8006edac>] do_gettimeofday+0x40/0x90
Jun 24 19:39:04 node1 kernel:  [<ffffffff8005ab4c>] getnstimeofday+0x10/0x29
Jun 24 19:39:04 node1 kernel:  [<ffffffff800637de>] io_schedule+0x3f/0x67
Jun 24 19:39:04 node1 kernel:  [<ffffffff800fa947>] __blockdev_direct_IO+0x8d5/0xa88
Jun 24 19:39:04 node1 kernel:  [<ffffffff800ea56c>] blkdev_direct_IO+0x32/0x37
Jun 24 19:39:04 node1 kernel:  [<ffffffff800ea4a4>] blkdev_get_blocks+0x0/0x96
Jun 24 19:39:04 node1 kernel:  [<ffffffff800ca791>] generic_file_direct_IO+0x108/0x122
Jun 24 19:39:04 node1 kernel:  [<ffffffff8001f52b>] generic_file_direct_write+0x60/0xf2
Jun 24 19:39:04 node1 kernel:  [<ffffffff80016692>] __generic_file_aio_write_nolock+0x2b8/0x3b6
Jun 24 19:39:04 node1 kernel:  [<ffffffff800ca916>] generic_file_aio_write_nolock+0x20/0x6c
Jun 24 19:39:04 node1 kernel:  [<ffffffff800caced>] generic_file_write_nolock+0x8f/0xa8
Jun 24 19:39:04 node1 kernel:  [<ffffffff800a3f62>] autoremove_wake_function+0x0/0x2e
Jun 24 19:39:04 node1 kernel:  [<ffffffff801be6e7>] raw_file_write+0x1a/0x1f
Jun 24 19:39:04 node1 kernel:  [<ffffffff80016b4b>] vfs_write+0xce/0x174
Jun 24 19:39:04 node1 kernel:  [<ffffffff80044298>] sys_pwrite64+0x50/0x70
Jun 24 19:39:04 node1 kernel:  [<ffffffff8005d23a>] tracesys+0x71/0xdf
Jun 24 19:39:04 node1 kernel:  [<ffffffff8005d29e>] tracesys+0xd5/0xdf
Jun 24 19:39:04 node1 kernel: 
Jun 24 19:39:04 node1 kernel: INFO: task ocssd.bin:17063 blocked for more than 120 seconds.
Jun 24 19:39:04 node1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 24 19:39:04 node1 kernel: ocssd.bin     D ffffffff801582dc     0 17063  10349         20098 17050 (NOTLB)
Jun 24 19:39:04 node1 kernel:  ffff81147c219b98 0000000000000086 ffff81101a568800 ffff81101a5688f0
Jun 24 19:39:04 node1 kernel:  0000000000000000 0000000000000001 ffff81147e825080 ffff8130200dc830
Jun 24 19:39:04 node1 kernel:  00025df15d0fea43 000000000000636c ffff81147e825268 0000001200000000
Jun 24 19:39:04 node1 kernel: Call Trace:
Jun 24 19:39:04 node1 kernel:  [<ffffffff8006edac>] do_gettimeofday+0x40/0x90
Jun 24 19:39:04 node1 kernel:  [<ffffffff800637de>] io_schedule+0x3f/0x67
Jun 24 19:39:04 node1 kernel:  [<ffffffff800fa947>] __blockdev_direct_IO+0x8d5/0xa88
Jun 24 19:39:04 node1 kernel:  [<ffffffff800ea56c>] blkdev_direct_IO+0x32/0x37
Jun 24 19:39:04 node1 kernel:  [<ffffffff800ea4a4>] blkdev_get_blocks+0x0/0x96
Jun 24 19:39:04 node1 kernel:  [<ffffffff800ca791>] generic_file_direct_IO+0x108/0x122
Jun 24 19:39:04 node1 kernel:  [<ffffffff8001f52b>] generic_file_direct_write+0x60/0xf2
Jun 24 19:39:04 node1 kernel:  [<ffffffff80016692>] __generic_file_aio_write_nolock+0x2b8/0x3b6
Jun 24 19:39:04 node1 kernel:  [<ffffffff800ca916>] generic_file_aio_write_nolock+0x20/0x6c
Jun 24 19:39:04 node1 kernel:  [<ffffffff800caced>] generic_file_write_nolock+0x8f/0xa8
Jun 24 19:39:04 node1 kernel:  [<ffffffff800a3f62>] autoremove_wake_function+0x0/0x2e
Jun 24 19:39:04 node1 kernel:  [<ffffffff801be6e7>] raw_file_write+0x1a/0x1f
Jun 24 19:39:04 node1 kernel:  [<ffffffff80016b4b>] vfs_write+0xce/0x174
Jun 24 19:39:04 node1 kernel:  [<ffffffff80044298>] sys_pwrite64+0x50/0x70
Jun 24 19:39:04 node1 kernel:  [<ffffffff8005d23a>] tracesys+0x71/0xdf
Jun 24 19:39:04 node1 kernel:  [<ffffffff8005d29e>] tracesys+0xd5/0xdf
Jun 24 19:39:04 node1 kernel:
Jun 24 19:39:04 node1 kernel: INFO: task ocssd.bin:20098 blocked for more than 120 seconds.
Jun 24 19:39:04 node1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 24 19:39:04 node1 kernel: ocssd.bin     D ffffffff801582dc     0 20098  10349         20123 17063 (NOTLB)
Jun 24 19:39:04 node1 kernel:  ffff81147bb9fcb8 0000000000000086 ffff811019e0fa00 ffff811019e0faf0
Jun 24 19:39:04 node1 kernel:  0000000000000000 0000000000000001 ffff81201e65c7b0 ffff81401fe8b830
Jun 24 19:39:04 node1 kernel:  00025de87c9dade4 000000000000b9fe ffff81201e65c998 0000003600000000
Jun 24 19:39:04 node1 kernel: Call Trace:
Jun 24 19:39:04 node1 kernel:  [<ffffffff8006edac>] do_gettimeofday+0x40/0x90
Jun 24 19:39:04 node1 kernel:  [<ffffffff800637de>] io_schedule+0x3f/0x67
Jun 24 19:39:04 node1 kernel:  [<ffffffff800fa947>] __blockdev_direct_IO+0x8d5/0xa88
Jun 24 19:39:04 node1 kernel:  [<ffffffff800ea56c>] blkdev_direct_IO+0x32/0x37
Jun 24 19:39:04 node1 kernel:  [<ffffffff800ea4a4>] blkdev_get_blocks+0x0/0x96
Jun 24 19:39:04 node1 kernel:  [<ffffffff8000c66b>] __generic_file_aio_read+0xb8/0x198
Jun 24 19:39:04 node1 kernel:  [<ffffffff80132500>] inode_has_perm+0x56/0x63
Jun 24 19:39:04 node1 kernel:  [<ffffffff800cac45>] generic_file_read+0xac/0xc5
Jun 24 19:39:04 node1 kernel:  [<ffffffff800a3f62>] autoremove_wake_function+0x0/0x2e
Jun 24 19:39:04 node1 kernel:  [<ffffffff8005a866>] hrtimer_cancel+0xc/0x16
Jun 24 19:39:04 node1 kernel:  [<ffffffff801350e9>] selinux_file_permission+0x9f/0xb5
Jun 24 19:39:04 node1 kernel:  [<ffffffff8000b735>] vfs_read+0xcb/0x171
Jun 24 19:39:04 node1 kernel:  [<ffffffff8001376f>] sys_pread64+0x50/0x70
Jun 24 19:39:04 node1 kernel:  [<ffffffff8005d23a>] tracesys+0x71/0xdf
Jun 24 19:39:04 node1 kernel:  [<ffffffff8005d29e>] tracesys+0xd5/0xdf

Oracle complains about being unable to access voting files and voting disks:

Jun 24 19:39:28 node1 oraAlert_CRS: 2014-06-24 19:39:28.306
Jun 24 19:39:28 node1 oraAlert_CRS: [cssd(12132)]CRS-1606:CSSD Insufficient voting files available [1 of 3]. Details in /oracle/crs/product/10.2.0/log/node1/cssd/ocssd.log.
Jun 24 19:41:49 node1 oraAlert_CRS: 2014-06-24 19:41:48.968
Jun 24 19:41:49 node1 oraAlert_CRS: [cssd(12132)]CRS-1604:CSSD voting file is offline: /dev/raw/votingdisk1. Details in /oracle/crs/product/10.2.0/log/node1/cssd/ocssd.log.
Jun 24 19:41:52 node1 oraAlert_ORADB2-1: Tue Jun 24 19:41:52 GMT 2014
Jun 24 19:41:52 node1 oraAlert_ORADB2-1: IPC Send timeout detected.Sender: ospid 10224
Jun 24 19:41:52 node1 oraAlert_ORADB2-1: Receiver: inst 3 binc 429825219 ospid 14628Jun 24 19:42:00 node1 oraAlert_ORADB2-1: Tue Jun 24 19:41:59 GMT 2014
Jun 24 19:42:00 node1 oraAlert_ORADB2-1: IPC Send timeout detected.Sender: ospid 5493
Jun 24 19:42:00 node1 oraAlert_ORADB2-1: Receiver: inst 3 binc 429825219 ospid 14628

Oracle cssd reboots the node and it restarts:

Jun 24 19:42:00 node1 kernel: SysRq : Resetting
Jun 24 19:53:19 node1 syslogd 1.4.1: restart.

Oracle reports that the voting disk /dev/raw/votingdisk1 is offline, however this device is not one of the devices that had a failed path:

The message related to /dev/raw/votingdisk1 failing:

Jun 24 19:41:49 node1 oraAlert_CRS: [cssd(12132)]CRS-1604:CSSD voting
   file is offline: /dev/raw/votingdisk1. Details in /oracle/crs/product/10.2.0/log/node1/cssd/ocssd.log.

/dev/raw/votingdisk1 appears to be UUID 360000970000192604738533038333430p1:

$ grep votingdisk1 /etc/udev/rules.d/10-oracle-raw.rules 
PROGRAM="/bin/mv /dev/raw/raw1 /dev/raw/votingdisk1"
$ grep raw /etc/udev/rules.d/10-oracle-raw.rules
ACTION=="add", KERNEL=="dm-[0-9]*", RUN+="/bin/bash -c '/sbin/dmsetup info -c --noheadings -o name -j %M -m %m \
        | grep 360000970000192604738533038333430p1 && /bin/raw /dev/raw/raw1 /dev/%k'"

This UUID does not report a failed path in /var/log/messages:

$ grep 360000970000192604738533038333430 /var/log/messages 
Jun 24 19:53:32 node1 multipathd: 360000970000192604738533038333430: event checker started  <-- after the reboot

victoruu

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
All nodes in Oracle RAC cluster reboot when SAN controller fails on Red Hat Enterprise Linux 5.10

EnvironmentRed Hat Enterprise Linux 5.10 with kernel-2.6.18-371.6.1.el5 or later.Oracle RAC 10G environment with two or more nodesOracle RAC's voting disk timeout is 200 secondsDevice-mapper-multipath...
复制链接

扫一扫