计算机中丢失oraocr,守得云开见月明:一次ASM存储高可用故障解决过程分析

2018-08-16T14:56:04.957199+08:00 -->差不多和DB同时检测到IO错误

WARNING: Write Failed. group:1 disk:1 AU:1 offset:4190208 size:4096

path:/dev/rdisk/MES1_p4_moddb_ocrvote

incarnation:0xf0996d00 asynchronous result:'I/O error'

subsys:System krq:0x9ffffffffd285710 bufp:0x9ffffffffd299000 osderr1:0x69b7 osderr2:0x0

IO elapsed time: 0 usec Time waited on I/O: 0 usec

……

2018-08-16T14:56:34.400892+08:00

WARNING: Disk 4 (MES1_REDO02) in group 2 mode 0x7f is now being offlined

……

ORA-27072: File I/O error

HPUX-ia64 Error: 11: Resource temporarily unavailable

Additional information: 4

Additional information: 8188

Additional information: 4294967295

GMON checking disk modes for group 2 at 2460 for pid 49, osid 13802

2018-08-16T14:56:35.056049+08:00

NOTE: checking PST for grp 2 done.

2018-08-16T14:56:35.056382+08:00

Errors in file /grid/crs_base/diag/asm/+asm/+ASM1/trace/+ASM1_gmon_18741.trc:

ORA-27072: File I/O error

HPUX-ia64 Error: 11: Resource temporarily unavailable

Additional information: 4

Additional information: 8188

Additional information: 4294967295

ORA-27072: File I/O error

HPUX-ia64 Error: 11: Resource temporarily unavailable

Additional information: 4

Additional information: 8188

Additional information: 4294967295

ORA-27072: File I/O error

HPUX-ia64 Error: 11: Resource temporarily unavailable

Additional information: 4

Additional information: 8188

Additional information: 4294967295

2018-08-16T14:56:35.056592+08:00

GMON checking disk modes for group 2 at 2461 for pid 19, osid 18728

NOTE: initiating PST update: grp 2 (DG_REDO_MOD), dsk = 4/0xf0996e2a, mask = 0x6a, op = clear mandatory

GMON checking disk modes for group 1 at 2462 for pid 57, osid 24043

2018-08-16T14:56:35.057277+08:00

NOTE: checking PST for grp 2 done.

NOTE: initiating PST update: grp 2 (DG_REDO_MOD), dsk = 4/0xf0996e2a, mask = 0x6a, op = clear mandatory

2018-08-16T14:56:35.059431+08:00

NOTE: checking PST for grp 1 done.

2018-08-16T14:56:35.059697+08:00

NOTE: sending set offline flag message (3413962691) to 1 disk(s) in group 1

2018-08-16T14:56:35.059796+08:00

GMON checking disk modes for group 4 at 2463 for pid 59, osid 24045

2018-08-16T14:56:36.205496+08:00

NOTE: Attempting voting file refresh on diskgroup DG_CRS_MOD

WARNING: Read Failed. group:1 disk:1 AU:0 offset:0 size:4096

path:/dev/rdisk/MES1_p4_moddb_ocrvote

incarnation:0xf0996d00 asynchronous result:'I/O error'

subsys:System krq:0x9ffffffffd4a4068 bufp:0x9ffffffffcf91000 osderr1:0x534b4950 osderr2:0x0

IO elapsed time: 0 usec Time waited on I/O: 0 usec

NOTE: Refresh completed on diskgroup DG_CRS_MOD. Found 3 voting file(s).

NOTE: Voting file relocation is required in diskgroup DG_CRS_MOD

NOTE: Attempting voting file relocation on diskgroup DG_CRS_MOD

WARNING: Read Failed. group:1 disk:1 AU:0 offset:0 size:4096

path:/dev/rdisk/MES1_p4_moddb_ocrvote

incarnation:0xf0996d00 asynchronous result:'I/O error'

subsys:System krq:0x9ffffffffd4a4068 bufp:0x9ffffffffcf84000 osderr1:0x534b4950 osderr2:0x0

IO elapsed time: 0 usec Time waited on I/O: 0 usec

NOTE: Successful voting file relocation on diskgroup DG_CRS_MOD

2018-08-16T14:56:52.061661+08:00

System state dump requested by (instance=1, osid=3906536), summary=[SYSTEMSTATE_GLOBAL: global system state dump request (kjdgdss_g)].

System State dumped to trace file /grid/crs_base/diag/asm/+asm/+ASM1/trace/+ASM1_diag_18708_20180816145652.trc

2018-08-16T14:56:55.250804+08:00

WARNING: Read Failed. group:2 disk:4 AU:0 offset:0 size:4096

2018-08-16T14:56:55.250889+08:00

WARNING: Read Failed. group:2 disk:4 AU:7 offset:245760 size:4096

2018-08-16T14:56:55.251279+08:00

path:/dev/rdisk/MES1_p4_moddb_redo02

2018-08-16T14:56:55.251590+08:00

incarnation:0xf0996e2a synchronous result:'I/O error'

subsys:System krq:0x9ffffffffd4b2f80 bufp:0x9ffffffffd407000 osderr1:0x69b7 osderr2:0x0

2018-08-16T14:56:55.251753+08:00

path:/dev/rdisk/MES1_p4_moddb_redo02

IO elapsed time: 0 usec Time waited on I/O: 0 usec

2018-08-16T14:56:55.252112+08:00

incarnation:0xf0996e2a synchronous result:'I/O error'

subsys:System krq:0x9ffffffffd225ec0 bufp:0x9ffffffffcc0f000 osderr1:0x69c0 osderr2:0x0

IO elapsed time: 0 usec Time waited on I/O: 0 usec

WARNING: cache failed reading from group=2(DG_REDO_MOD) fn=8 blk=60 count=1 from disk=4 (MES1_REDO02) mirror=0 kfkist=0x20 status=0x07 osderr=0x69c0 file=kfc.c line=12986

WARNING: cache succeeded reading from group=2(DG_REDO_MOD) fn=8 blk=60 count=1 from disk=3 (MES2_REDO01) mirror=1 kfkist=0x20 status=0x01 osderr=0x0 file=kfc.c line=13031

2018-08-16T14:56:55.274598+08:00

SUCCESS: extent 0 of file 8 group 2 - all online mirror sides found readable, no repair required ---> 所有online mirror side是完好的