rac 启动其中一个node_rac双节点，其中一个节点开不了数据库

node1正常打开数据库

[grid@node1 ~]$ crs_stat -t

Name           Type           Target    State     Host

------------------------------------------------------------

ora.DATA.dg    ora....up.type ONLINE    ONLINE    node1

ora.DG1.dg     ora....up.type ONLINE    ONLINE    node1

ora.DG2.dg     ora....up.type ONLINE    ONLINE    node1

ora....ER.lsnr ora....er.type ONLINE    ONLINE    node1

ora....N1.lsnr ora....er.type ONLINE    ONLINE    node2

ora.asm        ora.asm.type   ONLINE    ONLINE    node1

ora.cvu        ora.cvu.type   ONLINE    ONLINE    node1

ora.db.db      ora....se.type ONLINE    ONLINE    node1

ora.gsd        ora.gsd.type   OFFLINE   OFFLINE

ora....network ora....rk.type ONLINE    ONLINE    node1

ora....SM1.asm application    ONLINE    ONLINE    node1

ora....E1.lsnr application    ONLINE    ONLINE    node1

ora.node1.gsd  application    OFFLINE   OFFLINE

ora.node1.ons  application    ONLINE    ONLINE    node1

ora.node1.vip  ora....t1.type ONLINE    ONLINE    node1

ora....SM2.asm application    ONLINE    ONLINE    node2

ora....E2.lsnr application    ONLINE    ONLINE    node2

ora.node2.gsd  application    OFFLINE   OFFLINE

ora.node2.ons  application    ONLINE    ONLINE    node2

ora.node2.vip  ora....t1.type ONLINE    ONLINE    node2

ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    node1

ora.ons        ora.ons.type   ONLINE    ONLINE    node1

ora.scan1.vip  ora....ip.type ONLINE    ONLINE    node2

[grid@node1 ~]$ su - oracle

Password:

[oracle@node1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Mon Jan 27 12:38:47 2014

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> select status from v$instance;

STATUS

------------

OPEN

node2打开出错，提出VOL3磁盘不能访问

[oracle@node2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Mon Jan 27 11:55:37 2014

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup

ORACLE instance started.

Total System Global Area  835104768 bytes

Fixed Size            2231088 bytes

Variable Size          553649360 bytes

Database Buffers      272629760 bytes

Redo Buffers            6594560 bytes

ORA-03113: end-of-file on communication channel

Process ID: 5491

Session ID: 125 Serial number: 3

[oracle@node2 trace]$ tail -50 alert_db2.log

Global Resource Directory frozen

* allocate domain 0, invalid = TRUE

Communication channels reestablished

* domain 0 valid = 0 according to instance 1

Master broadcasted resource hash value bitmaps

Non-local Process blocks cleaned out

LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived

Set master node info

Submitted all remote-enqueue requests

Dwn-cvts replayed, VALBLKs dubious

All grantable enqueues granted

Post SMON to start 1st pass IR

Submitted all GCS remote-cache requests

Post SMON to start 1st pass IR

Fix write in gcs resources

Reconfiguration complete

Mon Jan 27 12:17:18 2014

LCK0 started with pid=30, OS id=5430

Starting background process RSMN

Mon Jan 27 12:17:18 2014

RSMN started with pid=31, OS id=5432

ORACLE_BASE from environment = /u02/app

Mon Jan 27 12:18:03 2014

ALTER DATABASE   MOUNT

This instance was first to mount

Mon Jan 27 12:18:08 2014

NOTE: Loaded library: System

ORA-15025: could not open disk "/dev/oracleasm/disks/VOL3"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 9

Mon Jan 27 12:18:08 2014

SUCCESS: diskgroup DG1 was mounted

Mon Jan 27 12:18:09 2014

NOTE: dependency between database db and diskgroup resource ora.DG1.dg is established

Errors in file /u02/app/diag/rdbms/db/db2/trace/db2_ckpt_5403.trc  (incident=20577):

ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []

Incident details in: /u02/app/diag/rdbms/db/db2/incident/incdir_20577/db2_ckpt_5403_i20577.trc

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Mon Jan 27 12:18:19 2014

Dumping diagnostic data in directory=[cdmp_20140127121819], requested by (instance=2, osid=5403 (CKPT)), summary=[incident=20577].

Mon Jan 27 12:18:21 2014

ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 5403

Mon Jan 27 12:18:22 2014

PMON (ospid: 5365): terminating the instance due to error 469

System state dump requested by (instance=2, osid=5365 (PMON)), summary=[abnormal instance termination].

System State dumped to trace file /u02/app/diag/rdbms/db/db2/trace/db2_diag_5375.trc

Dumping diagnostic data in directory=[cdmp_20140127121822], requested by (instance=2, osid=5365 (PMON)), summary=[abnormal instance termination].

Instance terminated by PMON, pid = 5365

[oracle@node2 trace]$ id

uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),505(asmdba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

2个节点磁盘检查，磁盘权限时一样的，为什么node1可以访问，node2不行？

[root@node1 disks]# ll

total 0

brw-rw----. 1 grid asmadmin 8, 17 Jan 27 12:22 VOL1

brw-rw----. 1 grid asmadmin 8, 33 Jan 27 11:49 VOL2

brw-rw----. 1 grid asmadmin 8, 49 Jan 27 12:21 VOL3

brw-rw----. 1 grid asmadmin 8, 65 Jan 27 12:21 VOL4

[oracle@node1 ~]$ id

uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),505(asmdba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

[root@node2 disks]# ll

total 0

brw-rw----. 1 grid asmadmin 8, 17 Jan 27 12:19 VOL1

brw-rw----. 1 grid asmadmin 8, 33 Jan 27 12:18 VOL2

brw-rw----. 1 grid asmadmin 8, 49 Jan 27 12:19 VOL3

brw-rw----. 1 grid asmadmin 8, 65 Jan 27 12:19 VOL4

[oracle@node2 ~]$ id

uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),505(asmdba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023