RAC 10.2.0.4 Linux AS 5.3 X86_64
3 NODES : 1 , 3 , 4
节点1 Server莫名 重新启动,导致节点3 实例也自动重启了 ,注意节点1,3,4 是节点号码 。
因为当初节点2 损坏了 。
难道是open max file 太多 ? 大家有没有碰到过 ,紧急啊 。
节点1的信息:
Fri Jan 22 19:45:29 2010
Thread 1 advanced to log sequence 5981 (LGWR switch)
Current log# 30 seq# 5981 mem# 0: /ocfs_ctrl_redo/mxdell/redo30.log
Current log# 30 seq# 5981 mem# 1: /ocfs_data/mxdell/redo30.log
Fri Jan 22 19:58:29 2010
Thread 1 advanced to log sequence 5982 (LGWR switch)
Current log# 29 seq# 5982 mem# 0: /ocfs_ctrl_redo/mxdell/redo29.log
Current log# 29 seq# 5982 mem# 1: /ocfs_data/mxdell/redo29.log
Fri Jan 22 20:16:45 2010
Thread 1 advanced to log sequence 5983 (LGWR switch)
Current log# 25 seq# 5983 mem# 0: /ocfs_ctrl_redo/mxdell/redo25.log
Current log# 25 seq# 5983 mem# 1: /ocfs_data/mxdell/redo25.log
Sat Jan 23 03:18:02 2010
Errors in file /u01/product/admin/mxdell/udump/mxdell1_ora_30566.trc:
ORA-00603: Message 603 not found; No message file for product=RDBMS, facility=ORA
ORA-27504: Message 27504 not found; No message file for product=RDBMS, facility=ORA
ORA-27300: Message 27300 not found; No message file for product=RDBMS, facility=ORA; arguments: [ioctl] [23]
ORA-27301: Message 27301 not found; No message file for product=RDBMS, facility=ORA; arguments: [Too many open files in system]
ORA-27302: Message 27302 not found; No message file for product=RDBMS, facility=ORA; arguments: [skgxpvaddr1]
Sat Jan 23 03:18:02 2010
Errors in file /u01/product/admin/mxdell/udump/mxdell1_ora_30568.trc:
ORA-00603: Message 603 not found; No message file for product=RDBMS, facility=ORA
ORA-27504: Message 27504 not found; No message file for product=RDBMS, facility=ORA
ORA-27300: Message 27300 not found; No message file for product=RDBMS, facility=ORA; arguments: [ioctl] [23]
ORA-27301: Message 27301 not found; No message file for product=RDBMS, facility=ORA; arguments: [Too many open files in system]
ORA-27302: Message 27302 not found; No message file for product=RDBMS, facility=ORA; arguments: [skgxpvaddr1]
Fri Jan 22 20:33:55 2010
Errors in file /u01/product/admin/mxdell/udump/mxdell1_ora_4613.trc:
ORA-07445: exception encountered: core dump [rpiswu2()+1600] [SIGSEGV] [Address not mapped to object] [0x000000A20] [] []
ORA-00600: internal error code, arguments: [ksnpost:ksnigb], [], [], [], [], [], [], []
ORA-01116: error in opening database file 81
ORA-01110: data file 81: '/ocfs_data/mxdell/wip_data03.dbf'
ORA-27041: unable to open file
Linux-x86_64 Error: 23: Too many open files in system
Additional information: 3
ORA-01116: error in opening database file 112
ORA-01110: data file 112: '/ocfs_index/mxdell/wip_idx06.dbf'
ORA-27041: unable to open file
Linux-x86_64 Error: 23: Too many open files in system
Additional information: 3
ORA-01116: error in opening database file 112
ORA-01110: data file 112: '/ocfs_index/mxdell/wip_idx06.dbf'
ORA-27041: unable to open file
Linux-x86_64 Error: 23: Too many open files in system
Additional information: 3
Fri Jan 22 20:33:55 2010
Trace dumping is performing id=[cdmp_20100122203355]
Fri Jan 22 20:33:55 2010
Trace dumping is performing id=[cdmp_20100122203355]
Fri Jan 22 20:34:31 2010
Trace dumping is performing id=[cdmp_20100122203431]
Sat Jan 23 03:34:37 2010
Errors in file /u01/product/admin/mxdell/udump/mxdell1_ora_5671.trc:
ORA-00603: Message 603 not found; No message file for product=RDBMS, facility=ORA
ORA-27544: Message 27544 not found; No message file for product=RDBMS, facility=ORA
ORA-27300: Message 27300 not found; No message file for product=RDBMS, facility=ORA; arguments: [socket] [23]
ORA-27301: Message 27301 not found; No message file for product=RDBMS, facility=ORA; arguments: [Too many open files in system]
ORA-27302: Message 27302 not found; No message file for product=RDBMS, facility=ORA; arguments: [sskgxpcre1]
Sat Jan 23 03:34:37 2010
Errors in file /u01/product/admin/mxdell/udump/mxdell1_ora_5673.trc:
ORA-00603: Message 603 not found; No message file for product=RDBMS, facility=ORA
ORA-27544: Message 27544 not found; No message file for product=RDBMS, facility=ORA
ORA-27300: Message 27300 not found; No message file for product=RDBMS, facility=ORA; arguments: [socket] [23]
ORA-27301: Message 27301 not found; No message file for product=RDBMS, facility=ORA; arguments: [Too many open files in system]
ORA-27302: Message 27302 not found; No message file for product=RDBMS, facility=ORA; arguments: [sskgxpcre1]
Fri Jan 22 20:34:37 2010
Trace dumping is performing id=[cdmp_20100122203437]
Fri Jan 22 20:35:50 2010
Thread 1 advanced to log sequence 5984 (LGWR switch)
Current log# 26 seq# 5984 mem# 0: /ocfs_ctrl_redo/mxdell/redo26.log
Current log# 26 seq# 5984 mem# 1: /ocfs_data/mxdell/redo26.log
Fri Jan 22 20:42:58 2010
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 192.168.1.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.13.67.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
processes = 1500
sessions = 1655
resource_limit = TRUE
__shared_pool_size = 4060086272
__large_pool_size = 50331648
__java_pool_size = 100663296
__streams_pool_size = 16777216
spfile = /ocfs_data/mxdell/spfilemxdell.ora
sga_target = 21474836480
control_files = /ocfs_data/mxdell/control01.ctl, /ocfs_data/mxdell/control02.ctl, /ocfs_data/mxdell/control03.ctl
db_block_size = 16384
__db_cache_size = 17230200832
compatible = 10.2.0.3.0
log_archive_dest_1 = location=/ocfs_data/mxdell/arch valid_for=(all_logfiles,all_roles) db_unique_name=mxdell
log_archive_dest_2 =
log_archive_dest_state_2 = enable
log_archive_format = %t_%s_%r.arc
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 3
db_recovery_file_dest = /ocfs_data
db_recovery_file_dest_size= 2147483648
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
undo_retention = 3600
remote_login_passwordfile= EXCLUSIVE
db_domain =
------------------------------------
NODE3 ERROR :
Fri Jan 22 20:33:55 2010
Trace dumping is performing id=[cdmp_20100122203355]
Fri Jan 22 20:33:59 2010
Trace dumping is performing id=[cdmp_20100122203355]
Fri Jan 22 20:34:32 2010
Trace dumping is performing id=[cdmp_20100122203431]
Fri Jan 22 20:34:37 2010
Trace dumping is performing id=[cdmp_20100122203437]
Fri Jan 22 20:35:07 2010
Thread 3 advanced to log sequence 4924 (LGWR switch)
Current log# 22 seq# 4924 mem# 0: /ocfs_ctrl_redo/mxdell/redo22.log
Current log# 22 seq# 4924 mem# 1: /ocfs_data/mxdell/redo22.log
Fri Jan 22 20:35:09 2010
Trace dumping is performing id=[cdmp_20100122203509]
Fri Jan 22 20:39:51 2010
Reconfiguration started (old inc 4, new inc 6)
List of nodes:
2 3
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
* domain 0 not valid according to instance 3
Fri Jan 22 20:39:51 2010
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Fri Jan 22 20:39:52 2010
LMS 2: 14 GCS shadows cancelled, 11 closed
Fri Jan 22 20:39:52 2010
LMS 0: 27 GCS shadows cancelled, 13 closed
Fri Jan 22 20:39:52 2010
LMS 5: 40 GCS shadows cancelled, 6 closed
Fri Jan 22 20:39:52 2010
LMS 4: 30 GCS shadows cancelled, 5 closed
Fri Jan 22 20:39:52 2010
LMS 3: 27 GCS shadows cancelled, 11 closed
Fri Jan 22 20:39:52 2010
LMS 1: 16 GCS shadows cancelled, 7 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Fri Jan 22 20:39:52 2010
Instance recovery: looking for dead threads
Fri Jan 22 20:39:52 2010
LMS 5: 23944 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 4: 23825 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 3: 23507 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 2: 23628 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 0: 22673 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 1: 23426 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 5: 24051 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 2: 23083 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 0: 23349 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 4: 24077 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 3: 23872 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 1: 23249 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 5: 23931 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
Fri Jan 22 20:39:52 2010
LMS 4: 23949 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 3: 23467 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 2: 22756 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 1: 23397 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 5: 24238 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 4: 23764 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 0: 22997 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 3: 23590 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 2: 23469 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 5: 232 GCS shadows traversed, 36 replayed
Fri Jan 22 20:39:52 2010
LMS 4: 1402 GCS shadows traversed, 237 replayed
Fri Jan 22 20:39:52 2010
LMS 3: 2504 GCS shadows traversed, 419 replayed
Fri Jan 22 20:39:52 2010
LMS 0: 4065 GCS shadows traversed, 682 replayed
Fri Jan 22 20:39:52 2010
LMS 1: 23162 GCS shadows traversed, 4001 replayed
Fri Jan 22 20:39:52 2010
LMS 2: 4365 GCS shadows traversed, 753 replayed
Fri Jan 22 20:39:52 2010
LMS 1: 4199 GCS shadows traversed, 706 replayed
Fri Jan 22 20:39:52 2010
Fri Jan 22 20:39:52 2010
LMS 2: 4365 GCS shadows traversed, 753 replayed
Fri Jan 22 20:39:52 2010
LMS 1: 4199 GCS shadows traversed, 706 replayed
Fri Jan 22 20:39:52 2010
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
Fri Jan 22 20:39:56 2010
Beginning instance recovery of 1 threads
parallel recovery started with 16 processes
Fri Jan 22 20:39:58 2010
Started redo scan
Fri Jan 22 20:39:58 2010
Completed redo scan
62065 redo blocks read, 7061 data blocks need recovery
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p001_5448.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p008_5462.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p007_5460.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p005_5456.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p002_5450.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p011_5473.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p000_5439.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p004_5454.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p010_5471.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p015_5483.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p013_5479.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p009_5469.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p006_5458.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p014_5481.trc:
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p006_5458.trc:
ORA-01116: error in opening database file 34
ORA-01110: data file 34: '/ocfs_index/mxdell/part_i_ts1_4.dbf'
ORA-27041: unable to open file
Linux-x86_64 Error: 23: Too many open files in system
Additional information: 3
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_p013_5479.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-27544: Failed to map memory region for export
ORA-27300: OS system dependent operation:socket failed with status: 23
ORA-27301: OS failure message: Too many open files in system
ORA-27302: failure occurred at: sskgxpcre1
ORA-01116: error in opening database file 115
ORA-01110: data file 115: '/ocfs_index/mxdell/sn_idx07.dbf'
ORA-27041: unable to open file
Linux-x86_64 Error: 23: Too many open files in system
Additional information: 3
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_smon_12898.trc:
ORA-12801: error signaled in parallel query server P001, instance mxrac03:mxdell3 (3)
ORA-01116: error in opening database file 127
ORA-01110: data file 127: '/ocfs_index/mxdell/wip_idx07.dbf'
ORA-27041: unable to open file
Linux-x8
SMON: terminating instance due to error 12801
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_dbw2_12892.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lms0_12862.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lms3_12874.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lmd0_12860.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lms2_12870.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lms4_12878.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_dbw0_12888.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lms5_12882.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_pmon_12852.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_dbw1_12890.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lmon_12858.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
System state dump is made for local instance
System State dumped to trace file /u01/product/admin/mxdell/bdump/mxdell3_diag_12854.trc
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_j000_14104.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:39:58 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_mman_12886.trc:
Fri Jan 22 20:39:59 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_lgwr_12894.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:40:00 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_q001_13418.trc:
ORA-00604: error occurred at recursive SQL level 2
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:40:00 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_ckpt_12896.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:40:00 2010
Errors in file /u01/product/admin/mxdell/bdump/mxdell3_psp0_12856.trc:
ORA-12801: error signaled in parallel query server
Fri Jan 22 20:40:01 2010
Shutting down instance (abort)
License high water mark = 1035
Fri Jan 22 20:40:11 2010
Instance terminated by SMON, pid = 12898
Fri Jan 22 20:40:16 2010
Instance terminated by USER, pid = 5519
Fri Jan 22 20:40:21 2010
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 192.168.1.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.13.67.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
processes = 1500
sessions = 1655
resource_limit = TRUE
__shared_pool_size = 5200936960
__large_pool_size = 16777216
__java_pool_size = 100663296
--------------------------------------------------------------------
/u01/product/admin/mxdell/bdump/mxdell3_p010_5471.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /u01/product/oracle
System name: Linux
Node name: mxrac03
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
Instance name: mxdell3
Redo thread mounted by this instance: 3
Oracle process number: 912
Unix process pid: 5471, image: oracle@mxrac03 (P010)
*** SERVICE NAME
SYS$BACKGROUND) 2010-01-22 20:39:58.253
*** SESSION ID
1323.8044) 2010-01-22 20:39:58.253
*** 2010-01-22 20:39:58.253
Start recovery for domain 0, valid = 0, flags = 0x1
ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call
Additional information: 3
Additional information: 128
Additional information: 65536
Slave exiting with ORA-1116 exception
Parallel recovery context dump:SERVER
kcrpxflg = 0x13
kcrpxsta = R
kcrpxmbs = 1912
--- Dump of IPC context
kcrpixmb = 0x42eff04b0
kcrpixsta = I
kcrpixof = 672
u.kcrpixcm = 576
ORA-01116: error in opening database file 97
ORA-01110: data file 97: '/ocfs_index/mxdell/part_i_ts2_11.dbf'
ORA-27041: unable to open file
Linux-x86_64 Error: 23: Too many open files in system
Additional information: 3
----- Recovery Hash Table Statistics ---------
Hash table buckets = 65536
Longest hash chain = 1
Average hash chain = 45/45 = 1.0
Max compares per lookup = 0
Avg compares per lookup = 0/45 = 0.0
----------------------------------------------
ksupop : Instance termination detected