数据库实例节点重启问题分析:

最新推荐文章于 2022-05-19 11:59:20 发布

1ihj2015

最新推荐文章于 2022-05-19 11:59:20 发布

阅读量2.7k

点赞数

分类专栏： Oracle Troubleshooting

本文链接：https://blog.csdn.net/itdba/article/details/38441231

版权

Oracle Troubleshooting 专栏收录该内容

37 篇文章 0 订阅

订阅专栏

alertoa-db2.log
2014-07-13 02:29:09.132
[cssd(2752638)]CRS-1612:Network communication with node oa-db1 (1) missing for 50% of timeout interval. Removal of this node from cluster in 14.443 seconds
2014-07-13 02:29:09.723
[cssd(2752638)]CRS-1604:CSSD voting file is offline: /dev/asmdisk5; details at (:CSSNM00058:) in /oracle/product/11.2.0/grid11g/log/oa-db2/cssd/ocssd.log.
2014-07-13 02:29:09.724
[cssd(2752638)]CRS-1606:The number of voting files available, 0, is less than the minimum number of voting files required, 1, resulting in CSSD termination
to ensure data integrity; details at (:CSSNM00018:) in /oracle/product/11.2.0/grid11g/log/oa-db2/cssd/ocssd.log
2014-07-13 02:29:09.724
[cssd(2752638)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /oracle/product/11.2.0/grid11g/log/oa-db2/cssd/ocss
d.

--磁盘心跳出现问题：
2014-07-13 02:29:05.193: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2014-07-13 02:29:09.133: [ CSSD][3605]clssnmPollingThread: node oa-db1 (1) at 50% heartbeat fatal, removal in 14.443 seconds
2014-07-13 02:29:09.133: [ CSSD][3605]clssnmPollingThread: node oa-db1 (1) is impending reconfig, flag 2229260, misstime 15557
2014-07-13 02:29:09.133: [ CSSD][3605]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status
(1)
2014-07-13 02:29:09.210: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2014-07-13 02:29:09.210: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2014-07-13 02:29:09.723: [ CSSD][2834](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 32528 ms for voting file /dev/asmdisk5)
2014-07-13 02:29:09.724: [ CSSD][2834]clssnmvDiskAvailabilityChange: voting file /dev/asmdisk5 now offline
2014-07-13 02:29:09.724: [ CSSD][2834](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2014-07-13 02:29:09.724: [ CSSD][2834]###################################
2014-07-13 02:29:09.724: [ CSSD][2834]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2014-07-13 02:29:09.724: [ CSSD][2834]###################################
2014-07-13 02:29:09.724: [ CSSD][2834](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2014-07-13 02:29:09.724: [ CSSD][2834]

----- Call Stack Trace -----
2014-07-13 02:29:09.728: [ CSSD][2834]calling call entry argument values in hex
2014-07-13 02:29:09.728: [ CSSD][2834]location type point (? means dubious value)
2014-07-13 02:29:09.728: [ CSSD][2834]-------------------- -------- -------------------- ----------------------------
2014-07-13 02:29:09.729: [ CSSD][2834]clssscExit()+708 bl 10006ddd4 67800000678 ? 1000F3828 ?
2014-07-13 02:29:09.729: [ CSSD][2834] 11008F270 ? 900000000AB3B94 ?
2014-07-13 02:29:09.729: [ CSSD][2834] 000000000 ? 01105008D ?
2014-07-13 02:29:09.729: [ CSSD][2834] 01105008D ? 000000000 ?
2014-07-13 02:29:09.729: [ CSSD][2834]clssnmvDiskCheck()+ call clssscExit() 1115D74F0 ? 200000002 ?
2014-07-13 02:29:09.729: [ CSSD][2834]2744 000000001 ? 1000FDD98 ?
2014-07-13 02:29:09.729: [ CSSD][2834] 11008F270 ? 01105008D ?
2014-07-13 02:29:09.729: [ CSSD][2834] 01105008D ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834]clssnmvDiskPingMoni call clssnmvDiskCheck() 1115D74F0 ? 1111AF850 ?
2014-07-13 02:29:09.730: [ CSSD][2834]torThread()+416 112584680 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834] 01105008D ? 0137F0440 ?
2014-07-13 02:29:09.730: [ CSSD][2834]clssscthrdmain()+20 call clssnmvDiskPingMoni 1115D74F0 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834]8 torThread() 000000000 ? 000000000 ?
2014-07-13 02:29:09.730: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834]_pthread_body()+240 call clssscthrdmain() 1115D74F0 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834]fffffffffffffffc call _pthread_body() 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.731: [ CSSD][2834] 000000000 ? 000000000 ?
2014-07-13 02:29:09.732: [ CSSD][2834] 000000000 ? 000000000 ?

[oracle@oa2 /home/oracle]$errpt -aj A924A5FC|more
---------------------------------------------------------------------------
LABEL: CORE_DUMP
IDENTIFIER: A924A5FC

Date/Time: Sun Jul 13 02:29:22 GMT+08:00 2014
Sequence Number: 35060
Machine Id: 00C7BA7D4C00
Node Id: OA-DB2
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSPROC

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SOFTWARE PROGRAM

User Causes
USER GENERATED SIGNAL

Recommended Actions
CORRECT THEN RETRY

Failure Causes
SOFTWARE PROGRAM

Recommended Actions
RERUN THE APPLICATION PROGRAM
IF PROBLEM PERSISTS THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
6
USER'S PROCESS ID:
2752638
FILE SYSTEM SERIAL NUMBER
13
INODE NUMBER
0 12354
CORE FILE NAME
/oracle/product/11.2.0/grid11g/log/oa-db2/cssd/core
PROGRAM NAME
ocssd.bin
STACK EXECUTION DISABLED

[oracle@oa2 /oracle/product/11.2.0/grid11g/log/oa-db2/cssd]$pwd

/oracle/product/11.2.0/grid11g/log/oa-db2/cssd/core 需要分析 /dev/asmdisk5 offline原因

由于权限原因无法查询asm运行日志，不过猜测应该没问题。

CSSD 磁盘心跳异常引起crs重启！