linux vcs oracle启动失败,VCS引起的oracle数据库异常重新启动一例

PAN YAO

于 2021-05-07 23:48:57 发布

阅读量386

点赞数

文章标签： linux vcs oracle启动失败

1. 环境描写叙述

操作系统版本号：SUSE Linux Enterprise Server 10 sp2 (x86_64)

数据库版本号：Oracle 11.1.0.7.16

VCS版本号：5.1

2. 问题现象及分析

凌晨3:46左右。手机短信收到数据库异常告警。

(1)查看数据库alert日志

Sun Nov 16 03:46:51 2014

Stopping background process SMCO

Stopping background process FBDA

Shutting down instance: further logons disabled

Sun Nov 16 03:46:54 2014

kkjcre1p: unable to spawn jobq slave process, slot 0, error 1089

Stopping background process CJQ0

Stopping background process QMNC

Stopping background process MMNL

Stopping background process MMON

Sun Nov 16 03:47:03 2014

Shutting down instance (immediate)

License high water mark = 366

Sun Nov 16 03:51:52 2014

License high water mark = 366

USER (ospid: 8654): terminating the instance

Sun Nov 16 03:51:52 2014

ORA-1092 : opiodr aborting process unknown ospid (28656_47367034883680)

Sun Nov 16 03:51:52 2014

LGWR waiting for instance termination

并未发现较为明显的异常原因。

同一时候也查看了那个时间段的报告，发现数据库压力不大。也没有什么异常的SQL存在。

(2)检查OS资源使用

MyJxyBZX5n

watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvSFdfTGlCbw==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="">

系统资源使用正常。IO也不高。

(3)查看messages日志

Nov 16 03:42:46 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13027 Thread(4149214112) Resource(ora_vg) - monitor procedure did not complete within the expected time.

Nov 16 03:42:46 boscodb Had[10258]: VCS ERROR V-16-2-13027 (boscodb) Resource(ora_vg) - monitor procedure did not complete within the expected time.

Nov 16 03:43:01 boscodb su: (to root) root on none

……

Nov 16 03:45:39 boscodb su: (to oracle) root on none

Nov 16 03:45:47 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13210 Thread(4154813344) Agent is calling clean for resource(ora_vg) because 4 successive invocations of the monitor procedure did not complete within the expected time.

Nov 16 03:45:47 boscodb Had[10258]: VCS ERROR V-16-2-13210 (boscodb) Agent is calling clean for resource(ora_vg) because 4 successive invocations of the monitor procedure did not complete within the expected time.

……

Nov 16 03:46:48 boscodb AgentFramework[10894]: VCS ERROR V-16-2-13006 Thread(4153760672) Resource(ora_vg): clean procedure did not complete within the expected time.

Nov 16 03:46:48 boscodb Had[10258]: VCS ERROR V-16-2-13006 (boscodb) Resource(ora_vg): clean procedure did not complete within the expected time.

Nov 16 03:46:49 boscodb AgentFramework[10899]: VCS ERROR V-16-2-13067 Thread(4155194272) Agent is calling clean for resource(ora_archive) because the resource became OFFLINE unexpectedly, on its own.

Nov 16 03:46:50 boscodb Had[10258]: VCS ERROR V-16-2-13067 (boscodb) Agent is calling clean for resource(ora_archive) because the resource became OFFLINE unexpectedly, on its own.

Nov 16 03:46:51 boscodb AgentFramework[10899]: VCS ERROR V-16-2-13068 Thread(4155194272) Resource(ora_archive) - clean completed successfully.

原来是VCS出现了错误，监控vg超时了，这些错误信息在VCS的日志中也有记录。

3. 解决的方法

检查了监控vg的moniter脚本。发现也没有异常，并且脚本在其它库，压力都比較大的都不曾出问题。

当前的配置为：

Application ora_vg (

StartProgram = "/opt/BOSCO_ora/bin/vg_start.sh"

StopProgram = "/opt/BOSCO_ora/bin/vg_stop.sh"

CleanProgram = "/opt/BOSCO_ora/bin/vg_clean.sh"

MonitorProgram = "/opt/BOSCO_ora/bin/vg_monitor.sh"

MonitorInterval = 30

MonitorTimeout = 30