AIX6调整系统时间，引发asm报错实例宕机

环境：22.188.20.196 早上8：30左右SIT测试进行系统时间切换，机器时间由2014-9-21切换到了2014-9-30
AIX版本：6100-07-05-1228
ORACLE版本：11.2.0.3.0
现象：早上8：30左右项目组反映数据库实例重启了，导致应用报错

日志信息：
完整日志：

其中摘取的部分日志信息
ASM中alert日志
Sat Sep 20 23:59:00 2014
Time drift detected. Please check VKTM trace file for more details.
Sun Sep 21 01:10:00 2014
Time drift detected. Please check VKTM trace file for more details.
Sun Sep 21 04:15:00 2014
Time drift detected. Please check VKTM trace file for more details.
Sun Sep 21 05:30:00 2014
Time drift detected. Please check VKTM trace file for more details.
Sun Sep 21 07:25:00 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Sep 30 08:29:45 2014
WARNING: client [orau11g:orau11g] not responsive for 777586s; state=0x1. killing pid 38076510
Tue Sep 30 08:29:57 2014
Starting background process ASMB
Tue Sep 30 08:29:57 2014
ASMB started with pid=19, OS id=31129644
Tue Sep 30 08:29:57 2014
NOTE: client +ASM:+ASM registered, osid 41549842, mbr 0x0
Tue Sep 30 08:30:01 2014
NOTE: client orau11g:orau11g registered, osid 40828940, mbr 0x1
Tue Sep 30 08:35:02 2014
NOTE: ASMB process exiting due to lack of ASM file activity for 305 seconds
Tue Sep 30 08:50:00 2014
Time drift detected. Please check VKTM trace file for more details.

orau1g中alert日志
Sun Sep 21 07:11:47 2014
Thread 1 advanced to log sequence 1367 (LGWR switch)
  Current log# 2 seq# 1367 mem# 0: +DATA1/orau11g/onlinelog/group_2.262.818505661
Sun Sep 21 07:45:00 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Sep 30 08:29:45 2014
Closing scheduler window
Closing Resource Manager plan via scheduler window
Clearing Resource Manager plan via parameter
Tue Sep 30 08:29:46 2014
NOTE: ASMB terminating
Errors in file /oracle/diag/rdbms/orau11g/orau11g/trace/orau11g_asmb_40960218.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 1153 Serial number: 51
Errors in file /oracle/diag/rdbms/orau11g/orau11g/trace/orau11g_asmb_40960218.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 1153 Serial number: 51
ASMB (ospid: 40960218): terminating the instance due to error 15064
Tue Sep 30 08:29:46 2014
opiodr aborting process unknown ospid (32309392) as a result of ORA-1092
Tue Sep 30 08:29:46 2014
opiodr aborting process unknown ospid (38535384) as a result of ORA-1092
Tue Sep 30 08:29:47 2014
System state dump requested by (instance=1, osid=40960218 (ASMB)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/diag/rdbms/orau11g/orau11g/trace/orau11g_diag_41549836.trc
Tue Sep 30 08:29:47 2014
ORA-1092 : opitsk aborting process
Tue Sep 30 08:29:47 2014
ORA-1092 : opitsk aborting process
Tue Sep 30 08:29:47 2014
License high water mark = 143
Instance terminated by ASMB, pid = 40960218
USER (ospid: 36372532): terminating the instance
Instance terminated by USER, pid = 36372532
Tue Sep 30 08:29:57 2014
Adjusting the default value of parameter parallel_max_servers
from 960 to 585 due to the value of parameter processes (600)
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /oracle/product/11g/rdbms/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =102
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
ORACLE_HOME = /oracle/product/11g/rdbms
System name: AIX
Node name: ZHS3BB306
Release: 1
Version: 6

问题分析
可能！理论上vktm就是Oracle所有进程的时间服务器。应该隔离系统date变化的。可能是在特定时点触发了这个问题。于是报SR。

Oracle回复如下：
这个问题只发生在当Oracle需要释放一些shared pool的LRU list里的空闲块的时候，由于两个chunk会使用到一个相同的duration time ，当2个块之间释放的时间超时的时候才会出现您的问题。
这个发生的几率确实不是100%的。
Oracle已经把这个问题定位成了一个bug : Bug 13914613 - DATABASE CRASHED DUE TO ORA-240 AND ORA-15064
如果您的时间调整的机制不能修改，为了避免这个问题您可以做以下操作来避免该问题：
1.升级数据库版本到12.1 或者11.2.0.3.6 ，这个问题在12.1及11.2.0.3.6 上已经做了修复；
2.您可以在11.2.0.3.5的基础上打补丁13914613 ，这个补丁您可以通过以下链接下载到：
https://updates.oracle.com/download/13914613.html
3.我们还可以通过隐含参数来屏蔽这个问题
_enable_shared_pool_durations=false

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/15711267/viewspace-1082896/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/15711267/viewspace-1082896/