报告概述
2014年9月4号7点40分左右，客户收到数据库告警，物理内存使用率达到99%左右，且数据库已经hang住，无法通过sqlplus登陆， 后通过添加参数-prelium进入数据库，使用oradebug生成systemstate trace文件，客户为了减少对于业务的影响，在操作系统层面直接杀掉了一些数据库会话，然后通过HA环境切换到其它的主机。
事后与我方联系，希望全面诊断分析为何数据库产生hang，以期后期作出有效的安排，消减对于业务的影响。经我司工程师与客户方工程师沟通，了解业务应用运行场景，分析故障时段的相关日志，可以定位是因为ORACLE bug引发。详见下述内容。
故障现象
故障原因分析
检查alert数据库告警日志，2014-09-04 01:02:38换页换出活动比较严重
Thu Sep 04 01:02:38 2014
WARNING: Heavy swapping observed on system in last 5 mins.
pct of memory swapped in [7.09%] pct of memory swapped out [10.88%].
Please make sure there is no memory pressure and the SGA and PGA
are configured correctly. Look at DBRM trace file for more details.
查看dbrm进程产生的对应时间点的trc文件
从标红的dbrm进程的trc文件，产生换页换出活动明细如下：
total swpin [58.47%][38017708K], total swpout [58.77%][38208639K]total swpin [62.53%][40653120K], total swpout [62.76%][40804572K]total swpin [81.93%][53266399K], total swpout [82.19%][53438888K]total swpin [48.39%][31460400K], total swpout [48.66%][31636188K]total swpin [54.85%][35659752K], total swpout [59.35%][38587500K]total swpin [100.01%][65021115K], total swpout [104.51%][67950187K]total swpin [ 9.56%][6217944K], total swpout [15.83%][10297296K]total swpin [17.63%][11466475K], total swpout [38.34%][24929148K]total swpin [17.87%][11617927K], total swpout [40.06%][26048475K]total swpin [24.84%][16150484K], total swpout [41.24%][26815548K]total swpin [27.23%][17704927K], total swpout [44.07%][28656408K]total swpin [35.51%][23087827K], total swpout [44.06%][28644416K]total swpin [ 3.34%][2175739K], total swpout [ 7.36%][4786932K]total swpin [ 8.80%][5721636K], total swpout [14.87%][9672763K]total swpin [ 9.15%][5954107K], total swpout [18.41%][11974656K]total swpin [ 8.21%][5342696K], total swpout [18.08%][11756779K]total swpin [ 8.04%][5228895K], total swpout [17.86%][11612688K]total swpin [ 9.32%][6059396K], total swpout [20.06%][13043300K]total swpin [ 7.97%][5187559K], total swpout [19.77%][12854515K]total swpin [ 8.00%][5207284K], total swpout [20.62%][13405752K]total swpin [ 7.95%][5169660K], total swpout [21.22%][13796160K]total swpin [ 8.01%][5208399K], total swpout [21.24%][13814424K]total swpin [ 8.04%][5228000K], total swpout [22.58%][14683368K]total swpin [ 7.98%][5192619K], total swpout [24.08%][15655620K]total swpin [ 8.06%][5240836K], total swpout [25.41%][16524568K]total swpin [ 9.44%][6142620K], total swpout [28.23%][18357684K]total swpin [ 8.20%][5336352K], total swpout [27.78%][18061812K]total swpin [ 9.06%][5890267K], total swpout [28.58%][18582559K]total swpin [ 3.32%][2164588K], total swpout [21.08%][13705216K]total swpin [ 2.64%][1720075K], total swpout [18.95%][12323887K]total swpin [ 2.64%][1716824K], total swpout [18.63%][12115927K]total swpin [ 3.88%][2527308K], total swpout [19.55%][12715412K]total swpin [ 2.63%][1712484K], total swpout [18.82%][12240727K]total swpin [ 2.62%][1707147K], total swpout [18.26%][11876140K]total swpin [ 2.66%][1732372K], total swpout [16.88%][10976643K]total swpin [ 2.42%][1576764K], total swpout [ 7.92%][5152440K]total swpin [ 2.62%][1705567K], total swpout [ 7.74%][5033563K]total swpin [ 2.46%][1601160K], total swpout [ 6.86%][4465696K]total swpin [ 2.41%][1572772K], total swpout [ 6.97%][4536284K]total swpin [ 2.50%][1631620K], total swpout [ 7.05%][4586707K]total swpin [ 2.57%][1675296K], total swpout [ 7.13%][4637684K]total swpin [ 2.43%][1582743K], total swpout [ 7.08%][4603996K]total swpin [ 2.47%][1611916K], total swpout [ 7.02%][4567280K]total swpin [ 2.71%][1767404K], total swpout [ 7.10%][4616200K]total swpin [ 2.38%][1550779K], total swpout [ 5.78%][3764071K]total swpin [ 2.48%][1616448K], total swpout [ 6.10%][3969895K]total swpin [ 2.57%][1673640K], total swpout [ 6.35%][4132444K]total swpin [ 2.55%][1657951K], total swpout [ 6.24%][4062136K]total swpin [ 2.68%][1743072K], total swpout [ 6.24%][4062200K]total swpin [ 2.55%][1657807K], total swpout [ 5.12%][3328888K]total swpin [ 2.65%][1724155K], total swpout [ 5.14%][3346084K]total swpin [ 2.53%][1648572K], total swpout [ 4.63%][3010800K]total swpin [ 2.68%][1742712K], total swpout [ 4.20%][2734971K]total swpin [ 2.61%][1697656K], total swpout [ 3.57%][2323956K]total swpin [ 3.06%][1992624K], total swpout [ 0.03%][24096K]total swpin [51.63%][33570871K], total swpout [51.93%][33763540K]total swpin [51.75%][33645988K], total swpout [51.90%][33743743K]total swpin [52.89%][34385775K], total swpout [53.15%][34556400K]total swpin [14.18%][9220351K], total swpout [14.43%][9386676K]total swpin [35.51%][23089704K], total swpout [35.80%][23275395K]total swpin [ 3.12%][2029576K], total swpout [ 4.95%][3219315K]total swpin [10.31%][6703920K], total swpout [23.34%][15175676K]total swpin [12.85%][8356023K], total swpout [23.07%][15003048K]total swpin [ 8.17%][5317844K], total swpout [16.82%][10938912K]total swpin [12.23%][7953403K], total swpout [21.68%][14095839K]total swpin [13.28%][8637604K], total swpout [25.13%][16338400K]total swpin [ 4.38%][2849808K], total swpout [ 5.62%][3658176K]total swpin [ 7.19%][4676559K], total swpout [10.61%][6897919K]total swpin [ 6.47%][4206584K], total swpout [ 9.67%][6291619K]total swpin [ 7.17%][4664044K], total swpout [10.02%][6516264K]total swpin [ 4.13%][2690720K], total swpout [ 0.07%][51664K]total swpin [ 2.67%][1737403K], total swpout [ 0.16%][108835K]total swpin [ 7.22%][4699988K], total swpout [10.87%][7070427K]total swpin [ 5.88%][3829160K], total swpout [10.34%][6722767K]total swpin [ 5.46%][3549771K], total swpout [ 8.60%][5593924K]total swpin [ 7.28%][4735540K], total swpout [10.73%][6980047K]total swpin [ 7.09%][4610592K], total swpout [10.88%][7074692K]
由上可见，共计产生77次换页换出活动。经查阅oracle mos,发现如下文章：
WARNING: Heavy Swapping Observed On System In Last 5 Mins.After upgrade to 11.2.0.3 [ID 1452790.1]
综上所述，产生操作系统换面换出活动，只是提示性信息，不会对数据库产生根本性的影响，可以忽略不计。
04:00:47左右触发ora-07445错误
Thu Sep 04 04:00:47 2014
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x642028544E53206E] [PC:0x10123F5BC, kglic0()+764] [flags: 0x0, count: 1]
Errors in file /home/oracle/app/oracle/diag/rdbms/satisfaction/Satisfaction/trace/Satisfaction_m000_18939952.trc (incident=101489):
ORA-07445: exception encountered: core dump [kglic0()+764] [SIGSEGV] [ADDR:0x642028544E53206E] [PC:0x10123F5BC] [Address not mapped to object] []
Incident details in: /home/oracle/app/oracle/diag/rdbms/satisfaction/Satisfaction/incident/incdir_101489/Satisfaction_m000_18939952_i101489.trc
Use ADRCI or Support Workbench to package the incident.
09:09:40手工生成3次system state trace
Thu Sep 04 09:09:40 2014
System State dumped to trace file /home/oracle/app/oracle/diag/rdbms/satisfaction/Satisfaction/trace/Satisfaction_ora_37158974.trc
Thu Sep 04 09:10:31 2014
System State dumped to trace file /home/oracle/app/oracle/diag/rdbms/satisfaction/Satisfaction/trace/Satisfaction_ora_27525134.trc
Thu Sep 04 09:20:18 2014
System State dumped to trace file /home/oracle/app/oracle/diag/rdbms/satisfaction/Satisfaction/trace/Satisfaction_ora_12976360.trc
Thu Sep 04 09:21:13 2014
System State dumped to trace file /home/oracle/app/oracle/diag/rdbms/satisfaction/Satisfaction/trace/Satisfaction_ora_57344250.trc
10:23:29 W006进程出现故障
Thu Sep 04 10:23:39 2014
Errors in file /home/oracle/app/oracle/diag/rdbms/satisfaction/Satisfaction/trace/Satisfaction_smco_44499044.trc (incident=101433):
ORA-00445: background process "W006" did not start after 120 seconds
Incident details in: /home/oracle/app/oracle/diag/rdbms/satisfaction/Satisfaction/incident/incdir_101433/Satisfaction_smco_44499044_i101433.trc
Thu Sep 04 10:23:42 2014
Dumping diagnostic data in directory=[cdmp_20140904102342], requested by (instance=1, osid=44499044 (SMCO)), summary=[incident=101433].
10:24:39 由于474错误（注：smon进程异常中断）pmon强制中止了数据库实例
Thu Sep 04 10:24:39 2014
PMON (ospid: 62652482): terminating the instance due to error 474
Thu Sep 04 10:24:39 2014
System state dump requested by (instance=1, osid=62652482 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /home/oracle/app/oracle/diag/rdbms/satisfaction/Satisfaction/trace/cdmp_20140904102342/Satisfaction_ora_21102802_bucket.trc
Dumping diagnostic data in directory=[cdmp_20140904102443], requested by (instance=1, osid=62652482 (PMON)), summary=[abnormal instance termination].
Instance terminated by PMON, pid = 62652482
下面我们对数据库hang时段的system state trace文件进行分析：
提取上述3个system state trace生成时间最早的文件/home/oracle/app/oracle/diag/rdbms/satisfaction/Satisfaction/trace/Satisfaction_ora_37158974.trc进行诊断。
System state trace文件内容主要包括内容如下：
l trace文件包括数据库进程信息，包括前台及后台进程
l 进程包括会话信息
l 会话包含阻塞会话信息及被阻塞会话信息
l 进程及会话会创建属于们的状态对象信息，状态对象为适于oracle kernel高效管理数据库资源的一种内存结构体。
通过ultraedit软件在上述的trc文件，查找’PROCESS ‘关键字,获取结果如下，包括数据库后台进程和用户进程,共计631个进程
OSD pid info: Unix process pid: 0, image: PSEUDO
OSD pid info: Unix process pid: 62652482, image: oracle@720db2 (PMON)
OSD pid info: Unix process pid: 61145206, image: oracle@720db2 (PSP0)
OSD pid info: Unix process pid: 61472784, image: oracle@720db2 (VKTM)
OSD pid info: Unix process pid: 63176776, image: oracle@720db2 (GEN0)
OSD pid info: Unix process pid: 57933988, image: oracle@720db2 (DIAG)
OSD pid info: Unix process pid: 60883088, image: oracle@720db2 (DBRM)
OSD pid info: Unix process pid: 62718034, image: oracle@720db2 (DIA0)
OSD pid info: Unix process pid: 43319514, image: oracle@720db2 (MMAN)
OSD pid info: Unix process pid: 63438870, image: oracle@720db2 (DBW0)
OSD pid info: Unix process pid: 64094244, image: oracle@720db2 (DBW1)
OSD pid info: Unix process pid: 63766714, image: oracle@720db2 (DBW2)
OSD pid info: Unix process pid: 66256994, image: oracle@720db2 (LGWR)
OSD pid info: Unix process pid: 63242404, image: oracle@720db2 (CKPT)
OSD pid info: Unix process pid: 64487578, image: oracle@720db2 (SMON)
OSD pid info: Unix process pid: 55836728, image: oracle@720db2 (RECO)
OSD pid info: Unix process pid: 63307970, image: oracle@720db2 (MMON)
OSD pid info: Unix process pid: 62849044, image: oracle@720db2 (MMNL)
OSD pid info: Unix process pid: 63570012, image: oracle@720db2 (D000)
OSD pid info: Unix process pid: 62455936, image: oracle@720db2 (S000)
OSD pid info: Unix process pid: 38011030, image: oracle@720db2
OSD pid info: Unix process pid: 60555360, image: oracle@720db2 (ARC0)
OSD pid info: Unix process pid: 46006304, image: oracle@720db2 (ARC1)
OSD pid info: Unix process pid: 917876, image: oracle@720db2 (ARC2)
OSD pid info: Unix process pid: 62390396, image: oracle@720db2 (ARC3)
OSD pid info: Unix process pid: 36700176, image: oracle@720db2
OSD pid info: Unix process pid: 63111358, image: oracle@720db2 (QMNC)
OSD pid info: Unix process pid: 61604020, image: oracle@720db2 (CJQ0)
OSD pid info: Unix process pid: 26804452, image: oracle@720db2
OSD pid info: Unix process pid: 49086710, image: oracle@720db2
OSD pid info: Unix process pid: 18939912, image: oracle@720db2
OSD pid info: Unix process pid: 52953292, image: oracle@720db2
OSD pid info: Unix process pid: 46792858, image: oracle@720db2 (Q000)
OSD pid info: Unix process pid: 61997182, image: oracle@720db2 (Q001)
OSD pid info: Unix process pid: 44499044, image: oracle@720db2 (SMCO)
OSD pid info: Unix process pid: 39190722, image: oracle@720db2 (J000)
OSD pid info: Unix process pid: 28049450, image: oracle@720db2
OSD pid info: Unix process pid: 61538404, image: oracle@720db2
OSD pid info: Unix process pid: 23920750, image: oracle@720db2
OSD pid info: Unix process pid: 42664146, image: oracle@720db2
OSD pid info: Unix process pid: 17825882, image: oracle@720db2 (J001)
OSD pid info: Unix process pid: 36831360, image: oracle@720db2 (M000)
OSD pid info: Unix process pid: 31981778, image: oracle@720db2 (W005)
OSD pid info: Unix process pid: 14155838, image: oracle@720db2
OSD pid info: Unix process pid: 62587048, image: oracle@720db2 (W001)
OSD pid info: Unix process pid: 60489868, image: oracle@720db2
OSD pid info: Unix process pid: 52428844, image: oracle@720db2
OSD pid info: Unix process pid: 50724972, image: oracle@720db2 (J002)
OSD pid info: Unix process pid: 20119676, image: oracle@720db2
OSD pid info: Unix process pid: 42860562, image: oracle@720db2
OSD pid info: Unix process pid: 18284752, image: oracle@720db2
OSD pid info: Unix process pid: 54001806, image: oracle@720db2 (TNS V1-V3)
OSD pid info: Unix process pid: 21495908, image: oracle@720db2 (TNS V1-V3)
OSD pid info: Unix process pid: 18350330, image: oracle@720db2 (TNS V1-V3)
OSD pid info: Unix process pid: 55116026, image: oracle@720db2 (TNS V1-V3)
OSD pid info: Unix process pid: 56230066, image: oracle@720db2 (TNS V1-V3)
OSD pid info: Unix process pid: 27394114, image: oracle@720db2 (TNS V1-V3)
OSD pid info: Unix process pid: 42795208, image: oracle@720db2 (TNS V1-V3)
OSD pid info: Unix process pid: 52822056, image: oracle@720db2 (TNS V1-V3)
OSD pid info: Unix process pid: 0, image: PSEUDO
在trc文件查找waited for,查看等待事件的分布情况,可见主要是等待ibrary cache: mutex X
0: waited for 'pmon timer'
1: waited for 'pmon timer'
2: waited for 'pmon timer'
3: waited for 'pmon timer'
4: waited for 'pmon timer'
5: waited for 'pmon timer'
6: waited for 'pmon timer'
7: waited for 'pmon timer'
8: waited for 'pmon timer'
9: waited for 'pmon timer'
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
waited for 'latch free', seq_num: 49172
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
0: waited for 'DIAG idle wait'
1: waited for 'DIAG idle wait'
2: waited for 'DIAG idle wait'
3: waited for 'DIAG idle wait'
4: waited for 'DIAG idle wait'
5: waited for 'DIAG idle wait'
6: waited for 'DIAG idle wait'
7: waited for 'DIAG idle wait'
8: waited for 'DIAG idle wait'
9: waited for 'DIAG idle wait'
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
0: waited for 'DIAG idle wait'
1: waited for 'DIAG idle wait'
2: waited for 'DIAG idle wait'
3: waited for 'DIAG idle wait'
4: waited for 'DIAG idle wait'
5: waited for 'DIAG idle wait'
6: waited for 'DIAG idle wait'
7: waited for 'DIAG idle wait'
8: waited for 'DIAG idle wait'
9: waited for 'DIAG idle wait'
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
0: waited for 'control file parallel write'
1: waited for 'rdbms ipc message'
2: waited for 'control file parallel write'
3: waited for 'rdbms ipc message'
4: waited for 'control file parallel write'
5: waited for 'control file sequential read'
6: waited for 'control file sequential read'
7: waited for 'control file sequential read'
8: waited for 'control file sequential read'
9: waited for 'rdbms ipc message'
0: waited for 'smon timer'
1: waited for 'smon timer'
2: waited for 'smon timer'
3: waited for 'smon timer'
4: waited for 'db file sequential read'
5: waited for 'smon timer'
6: waited for 'smon timer'
7: waited for 'smon timer'
8: waited for 'smon timer'
9: waited for 'smon timer'
waited for 'latch: shared pool', seq_num: 12582
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
waited for 'latch: shared pool', seq_num: 65449
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'SQL*Net message from client'
waited for 'library cache: mutex X', seq_num: 82
0: waited for 'control file sequential read'
1: waited for 'control file sequential read'
2: waited for 'control file sequential read'
3: waited for 'control file sequential read'
4: waited for 'control file sequential read'
5: waited for 'control file sequential read'
6: waited for 'control file sequential read'
7: waited for 'rdbms ipc message'
8: waited for 'control file sequential read'
9: waited for 'control file sequential read'
0: waited for 'control file sequential read'
1: waited for 'control file sequential read'
2: waited for 'control file sequential read'
3: waited for 'control file sequential read'
4: waited for 'control file sequential read'
5: waited for 'control file sequential read'
6: waited for 'control file sequential read'
7: waited for 'rdbms ipc message'
8: waited for 'control file sequential read'
9: waited for 'control file sequential read'
0: waited for 'control file sequential read'
1: waited for 'control file sequential read'
2: waited for 'control file sequential read'
3: waited for 'control file sequential read'
4: waited for 'control file sequential read'
5: waited for 'control file sequential read'
6: waited for 'control file sequential read'
7: waited for 'control file sequential read'
8: waited for 'control file sequential read'
9: waited for 'control file sequential read'
0: waited for 'control file sequential read'
1: waited for 'control file sequential read'
2: waited for 'control file sequential read'
3: waited for 'control file sequential read'
4: waited for 'control file sequential read'
5: waited for 'control file sequential read'
6: waited for 'control file sequential read'
7: waited for 'rdbms ipc message'
8: waited for 'control file sequential read'
9: waited for 'control file sequential read'
0: waited for 'SQL*Net break/reset to client'
1: waited for 'SQL*Net break/reset to client'
2: waited for 'SQL*Net break/reset to client'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'SQL*Net message from client'
6: waited for 'SQL*Net message to client'
7: waited for 'SQL*Net message from client'
8: waited for 'SQL*Net message to client'
9: waited for 'SQL*Net message from client'
waited for 'latch: shared pool', seq_num: 518
0: waited for 'Streams AQ: qmn coordinator idle wait'
1: waited for 'Streams AQ: qmn coordinator idle wait'
2: waited for 'Streams AQ: qmn coordinator idle wait'
3: waited for 'Streams AQ: qmn coordinator idle wait'
4: waited for 'Streams AQ: qmn coordinator idle wait'
5: waited for 'Streams AQ: qmn coordinator idle wait'
6: waited for 'Streams AQ: qmn coordinator idle wait'
7: waited for 'Streams AQ: qmn coordinator idle wait'
8: waited for 'Streams AQ: qmn coordinator idle wait'
9: waited for 'Streams AQ: qmn coordinator idle wait'
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'os thread startup'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
waited for 'latch: shared pool', seq_num: 58662
0: waited for 'SQL*Net message from client'
1: waited for 'SQL*Net message to client'
2: waited for 'SQL*Net message from client'
3: waited for 'SQL*Net message to client'
4: waited for 'SQL*Net message from client'
5: waited for 'SQL*Net message to client'
6: waited for 'SQL*Net message from client'
7: waited for 'SQL*Net message to client'
8: waited for 'SQL*Net message from client'
9: waited for 'SQL*Net message to client'
waited for 'latch: shared pool', seq_num: 81
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 138
0: waited for 'SQL*Net message from client'
1: waited for 'SQL*Net message to client'
2: waited for 'SQL*Net message from client'
3: waited for 'SQL*Net message to client'
4: waited for 'SQL*Net message from client'
5: waited for 'SQL*Net message to client'
6: waited for 'SQL*Net message from client'
7: waited for 'SQL*Net message to client'
8: waited for 'SQL*Net message from client'
9: waited for 'SQL*Net message to client'
waited for 'latch: shared pool', seq_num: 73
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'SQL*Net message from client'
waited for 'library cache: mutex X', seq_num: 24
0: waited for 'Streams AQ: qmn slave idle wait'
1: waited for 'Streams AQ: qmn slave idle wait'
2: waited for 'Streams AQ: qmn slave idle wait
3: waited for 'Streams AQ: qmn slave idle wait'
4: waited for 'Streams AQ: qmn slave idle wait'
5: waited for 'Streams AQ: qmn slave idle wait'
6: waited for 'Streams AQ: qmn slave idle wait'
7: waited for 'Streams AQ: qmn slave idle wait'
8: waited for 'Streams AQ: qmn slave idle wait'
9: waited for 'Streams AQ: qmn slave idle wait'
0: waited for 'class slave wait'
0: waited for 'rdbms ipc message'
1: waited for 'rdbms ipc message'
2: waited for 'rdbms ipc message'
3: waited for 'rdbms ipc message'
4: waited for 'rdbms ipc message'
5: waited for 'rdbms ipc message'
6: waited for 'rdbms ipc message'
7: waited for 'rdbms ipc message'
8: waited for 'rdbms ipc message'
9: waited for 'rdbms ipc message'
waited for 'latch: shared pool', seq_num: 59477
0: waited for 'jobq slave wait'
1: waited for 'jobq slave wait'
2: waited for 'jobq slave wait'
3: waited for 'jobq slave wait'
4: waited for 'jobq slave wait'
5: waited for 'jobq slave wait'
6: waited for 'jobq slave wait'
7: waited for 'jobq slave wait'
8: waited for 'jobq slave wait'
9: waited for 'jobq slave wait'
waited for 'latch: shared pool', seq_num: 61
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'SQL*Net break/reset to client'
9: waited for 'SQL*Net break/reset to client'
waited for 'library cache: mutex X', seq_num: 324
0: waited for 'SQL*Net break/reset to client'
1: waited for 'SQL*Net break/reset to client'
2: waited for 'SQL*Net break/reset to client'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'cursor: pin S wait on X'
waited for 'latch: shared pool', seq_num: 161
0: waited for 'SQL*Net message from client'
1: waited for 'SQL*Net message to client'
2: waited for 'SQL*Net message from client'
3: waited for 'SQL*Net message to client'
4: waited for 'log file sync'
5: waited for 'SQL*Net message from client'
6: waited for 'SQL*Net message to client'
7: waited for 'SQL*Net message from client'
8: waited for 'SQL*Net message to client'
9: waited for 'SQL*Net message from client'
waited for 'latch: shared pool', seq_num: 388
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'SQL*Net break/reset to client'
9: waited for 'SQL*Net break/reset to client'
waited for 'library cache: mutex X', seq_num: 235
0: waited for 'jobq slave wait'
1: waited for 'jobq slave wait'
2: waited for 'jobq slave wait'
3: waited for 'jobq slave wait'
4: waited for 'jobq slave wait'
5: waited for 'jobq slave wait'
6: waited for 'jobq slave wait'
7: waited for 'jobq slave wait'
8: waited for 'jobq slave wait'
9: waited for 'jobq slave wait'
waited for 'latch: shared pool', seq_num: 123
0: waited for 'db file sequential read'
1: waited for 'db file sequential read'
2: waited for 'db file sequential read'
3: waited for 'db file sequential read'
4: waited for 'db file sequential read'
5: waited for 'db file sequential read'
6: waited for 'db file sequential read'
7: waited for 'control file sequential read'
8: waited for 'control file sequential read'
9: waited for 'control file sequential read'
0: waited for 'Space Manager: slave idle wait'
1: waited for 'Space Manager: slave idle wait'
2: waited for 'Space Manager: slave idle wait'
3: waited for 'Space Manager: slave idle wait'
4: waited for 'Space Manager: slave idle wait'
5: waited for 'Space Manager: slave idle wait'
6: waited for 'Space Manager: slave idle wait'
7: waited for 'Space Manager: slave idle wait'
8: waited for 'Space Manager: slave idle wait'
9: waited for 'Space Manager: slave idle wait'
waited for 'latch: shared pool', seq_num: 703
0: waited for 'SQL*Net message to client'
1: waited for 'SQL*Net message from client'
2: waited for 'SQL*Net message to client'
0: waited for 'Space Manager: slave idle wait'
1: waited for 'Space Manager: slave idle wait'
2: waited for 'Space Manager: slave idle wait'
3: waited for 'Space Manager: slave idle wait'
4: waited for 'Space Manager: slave idle wait'
5: waited for 'Space Manager: slave idle wait'
6: waited for 'Space Manager: slave idle wait'
7: waited for 'Space Manager: slave idle wait'
8: waited for 'Space Manager: slave idle wait'
9: waited for 'Space Manager: slave idle wait'
waited for 'latch: shared pool', seq_num: 130
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'SQL*Net message from client'
8: waited for 'SQL*Net message to client'
9: waited for 'SQL*Net message from client'
waited for 'library cache: mutex X', seq_num: 12497
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'SQL*Net message from client'
9: waited for 'SQL*Net message to client'
waited for 'library cache: mutex X', seq_num: 13055
0: waited for 'jobq slave wait'
1: waited for 'jobq slave wait'
2: waited for 'jobq slave wait'
3: waited for 'jobq slave wait'
4: waited for 'jobq slave wait'
5: waited for 'jobq slave wait'
6: waited for 'jobq slave wait'
7: waited for 'jobq slave wait'
8: waited for 'jobq slave wait'
9: waited for 'jobq slave wait'
waited for 'latch: shared pool', seq_num: 61
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
waited for 'latch: shared pool', seq_num: 3
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'SQL*Net message from client'
1: waited for 'SQL*Net message to client'
2: waited for 'SQL*Net message from client'
3: waited for 'SQL*Net message to client'
4: waited for 'SQL*Net message from client'
5: waited for 'SQL*Net message to client'
6: waited for 'SQL*Net message from client'
7: waited for 'SQL*Net message to client'
8: waited for 'SQL*Net message from client'
9: waited for 'SQL*Net message to client'
waited for 'latch: shared pool', seq_num: 12898
0: waited for 'library cache: mutex X'
1: waited for 'SQL*Net message from client'
2: waited for 'SQL*Net message to client'
3: waited for 'SQL*Net message from client'
4: waited for 'SQL*Net message to client'
5: waited for 'SQL*Net message from client'
6: waited for 'SQL*Net message to client'
7: waited for 'SQL*Net message from client'
8: waited for 'SQL*Net message to client'
9: waited for 'SQL*Net message from client'
waited for 'latch: shared pool', seq_num: 14290
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'SQL*Net message to client'
1: waited for 'SQL*Net message from client'
2: waited for 'SQL*Net message to client'
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'SQL*Net message to client'
1: waited for 'SQL*Net message from client'
2: waited for 'SQL*Net message to client'
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'SQL*Net message to client'
1: waited for 'SQL*Net message from client'
2: waited for 'SQL*Net message to client'
0: waited for 'cursor: pin S wait on X'
1: waited for 'cursor: pin S wait on X'
2: waited for 'SQL*Net message from client'
3: waited for 'SQL*Net message to client'
4: waited for 'SQL*Net message from client'
5: waited for 'SQL*Net message to client'
waited for 'cursor: pin S wait on X', seq_num: 7
waited for 'cursor: pin S wait on X', seq_num: 6
0: waited for 'cursor: pin S wait on X'
1: waited for 'cursor: pin S wait on X'
2: waited for 'cursor: pin S wait on X'
3: waited for 'SQL*Net message from client'
4: waited for 'SQL*Net message to client'
5: waited for 'SQL*Net message from client'
6: waited for 'SQL*Net message to client'
waited for 'cursor: pin S wait on X', seq_num: 8
waited for 'cursor: pin S wait on X', seq_num: 7
waited for 'cursor: pin S wait on X', seq_num: 6
0: waited for 'cursor: pin S wait on X'
1: waited for 'cursor: pin S wait on X'
2: waited for 'cursor: pin S wait on X'
3: waited for 'cursor: pin S wait on X'
4: waited for 'cursor: pin S wait on X'
5: waited for 'SQL*Net message from client'
6: waited for 'SQL*Net message to client'
7: waited for 'SQL*Net message from client'
8: waited for 'SQL*Net message to client'
waited for 'cursor: pin S wait on X', seq_num: 10
waited for 'cursor: pin S wait on X', seq_num: 9
waited for 'cursor: pin S wait on X', seq_num: 8
0: waited for 'SQL*Net message from client'
1: waited for 'SQL*Net message to client'
2: waited for 'SQL*Net message from client'
3: waited for 'SQL*Net message to client'
4: waited for 'SQL*Net message from client'
5: waited for 'SQL*Net message to client'
6: waited for 'SQL*Net message from client'
7: waited for 'SQL*Net message to client'
8: waited for 'SQL*Net message from client'
9: waited for 'SQL*Net message to client'
waited for 'latch: shared pool', seq_num: 958
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'SQL*Net message to client'
1: waited for 'SQL*Net message from client'
2: waited for 'SQL*Net message to client'
0: waited for 'SQL*Net message to client'
1: waited for 'SQL*Net message from client'
2: waited for 'SQL*Net message to client'
3: waited for 'SQL*Net message from client'
4: waited for 'SQL*Net message to client'
5: waited for 'SQL*Net message from client'
6: waited for 'SQL*Net message to client'
7: waited for 'SQL*Net message from client'
8: waited for 'SQL*Net message to client'
9: waited for 'SQL*Net message from client'
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 13
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
0: waited for 'library cache: mutex X'
1: waited for 'library cache: mutex X'
2: waited for 'library cache: mutex X'
3: waited for 'library cache: mutex X'
4: waited for 'library cache: mutex X'
5: waited for 'library cache: mutex X'
6: waited for 'library cache: mutex X'
7: waited for 'library cache: mutex X'
8: waited for 'library cache: mutex X'
9: waited for 'library cache: mutex X'
waited for 'library cache: mutex X', seq_num: 12
waited for 'latch: shared pool', seq_num: 1
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
waited for 'latch: shared pool', seq_num: 1
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
waited for 'latch: shared pool', seq_num: 1
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
waited for 'latch: shared pool', seq_num: 1
waited for 'null event', seq_num: 0
waited for 'latch: shared pool', seq_num: 1
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
waited for 'null event', seq_num: 0
摘取其中几个ibrary cache: mutex X进行钻取分析，发生等待此事件的会话全被如下进程持有资源，即如下进程占据了资源。
state=busy [holder orapid=35] wlstate=free [value=0]
state=busy [holder orapid=91] wlstate=free [value=0]
state=busy [holder orapid=94] wlstate=free [value=0]
state=busy(shared) [value=0x3] wlstate=free [value=0]
state=busy [holder orapid=114] wlstate=free [value=0]
查看35进程,注意2个标红处，可知91进程阻塞35进程
PROCESS 35: SMCO
----------------------------------------
SO: 0x700000928d90210, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x700000928d90210, name=process, file=ksu.h LINE:12616 ID:, pg=0
(process) Oracle pid:35, ser:1, calls cur/top: 0x7000009714873a0/0x7000009714873a0
flags : (0x2) SYSTEM
flags2: (0x0), flags3: (0x10)
intr error: 0, call error: 0, sess error: 0, txn error 0
intr queue: empty
ksudlp FALSE at location: 0
(post info) last post received: 0 0 69
last post received-location: kso2.h LINE:472 ID:ksoreq_reply
last process to post me: 700000970d1f7a0 130 2
last post sent: 258 0 2
last post sent-location: ksl2.h LINE:2362 ID:kslpsr
last process posted by me: 700000918d1ca30 84 2
(latch info) wait_event=0 bits=10
Location from where call was made: kgh.h LINE:6407 ID:kghalo:
waiting for 7000000001074d0 Child shared pool level=7 child#=4
Location from where latch is held: kgh.h LINE:6407 ID:kghalo:
Context saved from call: 0
state=busy [holder orapid=91] wlstate=free [value=0]
waiters [orapid (seconds since: put on list, posted, alive check)]:
28 (7700, 1409792981, 1)
92 (7695, 1409792981, 1)
93 (7695, 1409792981, 1)
95 (7695, 1409792981, 1)
48 (7690, 1409792981, 1)
109 (7652, 1409792981, 1)
111 (7652, 1409792981, 1)
114 (7652, 1409792981, 1)
35 (7425, 1409792981, 1)
43 (6825, 1409792981, 1)
49 (6518, 1409792981, 1)
31 (6376, 1409792981, 1)
15 (6225, 1409792981, 1)
waiter count=13
gotten 782658701 times wait, failed first 90218 sleeps 79574
gotten 0 times nowait, failed: 0
possible holder pid = 91 ospid=13893742
on wait list for 7000000001074d0
holding (efd=6) 70000000000fbb8 OS process allocation level=4
Location from where latch is held: kso2.h LINE:317 ID:ksoreq_submit:
Context saved from call: 504403198262175032
state=busy [holder orapid=35] wlstate=free [value=0]
waiters [orapid (seconds since: put on list, posted, alive check)]:
3 (7424, 1409792981, 1)
waiter count=1
Process Group: DEFAULT, pseudo proc: 0x700000920ea0868
O/S info: user: oracle, term: UNKNOWN, ospid: 44499044
OSD pid info: Unix process pid: 44499044, image: oracle@720db2 (SMCO)
同理，查看91进程，注意标红部分，可见2329会话阻塞91进程
PROCESS 91:
----------------------------------------
SO: 0x700000918d24f30, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x700000918d24f30, name=process, file=ksu.h LINE:12616 ID:, pg=0
(process) Oracle pid:91, ser:50, calls cur/top: 0x0/0x70000092ff12f88
flags : (0x0) -
flags2: (0x0), flags3: (0x10)
intr error: 0, call error: 0, sess error: 0, txn error 0
intr queue: empty
ksudlp FALSE at location: 0
(post info) last post received: 0 0 0
last post received-location: No post
last process to post me: none
last post sent: 0 0 0
last post sent-location: No post
last process posted by me: none
(latch info) wait_event=0 bits=80
holding (efd=6) 7000000001074d0 Child shared pool level=7 child#=4
Location from where latch is held: kgh.h LINE:6407 ID:kghalo:
Context saved from call: 0
state=busy [holder orapid=91] wlstate=free [value=0]
waiters [orapid (seconds since: put on list, posted, alive check)]:
28 (7702, 1409792983, 0)
92 (7697, 1409792983, 0)
93 (7697, 1409792983, 0)
95 (7697, 1409792983, 0)
48 (7692, 1409792983, 0)
109 (7654, 1409792983, 0)
111 (7654, 1409792983, 0)
114 (7654, 1409792983, 0)
35 (7427, 1409792983, 0)
43 (6827, 1409792983, 0)
49 (6520, 1409792983, 0)
31 (6378, 1409792983, 0)
15 (6227, 1409792983, 0)
waiter count=13
Process Group: DEFAULT, pseudo proc: 0x700000920ea0868
O/S info: user: oracle, term: UNKNOWN, ospid: 13893742
OSD pid info: Unix process pid: 13893742, image: oracle@720db2
----------------------------------------
SO: 0x70000093137b3a0, type: 4, owner: 0x700000918d24f30, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x700000918d24f30, name=session, file=ksu.h LINE:12624 ID:, pg=0
(session) sid: 2452 ser: 35859 trans: 0x0, creator: 0x700000918d24f30
flags: (0x1) USR/- flags_idl: (0x0) -/-/-/-/-/-
flags2: (0x0) -/-/-
DID: , short-term DID:
txn branch: 0x0
oct: 0, prv: 0, sql: 0x0, psql: 0x700000944d23920, user: 0/SYS
ksuxds FALSE at location: 0
client details:
O/S info: user: , term: , ospid:
machine: program:
Current Wait Stack:
0: waiting for 'library cache: mutex X'
idn=0xbddaffd0, value=0x91900000000, where=0x4f
wait_id=11 seq_num=12 snap_id=1
wait times: snap=7 min 29 sec, exc=7 min 29 sec, total=7 min 29 sec
wait times: max=infinite, heur=128 min 21 sec
wait counts: calls=44693 os=44693
in_wait=1 iflags=0x11b2
There is at least one session blocking this session.
Dumping 1 direct blocker(s):
inst: 1, sid: 2329, ser: 53609
Dumping final blocker:
inst: 1, sid: 2329, ser: 53609
查看2329会话所属信息
SO: 0x700000919335228, type: 4, owner: 0x700000920d446f8, flag: INIT/-/-/0x00 if: 0x3 c:0x3
proc=0x700000920d446f8, name=session, file=ksu.h LINE:12624 ID:, pg=0
(session) sid: 2329 ser: 53609 trans: 0x0, creator: 0x700000920d446f8
flags: (0x51) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x40409) -/-/INC
DID: , short-term DID:
txn branch: 0x0
oct: 0, prv: 0, sql: 0x0, psql: 0x700000966d83ac0, user: 0/SYS
会话隶属于进程，根据上述会话的owner:0x700000920d446f8获取所属的进程，定位到后台进程M000
PROCESS 42: M000
----------------------------------------
SO: 0x700000920d446f8, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x700000920d446f8, name=process, file=ksu.h LINE:12616 ID:, pg=0
(process) Oracle pid:42, ser:95, calls cur/top: 0x70000092f1209e0/0x70000092d9a6960
flags : (0x2) SYSTEM
flags2: (0x30), flags3: (0x10)
intr error: 0, call error: 0, sess error: 0, txn error 0
intr queue: empty
ksudlp FALSE at location: 0
(post info) last post received: 0 0 43
last post received-location: ksv2.h LINE:1655 ID:ksvpst: run
last process to post me: 700000928d8d030 1 2
last post sent: 0 0 26
last post sent-location: ksa2.h LINE:285 ID:ksasnd
last process posted by me: 700000918d17710 1 6
(latch info) wait_event=0 bits=0
Process Group: DEFAULT, pseudo proc: 0x700000920ea0868
O/S info: user: oracle, term: UNKNOWN, ospid: 36831360
OSD pid info: Unix process pid: 36831360, image: oracle@720db2 (M000)
同理，分析其它的等待事件，分析方法同上，最终定位到进程M000，即数据库等待事件产生的根源是M000，即在2014-09-04 04:00:47 M000进程报错，查找并分析对应的trc文件Satisfaction_m000_18939952_i101489.trc，诊断为何M000出现故障，由下标红部分可见，M000是在采集统计信息时出现故障引发了ora-07445错误。
Machine: 00F73B824C00
Instance name: Satisfaction
Redo thread mounted by this instance: 1
Oracle process number: 42
Unix process pid: 18939952, image: oracle@720db2 (M000)
*** 2014-09-04 04:00:47.561
*** SESSION ID:(2329.53305) 2014-09-04 04:00:47.561
*** CLIENT ID:() 2014-09-04 04:00:47.561
*** SERVICE NAME:(SYS$BACKGROUND) 2014-09-04 04:00:47.561
*** MODULE NAME:(MMON_SLAVE) 2014-09-04 04:00:47.561
*** ACTION NAME:(Auto-Flush Slave Action) 2014-09-04 04:00:47.561
Dump continued from file: /home/oracle/app/oracle/diag/rdbms/satisfaction/Satisfaction/trace/Satisfaction_m000_18939952.trc
ORA-07445: exception encountered: core dump [kglic0()+764] [SIGSEGV] [ADDR:0x642028544E53206E] [PC:0x10123F5BC] [Address not mapped to object] []
经查阅oracle MOS的大量相关文献，发现如下文章：
MMON Slave Process Reports ORA-7445 [kglic0]/[kksIterCursorStat] Error, Plus Database Hangs
(文档ID 1487720.1)
文章具体内容如下，详细展开分析：
客户数据库版本为11.2.0.3，版本匹配。
APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.1 to 11.2.0.3 [Release 11.2]
Information in this document applies to any platform.
从表现症状来看，ora-07445错误的参数及call stack也是完全匹配，并且在报错期间数据库处于hang状态。
SYMPTOMS
When the MMON_SLAVE is attempting to perform an "Auto-Flush Slave Action",?one of the following errors may occur.
ORA-07445: exception encountered: core dump [kglic0()+1075] [SIGSEGV] [ADDR:0x18] [PC:0x90F1CFF] [Address not mapped to object] []
OR
ORA-07445: exception encountered: core dump [kksIterCursorStat()+596] [SIGSEGV] [ADDR:0x17] [PC:0x5562FEC] [Address not mapped to object] []
Subsequently, the following message can be reported in the Alert log:
PMON failed to acquire latch...
The Call Stack in the MMON_SLAVE trace file at the time of the error shows the following Oracle functions:
... kksIterCursorStat kewrrtsq_rank_topsql kewrbtsq_build_topsql kewrftsq_flush_topsql kewrft_flush_table kewrftec_flush_table_ehdlcx kewrfat_flush_all_tables kewrfos_flush_onesnap kewrfsc_flush_snapshot_c kewrafs_auto_flush_slave...
In the case of the ORA-7445 [kglic0] error, the "kksIterCursorStat" function is preceded by the "kglic0" function in the Call Stack:
... kglic0 kksIterCursorStat kewrrtsq_rank_topsql kewrbtsq_build_topsql kewrftsq_flush_topsql ...
During this time, a database hang may be experienced
产生M000报错ora-0745的原因，可能是bug:13550185(注：additional部分描述的bug和客户数据库报错不一致故不再详述，且此2个bug不会引发oracle hang)
CAUSE
If the Shared Pool has multiple subpools, and literal replacement is being used, then the ORA-7445 [kglic0] or ORA-7445 [kksIterCursorStat] error can occur.??This has been identified as:?BUG 13550185 - QUERIES ON V$SQLSTATS MAY CAUSE CRASH OR HANG ?where the situation should match on the following criteria;
? Using Oracle Database Versions >= 11.2.0.2 but BELOW 12.1, plus the fix for Bug 13550185 should not already be applied,
? Reporting the ORA-7445 [kglic0] or ORA-7445 [kksIterCursorStat] error,
? Oracle function kksIterCursorStat is on the Call Stack,
? There is >1 Subpool (i.e. _kghdsidx_count=2 or more),
? Literal replacements are being used (i.e. CURSOR_SHARING != EXACT or a select count(*) from v$sql_bind_capture where name like ':SYS%'; returns a count >0 ),
? A database hang may be experienced.
This is as per the Bug Note:?Note 13550185.8 Bug 13550185 - Hang / SGA memory corruption / ORA-7445 [kglic0] with CURSOR_SHARING using multiple shared pool subpools
ADDITIONAL?Please note that there are also the following similar bugs;?Note 12340939.8 Bug 12340939 - ORA-7445 [kglic0] can occur capturing cursor stats for V$SQLSTATS?Note 12409916.8 Bug 12409916 - ORA-7445 [kglic0] accessing V$SQL or V$SQLSTATS?where the symptoms are;
? Process May Dump (ORA-7445) / Abend / Abort
? Dump in or under kglic0
? Stack is likely to include kksIterCursorStat
These bugs differ from Bug 13550185 in that;
? No database hang is experienced,
? Literal replacement is not involved,
? The work-around is to disable the code path that leads to the crash by setting the hidden parameter "_cursor_stats_enabled"=FALSE.
基于上述BUG给出的解决方案如下：下载临时中间补丁patch 13550185或者应用PSU 11.2.0.3.4或者直接升级到11.2.0.4或者设置隐含参数只启用一个共享子池。
solution
? Download and apply interim Patch 13550185, if available for your platform and RDBMS release. If Patch 13550185 is not available for your version & platform, log a new Service Request and request a One-Off Back-port.
OR
? Apply PSU 11.2.0.3.4, where the fix for Bug 13550185 has already been included (PSU Patch 14275605)
OR
? Upgrade to 11.2.0.4 (when available) where the fix for Bug 13550185 has already been included.
OR
WORKAROUND: Use only 1 Subpool (alter system set "_kghdsidx_count"=1 scope=spfile; then restart the instance)
经查阅客户数据库相关信息，可见数据库配置了6个共享子池
select x.inst_id as inst_id,
ksppinm as name,
ksppstvl as value_1,
ksppdesc as desc_1
from x$ksppi x, x$ksppcv y
where (x.indx = y.indx) and
(x.inst_id=y.inst_id) and
lower(x.ksppinm)='_kghdsidx_count'
INST_ID NAME VALUE_1 DESC_1
---------- ------------------------- -------------------- ---------------
1 _kghdsidx_count 6 max kghdsidx count
由下可知，客户数据库已应用11.2.0.3.6 PSU,可见已经修复了上述的BUG
720db1>#$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.
Oracle Home : /home/oracle/app/oracle/product/11.2.0
Central Inventory : /home/oracle/app/oraInventory
from : /home/oracle/app/oracle/product/11.2.0/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.3.0
Log file location : /home/oracle/app/oracle/product/11.2.0/cfgtoollogs/opatch/opatch2014-09-10_15-07-03PM_1.log
Lsinventory Output file location : /home/oracle/app/oracle/product/11.2.0/cfgtoollogs/opatch/lsinv/lsinventory2014-09-10_15-07-03PM.txt
------------------------------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Database 11g 11.2.0.3.0
There are 1 products installed in this Oracle Home.
List of Bugs fixed by Installed Patches:
Bug Fixed by Installed at Description
Patch
--- -------- ------------ -----------
8547978 16056266 Wed Aug 14 06:11:52 GMT+08:00 2013DROP USER GIVES ORA-00600[KQD-OBJERROR$] AFTER
ONLINE REDEFINITION OF TABLE
因篇幅原因，中间略
13528551 13343438 Wed Aug 14 06:07:27 GMT+08:00 2013DB-11.2.0.3-MOLECULE-002-CPUJAN2012
13493847 13493847 Wed Jul 03 12:34:14 GMT+08:00 2013ORA-600 [15709], [29], [3]
------------------------------------------------------------------------------------------------------
OPatch succeeded.
由上可知，已经安装了修复bug 13550185的patch
13550185 14275605 Wed Aug 14 06:10:04 GMT+08:00 2013QUERIES ON V$SQLSTATS MAY CAUSE CRASH OR HANG
继续查阅ORACLE MOS,发现如下文章：
Instance Hang And Crash With ORA-07445: [kglic0()] Even With Patch 13550185 Installed When
Dispatchers Are Being Used (文档 ID 1526720.1)
其文章内容如下：
A从数据库版本而言，完全匹配一致S TO
APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.
从表现症状看，客户数据库也是M000在收集统计信息时出现故障，虽然报错的SQL不一样，原理是一样的。
SYMPTOMS
Your database jobs are throwing errors during the execution of StatsPack job
ORA-07445: exception encountered: core dump [kglic0()+1205] [SIGSEGV] [ADDR:0x18] [PC:0x2DA656B] [Address not mapped to object] []
The trace file shows the following SQL being performed:
INSERT INTO STATS$SQL_STATISTICS ( SNAP_ID , DBID , INSTANCE_NUMBER , TOTAL_SQL , TOTAL_SQL_MEM , SINGLE_USE_SQL , SINGLE_USE_SQL_MEM , TOTAL_CURSORS ) SELECT :B3 , :B2 , :B1 , COUNT(1) , SUM(SHARABLE_MEM) , SUM(DECODE(EXECUTIONS, 1, 1, 0)) , SUM(DECODE(EXECUTIONS, 1, SHARABLE_MEM, 0)) , SUM(VERSION_COUNT) FROM STATS$V$SQLSTATS_SUMMARY WHERE SHARABLE_MEM > 0
从堆栈跟踪信息看，基本相同，当然并不完全匹配，匹配度为80%左右。
Stack trace
skdstdst ksedst1 ksedst dbkedDefDump ksedmp ssexhd sighandler kglic0 kksIterCursorStat kksFetchCursorStat qerfxFetch qerghFetch qerflFetchOutside qervwFetch qergsFetch rwsfcd qerltcFetch insexe opiexe opipls opiodr rpidrus skgmstack rpiswu2 rpidrv psddr0 psdnal pevm_EXECC pfrinstr_EXECC pfrrun_no_tool pfrrun plsql_run peicnt kkxexe opiexe opiodr rpidrus skgmstack rpiswu2 rpidrv rpiexe kkjex1e kkjsexe kkjrdp opirip opidrv sou2o opimai_real ssthrdmain main libc_start_main start
下面这句很关键，上述的所有描述非常匹配bug:13550185,但数据库已安装此bug patch,但bug:13550185的表现症状依旧存在。
This behavior and stack trace points to Bug:13550185, but the fix for 13550185 has already been installed, but the behavior continues.
引发的原因源于一个未公布的bug:13853126,并且确认会影响11.2.0.3数据库版本。而且如果数据库配置的共享服务器模式及配置了调度器进程或启用了并行查询服务器进程，即会引发此未公布的BUG。
CAUSE
This has been identified as unpublished Bug:13853126 which is known to affect the 11.2.0.3 release.
When using dispatchers (MTS) and/or new PQ slaves, if the shared pool is under pressure we could ask KGL to release memory but if the process hadn't established a session we might acquire the SAME mutex as another process, also with no session established, and this could then lead to SGA corruptions.??Although it shows similarities to bugs 13550185, the fix for 13550185 is already installed and is also included in the 11.2.0.3.4 PSU and higher.?The difference in this case, is that you have processes that might operate without a session established (eg., dispatchers, spawning PQ slaves etc.).
客户数据库确实配置了共享服务器模式，并且启用了调度器进程。
SQL> show parameter shared_server
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
max_shared_servers integer
shared_server_sessions integer
shared_servers integer 1
SQL> show parameter dispatcher
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
dispatchers string (PROTOCOL=TCP) (SERVICE=Satisf
actionXDB)
max_dispatchers integer
解决方案如下：应用patch:13853126或者禁用共享服务器模式，改用专用服务器模式，从风险评估及工作量考量，强烈建议采用第2种方法。
SOLUTION
1. If available for your platform and version, download and apply Patch:13853126.
OR
2. Disable the use of MTS and use dedicated connections.
基于上述分析结合alert数据库报警日志，我司列举出客户数据库故障时间时谱表：
| 序号 | 时间 | 事件 | 备注 |
| 1 | 2014-09-04 01:02:38 | 操作系统发生换页换出活动 |
|
| 2 | 2014-09-04 04:00:47 | 进程M000出现故障报ora-07445，数据产生大量library cache:mutex x，导致数据库hang |
|
| 3 | 2014-09-04 10:23:29 | W006进程因故无法正常启动报ORA-00445错误 |
|
| 4 | 2014-09-04 10:24:39 | PMON关闭了数据库实例 |
|
后续建议
建议根据MOS文档：
Instance Hang And Crash With ORA-07445: [kglic0()] Even With Patch 13550185 Installed When Dispatchers Are Being Used (文档ID 1526720.1)。
禁用共享服务器模式改用专用服务器模式,具体操作命令如下：
禁用共享服务器
alter system set shared_servers=0;
查看共享服务器是否禁用
show parameter shared_servers
关闭调度器进程
alter system set dispatchers=’’;
查看调度器进程是否关闭
show parameter dispatchers;
个人简介：
项目经验:
来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/9240380/viewspace-1270320/，如需转载，请注明出处，否则将追究法律责任。
转载于:http://blog.itpub.net/9240380/viewspace-1270320/
2308
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
