系统内存不足导致的一次数据异常关闭

今天数据库突然挂掉了,查看alert日志发现:

Thu Jun 25 01:07:14 2015
Thread 1 advanced to log sequence 262 (LGWR switch)
  Current log# 1 seq# 262 mem# 0: /oracle/oradata/sip/redo01.log
Archived Log entry 257 added for thread 1 sequence 261 ID 0x47d3d707 dest 1:

Thu Jun 25 01:08:11 2015
System state dump requested by (instance=1, osid=30314 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/diag/rdbms/sip/sip/trace/sip_diag_30332_20150625010811.trc
Thu Jun 25 01:08:11 2015
PMON (ospid: 30314): terminating the instance due to error 822
Thu Jun 25 01:08:15 2015
ORA-1092 : opitsk aborting process
Instance terminated by PMON, pid = 30314


可见之前还正常,突然被终结了。


查看系统日志:

Jun 23 18:04:51 SIP-DB yum[2801]: Installed: ftp-0.17-54.el6.x86_64
Jun 25 01:08:09 SIP-DB kernel: oracle invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jun 25 01:08:09 SIP-DB kernel: oracle cpuset=/ mems_allowed=0
Jun 25 01:08:09 SIP-DB kernel: Pid: 27179, comm: oracle Not tainted 2.6.32-431.23.3.el6.x86_64 #1
Jun 25 01:08:09 SIP-DB kernel: Call Trace:
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff810d0431>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81122810>] ? dump_header+0x90/0x1b0
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8122833c>] ? security_real_capable_noaudit+0x3c/0x70
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81122c92>] ? oom_kill_process+0x82/0x2a0
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81122bd1>] ? select_bad_process+0xe1/0x120
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff811230d0>] ? out_of_memory+0x220/0x3c0
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8112f9ef>] ? __alloc_pages_nodemask+0x89f/0x8d0
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff811678ea>] ? alloc_pages_current+0xaa/0x110
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8111fc07>] ? __page_cache_alloc+0x87/0x90
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8111f5ee>] ? find_get_page+0x1e/0xa0
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81121545>] ? generic_file_aio_read+0x585/0x700
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81188cda>] ? do_sync_read+0xfa/0x140
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8105de8d>] ? thread_group_times+0x3d/0x120
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81226d76>] ? security_file_permission+0x16/0x20
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81189695>] ? vfs_read+0xb5/0x1a0
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81091d2b>] ? do_sys_times+0xab/0xd0
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff811899c2>] ? sys_pread64+0x82/0xa0
Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Jun 25 01:08:09 SIP-DB kernel: Mem-Info:
Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA per-cpu:
Jun 25 01:08:09 SIP-DB kernel: CPU    0: hi:    0, btch:   1 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    1: hi:    0, btch:   1 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    2: hi:    0, btch:   1 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    3: hi:    0, btch:   1 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    4: hi:    0, btch:   1 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    5: hi:    0, btch:   1 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    6: hi:    0, btch:   1 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    7: hi:    0, btch:   1 usd:   0
Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA32 per-cpu:
Jun 25 01:08:09 SIP-DB kernel: CPU    0: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    1: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    2: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    3: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    4: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    5: hi:  186, btch:  31 usd:  60
Jun 25 01:08:09 SIP-DB kernel: CPU    6: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    7: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: Node 0 Normal per-cpu:
Jun 25 01:08:09 SIP-DB kernel: CPU    0: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    1: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    2: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    3: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    4: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    5: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    6: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: CPU    7: hi:  186, btch:  31 usd:   0
Jun 25 01:08:09 SIP-DB kernel: active_anon:786810 inactive_anon:598780 isolated_anon:0
Jun 25 01:08:09 SIP-DB kernel: active_file:145938 inactive_file:191123 isolated_file:141
Jun 25 01:08:09 SIP-DB kernel: unevictable:0 dirty:190674 writeback:483 unstable:0
Jun 25 01:08:09 SIP-DB kernel: free:24940 slab_reclaimable:25814 slab_unreclaimable:12349
Jun 25 01:08:09 SIP-DB kernel: mapped:684185 shmem:992477 pagetables:203816 bounce:0
Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA free:15728kB min:124kB low:152kB high:184kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15340kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jun 25 01:08:09 SIP-DB kernel: lowmem_reserve[]: 0 3768 8060 8060
Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA32 free:48152kB min:31532kB low:39412kB high:47296kB active_anon:1500552kB inactive_anon:1113036kB active_file:224128kB inactive_file:369116kB unevictable:0kB isolated(anon):0kB isolated(file):256kB present:3858656kB mlocked:0kB dirty:367384kB writeback:988kB mapped:1592520kB shmem:2020796kB slab_reclaimable:43944kB slab_unreclaimable:13100kB kernel_stack:1848kB pagetables:263196kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:138688 all_unreclaimable? no
Jun 25 01:08:09 SIP-DB kernel: lowmem_reserve[]: 0 0 4292 4292
Jun 25 01:08:09 SIP-DB kernel: Node 0 Normal free:35880kB min:35920kB low:44900kB high:53880kB active_anon:1646688kB inactive_anon:1282084kB active_file:359624kB inactive_file:395376kB unevictable:0kB isolated(anon):0kB isolated(file):308kB present:4395520kB mlocked:0kB dirty:395312kB writeback:944kB mapped:1144220kB shmem:1949112kB slab_reclaimable:59312kB slab_unreclaimable:36296kB kernel_stack:2720kB pagetables:552068kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:460416 all_unreclaimable? no
Jun 25 01:08:09 SIP-DB kernel: lowmem_reserve[]: 0 0 0 0
Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA: 2*4kB 1*8kB 2*16kB 2*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15728kB
Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA32: 5148*4kB 1834*8kB 189*16kB 34*32kB 20*64kB 27*128kB 4*256kB 4*512kB 2*1024kB 0*2048kB 0*4096kB = 49232kB
Jun 25 01:08:09 SIP-DB kernel: Node 0 Normal: 2163*4kB 970*8kB 396*16kB 187*32kB 67*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 36348kB
Jun 25 01:08:09 SIP-DB kernel: 1329619 total pagecache pages
Jun 25 01:08:09 SIP-DB kernel: 0 pages in swap cache
Jun 25 01:08:09 SIP-DB kernel: Swap cache stats: add 0, delete 0, find 0/0
Jun 25 01:08:09 SIP-DB kernel: Free swap  = 16777208kB
Jun 25 01:08:09 SIP-DB kernel: Total swap = 16777208kB
Jun 25 01:08:09 SIP-DB kernel: 2097151 pages RAM
Jun 25 01:08:09 SIP-DB kernel: 82637 pages reserved
Jun 25 01:08:09 SIP-DB kernel: 9350672 pages shared
Jun 25 01:08:09 SIP-DB kernel: 991485 pages non-shared
Jun 25 01:08:09 SIP-DB kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Jun 25 01:08:09 SIP-DB kernel: [  435]     0   435     2670      201   0     -17         -1000 udevd
Jun 25 01:08:09 SIP-DB kernel: [ 1083]     0  1083    62797      309   0       0             0 rsyslogd
Jun 25 01:08:09 SIP-DB kernel: [ 1141]    38  1141     6648      289   0       0             0 ntpd
Jun 25 01:08:09 SIP-DB kernel: [ 1151]     0  1151    29332      282   0       0             0 crond
Jun 25 01:08:09 SIP-DB kernel: [ 1180]     0  1180     7905       17   0       0             0 gshelld
Jun 25 01:08:09 SIP-DB kernel: [ 1197]     0  1197     1016       91   6       0             0 mingetty
Jun 25 01:08:09 SIP-DB kernel: [ 1199]     0  1199     1016       90   2       0             0 mingetty
Jun 25 01:08:09 SIP-DB kernel: [ 1201]     0  1201     1016       90   4       0             0 mingetty
Jun 25 01:08:09 SIP-DB kernel: [ 1203]     0  1203     1016       91   4       0             0 mingetty
Jun 25 01:08:09 SIP-DB kernel: [ 1207]     0  1207     1016       90   6       0             0 mingetty
Jun 25 01:08:09 SIP-DB kernel: [ 1209]     0  1209     1016       91   4       0             0 mingetty
Jun 25 01:08:09 SIP-DB kernel: [ 1230]     0  1230   118751      961   5       0             0 AliYunDunUpdate
Jun 25 01:08:09 SIP-DB kernel: [ 1284]     0  1284   194826     1665   5       0             0 AliYunDun
Jun 25 01:08:09 SIP-DB kernel: [ 1299]     0  1299   209991     1979   4       0             0 AliHids
Jun 25 01:08:09 SIP-DB kernel: [24128]     0 24128    16541      274   0     -17         -1000 sshd
Jun 25 01:08:09 SIP-DB kernel: [24337]    28 24337   157649      289   1       0             0 nscd
Jun 25 01:08:09 SIP-DB kernel: [29259]   500 29259    54234     1895   0       0             0 tnslsnr
Jun 25 01:08:09 SIP-DB kernel: [30314]   500 30314  1085477     8955   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30318]   500 30318  1084863     4294   2       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30322]   500 30322  1084863     4104   1       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30328]   500 30328  1084863     4171   1       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30332]   500 30332  1084863     4112   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30336]   500 30336  1085318    11623   1       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30340]   500 30340  1087551    13604   2       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30344]   500 30344  1084863   336329   1       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30348]   500 30348  1092190   185397   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30352]   500 30352  1089018     8587   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30356]   500 30356  1084991    24525   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30360]   500 30360  1086546    47594   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30364]   500 30364  1085262     8653   5       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30368]   500 30368  1086414    50695   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30372]   500 30372  1085286    19614   1       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30376]   500 30376  1085391     4094   2       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30380]   500 30380  1085129     3843   6       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30438]   500 30438  1096943    11318   3       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30442]   500 30442  1092847    10977   5       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30446]   500 30446  1090799     4410   1       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30450]   500 30450  1096943    11759   1       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30458]   500 30458  1084863     4523   5       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30486]   500 30486  1086046    37927   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30498]   500 30498  1086259    13676   3       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [30766]   500 30766  1086559     6460   5       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [10304]     0 10304    25204      669   0       0             0 sshd
Jun 25 01:08:09 SIP-DB kernel: [10310]     0 10310    14585      646   0       0             0 sftp-server
Jun 25 01:08:09 SIP-DB kernel: [12277]   500 12277  1085703    12118   5       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16173]   500 16173  1085550     7933   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16177]   500 16177  1085551     8104   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16181]   500 16181  1085550     8792   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16185]   500 16185  1085561     8483   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16189]   500 16189  1085549     8393   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16197]   500 16197  1085162     5966   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16201]   500 16201  1085293     7335   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16205]   500 16205  1085551     7821   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16209]   500 16209  1085162     5818   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16213]   500 16213  1085162     5804   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16247]   500 16247  1085294     7358   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16251]   500 16251  1085551     8019   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16255]   500 16255  1085294     7220   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16259]   500 16259  1085294     7421   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [16263]   500 16263  1085551     8047   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17229]   500 17229  1085555    13604   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17233]   500 17233  1085826    18918   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17237]   500 17237  1085558    13959   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17241]   500 17241  1085430    13661   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17245]   500 17245  1085558    39491   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17265]   500 17265  1085555    15481   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17277]   500 17277  1085817    18358   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17305]   500 17305  1085557    12906   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17325]   500 17325  1085556    32133   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17343]   500 17343  1085557    42001   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17347]   500 17347  1085557    12639   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17355]   500 17355  1085558    12568   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17359]   500 17359  1085557    16684   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17365]   500 17365  1085556    14139   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17369]   500 17369  1085822    37259   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17377]   500 17377  1085557    12854   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17389]   500 17389  1085821    15934   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17393]   500 17393  1085558    35179   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17401]   500 17401  1086071   113320   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17405]   500 17405  1086455    80434   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17409]   500 17409  1085561   150402   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17413]   500 17413  1086245   128282   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17417]   500 17417  1086587   115121   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17421]   500 17421  1085558    12387   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17433]   500 17433  1085556    12449   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17441]   500 17441  1085558    42395   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17453]   500 17453  1085554    13040   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17457]   500 17457  1085558    41053   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17467]   500 17467  1085556    13095   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17475]   500 17475  1085558    13590   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17479]   500 17479  1085557    12963   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17491]   500 17491  1085555    11402   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17495]   500 17495  1085557    58069   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17499]   500 17499  1085558    62538   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17503]   500 17503  1085558    44952   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17507]   500 17507  1085560    61911   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17523]   500 17523  1085558    32448   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17531]   500 17531  1085556    30449   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17545]   500 17545  1085558    13626   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17557]   500 17557  1085813    67773   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17561]   500 17561  1085566    40153   1       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17565]   500 17565  1085558    12087   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17569]   500 17569  1085557    22287   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17575]   500 17575  1085557    12840   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17583]   500 17583  1085557    13566   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17597]   500 17597  1085556    13167   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17605]   500 17605  1085556    13977   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17609]   500 17609  1085557    14357   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17613]   500 17613  1085556    13650   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17617]   500 17617  1085555    13180   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17625]   500 17625  1085557    54014   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17629]   500 17629  1085558    67111   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17633]   500 17633  1085559   112836   4       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17637]   500 17637  1085558    63563   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17641]   500 17641  1085819   118554   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17645]   500 17645  1086082    66722   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17649]   500 17649  1086199    54099   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17653]   500 17653  1085557    65583   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17657]   500 17657  1086455    93988   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17661]   500 17661  1085558    64335   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17665]   500 17665  1086078    81023   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17669]   500 17669  1085557    12829   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17673]   500 17673  1085557    63520   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17675]   500 17675  1085563   163201   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17693]   500 17693  1085556    12342   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17697]   500 17697  1085558    12438   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17791]   500 17791  1085558    31065   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17795]   500 17795  1085557    31178   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17817]   500 17817  1085554    12821   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17821]   500 17821  1085556    50532   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17839]   500 17839  1085558    37189   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17847]   500 17847  1085555    12526   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17867]   500 17867  1085429    41098   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17873]   500 17873  1085557    30435   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17877]   500 17877  1085429    30619   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17881]   500 17881  1085556    31063   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17895]   500 17895  1085557    13867   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17911]   500 17911  1085556    30758   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17915]   500 17915  1085555    31021   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17927]   500 17927  1085428    13912   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17931]   500 17931  1085429    13399   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17945]   500 17945  1085558    13490   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [17953]   500 17953  1085558    13058   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18125]   500 18125  1085430    30449   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18129]   500 18129  1085428    12266   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18137]   500 18137  1085558    12604   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18143]   500 18143  1085557    13188   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18155]   500 18155  1085555    12551   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18159]   500 18159  1085558    12885   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18163]   500 18163  1085558    13632   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18167]   500 18167  1085557    12442   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18177]   500 18177  1085554    40984   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18181]   500 18181  1085556    12490   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18185]   500 18185  1085557    53961   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18209]   500 18209  1085558    31167   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18213]   500 18213  1085430    12828   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18219]   500 18219  1085428    30720   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18223]   500 18223  1085557    12875   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18239]   500 18239  1085557    13875   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18243]   500 18243  1085556    13570   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18247]   500 18247  1085430    13620   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18259]   500 18259  1085555    12676   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18285]   500 18285  1085556    12979   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18289]   500 18289  1085558    13020   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18307]   500 18307  1085557    12280   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18311]   500 18311  1085558    13073   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18333]   500 18333  1085558    12827   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18337]   500 18337  1085429    12382   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18341]   500 18341  1085430    13503   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18351]   500 18351  1085557    13204   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18359]   500 18359  1085558    12559   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18371]   500 18371  1085556    13008   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18383]   500 18383  1085556    13192   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18387]   500 18387  1085557    51934   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18405]   500 18405  1085556    13051   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18409]   500 18409  1085555    13010   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18413]   500 18413  1085556    11905   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18423]   500 18423  1085557    13839   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18435]   500 18435  1085558    13624   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18443]   500 18443  1085559    31639   4       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18455]   500 18455  1085556    12889   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18461]   500 18461  1085556    12916   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18469]   500 18469  1085427    12811   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18473]   500 18473  1085556    45938   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18481]   500 18481  1085554    12550   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18485]   500 18485  1085557    12517   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18493]   500 18493  1085427    12406   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18503]   500 18503  1085555    12110   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18507]   500 18507  1085428    41926   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18511]   500 18511  1085429    46833   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18523]   500 18523  1085557    12284   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18539]   500 18539  1085556    12358   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18545]   500 18545  1085556    12891   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18549]   500 18549  1085558    30307   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18579]   500 18579  1085558    30664   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18605]   500 18605  1085555    30209   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18609]   500 18609  1085556    12558   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18615]   500 18615  1085556    12035   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18645]   500 18645  1085558    12461   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18650]   500 18650  1085557    41220   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18654]   500 18654  1085558    14081   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18658]   500 18658  1085557    12905   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18662]   500 18662  1085431    12694   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18666]   500 18666  1085557    32906   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18780]   500 18780  1085429    12761   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18784]   500 18784  1085557    41518   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18788]   500 18788  1085556    31325   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18794]   500 18794  1085555    12123   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18798]   500 18798  1085556    12617   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18802]   500 18802  1085555    11689   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18816]   500 18816  1085427    11444   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18820]   500 18820  1085556    12778   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [18826]   500 18826  1085558    12594   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19174]   500 19174  1085429    41253   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19178]   500 19178  1085558    13927   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19190]   500 19190  1085554    29965   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19194]   500 19194  1085556    31096   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19198]   500 19198  1085557    30335   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19202]   500 19202  1085557    31748   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19206]   500 19206  1085558    32826   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19210]   500 19210  1085557    31519   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19214]   500 19214  1085429    30709   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19218]   500 19218  1085557    49490   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19224]   500 19224  1085430    12430   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19228]   500 19228  1085558    30815   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19232]   500 19232  1085558    12180   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19236]   500 19236  1085558    13343   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19240]   500 19240  1085556    13531   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19246]   500 19246  1085556    12964   5       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19250]   500 19250  1085557    12055   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19254]   500 19254  1085428    12637   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19268]   500 19268  1085555    12174   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19272]   500 19272  1085555    11174   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19276]   500 19276  1085555    29864   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19284]   500 19284  1085555    13576   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19288]   500 19288  1085555    12498   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19292]   500 19292  1085430    12981   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19296]   500 19296  1085554    11902   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19300]   500 19300  1085557    12560   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19304]   500 19304  1085557    31831   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [19308]   500 19308  1085556    12863   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [20142]   500 20142  1085556    41578   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [20146]   500 20146  1085555    12236   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [20150]   500 20150  1085557    13443   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [20154]   500 20154  1085558    12453   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [20158]   500 20158  1085428    12460   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [20162]   500 20162  1085555    13408   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [20168]   500 20168  1085557    31637   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [20172]   500 20172  1085555    12439   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [20178]   500 20178  1085556    13113   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [21264]   500 21264  1085558    41565   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [21630]   500 21630  1085557    45061   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [21634]   500 21634  1085985   109203   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [21638]   500 21638  1085946   135602   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [21642]   500 21642  1085558    37078   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22477]   500 22477  1085557    30752   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22495]   500 22495  1085558    12364   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22499]   500 22499  1085554    12332   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22503]   500 22503  1085557    30919   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22563]   500 22563  1085557    30068   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22567]   500 22567  1085428    12130   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22573]   500 22573  1085555    12743   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22577]   500 22577  1085555    41735   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22581]   500 22581  1085555    12192   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22585]   500 22585  1085558    42835   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22589]   500 22589  1085557    31250   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22593]   500 22593  1085429    12481   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22597]   500 22597  1085555    12491   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22601]   500 22601  1085556    13819   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22605]   500 22605  1085557    13720   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22609]   500 22609  1085554    11855   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22613]   500 22613  1085558    12944   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22617]   500 22617  1085554    30163   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22623]   500 22623  1085557    30428   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22627]   500 22627  1085558    12718   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22633]   500 22633  1085429    30834   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22685]   500 22685  1085558    31137   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22689]   500 22689  1085428    12694   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22693]   500 22693  1085554    12004   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22697]   500 22697  1085556    12623   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22703]   500 22703  1085557    36371   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22707]   500 22707  1085556    12987   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22711]   500 22711  1085557    13038   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22715]   500 22715  1085554    41183   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22719]   500 22719  1085557    41557   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22723]   500 22723  1085557    37651   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22727]   500 22727  1085557    14376   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22731]   500 22731  1085557    11939   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22735]   500 22735  1085555    41536   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22739]   500 22739  1085555    12501   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22743]   500 22743  1085554    12068   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22749]   500 22749  1085557    30880   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22753]   500 22753  1085555    14220   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22757]   500 22757  1085557    42491   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22761]   500 22761  1085429    12769   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22767]   500 22767  1085429    32130   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22771]   500 22771  1085555    11716   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22775]   500 22775  1085557    13544   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22779]   500 22779  1085558    43748   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22783]   500 22783  1085557    31105   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22787]   500 22787  1085556    14735   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22791]   500 22791  1085429    12277   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22795]   500 22795  1085558    12722   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22799]   500 22799  1085558    14014   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22803]   500 22803  1085556    12220   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22809]   500 22809  1085558    30848   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22813]   500 22813  1085170     6899   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22817]   500 22817  1085557    12752   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22821]   500 22821  1085556    13173   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22825]   500 22825  1085554    12238   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22829]   500 22829  1085556    30503   4       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22885]   500 22885  1085554    11271   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22889]   500 22889  1085557    12351   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22901]   500 22901  1085557    43318   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22955]   500 22955  1085555    30844   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23323]   500 23323  1085558    12256   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23327]   500 23327  1085557    13352   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23331]   500 23331  1085556    11873   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23335]   500 23335  1085555    30041   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23339]   500 23339  1085555    12160   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23345]   500 23345  1085556    41322   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23349]   500 23349  1085558    12080   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23353]   500 23353  1085557    12148   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23357]   500 23357  1085556    30705   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23361]   500 23361  1085558    11996   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23365]   500 23365  1085555    30734   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23369]   500 23369  1085557    12099   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23373]   500 23373  1085555    42331   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23379]   500 23379  1085428    11944   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23383]   500 23383  1085556    11421   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23387]   500 23387  1085557    14116   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23391]   500 23391  1085558    31611   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23397]   500 23397  1085554    11753   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23403]   500 23403  1085556    11849   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23407]   500 23407  1085558    12469   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23411]   500 23411  1085555    11600   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23417]   500 23417  1085554    12683   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23421]   500 23421  1085554    29833   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23489]   500 23489  1085428    31228   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [26174]   500 26174  1085569    10892   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [26178]   500 26178  1085551     8455   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [26182]   500 26182  1085294     7532   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [26186]   500 26186  1085294     7714   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [26190]   500 26190  1085550     9570   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [ 7399]   500  7399  1085825    34706   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [20727]   500 20727  1084863     5047   1       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [20840]   500 20840  1085816    37158   3       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [22402]   500 22402  1086558     7226   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23116]   500 23116  1085560    25352   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23138]   500 23138  1085554    33359   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23182]   500 23182  1085815    45277   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23200]   500 23200  1085559    36379   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23204]   500 23204  1085556    24560   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23322]   500 23322  1085683    32003   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23631]   500 23631  1085815    37122   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23635]   500 23635  1085816    49491   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23639]   500 23639  1085560    53198   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23657]   500 23657  1085559    41668   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23665]   500 23665  1085815    41192   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23741]   500 23741  1085559    42575   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23767]   500 23767  1085814    45467   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23837]   500 23837  1085559    40316   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23863]   500 23863  1085559    41834   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [23875]   500 23875  1085814    40051   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [24843]   500 24843  1085162     5817   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [24847]   500 24847  1085162     5900   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [24851]   500 24851  1085162     6259   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [24855]   500 24855  1085162     5886   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [24859]   500 24859  1085162     5888   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [24869]   500 24869  1085680    27042   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [24873]   500 24873  1085162     5959   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [24877]   500 24877  1085162     5964   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [24881]   500 24881  1085162     5969   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [24885]   500 24885  1085162     5981   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [25069]   500 25069  1085559    34675   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [25073]   500 25073  1085546     9324   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [25077]   500 25077  1085555    20105   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [25087]   500 25087  1085556    24367   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [27133]     0 27133    35053      437   0       0             0 crond
Jun 25 01:08:09 SIP-DB kernel: [27134]     0 27134    40821      429   0       0             0 su
Jun 25 01:08:09 SIP-DB kernel: [27136]   500 27136    26526      309   2       0             0 rman_backup_lv1
Jun 25 01:08:09 SIP-DB kernel: [27152]   500 27152    34069    14062   1       0             0 rman
Jun 25 01:08:09 SIP-DB kernel: [27169]   500 27169  1093001    25319   6       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [27172]   500 27172  1085312     7537   1       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [27176]   500 27176  1092998    14943   5       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [27179]   500 27179  1092991    12997   0       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: [27340]   500 27340  1084862     4749   5       0             0 oracle
Jun 25 01:08:09 SIP-DB kernel: Out of memory: Kill process 30344 (oracle) score 54 or sacrifice child
Jun 25 01:08:09 SIP-DB kernel: Killed process 30344, UID 500, (oracle) total-vm:4339452kB, anon-rss:2132kB, file-rss:1343184kB


发现是内存不足导致系统把oracle进程杀掉了。

网上搜了一下oom


oom-killer 触发的原因都是内存不足导致的,而内存不足通常是由于内存超分配导致的。

关于linux内核的超分配是由overcommit_memory参数控制的(/proc/sys/vm/overcommit_memory),

关于参数的解释大神都有很详细的解释了,我就直接拿来用了:

  • 0:启发式策略,比较严重的Overcommit将不能得逞,比如你突然申请了128TB的内存。

    而轻微的Overcommit将被允许。另外,root能Overcommit的值比普通用户要稍微多些。

  • 1:永远允许Overcommit,这种策略适合那些不能承受内存分配失败的应用,比如某些科学计算应用。

  • 2:永远禁止Overcommit,在这个情况下,系统所能分配的内存不会超过swap+RAM*系数

    (/proc/sys/vm/overcmmit_ratio,默认50%,你可以调整), 如果这么多资源

    已经用光,那么后面任何尝试申请内存的行为都会返回错误,这通常意味着此时没法运行任何新程序。

查看了一下我的配置

[root@TESTDB ~]# cat  /proc/sys/vm/swappiness

0

果然是0,怪不得系统日志里看swap都没用到,就把oracle干掉了。

修改一下在观察观察吧。

echo 60 > /proc/sys/vm/swappiness



另外发现一个帖子也有可能用到:

http://www.xifenfei.com/3400.html



阅读更多
想对作者说点什么?

博主推荐

换一批

没有更多推荐了,返回首页