这个是用户另外一个项目DB服务器,DBA最近撤了
关键词:page in,内存不足引起大io,大iowait
环境:10.2 RAC ,AIX 5L
System Model: IBM,8844-IB6
Machine Serial Number: xxxxx
Processor Type: PowerPC_POWER
Number Of Processors: 4
Processor Clock Speed: 2498 MHz
CPU Type: 64-bit
Memory Size: 16064 MB
表现就是慢(用户说的),其中一个节点慢
自然要先看操作系统性能
[qasmdm2racle] /oracle/app/oracle/product/10.2.0/dbs> vmstat 1
System Configuration: lcpu=4 mem=16064MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 0 4258106 114674 0 15 0 0 0 0 838 91554 3085 42 11 38 8
2 0 4258100 114651 0 28 0 0 0 0 1167 58717 3838 39 7 43 10
2 0 4258100 114625 0 26 0 0 0 0 1011 50930 3323 44 5 42 10
6 0 4258208 114505 0 12 0 0 0 0 1202 66228 3846 40 6 43 12
2 1 4259282 113280 0 149 0 0 0 0 1346 67959 4000 44 10 25 21
1 1 4259300 113065 0 198 0 0 0 0 1124 46216 3314 42 8 22 28
1 3 4259257 112906 0 201 0 0 0 0 1638 68837 5059 39 8 25 28
2 0 4261895 110153 0 113 0 0 0 0 1194 49596 3650 47 7 26 20
从这里看到pi 比较多,这是内存不足的典型特征!
但是ps aux没看到什么特殊进程占内存(除了Oracle进程)
经历了半小时左右的瞎看.....
还被awr的top wait events误导了(做了个10分钟的awr):
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) ( ms) Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
latch: library cache 6,868 1,826 266 43.2 Concurrenc
latch: shared pool 6,866 1,770 258 41.8 Concurrenc
CPU time 351 8.3
enq: TX - row lock contention 709 340 480 8.0 Applicatio
latch: row cache objects 750 161 214 3.8 Concurrenc
经过再询问环境,发现:
1.这个节点上还有一个developer库.
2.检查RAC instance的内存分配:
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_target big integer 1596M
sga_max_size big integer 4800M
sga_target big integer 4800M
dev库的内存分配
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_target big integer 2G
sga_max_size big integer 12G
sga_target big integer 6G
这肯定有问题了,怪不得内存不够用呢
显然的问题: SGA设置的太大,物理内存不足造成页面交换,进而hdisk0,hdisk1 io经常为90%以上。
缩小dev库的sga_max_size,sga_target,pga_aggregate_target 之后重起instance之后恢复正常:
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 0 2918295 1027498 0 1 0 0 0 0 1461 42828 5236 53 8 27 12
1 0 2918291 1027499 0 0 0 0 0 0 2932 60337 6900 52 10 29 9
1 0 2918981 1026849 0 0 0 266 786 0 2713 47645 4803 57 10 23 10
2 0 2919057 1026772 0 0 0 0 0 0 2061 41243 5226 39 7 32 23
[ 本帖最后由 wa0362 于 2008-6-30 10:49 编辑 ]
关键词:page in,内存不足引起大io,大iowait
环境:10.2 RAC ,AIX 5L
System Model: IBM,8844-IB6
Machine Serial Number: xxxxx
Processor Type: PowerPC_POWER
Number Of Processors: 4
Processor Clock Speed: 2498 MHz
CPU Type: 64-bit
Memory Size: 16064 MB
表现就是慢(用户说的),其中一个节点慢
自然要先看操作系统性能
[qasmdm2racle] /oracle/app/oracle/product/10.2.0/dbs> vmstat 1
System Configuration: lcpu=4 mem=16064MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 0 4258106 114674 0 15 0 0 0 0 838 91554 3085 42 11 38 8
2 0 4258100 114651 0 28 0 0 0 0 1167 58717 3838 39 7 43 10
2 0 4258100 114625 0 26 0 0 0 0 1011 50930 3323 44 5 42 10
6 0 4258208 114505 0 12 0 0 0 0 1202 66228 3846 40 6 43 12
2 1 4259282 113280 0 149 0 0 0 0 1346 67959 4000 44 10 25 21
1 1 4259300 113065 0 198 0 0 0 0 1124 46216 3314 42 8 22 28
1 3 4259257 112906 0 201 0 0 0 0 1638 68837 5059 39 8 25 28
2 0 4261895 110153 0 113 0 0 0 0 1194 49596 3650 47 7 26 20
从这里看到pi 比较多,这是内存不足的典型特征!
但是ps aux没看到什么特殊进程占内存(除了Oracle进程)
经历了半小时左右的瞎看.....
还被awr的top wait events误导了(做了个10分钟的awr):
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) ( ms) Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
latch: library cache 6,868 1,826 266 43.2 Concurrenc
latch: shared pool 6,866 1,770 258 41.8 Concurrenc
CPU time 351 8.3
enq: TX - row lock contention 709 340 480 8.0 Applicatio
latch: row cache objects 750 161 214 3.8 Concurrenc
经过再询问环境,发现:
1.这个节点上还有一个developer库.
2.检查RAC instance的内存分配:
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_target big integer 1596M
sga_max_size big integer 4800M
sga_target big integer 4800M
dev库的内存分配
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_target big integer 2G
sga_max_size big integer 12G
sga_target big integer 6G
这肯定有问题了,怪不得内存不够用呢
显然的问题: SGA设置的太大,物理内存不足造成页面交换,进而hdisk0,hdisk1 io经常为90%以上。
缩小dev库的sga_max_size,sga_target,pga_aggregate_target 之后重起instance之后恢复正常:
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 0 2918295 1027498 0 1 0 0 0 0 1461 42828 5236 53 8 27 12
1 0 2918291 1027499 0 0 0 0 0 0 2932 60337 6900 52 10 29 9
1 0 2918981 1026849 0 0 0 266 786 0 2713 47645 4803 57 10 23 10
2 0 2919057 1026772 0 0 0 0 0 0 2061 41243 5226 39 7 32 23
[ 本帖最后由 wa0362 于 2008-6-30 10:49 编辑 ]
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/10701850/viewspace-526733/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/10701850/viewspace-526733/