当时场景是VM上iowait和%util都是100%。
物理机稍微好点, iowait 30%~50%, %util也近100%
后面实在不行就对物理机进行重启,启动进不去系统,准备重装时发现硬盘灯有黄灯。由于有raid,把坏硬盘拔掉,就工作正常了。
中间定位iowait的过程涉及命令有:
# iostat -x 1
Linux 2.6.32-220.el6.x86_64 (SN2012-04-010) 02/22/2013 _x86_64_ (24 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
4.76 0.00 1.25 0.00 0.00 93.99
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.01 15.93 0.21 10.29 24.87 177.08 19.23 0.00 0.31 0.11 0.11
....
# iostat -c 1
Linux 2.6.32-220.el6.x86_64 (SN2012-04-010) 02/22/2013 _x86_64_ (24 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
4.76 0.00 1.25 0.00 0.00 93.99
...
# vmstat -n 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 1002228 15545828 165784 12225632 0 0 1 4 0 0 5 1 94 0 0
1 0 1002228 15546048 165784 12225632 0 0 0 0 30811 31426 8 2 91 0 0
blockdump
/etc/init.d/syslog stop
echo 1 > /proc/sys/vm/block_dump
sleep 60
dmesg | awk '/(READ|WRITE|dirtied)/ {process[$1]++} END {for (x in process) \
print process[x],x}' |sort -nr |awk '{print $2 " " $1}' | \
head -n 10
echo 0 > /proc/sys/vm/block_dump
/etc/init.d/syslog start
其它杂七杂八
du -h, lsof -p 1234