转自： http://hi.baidu.com/clusterlee/blog/item/661665eb279071c0d439c972.html
如何查看具体进程的IO情况？

一.前言

   linux下有大量的系统命令vmstat、iostat等可以反映系统的总体io情况，但是不能监测具体进程的io情况，本文将介绍两种方法：

   1.如果内核版本大于2.6.20，通过 /proc/pid/io 便可以获取进程的io信息。

   2.通过echo 1 > /proc/sys/vm/block_dump ，来把 block 读写（WRITE/READ/DIRTY）状况 dump 到日志里，通过 dmesg 命令来查看。

 


二.  /proc/pid/io

(详细解释：http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/proc.txt;hb=HEAD)

 

示例： 
test:/tmp# dd if=/dev/zero of=/tmp/test.dat &
 [1] 3828
 
test:/tmp # cat /proc/3828/io
 rchar: 323934931         // 读出的总字节数，read或者pread()中的长度参数总和（pagecache中统计而来，不代表实际磁盘的读入）
 wchar: 323929600        // 写入的总字节数，write或者pwrite中的长度参数总和
 syscr: 632687               // read()或者pread()总的调用次数 
 syscw: 632675              // write()或者pwrite()总的调用次数 
 read_bytes: 0                      // 实际从磁盘中读取的字节总数   (这里if=/dev/zero 所以没有实际的读入字节数)
 write_bytes: 323932160      // 实际写入到磁盘中的字节总数
 cancelled_write_bytes: 0     // 由于截断pagecache导致应该发生而没有发生的写入字节数（可能为负数）
 
检测工具dstat (http://dag.wieers.com/home-made/dstat/dstat-0.7.2.tar.bz2 ) 的top_io 模块就是利用此功能实现。dstat的详细说明可以到这里http://dag.wieers.com/home-made/dstat/
 
 
三.block_dump
 
具体操作步骤是： 
#sysctl vm.block_dump=1
or
# echo 1 > /proc/sys/vm/block_dump

然后就可以通过 dmesg 就可以观察到各个进程 IO 活动的状况了：

#dmesg -c
kjournald(542): WRITE block 222528 on dm-0
kjournald(542): WRITE block 222552 on dm-0
bash(18498): dirtied inode 5892488 (ld-linux-x86-64.so.2) on dm-0
bash(18498): dirtied inode 5892482 (ld-2.5.so) on dm-0
dmesg(18498): dirtied inode 11262038 (ld.so.cache) on dm-0
dmesg(18498): dirtied inode 5892496 (libc.so.6) on dm-0
dmesg(18498): dirtied inode 5892489 (libc-2.5.so) on dm-0

通过一些脚本分析这些数据就可以得到各个进程的IO情况。下文便描述了这种方法（不同版本的linux下这里dmesg日志格式可能不同）

How to find per-process I/O statistics on Linux

 

Newer Linux kernels have per-process I/O accounting and you can use the iotop tool to find out what’s performing I/O, but in many cases I’m trying to find the source of an I/O problem in an older kernel. I found sort of a hack-ish way to do that today, while trying to figure out why a system was basically unresponsive.

 

I founda post on Stack Overflowthat showed a way you can get per process I/O statistics from the kernel even in older kernels. I adapted this to my needs, and wrote a little script.

Here’s how you use it. First, get it:

wget http://aspersa.googlecode.com/svn/trunk/iodump

Then turn on kernel messages about I/O:

echo 1 > /proc/sys/vm/block_dump

This makes the kernel start writing messages about every I/O operation that takes place. Now all you have to do is get those messages and feed them into my script:

while true; do sleep 1; dmesg -c; done | perl iodump

Wait a little while, then cancel the script. The results should look something like the following:

root@kanga:~# while true; do sleep 1; dmesg -c; done | perl iodump
^C# Caught SIGINT.
TASK                   PID      TOTAL       READ      WRITE      DIRTY DEVICES
firefox               4450       4538        251       4287          0 sda4, sda3
kjournald             2100        551          0        551          0 sda4
firefox              28452        185        185          0          0 sda4
kjournald              782         59          0         59          0 sda3
pdflush                 31         30          0         30          0 sda4, sda3
syslogd               2485          2          0          2          0 sda3
firefox              28414          2          2          0          0 sda4, sda3
firefox              28413          1          1          0          0 sda4
firefox              28410          1          1          0          0 sda4
firefox              28307          1          1          0          0 sda4
firefox              28451          1          1          0          0 sda4

I deliberately generated a bunch

of I/O by deleting my Firefox history and cache.