目录
相关命令
cachestat 和 cachetop都用到的一些字段,man解释如下
TIME Timestamp.
HITS Number of page cache hits.
MISSES Number of page cache misses.
DIRTIES
Number of dirty pages added to the page cache.
READ_HIT%
Read hit percent of page cache usage.
WRITE_HIT%
Write hit percent of page cache usage.
BUFFERS_MB
Buffers size taken from /proc/meminfo.
CACHED_MB
Cached amount of data in current page cache taken from /proc/meminfo.
Ubuntu 二进制安装这两个工具
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)
基于二进制安装pcstat
if [ $(uname -m) == "x86_64" ] ; then
curl -L -o pcstat https://github.com/tobert/pcstat/raw/2014-05-02-01/pcstat.x86_64
else
curl -L -o pcstat https://github.com/tobert/pcstat/raw/2014-05-02-01/pcstat.x86_32
fi
chmod 755 pcstat
执行pcstat的结果
pcstat /bin/cat hehe.log
|----------+----------------+------------+-----------+---------|
| Name | Size | Pages | Cached | Percent |
|----------+----------------+------------+-----------+---------|
| /bin/cat | 35064 | 9 | 0 | 000.000 |
| hehe.log | 25 | 1 | 0 | 000.000 |
|----------+----------------+------------+-----------+---------|
cat hehe.log
aaaaaaa
bbbbbbbbbb
ccccc
#第二次就执,数据就被缓存了
pcstat /bin/cat hehe.log
|----------+----------------+------------+-----------+---------|
| Name | Size | Pages | Cached | Percent |
|----------+----------------+------------+-----------+---------|
| /bin/cat | 35064 | 9 | 9 | 100.000 |
| hehe.log | 25 | 1 | 1 | 100.000 |
|----------+----------------+------------+-----------+---------|
/bin/cat 的大小是35064字节,一个页面大小是4K,所以 35064/(4*1024.0) = 8.5,也就是占用了9个页面
测试缓存命中情况
用dd写入一个文件,再反复读取这个文件
dd if=/dev/sda1 of=file bs=1M count=512
echo 3 > /proc/sys/vm/drop_caches
#这个时候缓存是空的
pcstat file
|----------+----------------+------------+-----------+---------|
| Name | Size | Pages | Cached | Percent |
|----------+----------------+------------+-----------+---------|
| file | 536870912 | 131072 | 0 | 000.000 |
|----------+----------------+------------+-----------+---------|
测试读取数据
dd if=file of=/dev/null bs=1M
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 5.04981 s, 106 MB/s
cachetop
PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT%
3928 root python 5 0 0 100.0% 0.0%
3972 root python 5 0 0 100.0% 0.0%
4066 root dd 86868 85505 0 50.4% 49.6%
#第二次读取
dd if=file of=/dev/null bs=1M
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 0.182855 s, 2.9 GB/s
cachetop
PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT%
4079 root bash 197 0 0 100.0% 0.0%
4079 root dd 131605 0 0 100.0% 0.0%
可以看到第二次读取的时候性能大幅度提升了,再看pcstat情况
pcstat file
|----------+----------------+------------+-----------+---------|
| Name | Size | Pages | Cached | Percent |
|----------+----------------+------------+-----------+---------|
| file | 536870912 | 131072 | 131072 | 100.000 |
|----------+----------------+------------+-----------+---------|
测试direct I/O
用dd读取一个文件,加入direct标志
dd if=file of=/dev/null bs=1M iflag=direct
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 4.91659 s, 109 MB/s
通过监控命令观察运行情况
cachetop 3
14:14:13 Buffers MB: 9 / Cached MB: 614 / Sort: HITS / Order: ascending
PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT%
4161 root python 1 0 0 100.0% 0.0%
4162 root dd 518 0 0 100.0% 0.0%
这里对 dd 监控的结果是每秒钟 HITS是 518,cachetop是3秒监控一次
518*4/1024.0/3.0,也就是每秒读取0.67M的数据
通过strace dd 看结果,再读 file 这个文件的时候,确实是用了 O_DIRECT标志
openat(AT_FDCWD, "file", O_RDONLY|O_DIRECT) = 3
dup2(3, 0) = 0
close(3) = 0
lseek(0, 0, SEEK_CUR) = 0
openat(AT_FDCWD, "/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
用dstat看,dd读取的那段时间,iowait也是很高的
把dd 的直接I/O选项去掉,再执行一次
echo 3 > /proc/sys/vm/drop_caches
dd if=file of=/dev/null bs=1M
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 4.91158 s, 109 MB/s
cachetop
PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT%
4397 root python 2 0 0 100.0% 0.0%
4398 root dd 34198 33027 0 50.9% 49.1%
这里对 dd 监控的结果是每秒钟的 HITS是34198,cachetop是3秒监控一次
34198*4/1024.0/3.0,也就是每秒钟读取44M的数据,这次是正常了
关于O_DIRECT 标志的说明
O_DIRECT (since Linux 2.4.10)
Try to minimize cache effects of the I/O to and from this
file. In general this will degrade performance, but it is
useful in special situations, such as when applications do
their own caching. File I/O is done directly to/from user-
space buffers. The O_DIRECT flag on its own makes an effort
to transfer data synchronously, but does not give the
guarantees of the O_SYNC flag that data and necessary metadata
are transferred. To guarantee synchronous I/O, O_SYNC must be
used in addition to O_DIRECT. See NOTES below for further
discussion.
直接I/O一般是上层应用有自己的缓存系统,就不需要操作系统级别的缓存了
直接读写磁盘一般是用于 存储系统的场合,比如数据库,文件系统,读写的时候可以绕过操作系统的文件系统这一层
参考