linux内存节省优化,Linux性能优化-内存问题排查

目录

相关命令

cachestat 和 cachetop都用到的一些字段，man解释如下

TIME Timestamp.

HITS Number of page cache hits.

MISSES Number of page cache misses.

DIRTIES

Number of dirty pages added to the page cache.

READ_HIT%

Read hit percent of page cache usage.

WRITE_HIT%

Write hit percent of page cache usage.

BUFFERS_MB

Buffers size taken from /proc/meminfo.

CACHED_MB

Cached amount of data in current page cache taken from /proc/meminfo.

Ubuntu 二进制安装这两个工具

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD

echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/iovisor.list

sudo apt-get update

sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)

基于二进制安装pcstat

if [ $(uname -m) == "x86_64" ] ; then

curl -L -o pcstat https://github.com/tobert/pcstat/raw/2014-05-02-01/pcstat.x86_64

else

curl -L -o pcstat https://github.com/tobert/pcstat/raw/2014-05-02-01/pcstat.x86_32

fi

chmod 755 pcstat

执行pcstat的结果

pcstat /bin/cat hehe.log

|----------+----------------+------------+-----------+---------|

| Name | Size | Pages | Cached | Percent |

|----------+----------------+------------+-----------+---------|

| /bin/cat | 35064 | 9 | 0 | 000.000 |

| hehe.log | 25 | 1 | 0 | 000.000 |

|----------+----------------+------------+-----------+---------|

cat hehe.log

aaaaaaa

bbbbbbbbbb

ccccc

#第二次就执，数据就被缓存了

pcstat /bin/cat hehe.log

|----------+----------------+------------+-----------+---------|

| Name | Size | Pages | Cached | Percent |

|----------+----------------+------------+-----------+---------|

| /bin/cat | 35064 | 9 | 9 | 100.000 |

| hehe.log | 25 | 1 | 1 | 100.000 |

|----------+----------------+------------+-----------+---------|

/bin/cat 的大小是35064字节，一个页面大小是4K，所以 35064/(4*1024.0) = 8.5，也就是占用了9个页面

测试缓存命中情况

用dd写入一个文件，再反复读取这个文件

dd if=/dev/sda1 of=file bs=1M count=512

echo 3 > /proc/sys/vm/drop_caches

#这个时候缓存是空的

pcstat file

|----------+----------------+------------+-----------+---------|

| Name | Size | Pages | Cached | Percent |

|----------+----------------+------------+-----------+---------|

| file | 536870912 | 131072 | 0 | 000.000 |

|----------+----------------+------------+-----------+---------|

测试读取数据

dd if=file of=/dev/null bs=1M

512+0 records in

512+0 records out

536870912 bytes (537 MB, 512 MiB) copied, 5.04981 s, 106 MB/s

cachetop

PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT%

3928 root python 5 0 0 100.0% 0.0%

3972 root python 5 0 0 100.0% 0.0%

4066 root dd 86868 85505 0 50.4% 49.6%

#第二次读取

dd if=file of=/dev/null bs=1M

512+0 records in

512+0 records out

536870912 bytes (537 MB, 512 MiB) copied, 0.182855 s, 2.9 GB/s

cachetop

PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT%

4079 root bash 197 0 0 100.0% 0.0%

4079 root dd 131605 0 0 100.0% 0.0%

可以看到第二次读取的时候性能大幅度提升了，再看pcstat情况

pcstat file

|----------+----------------+------------+-----------+---------|

| Name | Size | Pages | Cached | Percent |

|----------+----------------+------------+-----------+---------|

| file | 536870912 | 131072 | 131072 | 100.000 |

|----------+----------------+------------+-----------+---------|

测试direct I/O

用dd读取一个文件，加入direct标志

dd if=file of=/dev/null bs=1M iflag=direct

512+0 records in

512+0 records out

536870912 bytes (537 MB, 512 MiB) copied, 4.91659 s, 109 MB/s

通过监控命令观察运行情况

cachetop 3

14:14:13 Buffers MB: 9 / Cached MB: 614 / Sort: HITS / Order: ascending

PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT%

4161 root python 1 0 0 100.0% 0.0%

4162 root dd 518 0 0 100.0% 0.0%

这里对 dd 监控的结果是每秒钟 HITS是 518，cachetop是3秒监控一次

518*4/1024.0/3.0，也就是每秒读取0.67M的数据

通过strace dd 看结果，再读 file  这个文件的时候，确实是用了 O_DIRECT标志

openat(AT_FDCWD, "file", O_RDONLY|O_DIRECT) = 3

dup2(3, 0) = 0

close(3) = 0

lseek(0, 0, SEEK_CUR) = 0

openat(AT_FDCWD, "/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

用dstat看，dd读取的那段时间，iowait也是很高的

把dd 的直接I/O选项去掉，再执行一次

echo 3 > /proc/sys/vm/drop_caches

dd if=file of=/dev/null bs=1M

512+0 records in

512+0 records out

536870912 bytes (537 MB, 512 MiB) copied, 4.91158 s, 109 MB/s

cachetop

PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT%

4397 root python 2 0 0 100.0% 0.0%

4398 root dd 34198 33027 0 50.9% 49.1%

这里对 dd 监控的结果是每秒钟的 HITS是34198，cachetop是3秒监控一次

34198*4/1024.0/3.0，也就是每秒钟读取44M的数据，这次是正常了

关于O_DIRECT 标志的说明

O_DIRECT (since Linux 2.4.10)

Try to minimize cache effects of the I/O to and from this

file. In general this will degrade performance, but it is

useful in special situations, such as when applications do

their own caching. File I/O is done directly to/from user-

space buffers. The O_DIRECT flag on its own makes an effort

to transfer data synchronously, but does not give the

guarantees of the O_SYNC flag that data and necessary metadata

are transferred. To guarantee synchronous I/O, O_SYNC must be

used in addition to O_DIRECT. See NOTES below for further

discussion.

直接I/O一般是上层应用有自己的缓存系统，就不需要操作系统级别的缓存了

直接读写磁盘一般是用于 存储系统的场合，比如数据库，文件系统，读写的时候可以绕过操作系统的文件系统这一层

参考