主要针对Linux系统的Bootchart文件进行分析,关于怎么采集Bootchart可以查看官方文档。
前言
本文主要针对Bootchart生成的proc_diskstats.log文件进行分析
Bootchart参考文章:elinux-Bootchart
一、proc_diskstats.log文件含义
顾名思义,proc_diskstats主要代表磁盘状态,当开启采集bootchart时,会在/data/bootchart/目录下生成proc_diskstats.log文件。这个文件用于显示磁盘、分区和统计信息,比起常规的cat /proc/diskstats
会增加一行采集信息的时间。
z3q:/ # cat /proc/diskstats
1 0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 1 ram1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 2 ram2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 3 ram3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 4 ram4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 5 ram5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 6 ram6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 7 ram7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 8 ram8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 9 ram9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 10 ram10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 11 ram11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 12 ram12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 13 ram13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 14 ram14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 15 ram15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 0 loop0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 1 loop1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 2 loop2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 3 loop3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 4 loop4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 5 loop5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 6 loop6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 7 loop7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 0 sda 9267 171 938604 3934 13 0 56 6 0 1850 4220 0 0 0 0
8 1 sda1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 2 sda2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 3 sda3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 4 sda4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 5 sda5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 6 sda6 9264 171 938580 3933 7 0 56 5 0 1850 4220 0 0 0 0
8 16 sdb 7610 16 159552 2050 300 534 29696 550 0 880 2780 0 0 0 0
8 17 sdb1 111 0 900 53 3 0 24 2 0 20 70 0 0 0 0
8 18 sdb2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 19 sdb3 7497 16 158636 1996 285 534 29672 545 0 860 2710 0 0 0 0
proc_diskstats.log文件样式如下,其中507、528、549即为采集时间:
507
1 0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 1 ram1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
···
1 14 ram14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 15 ram15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7 0 loop0 35 1 3080 16 0 0 0 0 0 12 20 0 0 0 0
···
7 120 loop15 10 0 80 4 0 0 0 0 0 4 4 0 0 0 0
254 0 zram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 0 sda 5190 4845 255144 5611 16 0 80 8 0 984 5556 0 0 0 0
8 1 sda1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
···
259 18 sda34 1979 2543 89608 3023 3 0 24 0 0 260 3388 0 0 0 0
8 16 sdb 5 0 40 22 0 0 0 0 0 24 24 0 0 0 0
8 17 sdb1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8 32 sdc 5 0 40 6 0 0 0 0 0 8 8 0 0 0 0
8 48 sdd 5 0 40 3 0 0 0 0 0 4 4 0 0 0 0
8 64 sde 66 1 1080 42 0 0 0 0 0 28 32 0 0 0 0
···
8 80 sdf 5 0 40 4 0 0 0 0 0 0 0 0 0 0 0
···
8 90 sdf10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
31 0 mtdblock0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
252 0 dm-0 3311 0 105200 4568 0 0 0 0 0 380 4568 0 0 0 0
···
7 128 loop16 12 0 96 5 0 0 0 0 0 8 8 0 0 0 0
252 15 dm-15 6 0 48 4 0 0 0 0 0 4 4 0 0 0 0
7 136 loop17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
252 16 dm-16 8 0 64 4 0 0 0 0 0 4 4 0 0 0 0
528
1 0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
···
8 0 sda 5236 4845 255512 5632 16 0 80 8 0 1012 5596 0 0 0 0
···
252 20 dm-20 13 0 104 12 0 0 0 0 0 4 12 0 0 0 0
549
1 0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
二、proc_diskstats.log文件解析
1. 参数解释
cat /proc/diskstats
具体参数可以参考[Linux 运维 – 存储] /proc/diskstats详解和www.kernel.org
以sda举例
# sda
8 0 sda 9267 171 938604 3934 13 0 56 6 0 1850 4220 0 0 0 0
# 第1列为 为设备号 - 8
# 第2列为 次设备号 - 0
# 第3列为 设备名称 - sda
# 第4列为 成功完成读的总次数 - 9267
# 第5列为 合并读次数 - 171 (为了效率可能会合并相邻的读和写,从而两次4K的读在它最终被处理到磁盘上之前可能会变成一次8K的读,才被计数(和排队),因此只有一次I/O操作。)
# 第6列为 读扇区的次数 - 938604 (读扇区大小,成功读过的扇区总次数)
# 第7列为 读花的时间(ms) - 3934
# 第8~11分别是写, 与4~7读对应, 依次是 13 0 56 0
# 第12列为 I/O的当前进度 - 0 (只有这个域应该是0,如果这个值为0,同时write_complete read_complete io_processing 一直不变可能就就是IO hang了)
# 第13列为 输入输入花的时间(ms) - 1850 (花在I/O操作上的毫秒数,这个域会增长只要field 就不为0)
# 第14列为 输入/输出操作花费的加权毫秒数 - 4220
# 第15列为 丢弃完成的成功数 - 0
# 第16列为 丢弃合并数 - 0
# 第17列为 丢弃扇区数 - 0
# 第18列为 丢弃花费时间(ms) - 0
# quote https://www.kernel.org/doc/Documentation/ABI/testing/procfs-diskstats
What: /proc/diskstats
Date: February 2008
Contact: Jerome Marchand <jmarchan@redhat.com>
Description:
The /proc/diskstats file displays the I/O statistics
of block devices. Each line contains the following 14
fields:
== ===================================
1 major number
2 minor mumber
3 device name
4 reads completed successfully
5 reads merged
6 sectors read
7 time spent reading (ms)
8 writes completed
9 writes merged
10 sectors written
11 time spent writing (ms)
12 I/Os currently in progress
13 time spent doing I/Os (ms)
14 weighted time spent doing I/Os (ms)
== ===================================
Kernel 4.18+ appends four more fields for discard
tracking putting the total at 18:
== ===================================
15 discards completed successfully
16 discards merged
17 sectors discarded
18 time spent discarding
== ===================================
Kernel 5.5+ appends two more fields for flush requests:
== =====================================
19 flush requests completed successfully
20 time spent flushing
== =====================================
For more details refer to Documentation/admin-guide/iostats.rst
2. 文件解析
查看pybootchartgui源码,可以看到关于diskstats的解析,下面给出关键代码
def _parse_proc_disk_stat_log(file, numCpu):
"""
Parse file for disk stats, but only look at the whole device, eg. sda,
not sda1, sda2 etc. The format of relevant lines should be:
{major minor name rio rmerge rsect ruse wio wmerge wsect wuse running use aveq}
"""
disk_regex_re = re.compile ('^([hsv]d.|mtdblock\d|mmcblk\d|cciss/c\d+d\d+.*)$')
# this gets called an awful lot.
def is_relevant_line(linetokens):
if len(linetokens) < 14:
return False
disk = linetokens[2]
return disk_regex_re.match(disk)
disk_stat_samples = []
for time, lines in _parse_timed_blocks(file):
sample = DiskStatSample(time)
relevant_tokens = [linetokens for linetokens in map (lambda x: x.split(),lines) if is_relevant_line(linetokens)]
for tokens in relevant_tokens:
disk, rsect, wsect, use = tokens[2], int(tokens[5]), int(tokens[9]), int(tokens[12])
sample.add_diskdata([rsect, wsect, use])
disk_stat_samples.append(sample)
disk_stats = []
for sample1, sample2 in zip(disk_stat_samples[:-1], disk_stat_samples[1:]):
interval = sample1.time - sample2.time
if interval == 0:
interval = 1
sums = [ a - b for a, b in zip(sample1.diskdata, sample2.diskdata) ]
readTput = sums[0] / 2.0 * 100.0 / interval
writeTput = sums[1] / 2.0 * 100.0 / interval
util = float( sums[2] ) / 10 / interval / numCpu
util = max(0.0, min(1.0, util))
disk_stats.append(DiskSample(sample2.time, readTput, writeTput, util))
return disk_stats
- is_relevant_line 函数是找出相关的sda、sdb、hda、vda、mtdblock0、mmcblk0等,忽略ram、sda1等
disk_regex_re = re.compile ('^([hsv]d.|mtdblock\d|mmcblk\d|cciss/c\d+d\d+.*)$')
disk = 'sda'
print(disk_regex_re.match(disk))
<re.Match object; span=(0, 3), match='sda'>
disk = 'sda1'
print(disk_regex_re.match(disk))
None
disk = 'mmcblk0'
print(disk_regex_re.match(disk))
<re.Match object; span=(0, 7), match='mmcblk0'>
- _parse_timed_blocks 这个函数是将proc_diskstats.log进行分割,_parse_timed_blocks(file)返回一个列表,元素为元组形式,第一个元素为时间,第二个元素为按行读取的列表。
# 仅作示意
[(507, ['1 0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0','1 1 ram1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'],
[528, ['1 0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'])]
- 接下来解析出所有磁盘(sda\sdb\sdc所有的)本次采样时间的采样时间time、设备名disk、删除读取次数rsect、扇区写入次数wsect、 输入输出花的时间use
- 两次采样差值为变化量
interval = sample1.time - sample2.time # 采样时间差,ms
readTput = sums[0] / 2.0 * 100.0 / interval # sums[0] 为两次采样读扇区次数差
writeTput = sums[1] / 2.0 * 100.0 / interval # sums[1] 为两次采样写扇区次数差
util = float( sums[2] ) / 10 / interval / numCpu # sums[2] 为两次采样读写时间差
util = max(0.0, min(1.0, util))
disk_stats.append(DiskSample(sample2.time, readTput, writeTput, util))
扇区大小为512字节