详细解读Linux存储性能观测中的iostat命令

iostat(input/output statistics)报告中央处理器（CPU）统计信息和整个系统、适配器、tty 设备、磁盘和 CD-ROM 的输入/输出统计信息。

特点

• 不能单独对某个进程进行分析，仅能够分析系统的整体情况

安装方式：

#iostat属于sysstat软件包。
yum install sysstat 

iostat常用命令格式：

iostat [参数] [时间] [次数]

参数说明：

-c 显示CPU使用情况
-d 显示磁盘使用情况
-k 以K为单位显示
-m 以M为单位显示
-n 显示NFS使用情况
-N 显示磁盘阵列(LVM) 信息
-p 可以报告出每块磁盘的每个分区的使用情况
-t 显示每秒向终端读取和写入的字符数和CPU的信息
-x 显示详细信息

命令常用形式

$ iostat //显示所有设备负载情况
Linux 3.****.el7.x86_64  	2020年11月01日 	_x86_64_	(32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.27    0.00    0.30    0.14    0.00   96.29

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdb              28.90        38.34       636.19   51208534  849699418
sda               3.71       397.60       216.96  531032721  289779808
nvme1n1           1.31         0.02        25.64      33365   34245848
nvme0n1           0.00         0.00         0.00       5749       5560

cpu属性值说明：

• %user：CPU处在用户模式下的时间百分比。
• %nice：CPU处在带NICE值的用户模式下的时间百分比。
• %system：CPU处在系统模式下的时间百分比。
• %iowait：CPU等待输入输出完成时间的百分比。
• %steal：管理程序维护另一个虚拟处理器时，虚拟CPU的无意识等待时间百分比。
• %idle：CPU空闲时间百分比。

Device属性值说明：

• tps：设备每秒的传输次数,也就是每秒I/O次数（IOPS,磁盘连续读和连续写之和）
• kB_read/s：每秒从设备读出的数据量（kB/s）
• kB_wrtn/s：每秒向设备写入的数据量（kB/s）
• kB_read： 读出数据 的总量（kB）
• kB_wrtn：写入数据 的总量（kB）

$ iostat -x //显示详细信息
Linux 3.****.el7.x86_64  	2020年11月03日 	_x86_64_	(32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.28    0.00    0.30    0.14    0.00   96.28

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0.00     6.43    0.33   28.59    38.29   636.54    46.66     1.23   42.48    4.90   42.92   0.48   1.40
sda               0.00     0.72    1.77    1.94   397.06   216.68   330.99     0.98  264.50  278.53  251.63   1.80   0.67
nvme1n1           0.00     3.19    0.01    1.31     0.02    25.64    39.08     0.00    0.81    0.29    0.82   0.02   0.00
nvme0n1           0.00     0.00    0.00    0.00     0.00     0.00    18.11     0.00    0.06    0.08    0.01   0.03   0.00

Device详细属性值说明：

• rrqm/s：每秒合并到设备的读取请求数
• wrqm/s：每秒合并到设备的写请求数
• r/s：每秒向磁盘发起的读操作数
• w/s：每秒向磁盘发起的写操作数
• rkB/s：每秒读多少个K字节
• wkB/s: 每秒写多少个K字节
• avgrq-sz：平均每次设备I/O操作的数据大小
• avgqu-sz：平均I/O队列长度
• await：平均每次设备I/O操作的等待时间(ms)，一般，系统I/O响应时间应该低于5ms。若大于10ms就是较大了
• r_await：每个读操作平均所需的时间。（包括硬盘设备读操作的时间，在kernel队列中等待的时间）
• w_await：每个写操作平均所需的时间。（包括硬盘设备写操作的时间，在kernel队列中等待的时间）
• svctm：平均每次设备I/O操作的服务时间 (ms）
• %util：一秒中有百分之多少的时间用于I/O操作（被IO消耗的CPU百分比），一般地，如果是100%表示设备已经接近满负荷运行

#显示指定磁盘信息
$iostat -d /dev/sda
Linux 3.****.el7.x86_64  	2020年11月03日 	_x86_64_	(32 CPU)

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               3.70       395.65       215.94  531032721  289822720

# 每隔1秒刷新显示，显示2次
$iostat 1  2

观测分析

• %iowait过高的话，则表示硬盘存在I/O瓶颈。
• %idle很高但是系统响应很慢，可能是CPU等待分配内存导致的。应加大内存容量。
  -%idle如果持续低于10，则表明CPU处理能力相对较低，也就是说CPU是系统中的瓶颈了。
• %util差不多到100%，说明产生的I/O请求过多，I/O系统接近满负荷
• avgqu-sz：如果I/O请求压力持续超出磁盘处理能力，该值将增加，因为会造成拥塞。
  • 如果单块磁盘的队列长度持续超过2，一般认为该磁盘存在I/O性能问题。
  • 对于顺序读写频繁的，需要传输大块数据的任务，需要关注磁盘的吞吐量（ kB_read/s ，kB_wrtn/s）。
• 对于磁盘随机读写频繁的任务，需要关注（tps）

英文说明

$ man iostat
NAME
       iostat - Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.

SYNOPSIS
       iostat  [  -c  ]  [  -d ] [ -h ] [ -k | -m ] [ -N ] [ -t ] [ -V ] [ -x ] [ -y ] [ -z ] [ -j { ID | LABEL | PATH | UUID | ... } ] [ [ -T ] -g group_name ] [ -p [ device
       [,...] | ALL ] ] [ device [...] | ALL ] [ interval [ count ] ]

DESCRIPTION
       The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer  rates.
       The iostat command generates reports that can be used to change system configuration to better balance the input/output load between physical disks.

       The  first report generated by the iostat command provides statistics concerning the time since the system was booted, unless the -y option is used (in this case, this
       first report is omitted).  Each subsequent report covers the time since the previous report. All statistics are reported each time  the  iostat  command  is  run.  The
       report  consists  of  a  CPU header row followed by a row of CPU statistics. On multiprocessor systems, CPU statistics are calculated system-wide as averages among all
       processors. A device header row is displayed followed by a line of statistics for each device that is configured.

       The interval parameter specifies the amount of time in seconds between each report. The first report contains statistics for the  time  since  system  startup  (boot),
       unless  the  -y  option  is  used  (in this case, this report is omitted).  Each subsequent report contains statistics collected during the interval since the previous
       report. The count parameter can be specified in conjunction with the interval parameter. If the count parameter is specified, the value of count determines the  number
       of reports generated at interval seconds apart. If the interval parameter is specified without the count parameter, the iostat command generates reports continuously.

REPORTS
       The iostat command generates two types of reports, the CPU Utilization report and the Device Utilization report.

       CPU Utilization Report
              The first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all proces‐
              sors.  The report has the following format:

              %user
                     Show the percentage of CPU utilization that occurred while executing at the user level (application).

              %nice
                     Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.

              %system
                     Show the percentage of CPU utilization that occurred while executing at the system level (kernel).

              %iowait
                     Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

              %steal
                     Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.

              %idle
                     Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

       Device Utilization Report
              The second report generated by the iostat command is the Device Utilization Report. The device report provides statistics on a per physical device or  partition
              basis.  Block  devices  and partitions for which statistics are to be displayed may be entered on the command line.  If no device nor partition is entered, then
              statistics are displayed for every device used by the system, and providing that the kernel maintains statistics for it.  If the ALL keyword  is  given  on  the
              command  line,  then statistics are displayed for every device defined by the system, including those that have never been used.  Transfer rates are shown in 1K
              blocks by default, unless the environment variable POSIXLY_CORRECT is set, in which case 512-byte blocks are used.  The report may show  the  following  fields,
              depending on the flags used:

              Device:
                     This column gives the device (or partition) name as listed in the /dev directory.

              tps
                     Indicate  the number of transfers per second that were issued to the device. A transfer is an I/O request to the device. Multiple logical requests can be
                     combined into a single I/O request to the device. A transfer is of indeterminate size.

              Blk_read/s (kB_read/s, MB_read/s)
                     Indicate the amount of data read from the device expressed in a number of blocks (kilobytes, megabytes) per second. Blocks are equivalent to sectors  and
                     therefore have a size of 512 bytes.

              Blk_wrtn/s (kB_wrtn/s, MB_wrtn/s)
                     Indicate the amount of data written to the device expressed in a number of blocks (kilobytes, megabytes) per second.

              Blk_read (kB_read, MB_read)
                     The total number of blocks (kilobytes, megabytes) read.

              Blk_wrtn (kB_wrtn, MB_wrtn)
                     The total number of blocks (kilobytes, megabytes) written.

              rrqm/s
                     The number of read requests merged per second that were queued to the device.

              wrqm/s
                     The number of write requests merged per second that were queued to the device.

              r/s
                     The number (after merges) of read requests completed per second for the device.

              w/s
                     The number (after merges) of write requests completed per second for the device.

              rsec/s (rkB/s, rMB/s)
                     The number of sectors (kilobytes, megabytes) read from the device per second.

              wsec/s (wkB/s, wMB/s)
                     The number of sectors (kilobytes, megabytes) written to the device per second.

              avgrq-sz
                     The average size (in sectors) of the requests that were issued to the device.

              avgqu-sz
                     The average queue length of the requests that were issued to the device.

              await
                     The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time
                     spent servicing them.

              r_await
                     The average time (in milliseconds) for read requests issued to the device to be served. This includes the time spent by the requests  in  queue  and  the
                     time spent servicing them.

              w_await
                     The  average  time  (in milliseconds) for write requests issued to the device to be served. This includes the time spent by the requests in queue and the
                     time spent servicing them.

              svctm
                     The average service time (in milliseconds) for I/O requests that were issued to the device. Warning! Do not trust this field any more.  This  field  will
                     be removed in a future sysstat version.

              %util
                     Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this
                     value is close to 100%.

OPTIONS
       -c     Display the CPU utilization report.

       -d     Display the device utilization report.

       -g group_name { device [...] | ALL }
              Display statistics for a group of devices.  The iostat command reports statistics for each individual device in the list then a line of  global  statistics  for
              the  group  displayed  as group_name and made up of all the devices in the list. The ALL keyword means that all the block devices defined by the system shall be
              included in the group.

       -h     Make the Device Utilization Report easier to read by a human.

       -j { ID | LABEL | PATH | UUID | ... } [ device [...] | ALL ]
              Display persistent device names. Options ID, LABEL, etc. specify the type of the persistent name. These options are  not  limited,  only  prerequisite  is  that
              directory  with  required  persistent names is present in /dev/disk.  Optionally, multiple devices can be specified in the chosen persistent name type.  Because
              persistent device names are usually long, option -h is enabled implicitly with this option.

       -k     Display statistics in kilobytes per second.

       -m     Display statistics in megabytes per second.

       -N     Display the registered device mapper names for any device mapper devices.  Useful for viewing LVM2 statistics.

       -p [ { device [,...] | ALL } ]
              The -p option displays statistics for block devices and all their partitions that are used by the system.  If a device name is entered on the command line, then
              statistics  for it and all its partitions are displayed. Last, the ALL keyword indicates that statistics have to be displayed for all the block devices and par‐
              titions defined by the system, including those that have never been used. If option -j is defined before this option, devices entered on the command line can be
              specified with the chosen persistent name type.

       -T     This  option  must be used with option -g and indicates that only global statistics for the group are to be displayed, and not statistics for individual devices
              in the group.

       -t     Print the time for each report displayed. The timestamp format may depend on the value of the S_TIME_FORMAT environment variable (see below).

       -V     Print version number then exit.

       -x     Display extended statistics.

       -y     Omit first report with statistics since system boot, if displaying multiple records at given interval.

       -z     Tell iostat to omit output for any devices for which there was no activity during the sample period.

ENVIRONMENT
       The iostat command takes into account the following environment variables:

       S_TIME_FORMAT
              If this variable exists and its value is ISO then the current locale will be ignored when printing the date in the report header. The iostat  command  will  use
              the ISO 8601 format (YYYY-MM-DD) instead.  The timestamp displayed with option -t will also be compliant with ISO 8601 format.

       POSIXLY_CORRECT
              When this variable is set, transfer rates are shown in 512-byte blocks instead of the default 1K blocks.

EXAMPLES
       iostat
              Display a single history since boot report for all CPU and Devices.

       iostat -d 2
              Display a continuous device report at two second intervals.

       iostat -d 2 6
              Display six reports at two second intervals for all devices.

       iostat -x sda sdb 2 6
              Display six reports of extended statistics at two second intervals for devices sda and sdb.

       iostat -p sda 2 6
              Display six reports at two second intervals for device sda and all its partitions (sda1, etc.)

BUGS
       /proc filesystem must be mounted for iostat to work.

       Kernels older than 2.6.x are no longer supported.

       The  average  service  time  (svctm  field) value is meaningless, as I/O statistics are now calculated at block level, and we don't know when the disk driver starts to
       process a request. For this reason, this field will be removed in a future sysstat version.

FILES
       /proc/stat contains system statistics.

       /proc/uptime contains system uptime.

       /proc/diskstats contains disks statistics.

       /sys contains statistics for block devices.

       /proc/self/mountstats contains statistics for network filesystems.

       /dev/disk contains persistent device names.

AUTHOR
       Sebastien Godard (sysstat <at> orange.fr)

SEE ALSO
       sar(1), pidstat(1), mpstat(1), vmstat(8), nfsiostat(1), cifsiostat(1)

       http://pagesperso-orange.fr/sebastien.godard/

Linux                                                                            NOVEMBER 2012