浅析vmstat命令

vmstat命令从名称可以看出是关于虚拟内存统计信息的，本文针对aix环境下的vmstat命令进行简单语法说明。

命令格式：

usage: vmstat [ -fsviItlw ] [ [ -p | -P ]  pagesize | ALL ] [Drives] [ Interval [Count] ]

命令用途：

The vmstat command reports statistics about kernel threads, virtual memory, disks, traps and pocessor activity. Reports generated by the vmstat command can be used to balance system load activity. These system-wide statistics (among all processors) are calculated as averages for values expressed as percentages, and as sums otherwise. Thevmstat command might return inconsistent statistics because the statistics are not read atomically.

vmstat命令用来报告与内核线程，虚拟内存，磁盘，陷阱和处理器活动有关的统计信息。根据该命令生成的报告我们可以平衡系统的活动负载。这些系统范围的统计信息按照其平均值的百分比形式或者累积总值的形式展现出来。由于该命令在统计信息的收集时并不是原子性的，因此统计信息可能会存在不一致性。

If the vmstat command is invoked without flags, the report contains a summary of the virtual memory activity since system startup. 

The Interval parameter specifies the amount of time in seconds between each report. If the Interval parameter is not specified, the vmstat command generates a single report that contains statistics for the time since system startup and then exits. The Count parameter can only be specified with the Interval parameter. If the Count parameter is specified, its value determines the number of reports generated and the number of seconds apart. If the Interval parameter is specified without the Count parameter, reports are continuously generated. A Count parameter of 0 is not allowed.

如果没有指定任何选项，则vmstat命令返回自系统启动以来虚拟内存活动的总统计信息。

Interval参数指定了每次报告之间的时间间隔，如果没有指定interval参数，vmstat命令生成自系统启动以来所有统计信息的单一报告。count参数需要与interval参数配合使用，如果指定了count参数，该参数决定了报告的生成数量，如果指定了interval但是没有指定count，报告会一直生成下去。count的取值不可以为0.

AIX 4.3.3 and later contain enhancements to the method used to compute the percentage of processor time spent waiting on disk I/O (wio time). The method used in AIX 4.3.2 and earlier versions of the operating system can, under certain circumstances, give an inflated view of wio time on SMPs.

The method used in AIX 4.3.2 and earlier versions is as follows: At each clock interrupt on each processor (100 times a second per processor), a determination is made as to which of the four categories (usr/sys/wio/idle) to place the last 10 ms of time. If the processor was busy in usr mode at the time of the clock interrupt, then usr gets the clock tick added into its category. If the processor was busy in kernel mode at the time of the clock interrupt, then the sys category gets the tick. If the processor was not busy, a check is made to see if any I/O to disk is in progress. If any disk I/O is in progress, the wio category is incremented. If no disk I/O is in progress and the processor is not busy, the idle category gets the tick. The inflated view of wio time results from all idle processors being categorized as wio regardless of the number of threads waiting on I/O. For example, systems with just one thread doing I/O could report over 90 percent wio time regardless of the number of processors it has. The wio time is reported by the commands sar(%wio), vmstat (wa) and iostat (% iowait).

AIX 4.3.3及其后续版本对cpu花费在DISK I/O(wio)上的时间计算方法进行了增强，在以前的版本中，在特定的情况下，会过高估计SMPs环境下wio的数值。

在AIX 4.3.2和之前的版本中，计算方法如下：在每个处理器的时钟中断时（每个处理器没秒钟发生100次），会判定最后的10ms时间属于usr/sys/wio/idle四分类中的哪个分类，如果在时钟中断时，处理器处于usrmode，那么其从属与usr分类。如果处理器处于kernel mode，那么从属于sys分类。如果处理器是空闲的，会坚持时钟存在磁盘输入输出操作，如果存在磁盘输入输出操作，那么从属于wio分类，否则从属于idle分类。之所以会过高估计wio的值，是因为当多个处理器空闲时，不管存在多少等待输入输出的线程，这些处理器都会被判定为wio。例如在仅仅具有单一线程处理输入输出时，报告中的wio可能会达到90%的比例，而不考虑cpu的数量。

The kernel maintains statistics for kernel threads, paging, and interrupt activity, which the vmstat command accesses through the use of the the perfstat kernel extension. The disk input/output statistics are maintained by device drivers. For disks, the average transfer rate is determined by using the active time and number of transfers information. The percent active time is computed from the amount of time the drive is busy during the report.

The PhysicalVolume parameter can be used to specify one to four names. Transfer statistics are given for each specified drive in the order specified. This count represents requests to the physical device. It does not imply an amount of data that was read or written. Several logical requests can be combined into one physical request. If the PhysicalVolume parameter is used, the physical volume names are printed at the beginning of command execution.

vmstat通过perfstat内核扩展来维护内核线程，分页和中断活动的统计信息。而硬盘输入输入的统计信息是有磁盘启动类维护的，对于磁盘来说，平均传输速率是有活动时间和传输数量来决定的。活动时间是报告生成期间启动处于忙状态的时间。

我们可以指定1-4个物理卷，磁盘传输统计是根据指定的物理卷参数顺序来显示的。统计信息体现了对物理设备的请求，与发生的读或者写请求是不一致的，因为多个逻辑请求可能会合并为单个物理请求。

Beginning with AIX 5.3, the vmstat command reports the number of physical processors consumed (pc), and the percentage of entitlement consumed (ec), in Micro-Partitioning environments. These metrics will only be displayed on Micro-Partitioning environments.

下面是对各个统计信息字段的含义解释

kthr: information about kernel thread states.

r	Average number of runnable kernel threads over the sampling interval. Runnable refers to threads that are ready but waiting to run and to those threads already running. 如果超过cpu数量过多，则系统繁忙
b	Average number of kernel threads placed in the VMM wait queue (awaiting resource, awaiting input/output) over the sampling interval.

Memory: information about the usage of virtual and real memory. Virtual pages are considered active if they have been accessed. A page is 4096 bytes.

avm	Active virtual pages.
fre	Size of the free list. Note: A large portion of real memory is utilized as a cache for file system data. It is not unusual for the size of the free list to remain small.

Page: information about page faults and paging activity. These are averaged over the interval and given in units per second.

re	Pager input/output list.
pi	Pages paged in from paging space.
po	Pages paged out to paging space.
fr	Pages freed (page replacement).
sr	Pages scanned by page-replacement algorithm.
cy	Clock cycles by page-replacement algorithm.

Faults: trap and interrupt rate averages per second over the sampling interval.

in	Device interrupts.
sy	System calls.
cs	Kernel thread context switches.

Cpu: breakdown of percentage usage of processor time.

us	User time.
sy	System time.
id	Processor idle time.
wa	Processor idle time during which the system had outstanding disk/NFS I/O request(s). See detailed description above.
pc	Number of physical processors consumed. Displayed only if the partition is running with shared processor.
ec	The percentage of entitled capacity consumed. Displayed only if the partition is running with shared processor. Because the time base over which this data is computed can vary, the entitled capacity percentage can sometimes exceed 100%. This excess is noticeable only with small sampling intervals.

Disk: Provides the number of transfers per second to the specified physical volumes that occurred in the sample interval. 

If the -I flag is specified, an I/O oriented view is presented with the following column changes.

kthr	The column p will also be displayed besides columns r and b. p Number of threads waiting on I/O to raw devices per second.
page	New columns fi and fo will be displayed instead of re and cy columns. fi File page-ins per second. fo File page-outs per second.

If, while the vmstat command is running, there is a change in system configuration that will affect the output, vmstat prints a warning message about the configuration change. It then continues the output, after printing the updated system configuration information and the header.

If the -l flag is specified, an additional "large-page" section is displayed with the following columns:

alp

Indicates the number of large pages currently in use.

flp

Indicates the number of large pages on the large page freelist.

If the -p option is specified, additional lines of VMM statistics are displayed for the specified page sizes. With -I and -t options, the -p option produces an additional line for the specified page size. This line contains the following VMM statistics relevant to the specified page size:

• avm
• fre
• re
• fi
• fo
• pi
• po
• fr
• sr
• cy

Note:

The disply of the  re,  fi,  fo, and  cy options are affected by the  -I option.

These VMM statistics are preceded by a psz column and followed by an siz column. The description of these two columns follows:

psz

Page size (for example, 4K, 64K).

siz

Number of frames of the specified page size that exist on the system.

With the -s option, the -p option produces a separate stanza of output that contains only the statistics relevant to the specified page size. This additional stanza is preceded by a page size header.

The -P option produces the following report for the specified page size:

pgsz

Indicates the page size (for example, 4K, 64K).

Memory

Indicates the memory statistics for the specified page sizes.

siz

The number of frames of the specified page size that exist on the system.

avm

Active virtual pages applicable to the specified page size.

fre

Size of the free list for the specified page size.

Page

Indicates the relevant page faults and paging activity for the specified page size. The page related columns  re,  pi,  po,  fr,  sr,  cy,  fi, and  fo are also applicable to this report.

Flags

Note:

If the  -f (or  -s) flag is entered on the command line, then the system will only accept the  -f (or  -s) flag and will ignore other flags. If both  -f and  -s flags are specified, the system will accept only the first flag and ignore the second flag.

-f	Reports the number of forks since system startup.
-i	Displays the number of interrupts taken by each device since system startup. Note: The  -I,  -t,  -w, and  -l flags are ignored when they are specified with the  -i flag.
-I	Displays I/O oriented view with the new columns of output, p under heading kthr,and columns fi and fo under heading page instead of the columns re and cy in the page heading.
-l	Displays an additional "large-page" section with the alp and flp columns.
-p pagesize	Appends the VMM statistics for the specified page size to the regular vmstat output.
-P pagesize	Displays only the VMM statistics which are relevant for the specified page size.
-s	Writes to standard output the contents of the sum structure, which contains an absolute count of paging events since system initialization. The-s flag can only be used with the -v flag. These events are described as follows: address translation faults Incremented for each occurrence of an address translation page fault. I/O may or may not be required to resolve the page fault. Storage protection page faults (lock misses) are not included in this count. page ins Incremented for each page read in by the virtual memory manager. The count is incremented for page ins from page space and file space. Along with the page out statistic, this represents the total amount of real I/O initiated by the virtual memory manager. page outs Incremented for each page written out by the virtual memory manager. The count is incremented for page outs to page space and for page outs to file space. Along with the page in statistic, this represents the total amount of real I/O initiated by the virtual memory manager. paging space page ins Incremented for VMM initiated page ins from paging space only. paging space page outs Incremented for VMM initiated page outs to paging space only. total reclaims Incremented when an address translation fault can be satisfied without initiating a new I/O request. This can occur if the page has been previously requested by VMM, but the I/O has not yet completed; or if the page was pre-fetched by VMM's read-ahead algorithm, but was hidden from the faulting segment; or if the page has been put on the free list and has not yet been reused. zero-filled page faults Incremented if the page fault is to working storage and can be satisfied by assigning a frame and zero-filling it. executable-filled page faults Incremented for each instruction page fault. pages examined by the clock VMM uses a clock-algorithm to implement a pseudo least recently used (lru) page replacement scheme. Pages are  aged by being examined by the clock. This count is incremented for each page examined by the clock. revolutions of the clock hand Incremented for each VMM clock revolution (that is, after each complete scan of memory). pages freed by the clock Incremented for each page the clock algorithm selects to free from real memory.
	backtracks Incremented for each page fault that occurs while resolving a previous page fault. (The new page fault must be resolved first and then initial page faults can be  backtracked.) free frame waits Incremented each time a process requests a page frame, the free list is empty, and the process is forced to wait while the free list is replenished. extend XPT waits Incremented each time a process is waited by VMM due to a commit in progress for the segment being accessed. pending I/O waits Incremented each time a process is waited by VMM for a page-in I/O to complete. start I/Os Incremented for each read or write I/O request initiated by VMM. iodones Incremented at the completion of each VMM I/O request. CPU context switches Incremented for each processor context switch (dispatch of a new process). device interrupts Incremented on each hardware interrupt. software interrupts Incremented on each software interrupt. A software interrupt is a machine instruction similar to a hardware interrupt that saves some state and branches to a service routine. System calls are implemented with software interrupt instructions that branch to the system call handler routine. decrementer interrupts Incremented on each decrementer interrupt. mpc send interrupts Incremented on each mpc send interrupt mpc receive interrupts Incremented on each mpc receive interrupt phantom interrupts Incremented on each phantom interrupt traps Not maintained by the operating system. syscalls Incremented for each system call.
	Notes: When used with the -p pagesize option, the -s option appends the sum structure for the specified page size to the system-wide sum structure. This additional stanza is preceded by a page size header (for example, 4K pages). The following details are not be displayed in this pagesize-based stanza as these statistics are not related to page sizes: Processor context switches Device interrupts Software interrupts Decrementer interrupts MPC-sent interrupts MPC-received interrupts Phantom interrupts Traps Syscalls When the -s flag is used with the -l flag, the vmstat command displays the following metric: large-page hi water count Specifies the maximum value of the large-page inuse count.
-t	Prints the time-stamp next to each line of output of vmstat. The time-stamp is displayed in the HH:MM:SS format. Note: Time stamp will not be printed if  -f,  -s, or  -i flags are specified.
-v	Writes to standard output various statistics maintained by the Virtual Memory Manager. The -v flag can only be used with the -s flag. memory pages Size of real memory in number of 4 KB pages. lruable pages Number of 4 KB pages considered for replacement. This number excludes the pages used for VMM internal pages, and the pages used for the pinned part of the kernel text. free pages Number of free 4 KB pages. memory pools Tuning parameter (managed using  vmo) specifying the number of memory pools. pinned pages Number of pinned 4 KB pages. maxpin percentage Tuning parameter (managed using  vmo) specifying the percentage of real memory which can be pinned. minperm percentage Tuning parameter (managed using  vmo) in percentage of real memory. This specifies the point below which file pages are protected from the re-page algorithm. maxperm percentage Tuning parameter (managed using  vmo) in percentage of real memory. This specifies the point above which the page stealing algorithm steals only file pages. numperm percentage Percentage of memory currently used by the file cache. file pages Number of 4 KB pages currently used by the file cache. compressed percentage Percentage of memory used by compressed pages. compressed pages Number of compressed memory pages. numclient percentage Percentage of memory occupied by client pages. maxclient percentage Tuning parameter (managed using vmo) specifying the maximum percentage of memory which can be used for client pages. client pages Number of client pages. remote pageouts scheduled Number of pageouts scheduled for client filesystems. pending disk I/Os blocked with no pbuf Number of pending disk I/O requests blocked because no pbuf was available. Pbufs are pinned memory buffers used to hold I/O requests at the logical volume manager layer. paging space I/Os blocked with no psbuf Number of paging space I/O requests blocked because no psbuf was available. Psbufs are pinned memory buffers used to hold I/O requests at the virtual memory manager layer.
-v	(Statistics displayed by -v, continued): filesystem I/Os blocked with no fsbuf Number of filesystem I/O requests blocked because no fsbuf was available. Fsbuf are pinned memory buffers used to hold I/O requests in the filesystem layer. client filesystem I/Os blocked with no fsbuf Number of client filesystem I/O requests blocked because no fsbuf was available. NFS (Network File System) and VxFS (Veritas) are client filesystems. Fsbuf are pinned memory buffers used to hold I/O requests in the filesystem layer. external pager filesystem I/Os blocked with no fsbuf Number of external pager client filesystem I/O requests blocked because no fsbuf was available. JFS2 is an external pager client filesystem. Fsbuf are pinned memory buffers used to hold I/O requests in the filesystem layer. Virtualized Partition Memory Page Faults Total number of virtual partition memory page faults recorded for the virtualized partition. Time resolving virtualized partition memory page faults Total time (in seconds, with millisecond granularity) the virtual partition has been blocked waiting for the resolution of the virtual partition's memory page faults.
-w	Display the report in wide mode.

IBMP570A/>vmstat -w

System Configuration: lcpu=16 mem=31616MB

 kthr          memory                         page                       faults           cpu    
------- --------------------- ------------------------------------ ------------------ -----------
  r   b        avm        fre    re    pi    po    fr     sr    cy    in     sy    cs us sy id wa
  2   1    8301946      56037     0     2     5   191    907     0  2281  67381  7744 15  4 79  2
IBMP570A/>vmstat -w 2

System Configuration: lcpu=16 mem=31616MB

 kthr          memory                         page                       faults           cpu    
------- --------------------- ------------------------------------ ------------------ -----------
  r   b        avm        fre    re    pi    po    fr     sr    cy    in     sy    cs us sy id wa
  1   0    8302032      55913     0     0     0     0      0     0  1921  50607  6873 18  2 78  1
  2   0    8300623      57322     0     0     0     0      0     0  1931  58353  6869 22  2 74  1
  1   0    8301821      56124     0     0     0     0      0     0  1961  53174  6902 20  2 76  1
IBMP570A/>vmstat -wi
priority level    type   count module(handler)
    0       2   hardware 9975749 i_mpc_int_handler(1541520.16)
    0  589825   hardware     0 /usr/lib/drivers/pci/sisraid_dd(3fd0e70.16)
    0  589825   hardware     0 /usr/lib/drivers/planar_pal_chrp(3f30bd0.16)
    0  589825   hardware     0 /usr/lib/drivers/pci/sisraid_dd(3fd0e70.16)
    1  655360   hardware     1 /usr/lib/drivers/vconsdd(3f77918.16)
    1  655361   hardware    52 /usr/lib/drivers/vconsdd(3f77918.16)
    3      83   hardware    27 /usr/lib/drivers/pci/sisraid_dd(3fd0ea0.16)
    3     115   hardware -2120194071 /usr/lib/drivers/pci/goentdd(3ffd9d0.16)
    3     117   hardware 42240924 /usr/lib/drivers/pci/sisraid_dd(3fd0ea0.16)
    3     163   hardware    32 /usr/lib/drivers/pci/sisraid_dd(3fd0ea0.16)
    3     167   hardware 1090002304 /etc/drivers/pci/efcddpin(403fec8.16)
    3   65619   hardware    27 /usr/lib/drivers/pci/sisraid_dd(3fd0ea0.16)
    3   65653   hardware 32898602 /usr/lib/drivers/pci/sisraid_dd(3fd0ea0.16)
    3   65701   hardware -1514225260 /usr/lib/drivers/pci/goentdd(3ffd9d0.16)
    3   65703   hardware 830860 /etc/drivers/pci/efcddpin(403fec8.16)
    4      85   hardware 139288 /usr/lib/drivers/pci/hcd33103500(42f3aa0.16)
    4      85   hardware 140784 /usr/lib/drivers/pci/hcd33103500(42f3aa0.16)
    4   65621   hardware 139264 /usr/lib/drivers/pci/hcd33103500(42f3aa0.16)
    4   65621   hardware 139263 /usr/lib/drivers/pci/hcd33103500(42f3aa0.16)
IBMP570A/>vmstat -wI 2

System Configuration: lcpu=16 mem=31616MB

   kthr            memory                         page                       faults           cpu    
----------- --------------------- ------------------------------------ ------------------ -----------
  r   b   p        avm        fre    fi    fo    pi    po    fr     sr    in     sy    cs us sy id wa
  2   0   0    8307718      50224     0     0     0     0     0      0  1167  47566  5060 18  2 79  0
  2   0   0    8307587      50355     0     0     0     0     0      0  2376  65138  7929 23  3 74  0
  2   0   0    8302153      55789     0     0     0     0     0      0   849  47837  4501 20  2 77  1
  2   0   0    8301990      55951     0    12     0     0     0      0   670  49942  4049 24  2 73  0
IBMP570A/>vmstat -wl 2

System Configuration: lcpu=16 mem=31616MB

 kthr          memory                         page                       faults           cpu        large-page    
------- --------------------- ------------------------------------ ------------------ ----------- -----------------
  r   b        avm        fre    re    pi    po    fr     sr    cy    in     sy    cs us sy id wa      alp      flp
  1   0    8311552      46385     0     0     0     0      0     0  1659  25855  6214 11  2 85  1        0        0
IBMP570A/>wmvstat -w -p 4096
ksh: wmvstat:  not found
IBMP570A/>vmstat -w -p 4096

System Configuration: lcpu=16 mem=31616MB

 kthr          memory                         page                       faults           cpu    
------- --------------------- ------------------------------------ ------------------ -----------
  r   b        avm        fre    re    pi    po    fr     sr    cy    in     sy    cs us sy id wa
  2   1    8330991      26931     0     2     5   191    907     0  2281  67381  7744 15  4 79  2

    psz        avm        fre    re    pi    po    fr     sr    cy     siz
     4K    8026320      18194     0     2     5   191    907     0 7780288 
IBMP570A/>vmstat -w -P all

System configuration: mem=31616MB

pgsz            memory                           page                 
----- -------------------------- ------------------------------------ 
           siz      avm      fre    re    pi    po    fr     sr    cy 
   4K  7780288  7997083    47424     0     2     5   191    907     0 
  64K    19588    19052      536     0     0     0     0      0     0