Another introduction of vmstat (ZZ)

Reading vmstat in linux – Part 1

Linux comes with many tools to enable administrators to evaluate the performance of a system. One of these very useful tools is vmstat. Vmstat is available on most unix distributions and is included by default on many modern Linux distributions. As with most Linux commands, built in help is available via the command man vmstat.

First we will review what the different columns means and then we will review the output that would be consistent with a few common bottlenecks.

Sample vmstat output:

procs      ———–memory———-             —swap–       —–io—- –system–              —–cpu——
r  b        swpd   free   buff  cache                   si   so          bi    bo   in   cs               us sy id wa st
1  1       5888 140136 167700 1506440        0    0          10     2    0    0                   6  2 90  3  0
0  1       5888 147864 167724 1507492        0    0       1164   340 2092 4506          8  3 76 13  0
0  1       5888 147220 167760 1508584        0    0       1240   212 2059 2539          2  1 84 13  0

Procs

r: The number of processes waiting for run time.
b: The number of processes in uninterruptible sleep.

Uninterruptible sleep usually represents processes that are waiting on input to be processed.  This could either be because it is waiting for user input or disk i/o operations to complete.

Memory

swpd: the amount of virtual memory used.
free: the amount of idle memory.
buff: the amount of memory used as buffers.
cache: the amount of memory used as cache.
inact: the amount of inactive memory. (-a option)
active: the amount of active memory. (-a option)

Swap

si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).

IO

bi: Blocks received from a block device (blocks/s).
bo: Blocks sent to a block device (blocks/s).

System

in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.

CPU

These are percentages of total CPU time.
us: Time spent running non-kernel code. (user time, including nice time)
sy: Time spent running kernel code. (system time)
id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.
st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.

Reading vmstat in linux – Part 2

In the second part of this series, we are going to target 3 different types of potential bottlenecks that a system might suffer from.  Specifically these are:

• IO Bound
• CPU Bound
• Memory Bound

Most performance problems can be placed into one of these categories.  By using what we learned in Part 1, we can use the information from vmstat to provide more information about what aspect of a system is limiting it from handling a greater load….and therefore what should be upgraded to provided greater capacity.

IO Bound System

An IO bound system is a machine that is usually suffering from performance limitations due to disk speed.  While there are other types of IO a server could be waiting on, usually it is disk and disk speed that are the problems.  This can be culled from reviewing vmstat by looking at both the uninterruptable sleep column (b column) and a high value in the wa column.    An example from vmstate would look as follows:

#vmstat 5

procs      ———–memory———-             —swap–       —–io—- –system–                          —–cpu——
r  b        swpd   free   buff  cache                   si   so          bi    bo   in   cs                             us sy id wa st

1  8      5888 140136 167700 1506440        0    0        6783 17333  1738  16648                   6  2 9  83  0
1  6      5888 140136 167700 1506440        0    0        6885 18353  1831  18658                   8  4 9  77  0
1  6      5888 140136 167700 1506440        0    0        6482 12323  1534  17632                   16  12 9  63  0

In the above, you can clearly see both issues at work as predicted above.  The wa is quite high 63-83 and the number of processes in uninterpretable sleep is between 6 and 8.

Memory Bound – Swapping

The second type of system we will look at is a system that is attempting to allocate more memory than physically exists in the system.  It is normal for a linux/unix system to use swap to augment its available memory.   As such, any system can show some amount of memory allocated to swap.  However, this only becomes a problem when the system starts waiting for memory to be paged in and out of swap to satisfy its processes.  From a vmstat perspective, this would appear in 2 ways.  First is the swpd column.  However, the presence of large values in the swpd column does not, in and of itself, constitute swapping.  The more important demonstration of this is high values in siand so columns.  Here is a sample vmstat that shows some swapping.

#vmstat 5

procs      ———–memory———-                   —swap–       —–io—- –system–              —–cpu——

r  b        swpd   free   buff  cache                    si           so          bi    bo   in   cs               us sy id wa st
1  18   3794291 140136 167700 1506440     19851    9780        16783 27333  1738  1648        6    2 29  63  0
1  16   3797936 140136 167700 1506440     15913  30807        38353 28331  1658  3342        8  14 19  57  0
1  10   3847364 140136 167700 1506440     18334  22355        26482 32323  1534  1732      16  12 29  33  0

CPU Bound Machine

The last type of system we will review is a CPU-Bound machine.  A server that is experiencing this problem simply does not have enough processing power to properly perform its operations.  Usually this system will show a high number in the us column.  Again, remember the us column shows us the amount of time spent processing non-kernel code.  It is also quite normal in this scenario to have a larger number in the r column as they are queued up waiting for cpu time.

procs ———–memory———-      —swap–      —–io—-      –system–      —-cpu—-
r  b   swpd   free   buff  cache   si   so         bi    bo        in    cs      us sy id wa
5  1    648 755300  31460 2095012         0    0         76  3458 2013  1369      95  5  0  0
5  0    648 741196  31568 2103336         0    0         24  1852 2063  1481      88 12  0  0
8  0    648 706828  31616 2115936         0    0         50  2590 2118  1609      90 10  0  0
14  0    648 700628  31664 2133024       0    0       117  4320 2138  1290      90 10  0  0

zz from: http://www.helpmehost.com/linux/reading-vmstat-in-linux-part-1/