System Activity Reporter (sar)

source: http://www.softpanorama.org/Admin/Monitoring/sar.shtml

The SAR suite of utilities originated in Solaris. It became popular and now runs on most flavors of UNIX, including AIX, HP-UX, and Linux.  Sysstat package is installed by default in standard Red Hat installation. For Suse it is not installed by default and you need to install sysstat package manually (package is provided by Novell).

The reason for sar creation was that gathering system activity data from vmstat and iostat is pretty time-consuming. If you try to automate the gathering of system activity data, and creation of periodic repots you naturally come to creation of a tool like sar. To avoid reinventing the bicycle again and again, Sun engineers wrote sar sar (System Activity Reporter) and included it in standard Solaris distribution. The rest is history.


For details of linux implementation (which is a part of sysstat package) see Linux implementation of sar


System Activity Recorder can monitor half-dozen metrics related to overall system performance, for example:


cpu utilization (it's pretty effective tool for spotting CPU bottlenecks)
hard disk utilization
terminal IO
number of files open
processes running
It provides queuing, paging, CPU and many other metrics. Modern Unixes maintains a series of system activity counters that record various activities and provide the data that sar reports. The command merely extracts the values of the counters and saves them based on the sampling rate and the set of samples specified.


The package consists of two programs sarc and sar:


The sadc command is a data collecting part of the package. It writes in binary format to the specified output file. To run sar in real time type:
    sar -u 2 5


In this case the sar command calls sadc to access system data.
 


To report on previously captured data – type 
    sar -u -f filename > file
Two shell scripts /usr/lib/sa/sa1 and usr/ib/sa/sa2 are also provided. They can be run by the cron daemon and provide daily statistics and reports.
Sar is not enabled by default.  To enable sar, you must include invocation of two of its components (sadc and sar) at selected intervals  in cron. Usually sadc is evoked via sa1 script and sar via sa2 script). 


On Solaris 9 and 10 it is preinstalled, but you need un-comment lines in the start script (/etc/rc2.d/S21perf) and crontab file (/var/spool/cron/crontabs/sys) associated with the tool. If sar is activated the crontab for root should contain something like this:


# Collect measurements at 10-minute intervals
0,10,20,30,40,50   * * * *   /usr/lib/sa/sa1
# Create daily reports and purge old files
0 * * *   /usr/lib/sa/sa2 -A
For linux you need to  install and activate sysstat package (see also Linux implementation of sar) :


The sysstat package contains the sar, sadf, iostat, mpstat, and pidstat commands for Linux. The sar command collects and reports system activity information. The statistics reported by sar concern I/O transfer rates, paging activity, process-related activites, interrupts, network activity, memory and swap space utilization, CPU utilization, kernel activities, and TTY statistics, among others. The sadf command may be used to display data collected by sar in various formats. The iostat command reports CPU statistics and I/O statistics for tty devices and disks. The pidstat command reports statistics for Linux processes. The mpstat command reports global and per-processor statistics.


There are several alternative script/programs for reporting sar data. Among them:
ksar  -- ksar is a Java-based sar graphing tool that can graph Linux, Mac OS X, AIX, and Solaris sar output. A sar statistics graph can be output to a PDF file.
BSDsar  -- BSDsar is a Perl script that generates a history of usage on a FreeBSD machine. It logs data such as CPU usage, disk activity, network bandwidth usage and activity, NFS information, memory, and swap. It is similar to atsar (for Linux) and sar (for Solaris).
Sar P Plot This one is asar, but can be easily adapted for sar.
Sar P Plot is a simple Perl script which takes the output of the atsar application and puts it into Gnuplot data files. It can be useful on server systems for performance analysis.


There are also alternative implementations:


DAG Dstat Versatile resource statistics tool
Dstat gives you detailed selective information in columns and clearly indicates in what magnitude and unit the output is displayed. Less confusion, less mistakes. And most importantly, it makes it very easy to write plugins to collect your own counters and extend in ways you never expected.


Dstat's output by default is designed for being interpreted by humans in real-time, however you can export details to CSV output to a file to be imported later into Gnumeric or Excel to generate graphs.


Features
Combines vmstat, iostat, ifstat, netstat information and more
Shows stats in exactly the same timeframe
Enable/order counters as they make most sense during analysis/troubleshooting
Modular design
Written in python so easily extendable for the task at hand
Easy to extend, add your own counters (please contribute those)
Includes many external plugins to show how easy it is to add counters
Can summarize grouped block/network devices and give total numbers
Can show interrupts per device
Very accurate timeframes, no timeshifts when system is stressed
Shows exact units and limits conversion mistakes
Indicate different units with different colors
Show intermediate results when delay > 1
Allows to export CSV output, which can be imported in Gnumeric and Excel to make graphs
atsar (outdated: last updated in Jul 2004) . Atsar has the ability to show what is happening on the system at a given moment. It also keeps track of the past system load by maintaining history files from which information can be extracted. Statistics about the utilization of CPUs, disks and disk partitions, memory and swap, tty's, TCP/IP (v4/v6), NFS, and FTP/HTTP traffic are gathered. Most of the functionality of atsar has been incorporated in the atop project.
For more details see:


Linux implementation of sar
Solaris implementation of sar
Side effects


Any tool that collects performance data has some impact on system performance, but with sar, it seems to be minimal. Even one minute sampling usually does not cause any serious issues. That may not hold true on a system that is very busy.