How do I collect performance data with perf which is readable on another machine?
环境
- Red Hat Enterprise Linux 8
- Red Hat Enterprise Linux 7
- Red Hat Enterprise Linux 6
问题
- How do I collect performance data with
perf
which is readable on another machine? - System performance is slow, how can I identify where user processes and the kernel are spending CPU time?
决议
Install relevant packages
- Add the debuginfo channel to your system. (needed for installing the kernel-debuginfo package)
-
Install perf and kernel debugging symbols for the version you are running:
yum install bzip2 perf kernel-debuginfo-`uname -r`
Note: The command "perf archive" described later requires the bzip2 binary. In some versions of the perf package, the bzip2 binary is not a dependency and running perf archive will fail if it is not installed.
Collect data
-
(Optional) Create a custom directory to capture perf data:
mkdir -p /tmp/perf; cd $_
-
Capture performance data with
perf record
command:-
To capture data of a specific command, use:
perf record -g -- <command>
This will capture performance data for the whole execution of the specified command (including all potential child processes) or until interrupted.
-
To capture global data, use:
perf record -a -g -- sleep X
Where
X
is number of seconds, after which the recording should stop. Recommended:10
-60
seconds.
Global data collection can produce very large amount of data. Therefore it is not recommended to run global recording for long periods of time.
The
perf record
command will create aperf.data
file in the current active directory.Note: You can also run perf record without specifying any command. This way, however, the recording runs indefinitely and needs to be manually stopped, for example by interrupting it with Ctrl+C.
For more information see
man perf-record
. -
-
Generate archive containing debug symbols for the recorded data:
perf archive
This will pack all relevant debug symbols for the latest recorded data. These debug symbols are required for the data to be interpretable on another (different) machine. Without them the data would only show address pointers (hexadecimal numbers) of the recorded functions.
The
perf archive
command will create aperf.data.tar.gz
archive file in the current active directory.Note: The perf archive command does not archive also the actual data, although the name might suggest such.
For more information see
man perf-archive
.
Upload data for analysis
-
At this point you should have 2 files generated in the current active directory:
perf.data
- these are the actual recorded data;perf.data.tar.gz
- this archive contains debug symbols needed to interpret the data.
-
Important: In order to analyse the recorded data on another (different) machine, both these files need to be uploaded!
根源
perf
is a Performance analysis tool for Linux. The man page explains the tool:
Performance counters for Linux are a new kernel-based subsystem that
provide a framework for all things performance analysis. It covers
hardware level (CPU/PMU), Performance Monitoring Unit) features and
software features (software counters, tracepoints) as well.