性能分析工具一览(gprof,perf,vtune,nsight,tau,hpctoolkit)

开始编辑时间：2024/2/22；最后编辑时间：2024/2/22

性能分析工具用于分析程序性能，具体来说，程序员往往使用性能分析工具来分析瓶颈，找出代码中不合理部分来调整代码，从而优化性能。出现瓶颈的原因可能是因为代码设计的不合理、架构适配等。

了解性能分析工具的最快方案是去用一下，否则无法体会工具的差异，往往只能从平台上进行区分。

此处仅从高性能计算的角度来描述目前广泛使用的性能分析工具。

本博客包括如下性能分析工具：

1. gprof
2. Perf
3. Nsight Computing
4. Intel VTune Profiler
5. TAU
6. HPCtoolkit

---

gprof：GNU profile工具

适用语言：C、C++、Pascal、Fortran

平台：Linux等可以使用GNU的平台

介绍：用于程序的性能优化以及程序瓶颈问题的查找和解决。通过分析应用程序运行时产生的“flat profile”，可以得到每个函数的调用次数，每个函数消耗的处理器时间，也可以得到函数的“调用关系图”，包括函数调用的层次关系，每个函数调用花费了多少时间。

缺点：对并行程序支持较差。

使用步骤：

1. 用gcc、g++、xlC编译程序时，使用-pg参数，如：g++ -pg -o test.exe test.cpp。编译器会自动在目标代码中插入用于性能测试的代码片断，这些代码在程序运行时采集并记录函数的调用关系和调用次数，并记录函数自身执行时间和被调用函数的执行时间。
2. 执行编译后的可执行程序，如：./test.exe。该步骤运行程序的时间会稍慢于正常编译的可执行程序的运行时间。程序运行结束后，会在程序所在路径下生成一个缺省文件名为gmon.out的文件，这个文件就是记录程序运行的性能、调用关系、调用次数等信息的数据文件。

3. 使用gprof命令来分析记录程序运行信息的gmon.out文件，如：gprof test.exe gmon.out。则可以在显示器上看到函数调用相关的统计、分析信息。

使用实例：

运行代码如下所示:

#include<stdio.h>
#include<math.h>

void set_init(double *a, int length) {
    for(int i = 0; i < length; i++) {
        a[i] = sin(i * 0.03);
    }
}

double sum_one(double *a, int length) {
    double result = 0;
    for(int i = 0; i < length; i++) {
        result += a[i];
    }
    return result;
}

int main(){
    double a[10000];
    double result = 0;
    for(int i = 0; i < 10000; i++) {
        set_init(a, 10000);
        result += sum_one(a, 10000);
    }
    printf("result = %lf\n", result);
}

Makefile内容如下：

all:
	gcc -o test test.c -pg -lm

run:
	./test

gprof:
	gprof test gmon.out

依次执行下列语句：

make
make run
make gprof

得到输出如下所示：

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls  us/call  us/call  name    
 67.53      0.22     0.22    10000    22.28    22.28  sum_one
 33.76      0.33     0.11    10000    11.14    11.14  set_init

 %         the percentage of the total running time of the
time       program used by this function.

cumulative a running sum of the number of seconds accounted
 seconds   for by this function and those listed above it.

 self      the number of seconds accounted for by this
seconds    function alone.  This is the major sort for this
           listing.

calls      the number of times this function was invoked, if
           this function is profiled, else blank.

 self      the average number of milliseconds spent in this
ms/call    function per call, if this function is profiled,
           else blank.

 total     the average number of milliseconds spent in this
ms/call    function and its descendents per call, if this
           function is profiled, else blank.

name       the name of the function.  This is the minor sort
           for this listing. The index shows the location of
           the function in the gprof listing. If the index is
           in parenthesis it shows where it would appear in
           the gprof listing if it were to be printed.


Copyright (C) 2012-2020 Free Software Foundation, Inc.

Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.


                     Call graph (explanation follows)


granularity: each sample hit covers 2 byte(s) for 2.99% of 0.33 seconds

index % time    self  children    called     name
                                                 <spontaneous>
[1]    100.0    0.00    0.33                 main [1]
                0.22    0.00   10000/10000       sum_one [2]
                0.11    0.00   10000/10000       set_init [3]
-----------------------------------------------
                0.22    0.00   10000/10000       main [1]
[2]     66.7    0.22    0.00   10000         sum_one [2]
-----------------------------------------------
                0.11    0.00   10000/10000       main [1]
[3]     33.3    0.11    0.00   10000         set_init [3]
-----------------------------------------------

 This table describes the call tree of the program, and was sorted by
 the total amount of time spent in each function and its children.

 Each entry in this table consists of several lines.  The line with the
 index number at the left hand margin lists the current function.
 The lines above it list the functions that called this function,
 and the lines below it list the functions this one called.
 This line lists:
     index      A unique number given to each element of the table.
                Index numbers are sorted numerically.
                The index number is printed next to every function name so
                it is easier to look up where the function is in the table.

     % time     This is the percentage of the `total' time that was spent
                in this function and its children.  Note that due to
                different viewpoints, functions excluded by options, etc,
                these numbers will NOT add up to 100%.

     self       This is the total amount of time spent in this function.

     children   This is the total amount of time propagated into this
                function by its children.

     called     This is the number of times the function was called.
                If the function called itself recursively, the number
                only includes non-recursive calls, and is followed by
                a `+' and the number of recursive calls.

     name       The name of the current function.  The index number is
                printed after it.  If the function is a member of a
                cycle, the cycle number is printed between the
                function's name and the index number.


 For the function's parents, the fields have the following meanings:

     self       This is the amount of time that was propagated directly
                from the function into this parent.

     children   This is the amount of time that was propagated from
                the function's children into this parent.

     called     This is the number of times this parent called the
                function `/' the total number of times the function
                was called.  Recursive calls to the function are not
                included in the number after the `/'.

     name       This is the name of the parent.  The parent's index
                number is printed after it.  If the parent is a
                member of a cycle, the cycle number is printed between
                the name and the index number.

 If the parents of the function cannot be determined, the word
 `<spontaneous>' is printed in the `name' field, and all the other
 fields are blank.

 For the function's children, the fields have the following meanings:

     self       This is the amount of time that was propagated directly
                from the child into the function.

     children   This is the amount of time that was propagated from the
                child's children to the function.

     called     This is the number of times the function called
                this child `/' the total number of times the child
                was called.  Recursive calls by the child are not
                listed in the number after the `/'.

     name       This is the name of the child.  The child's index
                number is printed after it.  If the child is a
                member of a cycle, the cycle number is printed
                between the name and the index number.

 If there are any cycles (circles) in the call graph, there is an
 entry for the cycle-as-a-whole.  This entry shows who called the
 cycle (as parents) and the members of the cycle (as children.)
 The `+' recursive calls entry shows the number of function calls that
 were internal to the cycle, and the calls entry for each member shows,
 for that member, how many times it was called from other members of
 the cycle.


Copyright (C) 2012-2020 Free Software Foundation, Inc.

Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.


Index by function name

   [3] set_init                [2] sum_one

更多的内容请参见其他教程。

---

Perf

适用语言： C, C++
平台： Linux
特点： Perf 是 Linux内置的性能分析工具，可用于分析 CPU 使用率、内存访问、系统调用等。它是一个命令行工具。事实上，并不是所有的Linux都有Perf工具，有部分超算没有安装Perf。

Perf是一个很大的工具，此处仅展示分析某个应用的的用法。

使用步骤（使用gprof的那个可执行文件）：

1. perf record ./test

   部分性能参数需要root权限

   $ perf record ./test
   WARNING: Kernel address maps (/proc/{kallsyms,modules}) are restricted,
   check /proc/sys/kernel/kptr_restrict and /proc/sys/kernel/perf_event_paranoid.
   
   Samples in kernel functions may not be resolved if a suitable vmlinux
   file is not found in the buildid cache or in the vmlinux path.
   
   Samples in kernel modules won't be resolved at all.
   
   If some relocation was applied (e.g. kexec) symbols may be misresolved
   even with a suitable vmlinux or kallsyms file.
   
   Couldn't record kernel reference relocation symbol
   Symbol resolution may be skewed if relocation was used (e.g. kexec).
   Check /proc/kallsyms permission or run as root.
   result = 345672.099494
   [ perf record: Woken up 2 times to write data ]
   [ perf record: Captured and wrote 0.279 MB perf.data (7260 samples) ]
   

2. perf report
   输出如下，可以看到sin函数的时间是最长的

   

---

Nsight computing

适用语言：CUDA

平台：windows，Linux

使用方案：

1. 在windows上配置Nsight Computing
2. 指定需要执行的cuda程序，可以远程指定
3. 等待运行完成，查看分析结果

问题：

1. 不稳定，并不是所有程序都可以进行性能分析。

实例

一个2022年时在windows平台上运行的截图：

图1：

1. GPU计算和内存利用率。
2. 每部分的计算和内存利用率。
3. 使用Roofline模型分析的程序性能。

图二：各个计算部件的利用率，用户可以依据这个指标来调整代码的计算指令，来利用多个运算部件：

图三：访存上的性能数据，包括cache命中和各级传输的吞吐：

---

Intel VTune Profiler

适用语言： 多语言支持
平台： Windows、Linux
特点： Intel VTune Profiler 是一个功能强大的性能分析工具，可用于分析 CPU 使用率、内存访问、多线程性能等。适用于 Intel 处理器。

本博主仅在比赛中使用过，仅能提供部分当时的截图：

图1：

1. 总时间，使用vtune会导致程序总时间增加
2. 热点
3. 每个核心的利用率，共64核

图2：

1. 调用关系与时间，轻松分析程序的时间

---

TAU

适用语言： C、C++
平台： Linux
特点： 界面有些老了，让人不想看下去。

运行图如下所示，

待更新… …

---

HPCtoolkit

适用语言： C、C++、CUDA
平台： Linux

特点：支持CUDA

目前还没有尝试，

论文中的运行图如下所示：