python程序性能分析

最新推荐文章于 2024-07-12 14:01:48 发布

Johnson0722

最新推荐文章于 2024-07-12 14:01:48 发布

阅读量622

点赞数

分类专栏： python 文章标签： python 性能分析函数时间变了内存

本文链接：https://blog.csdn.net/John_xyz/article/details/106180396

版权

python 专栏收录该内容

31 篇文章 1 订阅

订阅专栏

这篇博客简单介绍一些python性能分析的常用工具，性能分析主要是代码运行的时间和内存分析，希望能给大家提供帮助

通过time模块

import time
def test(num_iterations):
    a = 1
    for i in range(num_iterations):
        a *= -1
    return a

num_iterations = 1_000_000_0
t1 = time.time()
res = test(num_iterations)
t2 = time.time()
print(t2 - t1)

这可能是最常用的方法，我们可以用这个方法测试某一行代码或者某一个函数的运行时间

定义一个装饰器

def time_wrapper(func):
    def measure_time(*args, **kwargs):
        t1 = time.time()
        result = func(*args, **kwargs)
        t2 = time.time()
        print(f"{func.__name__} took {t2 - t1} seconds")
        return result
    return measure_time
@time_wrapper
def test(num_iterations):
    a = 1
    for i in range(num_iterations):
        a *= -1
    return a
    
res = test(num_iterations)
# test took 0.4167180061340332 seconds

通过装饰器time_wrapper，我们可以在任意的函数上加上@time_wrapper，非常方便

用timeit模块测量执行速度

在juputer中，我们甚至还可以使用%timeit来测量速度

# 循环次数（-n 2）和重复次数（-r 2）
# timeit会对语句循环执行n次并计算平均值作为一个结果，重复r次选择最好的那个
%timeit -n 2 -r 2 test(num_iterations)
"""
test took 0.4153749942779541 seconds
test took 0.4038848876953125 seconds
test took 0.40583109855651855 seconds
test took 0.40905189514160156 seconds
409 ms ± 1.09 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)
"""

当然也可以在terminal中进行测试

$python -m timeit -n 2 -r 2 -s "import numpy as np" "np.array(10)"

# 2 loops, best of 2: 2.52 usec per loop

使用unix time命令进行简单的计时

%usr/bin/time -p python test.py

real记录了整体的耗时
user记录了CPU花在任务上的时间，不包括内核函数耗费的时间
sys记录了内核函数耗费的时间

使用cProfile模块

cProfile是一个标准的内建分析工具，它钩入CPython虚拟机来测量其每一个函数运行所花费的时间，这会引入巨大开销，但会获得更多的信息

def test1(value, num_iterations):
  res = 1
  for i in range(num_iterations):
    res += res * 2 % 10
  return value


def test2(num_iterations):
  res = 0
  for i in range(num_iterations):
    res += i*i % 10
  test1(1, num_iterations)
  return res

if __name__ == '__main__':
  #-s cumulative对每个函数累计花费时间进行排序
  """python -m cProfile -s cumulative test_cprofiler.py"""
  # 生成一个统计文件
  """python -m cProfile -o profile.stats test_cprofiler.py"""
  print('hello world')
  num_iterations = 1_000_000_0
  res1 = test1(1, num_iterations)
  res2 = test2(num_iterations)

"""
<pstats.Stats object at 0x1115f29d0>
Sun May 10 19:30:19 2020    profile.stats

         5 function calls in 2.221 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    2.221    2.221 {built-in method builtins.exec}
        1    0.000    0.000    2.221    2.221 test_cprofiler.py:1(<module>)
        1    1.303    1.303    1.303    1.303 test_cprofiler.py:8(test2)
        1    0.918    0.918    0.918    0.918 test_cprofiler.py:1(test1)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}


<pstats.Stats at 0x1115f29d0>
"""

for the total time spent in the given function (and excluding time made in calls to sub-functions)
cumulative time spent in this and all subfunctions (from invocation till exit). This figure is accurate even for recursive functions.

用line_profiler进行逐行分析

line_profiler是调查CPU密集型问题的最强大工具，它可以对函数进行逐行的分析。可以先用cProfiler找到需要分析的函数，然后用line_profiler对函数进行分析

def test1(value, num_iterations):
  res = 1
  for i in range(num_iterations):
    res += res * 2 % 10
  return value

@profile
def test2(num_iterations):
  res = 0
  for i in range(num_iterations):
    res += i*i % 10
  return res

if __name__ == '__main__':
  # 用装饰器@profile标记选中的函数，用kernprof.py来运行代码
  # -l代表逐行分析，-v用于显示输出
  """kernprof -l -v test_lineprofiler.py"""
  num_iterations = 1_000_000_0
  res1 = test1(1, num_iterations)
  res2 = test2(num_iterations)

"""
Wrote profile results to test_lineprofiler.py.lprof
Timer unit: 1e-06 s

Total time: 6.1838 s
File: test_lineprofiler.py
Function: test2 at line 7

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     7                                           @profile
     8                                           def test2(num_iterations):
     9         1        231.0    231.0      0.0    res = 0
    10  10000001    2488600.0      0.2     40.2    for i in range(num_iterations):
    11  10000000    3694972.0      0.4     59.8      res += i*i % 10
    12         1          1.0      1.0      0.0    return res
"""

Timer unit: 1e-06 s：时间单位；
Total time: 0.004891 s：总时间；
Hit：代码运行次数；
%Time：代码占了它所在函数的消耗的时间百分比，通常直接看这一列。

用memory_profiler诊断内存用量

memory_profiler的操作和line_profiler十分类似，但是运行速度要慢的多. 内存分析可以轻易让你的代码慢上10倍到100倍。所以实际操作可能只是偶尔使用memory_profiler而更多的使用line_profiler来进行CPU分析

import numpy as np
from memory_profiler import profile


@profile
def test2(num_iterations):
  res = 0
  for i in range(num_iterations):
    res += i*i % 10
  b = np.random.rand(1000, 1000)
  a = [0] * 1000000
  return res, b, a

if __name__ == '__main__':
  # 在函数前添加 @profile
  """python -m memory_profiler test_memoryprofiler.py"""
  num_iterations = 1_000
  res2 = test2(num_iterations)


"""
Line #    Mem usage    Increment   Line Contents
================================================
     5     55.6 MiB     55.6 MiB   @profile
     6                             def test2(num_iterations):
     7     55.6 MiB      0.0 MiB     res = 0
     8     55.6 MiB      0.0 MiB     for i in range(num_iterations):
     9     55.6 MiB      0.0 MiB       res += i*i % 10
    10     63.2 MiB      7.7 MiB     b = np.random.rand(1000, 1000)
    11     70.9 MiB      7.6 MiB     a = [0] * 1000000
    12     70.9 MiB      0.0 MiB     return res, b, a
"""

Mem usage : 运行到当前位置使用内存
Increment : 运行当前代码后，增加的内存