tracemalloc - Trace memory allocations - 跟踪内存分配

tracemalloc - Trace memory allocations - 跟踪内存分配

tracemalloc - 跟踪内存分配
https://docs.python.org/zh-cn/3.8/library/tracemalloc.html

tracemalloc - Trace memory allocations
https://docs.python.org/3.8/library/tracemalloc.html

Source code: Lib/tracemalloc.py

The tracemalloc module is a debug tool to trace memory blocks allocated by Python. It provides the following information:
tracemalloc 模块是一个调试工具，用于跟踪 Python 分配的内存块。它提供以下信息：

• Traceback where an object was allocated - 分配对象的回溯
• Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks - 每个文件和每行分配的内存块的统计信息：分配的内存块的总大小、数量和平均大小
• Compute the differences between two snapshots to detect memory leaks - 计算两个快照之间的差异以检测内存泄漏

traceback：n. 回溯

To trace most memory blocks allocated by Python, the module should be started as early as possible by setting the PYTHONTRACEMALLOC environment variable to 1, or by using -X tracemalloc command line option. The tracemalloc.start() function can be called at runtime to start tracing Python memory allocations.
跟踪由 Python 分配的大多数内存块，应通过将环境变量 PYTHONTRACEMALLOC 设置为 1 或使用 -X tracemalloc 命令行选项来尽早启动该模块。可以在运行时调用 tracemalloc.start() 函数来开始跟踪 Python 内存分配。

By default, a trace of an allocated memory block only stores the most recent frame (1 frame). To store 25 frames at startup: set the PYTHONTRACEMALLOC environment variable to 25, or use the -X tracemalloc=25 command line option.
默认情况下，分配的存储块的跟踪仅存储最近的帧 (1 frame)。要在启动时存储 25 帧，请执行以下操作：将环境变量 PYTHONTRACEMALLOC 设置为 25，或使用 -X tracemalloc=25 命令行选项。

1. Display the top 10

Display the 10 files allocating the most memory:

import tracemalloc

tracemalloc.start()

# ... run your application ...

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')

print("[ Top 10 ]")
for stat in top_stats[:10]:
    print(stat)

Example of output of the Python test suite:

[ Top 10 ]
<frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B
<frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 B
/usr/lib/python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B
/usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B
/usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B
/usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B
<frozen importlib._bootstrap>:1446: size=70.4 KiB, count=911, average=79 B
<frozen importlib._bootstrap>:1454: size=52.0 KiB, count=25, average=2131 B
<string>:5: size=49.7 KiB, count=148, average=344 B
/usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB

We can see that Python loaded 4855 KiB data (bytecode and constants) from modules and that the collections module allocated 244 KiB to build namedtuple types.
我们可以看到 Python 从模块中加载了 4855 KiB 数据 (字节码和常量)，并且 collections 模块分配了 244 KiB 来构建 namedtuple 类型。

2. Compute differences

Take two snapshots and display the differences:

import tracemalloc
tracemalloc.start()
# ... start your application ...

snapshot1 = tracemalloc.take_snapshot()
# ... call the function leaking memory ...
snapshot2 = tracemalloc.take_snapshot()

top_stats = snapshot2.compare_to(snapshot1, 'lineno')

print("[ Top 10 differences ]")
for stat in top_stats[:10]:
    print(stat)

Example of output before/after running some tests of the Python test suite:

[ Top 10 differences ]
<frozen importlib._bootstrap>:716: size=8173 KiB (+4428 KiB), count=71332 (+39369), average=117 B
/usr/lib/python3.4/linecache.py:127: size=940 KiB (+940 KiB), count=8106 (+8106), average=119 B
/usr/lib/python3.4/unittest/case.py:571: size=298 KiB (+298 KiB), count=589 (+589), average=519 B
<frozen importlib._bootstrap>:284: size=1005 KiB (+166 KiB), count=7423 (+1526), average=139 B
/usr/lib/python3.4/mimetypes.py:217: size=112 KiB (+112 KiB), count=1334 (+1334), average=86 B
/usr/lib/python3.4/http/server.py:848: size=96.0 KiB (+96.0 KiB), count=1 (+1), average=96.0 KiB
/usr/lib/python3.4/inspect.py:1465: size=83.5 KiB (+83.5 KiB), count=109 (+109), average=784 B
/usr/lib/python3.4/unittest/mock.py:491: size=77.7 KiB (+77.7 KiB), count=143 (+143), average=557 B
/usr/lib/python3.4/urllib/parse.py:476: size=71.8 KiB (+71.8 KiB), count=969 (+969), average=76 B
/usr/lib/python3.4/contextlib.py:38: size=67.2 KiB (+67.2 KiB), count=126 (+126), average=546 B

We can see that Python has loaded 8173 KiB of module data (bytecode and constants), and that this is 4428 KiB more than had been loaded before the tests, when the previous snapshot was taken. Similarly, the linecache module has cached 940 KiB of Python source code to format tracebacks, all of it since the previous snapshot.
我们可以看到 Python 已经加载了 8173 KiB 的模块数据 (字节码和常量)，这比在获取前一个快照时 (在测试之前) 加载的数据多了 4428 KiB。类似地，linecache 模块已经缓存了 940 KiB 的 Python 源代码以格式化回溯，所有这些都是自上次快照以来的。

If the system has little free memory, snapshots can be written on disk using the Snapshot.dump() method to analyze the snapshot offline. Then use the Snapshot.load() method reload the snapshot.
如果系统的可用内存很少，则可以使用 Snapshot.dump() 方法将快照写入磁盘，以离线分析快照。然后使用 Snapshot.load() 方法重新加载快照。

3. Get the traceback of a memory block

Code to display the traceback of the biggest memory block (显示最大内存块的回溯代码):

import tracemalloc

# Store 25 frames
tracemalloc.start(25)

# ... run your application ...

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('traceback')

# pick the biggest memory block
stat = top_stats[0]
print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024))
for line in stat.traceback.format():
    print(line)

Example of output of the Python test suite (traceback limited to 25 frames):

903 memory blocks: 870.1 KiB
  File "<frozen importlib._bootstrap>", line 716
  File "<frozen importlib._bootstrap>", line 1036
  File "<frozen importlib._bootstrap>", line 934
  File "<frozen importlib._bootstrap>", line 1068
  File "<frozen importlib._bootstrap>", line 619
  File "<frozen importlib._bootstrap>", line 1581
  File "<frozen importlib._bootstrap>", line 1614
  File "/usr/lib/python3.4/doctest.py", line 101
    import pdb
  File "<frozen importlib._bootstrap>", line 284
  File "<frozen importlib._bootstrap>", line 938
  File "<frozen importlib._bootstrap>", line 1068
  File "<frozen importlib._bootstrap>", line 619
  File "<frozen importlib._bootstrap>", line 1581
  File "<frozen importlib._bootstrap>", line 1614
  File "/usr/lib/python3.4/test/support/__init__.py", line 1728
    import doctest
  File "/usr/lib/python3.4/test/test_pickletools.py", line 21
    support.run_doctest(pickletools)
  File "/usr/lib/python3.4/test/regrtest.py", line 1276
    test_runner()
  File "/usr/lib/python3.4/test/regrtest.py", line 976
    display_failure=not verbose)
  File "/usr/lib/python3.4/test/regrtest.py", line 761
    match_tests=ns.match_tests)
  File "/usr/lib/python3.4/test/regrtest.py", line 1563
    main()
  File "/usr/lib/python3.4/test/__main__.py", line 3
    regrtest.main_in_temp_cwd()
  File "/usr/lib/python3.4/runpy.py", line 73
    exec(code, run_globals)
  File "/usr/lib/python3.4/runpy.py", line 160
    "__main__", fname, loader, pkg_name)

We can see that the most memory was allocated in the importlib module to load data (bytecode and constants) from modules: 870.1 KiB. The traceback is where the importlib loaded data most recently: on the import pdb line of the doctest module. The traceback may change if a new module is loaded.
我们可以看到，在 importlib 模块中分配了最多的内存，以从模块中加载数据 (字节码和常量)：870.1 KiB。回溯是 importlib 最近加载数据的位置：在 doctest 模块的 import pdb 行上。如果加载了新模块，则回溯可能会更改。

4. Functions

tracemalloc.clear_traces()
Clear traces of memory blocks allocated by Python.
清除由 Python 分配的内存块的痕迹。

tracemalloc.stop()
Stop tracing Python memory allocations: uninstall hooks on Python memory allocators. Also clears all previously collected traces of memory blocks allocated by Python.
停止跟踪 Python 内存分配：卸载 Python 内存分配器上的钩子。同时清除由 Python 分配的所有先前收集的内存块跟踪。

Call take_snapshot() function to take a snapshot of traces before clearing them.
调用 take_snapshot() 函数在清除痕迹之前对其进行快照。

statistics(key_type: str, cumulative: bool=False)
获取 Statistic 信息列表，按 key_type 分组排序：

If cumulative is True, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The cumulative mode can only be used with key_type equals to filename and lineno.
如果 cumulative 为 True，则累计跟踪回溯所有帧的大小和存储块数，而不仅仅是最近的帧。累积模式只能与 key_type 等于 filename and lineno 时一起使用。

The result is sorted from the biggest to the smallest by: Statistic.size, Statistic.count and then by Statistic.traceback.
结果按从大到小排序：Statistic.size, Statistic.count，然后按 Statistic.traceback。