python内存泄漏_诊断和修复Python中的内存泄漏

python内存泄漏

Fugue uses Python extensively throughout the Conductor and in our support tools, due to its ease-of-use, extensive package library, and powerful language tools. One thing we’ve learned from building complex software for the cloud is that a language is only as good as its debugging and profiling tools. Logic errors, CPU spikes, and memory leaks are inevitable, but a good debugger, CPU profiler, and memory profiler can make finding these errors significantly easier and faster, letting our developers get back to creating Fugue’s dynamic cloud orchestration and enforcement system. Let’s look at a case in point.

Fugue的易用性，广泛的软件包库和强大的语言工具，在整个Conductor中以及在我们的支持工具中都广泛使用Python。 我们从为云构建复杂的软件中学到的一件事是，一种语言仅与其调试和概要分析工具一样好。 逻辑错误，CPU尖峰和内存泄漏是不可避免的，但是良好的调试器，CPU探查器和内存探查器可以使发现这些错误变得更加轻松快捷，这使我们的开发人员可以重新创建Fugue的动态云编排和执行系统。 让我们来看一个例子。

In the fall, our metrics reported that a Python component of Fugue called the reflector was experiencing random restarts and instability after a few days of uptime. Looking at memory usage showed that the reflector’s memory footprint increased monotonically and continuously, indicating a memory leak. tracemalloc, a powerful memory tracking tool in the Python standard library, made it possible to quickly diagnose and fix the leak. We discovered that the memory leak was related to our use of requests, a popular third-party Python HTTP library. Rewriting the component to use urllib from the Python standard library eliminated the memory leak. In this blog, we’ll explore the details.

在秋天，我们的指标报告称，Fuge的Python组件“反射器”在正常运行几天后经历了随机重启和不稳定。 查看内存使用情况表明，反射器的内存占用量单调且连续增加，表明内存泄漏。 tracemalloc是Python标准库中的功能强大的内存跟踪工具，它使得快速诊断和修复泄漏成为可能。 我们发现内存泄漏与我们对requests的使用有关， requests是一种流行的第三方Python HTTP库。 重写组件以使用Python标准库中的urllib消除了内存泄漏。 在此博客中，我们将探索细节。

Metrics show the problem: Percentage of total system memory used by the reflector, using the requests library.

指标显示了问题：使用请求库，反射器使用的系统内存总量的百分比。

Python中的内存分配 (Memory Allocation in Python)

In most scenarios, there’s no need to understand memory management in Python beyond knowing that the interpreter manages memory for you. However, when writing large, complex Python programs with high stability requirements, it’s useful to peek behind the curtain to understand how to write code that interacts well with Python’s memory management algorithms. Notably, Python uses reference counting and garbage collection to free memory blocks, and only frees memory to the system when certain internal requirements are met. A pure Python script will never have direct control over memory allocation in the interpreter. If direct control over memory allocation is desired, the interpreter’s memory allocation can be bypassed by writing or using an extension. For example, numpy manages memory for large data arrays using its own memory allocator.

在大多数情况下，无需了解Python中的内存管理，只需要了解解释器即可为您管理内存即可。 但是，当编写具有高稳定性要求的大型，复杂的Python程序时，了解一下如何编写与Python的内存管理算法良好交互的代码是很有用的。 值得注意的是，Python使用引用计数和垃圾回收来释放内存块，并且仅在满足某些内部要求时才将内存释放给系统。 纯Python脚本将永远无法直接控制解释器中的内存分配。 如果需要直接控制内存分配，则可以通过写入或使用扩展名来绕过解释器的内存分配。 例如， numpy使用其自己的内存分配器管理大型数据阵列的内存。

Fundamentally, Python is a garbage-collected language that uses reference counting. The interpreter automatically allocates memory for objects as they are created and tracks the number of references to those objects in a data structure associated with the object itself. This memory will be freed when the reference count for those objects reaches zero. In addition, garbage collection will detect cycles and remove objects that are only referenced in cycles. Every byte of memory allocated within the Python interpreter is able to be freed between these two mechanisms, but no claims can be made about memory allocated in extensions.

从根本上讲，Python是一种使用引用计数的垃圾收集语言。 解释器在创建对象时自动为它们分配内存，并在与对象本身关联的数据结构中跟踪对这些对象的引用数。 当这些对象的引用计数达到零时，将释放该内存。 此外，垃圾回收将检测周期并删除仅在周期中引用的对象。 可以在这两种机制之间释放在Python解释器中分配的内存的每个字节，但是无法声明扩展中分配的内存。

Python manages its own heap, separate from the system heap. Memory is allocated in the Python interpreter by different methods according to the type of the object to be created. Scalar types, such as integers and floats, use different memory allocation methods than composite types, such as lists, tuples, and dictionaries