pprof 内存泄漏_我如何在大型代码库上使用pprof调查Go中的内存泄漏

pprof 内存泄漏

by Jonathan Levison

乔纳森·莱维森(Jonathan Levison)

我如何在大型代码库上使用pprof调查Go中的内存泄漏 (How I investigated memory leaks in Go using pprof on a large codebase)

I have been working with Go for the better part of the year, implementing a scalable blockchain infrastructure at Orbs, and it’s been an exciting year. Over the course of 2018, we researched on which language to choose for our blockchain implementation. This led us to choose Go because of our understanding that it has a good community and an amazing tool-set.

在今年的大部分时间里，我一直在与Go合作，在Orbs实施了可扩展的区块链基础架构，这是令人兴奋的一年。 在2018年的过程中，我们研究了为区块链实施选择哪种语言。 这使我们选择了Go，因为我们了解到Go具有良好的社区和出色的工具集。

In recent weeks we are entering the final stages of integration of our system. As in any large system, the later stage problems which include performance issues, in specific memory leaks, may occur. As we were integrating the system, we realized we found one. In this article I will touch the specifics of how to investigate a memory leak in Go, detailing the steps taken to find, understand and resolve it.

最近几周，我们进入了系统集成的最后阶段。 像在任何大型系统中一样，可能会出现后期问题，包括特定内存泄漏中的性能问题。 在集成系统时，我们意识到找到了一个。 在本文中，我将详细介绍如何调查Go中的内存泄漏，并详细介绍了查找，理解和解决内存泄漏的步骤。

The tool-set offered by Golang is exceptional but has its limitations. Touching these first, the biggest one is the limited ability to investigate full core dumps. A full core dump would be the image of the memory (or user-memory) taken by the process running the program.

Golang提供的工具集非常出色，但有其局限性。 首先涉及到这些，最大的是调查完整核心转储的能力有限。 完整的核心转储将是运行程序的进程占用的内存(或用户内存)的映像。

We can imagine the memory mapping as a tree, and traversing that tree would take us through the different allocations of objects and the relations. This means that whatever is at the root is the reason for ‘holding’ the memory and not GCing it (Garbage Collecting). Since in Go there is no simple way to analyze the full core dump, getting to the roots of an object that does not get GC-ed is difficult.

我们可以将内存映射想象成一棵树，遍历该树将带我们完成对象和关系的不同分配。 这意味着根本的原因是“保留”内存而不是对其进行GC(垃圾收集)的原因。 由于在Go中没有简单的方法来分析完整的内核转储，因此很难找到没有进行GC编辑的对象的根。

At the time of writing, we were unable to find any tool online that can assist us with that. Since there exists a core dump format and a simple enough way to export it from the debug package, it could be that there is one used at Google. Searching online it looks like it is in the Golang pipeline, creating a core dump viewer of such, but doesn’t look like anyone is working on it. Having said that, even without access to such a solution, with the existing tools we can usually get to the root cause.

在撰写本文时，我们在线找不到任何可以帮助我们的工具。 由于存在一种核心转储格式以及一种从调试包中导出该格式的足够简单的方法，因此可能是Google使用了一种。 在线搜索看起来像是在Golang管道中，创建了此类的核心转储查看器，但看起来没有人在使用它。 话虽如此，即使没有这种解决方案，使用现有工具通常也可以找到根本原因。

内存泄漏 (Memory Leaks)

Memory leaks, or memory pressure, can come in many forms throughout the system. Usually we address them as bugs, but sometimes their root cause may be in design decisions.

内存泄漏或内存压力可以在整个系统中以多种形式出现。 通常我们将它们视为错误，但有时其根本原因可能在于设计决策。

As we build our system under emerging design principles, such considerations are not believed to be of importance and that is okay. It is more important to build the system in a way that avoids premature optimizations and enables you to perform them later on as the code matures, rather than over engineer it from the get-go. Still, some common examples of seeing memory pressure issues materialize are:

当我们根据新兴的设计原则构建系统时，这些考虑因素并不重要，这没关系。 更为重要的是，以一种避免过早优化的方式构建系统，使您能够在代码成熟后稍后执行它们，而不是一开始就过度设计它。 尽管如此，看到内存压力问题的一些常见示例仍然是：

• Too many allocations, incorrect data representation
  分配过多，数据表示不正确
• Heavy usage of reflection or strings
  大量使用反射或弦
• Using globals
  使用全局变量
• Orphaned, never-ending goroutines
  孤立无尽的goroutines

In Go, the simplest way to create a memory leak is defining a global variable, array, and appending data to that array. This great blog post describes that case in a good way.

在Go中，创建内存泄漏的最简单方法是定义全局变量，数组并将数据附加到该数组。 这篇很棒的博客文章很好地描述了这种情况。

So why am I writing this post? When I was researching into this case I found many resources about memory leaks. Yet, in reality systems have more than 50 lines of code and a single struct. In such cases, finding the source of a memory issue is much more complex than what that example describes.

那我为什么写这篇文章呢？ 在研究这种情况时，我发现了很多有关内存泄漏的资源。 但是，实际上系统具有超过50行代码和一个结构。 在这种情况下，查找内存问题的根源比该示例所描述的复杂得多。

Golang gives us an amazing tool called pprof. This tool, when mastered, can assist in investigating and most likely finding any memory issue. Another purpose it has is for investigating CPU issues, but I will not go into anything related to CPU in this post.

Golang给了我们一个叫做pprof的神奇工具。 掌握此工具后，可以协助调查并最有可能发现任何内存问题。 它的另一个目的是调查CPU问题，但是在本文中，我将不涉及与CPU相关的任何内容。

去工具pprof (go tool pprof)

Covering everything that this tool does will require more than one blog post. One thing that took a while is finding out how to use this tool to get something actionable. I will concentrate this post on the memory related feature of it.

涵盖此工具所做的所有事情，将需要多个博客文章。 花费了一段时间的一件事是找出如何使用此工具来获得可操作的东西。 我将把这篇文章集中在与内存相关的功能上。

The pprof package creates a heap sampled dump file, which you can later analyze / visualize to give you a map of both:

pprof软件包将创建一个堆采样的转储文件，您可以稍后对其进行分析/可视化以提供这两者的映射：

• Current memory allocations
  当前的内存分配
• Total (cumulative) memory allocations
  总(累积)内存分配

The tool has the ability to compare snapshots. This can enable you to compare a time diff display of what happened right now and 30 seconds ago, for example. For stress scenarios this can be useful to assist in locating problematic areas of your code.

该工具可以比较快照。 例如，这可以使您比较当前和30秒前发生的时间差异显示。 对于压力很大的情况，这有助于查找代码中有问题的区域。

pprof的个人资料 (pprof profiles)

The way pprof works is using profiles.

pprof的工作方式是使用配置文件。

A Profile is a collection of stack traces showing the call sequences that led to instances of a particular event, such as allocation.

配置文件是堆栈跟踪的集合，显示了导致特定事件(例如分配)实例的调用序列。

The file runtime/pprof/pprof.go contains the detailed information and implementation of the profiles.

文件runtime / pprof / pprof.go包含配置文件的详细信息和实现。

Go has several built in profiles for us to use in common cases:

Go有一些内置的配置文件供我们在常见情况下使用：

• goroutine — stack traces of all current goroutines
  goroutine —当前所有goroutine的堆栈跟踪
• heap — a sampling of memory allocations of live objects
  堆—活动对象的内存分配的样本
• allocs — a sampling of all past memory allocations
  allocs —过去所有内存分配的样本
• threadcreate — stack traces that led to the creation of new OS threads
  threadcreate-导致创建新