C++内存调试技术

     说到C++调试想必大家会想到一堆调试中遇到的问题，而在我看来C++中最难也是最普遍的调试问题就出在内存上。为什么要这么说呢？可以想想你曾经碰到过的问题，内存泄露应该是最普遍的，其次是内存越界，野指针，这些碰到哪一个都是硬点子。特别是项目规模越来越大的时候，这些问题就成为骨中钉，肉中刺，膈应的开发人员什么想法都没有了。

    问题既然产生，那必然会有方法解决。我们从现在开始一点一点的剖析这些问题的产生原因再对症下药，保管药到病除啊。

 

1. 内存泄露

 

    内存泄露是个老掉牙的问题，从写程序的第一天就没离开过我的视野范围。有点程序基础的人都知道它是怎么产生的，我这里就不罗嗦。我只介绍几种内存泄露的检查方法。

 

1.1 如何检测内存泄露

    正常情况下，我们通过Virtual Studio 生成的程序，除MFC以外是不会报告内存泄露的，即使你确实泄露了。那么为什么是除MFC应用程序以外呢？这个问题就说到了MFC应用程序向导都为我们生成了些什么。

#ifdef _DEBUG #define new DEBUG_NEW #endif

如果你细心的话，应该会在你的项目里找到这么一段话。这段话的意思是，如果是DEBUG版本，则将 new 替换成 DEBUG_NEW。

那么DEBUG_NEW又是什么？

我们跟过去看一下它的定义

#define DEBUG_NEW new(THIS_FILE, __LINE__)

它只是在new后面加了两个参数 THIS_FILE, __LINE__

这两个参数都是编译器的预定义宏(THIS_FILE其实是重新定义的__FILE__)，分别表示，当前文件的文件名和行号，如果你的MFC程序发生了泄露，又正好被捕获到了，那么output窗口中显示的文件名和行号就是从这里来的。

那么new 怎么会有参数的呢？

秘密在于MFC重载了 operator new。当然所有的内存分配最后都会调用crt的malloc进行内存分配。

void* __cdecl operator new(size_t nSize, LPCSTR lpszFileName, int nLine) { return ::operator new(nSize, _NORMAL_BLOCK, lpszFileName, nLine); } void* __cdecl operator new[](size_t nSize, LPCSTR lpszFileName, int nLine) { return ::operator new[](nSize, _NORMAL_BLOCK, lpszFileName, nLine); } void __cdecl operator delete(void* pData, LPCSTR /* lpszFileName */, int /* nLine */) { ::operator delete(pData); } void __cdecl operator delete[](void* pData, LPCSTR /* lpszFileName */, int /* nLine */) { ::operator delete(pData); }

所以我们不一定要用MFC，CRT本身就自带了内存泄露的检测功能。我们需要做的，只是做一些小设置。

在CRT中我们可通过如下的函数调用来打开内存泄露报告。

#include <crtdbg.h> _CrtSetDbgFlag(_CrtSetDbgFlag(_CRTDBG_REPORT_FLAG) | _CRTDBG_LEAK_CHECK_DF);

如果发生了内存泄露则会有如下的输出

Detected memory leaks! Dumping objects -> {91} normal block at 0x00725BB0, 256 bytes long. Data: < > CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD Object dump complete.

 

报告是有了，但是报告中没有指明产生泄露的文件和行号，那么如何通过报告来找出内存泄露的地方呢？

可以看到，每个泄露的报告项中的最前面有一个{91}，这其实是一个分配的序号。无论你用new或者malloc都会导致这个序号增长。

当我们需要定位某个序号的内存泄露的时候可以通过如下代码

_CrtSetBreakAlloc(91);

当程序分配到91块内存的时候就会出现ASSERT断言，暂停程序的执行。这样做有一个好处，就是可以通过函数调用堆栈还原泄露时的现场，从而更具体的分析泄露出现的原因。

 

那么CRT真的无法像MFC一样打印出泄露的文件名和行号吗？其实这是一个很简单的事情，我们只要通过

new(_NORMAL_BLOCK, __FILE__, __LINE__) char[256]

来分配内存就可以得到像MFC的内存泄露检测报告。其实MFC也是通过CRT做的。现在我们看到的内存泄露检测报告应该是这样的

Detected memory leaks! Dumping objects -> d:/developed/directxlearn/directxlearn/directxlearn.cpp(21) : {91} normal block at 0x002B5BB0, 256 bytes long. Data: < > CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD Object dump complete.

 如此我们已经可以让内存泄露检测报告像MFC一样打印出文件名和行号了。

3. 更深入的讨论

CRT是如何将文件名和行号保存下来的？又是如何打印到我们的Output窗口的呢？

我们先看一下反汇编以后的new 代码

mov eax,dword ptr [`wWinMain'::`2'::__LINE__Var (2D801Ch)] add eax,3 push eax push offset string "d://developed//directxlearn//direct"... (2D6960h) push 1 push 100h call operator new[] (2D12EEh) add esp,10h mov dword ptr [ebp-0F8h],eax

从反汇编出来的代码中我们可以看到，文件名和行号都是存储在程序数据段中的，传入operator new 的只是文件名的地址而已。跟踪进入call operator new[] 我们可以看到如下代码

void *__CRTDECL operator new[]( size_t cb, int nBlockUse, const char * szFileName, int nLine ) _THROW1(_STD bad_alloc) { void *res = operator new(cb, nBlockUse, szFileName, nLine ); RTCCALLBACK(_RTC_Allocate_hook, (res, cb, 0)); return res; }

 operator new 的代码如下

void *__CRTDECL operator new( size_t cb, int nBlockUse, const char * szFileName, int nLine ) _THROW1(_STD bad_alloc) { /* _nh_malloc_dbg already calls _heap_alloc_dbg in a loop and calls _callnewh if the allocation fails. If _callnewh returns (very likely because no new handlers have been installed by the user), _nh_malloc_dbg returns NULL. */ void *res = _nh_malloc_dbg( cb, 1, nBlockUse, szFileName, nLine ); RTCCALLBACK(_RTC_Allocate_hook, (res, cb, 0)); /* if the allocation fails, we throw std::bad_alloc */ if (res == 0) { static const std::bad_alloc nomem; _RAISE(nomem); } return res; }   

可以看到，最终还是调用了_nh_malloc_dbg.如果我们继续跟踪下去的话会找到真正分配内存的地方，这里我着重讲一下CRT的内存结构。和很多内存池一样，CRT也在分配内存的前后加入了一些标记和内容。其中最重要的就是一个_CrtMemBlockHeader 的结构。

typedef struct _CrtMemBlockHeader { struct _CrtMemBlockHeader * pBlockHeaderNext; struct _CrtMemBlockHeader * pBlockHeaderPrev; char * szFileName; int nLine; #ifdef _WIN64 /* These items are reversed on Win64 to eliminate gaps in the struct * and ensure that sizeof(struct)%16 == 0, so 16-byte alignment is * maintained in the debug heap. */ int nBlockUse; size_t nDataSize; #else /* _WIN64 */ size_t nDataSize; int nBlockUse; #endif /* _WIN64 */ long lRequest; unsigned char gap[nNoMansLandSize]; /* followed by: * unsigned char data[nDataSize]; * unsigned char anotherGap[nNoMansLandSize]; */ } _CrtMemBlockHeader;

这个结构是一个双向链表，分别记录了前一个和后一个分配的内存块的首地址。同时，该结构也记录了产生这块内存的具体文件和行号。其次是内存的类型，以及请求的大小，以及分配内存时的分配序号。最后是一个4字节的上溢保护，通过这四个字节可以检测出大部分由负序数导致的数组上溢问题。

blockSize = sizeof(_CrtMemBlockHeader) + nSize + nNoMansLandSize;

从内存实际分配字节数的计算中我们还可以看出，除请求的内存量和_CrtMemBlockHeader所占用的内存量以外，还有一个4字节的下溢字节保护。上溢和下溢的保护字节一般被初始化为0xFD。

初始化内存信息的代码如下

pHead->pBlockHeaderNext = _pFirstBlock; pHead->pBlockHeaderPrev = NULL; pHead->szFileName = (char *)szFileName; pHead->nLine = nLine; pHead->nDataSize = nSize; pHead->nBlockUse = nBlockUse; pHead->lRequest = lRequest; /* link blocks together */ _pFirstBlock = pHead; /* fill in gap before and after real block */ memset((void *)pHead->gap, _bNoMansLandFill, nNoMansLandSize); memset((void *)(pbData(pHead) + nSize), _bNoMansLandFill, nNoMansLandSize); /* fill data with silly value (but non-zero) */ memset((void *)pbData(pHead), _bCleanLandFill, nSize);

 如此，在最终生成内存泄露检测报告的时候就可以根据内存块信息来得到文件名和行号了。

4. CRT的缺陷

CRT检查内存泄露的方法是比较直观的，但是也相应的存在缺陷。

首先，是你new的时候或者malloc的时候不可能都是用特殊版本的函数调用，只能通过定义宏来实现。宏这个东西我比较讨厌，因为在来来回回的包含中你不能确定哪些new 被替换了。所以一旦有没有被宏替换过的new那么你的报告就会出现一些没有地址和行号的内存泄露报告。

其次，CRT的内存泄露报告没有调用堆栈，有时候泄露很可能是一些和调用顺序相关的临界条件引起的，至少碰到这种情况就比较难查。

最后，假如你的程序中存在静态对象，恰好你的静态对象被析构的时候是在内存检测报告完成之后，那么内存检测报告就会发生误报。

以上种种，都促使我寻找一种更先进的检查内存泄露的方法。

 

5. Virutal Leak Detected

初见VLD的时候我认为没有比他更好的内存泄露检测方法了。从技术的角度讲，它所做的工作有点像黑客干的事情，因为它用到了一种技术——DLL补丁。

说起DLL补丁我们还要先讲一下EXE程序和DLL之间的关系，以及EXE如何调用DLL

当主程序加载DLL到自己的进程地址空间之前，操作系统首先要将整个DLL文件载入到内存中，然后根据需要重新映射动态链接库的地址，之后调整导出符号表中入口函数的地址，使之指向正确的函数入口。如果你跟踪汇编代码的话，会发现你Call 一个DLL的导出函数或者类函数的时候其实是先到了一个全是jmp指令的地方，然后才到达正确的代码地址。这个全是jmp指令的地方就是导出函数表。如果我们修改了表中的跳转地址的话，当你去call 的时候就会跳转到一个你指定的地方，比如说一个钩子函数。但是，这种修改跳转表的方法只对当前程序实例有效。

而VLD的核心思想是，拦截所有的内存分配函数的入口地址，使之转移到我们的入口函数中记录一些内容，比如调用时的指令指针EIP的值等。 

虽然看起来很复杂，但是vld的使用时非常简单的我们只需要在工程中设置vld.lib导入库的地址，在某一个头文件中包含vld.h就可以了。之后我们将vld.ini放在运行目录下

vld.ini有如下配置

There are a several configuration options that control specific aspects of VLD's operation. These configuration options are stored in the vld.ini configuration file. By default, the configuration file should be in the Visual Leak Detector installation directory. However, the configuration file can be copied to the program's working directory, in which case the configuration settings in that copy of vld.ini will apply only when debugging that one program. VLD This option acts as a master on/off switch. By default, this option is set to "on". To completely disable Visual Leak Detector at runtime, set this option to "off". When VLD is turned off using this option, it will do nothing but print a message to the debugger indicating that it has been turned off. AggregateDuplicates Normally, VLD displays each individual leaked block in detail. Setting this option to "yes" will make VLD aggregate all leaks that share the same size and call stack under a single entry in the memory leak report. Only the first leaked block will be reported in detail. No other identical leaks will be displayed. Instead, a tally showing the total number of leaks matching that size and call stack will be shown. This can be useful if there are only a few sources of leaks, but those few sources are repeatedly leaking a very large number of memory blocks. ForceIncludeModules In some rare cases, it may be necessary to include a module in leak detection, but it may not be possible to include vld.h in any of the module's sources. In such cases, this option can be used to force VLD to include those modules in leak detection. List the names of the modules (DLLs) to be forcefully included in leak detection. If you do use this option, it's advisable to also add vld.lib to the list of library modules in the linker options of your project's settings. Caution: Use this option only when absolutely necessary. In some situations, use of this option may result in unpredictable behavior including false leak reports and/or crashes. It's best to stay away from this option unless you are sure you understand what you are doing. MaxDataDump Set this option to an integer value to limit the amount of data displayed in memory block data dumps. When this number of bytes of data have been dumped, the dump will stop. This can be useful if any of the leaked blocks are very large and the debugger's output window becomes too cluttered. You can set this option to 0 (zero) if you want to suppress data dumps altogether. MaxTraceFrames By default, VLD will trace the call stack for each allocated block as far back as possible. Each frame traced adds additional overhead (in both CPU time and memory usage) to your debug executable. If you'd like to limit this overhead, you can define this macro to an integer value. The stack trace will stop when it has traced this number of frames. The frame count may include some of the "internal" frames which, by default, are not displayed in the debugger's output window (see TraceInternalFrames below). In some cases there may be about three or four "internal" frames at the beginning of the call stack. Keep this in mind when using this macro, or you may not see the number of frames you expect. ReportEncoding When the memory leak report is saved to a file, the report may optionally be Unicode encoded instead of using the default ASCII encoding. This might be useful if the data contained in leaked blocks is likely to consist of Unicode text. Set this option to "unicode" to generate a Unicode encoded report. ReportFile Use this option to specify the name and location of the file in which to save the memory leak report when using a file as the report destination, as specified by the ReportTo option. If no file is specified here, then VLD will save the report in a file named "memory_leak_report.txt" in the working directory of the program. ReportTo The memory leak report may be sent to a file in addition to, or instead of, the debugger. Use this option to specify which type of destination to use. Specify one of "debugger" (the default), "file", or "both". SelfTest VLD has the ability to check itself for memory leaks. This feature is always active. Every time you run VLD, in addition to checking your own program for memory leaks, it is also checking itself for leaks. Setting this option to "on" forces VLD to intentionally leak a small amount of memory: a 21-character block filled with the text "Memory Leak Self-Test". This provides a way to test VLD's ability to check itself for memory leaks and verify that this capability is working correctly. This option is usually only useful for debugging VLD itself. SlowDebuggerDump If enabled, this option causes Visual Leak Detector to write the memory leak report to the debugger's output window at a slower than normal rate. This option is specifically designed to work around a known issue with some older versions of Visual Studio where some data sent to the output window might be lost if it is sent too quickly. If you notice that some information seems to be missing from the memory leak report, try turning this on. StackWalkMethod Selects the method to be used for walking the stack to obtain call stacks for allocated memory blocks. The default "fast" method may not always be able to successfully trace completely through all call stacks. In such cases, the "safe" method may prove to be more reliable in obtaining the full stack trace. The disadvantage with the "safe" method is that it is significantly slower than the "fast" method and will probably result in very noticeable performance degradation of the program being debugged. In most cases it should be okay to leave this option set to "fast". If you experience problems getting VLD to show call stacks, you can try setting this option to "safe". If you do use the "safe" method, and notice a significant performance decrease, you may want to consider using the MaxTraceFrames option to limit the number of frames traced to a relatively small number. This can reduce the amount of time spent tracing the stack by a very large amount. StartDisabled Set this option to "yes" to disable memory leak detection initially. This can be useful if you need to be able to selectively enable memory leak detection from runtime, without needing to rebuild the executable; however, this option should be used with caution. Any memory leaks that may occur before memory leak detection is enabled at runtime will go undetected. For example, if the constructor of some global variable allocates memory before execution reaches a subsequent call to VLDEnable, then VLD will not be able to detect if the memory allocated by the global variable is never freed. Refer to the following section on controlling leak detection at runtime for details on using the runtime APIs which can be useful in conjunction with this option. TraceInternalFrames This option determines whether or not all frames of the call stack, including frames internal to the heap, are traced. There will always be a number of frames on the call stack which are internal to Visual Leak Detector and C/C++ or Win32 heap APIs that aren't generally useful for determining the cause of a leak. Normally these frames are skipped during the stack trace, which somewhat reduces the time spent tracing and amount of data collected and stored in memory. Including all frames in the stack trace, all the way down into VLD's own code can, however, be useful for debugging VLD itself.

 

我们先看一下VLD的初始化过程，这有助于我们加深对VLD工作机制的理解。我这里使用的是vld 1.9h的版本，支持VS2008及以下的编译器。最新的2.0a版本已经可以支持vs2010

 

6. 非泄露内存增长

什么是非泄露内存增长？举例来说，你的程序有一个列表，所有已分配的内存都被记录在这个列表中，但是列表中的内存在某些情况下没有被删除。所以，未释放的内存越来越多，直到最后内存分配失败。但是，你使用之前的方法检查内存泄露却发现，并没有任何日志。原来，在你正常退出程序的时候列表中未被释放的内存已经被你挨个释放了。

实际情况要比这个复杂的多，也隐晦的多。这种情况也是内存泄露的一种，属于运行时泄露，它的危害更为严重。对于这种问题的追查也是一件很恼人的事情。那么从现在开始，让这么麻烦的问题见鬼去吧。

6.1 UMDH简介

 UMDH 是windows debug tools 下的一款命令行工具，它的全名是User-Mode Dump Heap 这个工具会分析当前进程在堆上分配的内存，并有两种模式

1. 进程分析模式，这个模式会对进程分配的每一块内存做记录，其中包含分配的内存大小；内存分配地址；内存分配时的函数调用堆栈等。

2. 日志分析模式，该模式会比较几个不同的日志，找出内存增长的地方。

在使用UMDH做分析之前我们要先做一些准备工作。

首先，打开进程的栈捕捉标志。这个步骤通过一行命令来完成

gflags /i ImageName +ust

这个命令只影响新启动的进程，对已经在运行状态的进程不起作用。

其次，安装windows 的 symbol文件。如果不需要的话可以不用安装。

最后，设置环境变量

set _NT_SYMBOL_PATH=Path

通过这几个步骤后我们才可以使用UMDH来对进程内存做分析。

 

首先，我们先启动目标进程，稍等一会儿，让进程进入稳定的运行状态。

之后我们通过命令行 umdh -p:2230  -f:dump_allocations.txt 对进程进行分析。

其中-p后面的数字是进程号， -f 后面跟的是日志文件的文件名。

等待一段时间后我们就会收集到一个文件，里面记录了一些内存的分配信息。

之后我们重复以上步骤几次，最好给文件名编个号。比如 dump1.txt dump2.txt

 

最后，我们通过命令行对刚才的文件进行分析

umdh -v dump1.txt dump2.txt > memleak.txt

这样我们得到了一个描述内存增长的日志文件memleak.txt类似如下的格式

 

+ 5320 (f110 - 9df0) 3a allocs BackTrace00053 
Total increase == 5320

ntdll!RtlDebugAllocateHeap+0x000000FD
ntdll!RtlAllocateHeapSlowly+0x0000005A
ntdll!RtlAllocateHeap+0x00000808
MyApp!_heap_alloc_base+0x00000069
MyApp!_heap_alloc_dbg+0x000001A2
MyApp!_nh_malloc_dbg+0x00000023
MyApp!_nh_malloc+0x00000016
MyApp!operator new+0x0000000E
MyApp!LeakyFunc+0x0000001E
MyApp!main+0x0000002C
MyApp!mainCRTStartup+0x000000FC
KERNEL32!BaseProcessStart+0x0000003D