Debugging USER object leak

We had an odd user object leak happening in our product. One of the part of the product has an ActiveX control inside of a webpage that user can use to interact with the product. If user continuously reloads the page with the ActiveX control (by navigating between pages for example), we see a continuous USER objects counter growth against the IE process in the Task Manager.

We tried a few tools, such as DebugDiag (does not help - it often produces error messages during analysis), my domdbg WinDBG extension (with some additional commands that I’ll blog about later), GlowCode (very nice product but sometimes shows too many leaks which seem to be related to internal structures of the tool itself).

This article is mostly about using the second approach - bringing up WinDBG and going deeply into the Windows internals to find out where the memory leak is.

Follow up:

The first question that I had was - which object is it leaking? USER objects is too wide category. According to Microsoft, the following objects can be USER objects:

• Accelerator table
• Caret
• Cursor
• DDE conversation
• Hook
• Icon
• Menu
• Window
• Window position

From my experience though, there are many more types of objects, some of them are internal.

One interesting thing is that Icon and Cursor are overlapping:

	0:001> uf /c user32!CreateCursor
	USER32!CreateCursor (7707ccd0)
	USER32!CreateCursor+0x2d (7707ccfd):
	call to USER32!memset (77067a40)
	USER32!CreateCursor+0x54 (7707cd24):
	call to USER32!CreateIcoCurIndirect (7707cbd6)
	0:001> uf /c CreateIconIndirect
	USER32!CreateIconIndirect (7705c256)
	...
	USER32!CreateIconIndirect+0x1ad (7705c3e8):
	call to USER32!_SetCursorIconData (77060026)
	...
	USER32!CreateIconIndirect+0x1e6 (770885d1):
	call to USER32!NtUserDestroyCursor (77063583)

As you can see, there are cursor/icon functions which help manager both objects. It makes sense, as internally they are just structures containing bitmaps. This also gives a hint why there are many more object types - many complex objects wrap smaller helper objects (often GDI, but sometimes USER) which also need to be managed.

I could not find the API or any other way to tell which objects are being allocated in which amounts. I tried to figure out how Task Manager gets the counter, and found this tool which is similar to process explorer - Process Hacker. It has a source code so soon I found out that USER objects are retrieved using function GetGuiResources from user32. Unfortunately, that only returns USER and GDI objects count, but does not give more details. However, I decided to look at its code to see if I can spot more numbers in vicinity.

Disassembling GetGuiResources gives us information about the kernel entry point:

	0:001> uf user32!NtUserGetGuiResources
	USER32!NtUserGetGuiResources:
	770b3a3b b8a3110000      mov     eax,11A3h
	770b3a40 ba0003fe7f      mov     edx,offset SharedUserData!SystemCallStub (7ffe0300)
	770b3a45 ff12            call    dword ptr [edx]
	770b3a47 c20800          ret     8

If you want to get the detailed information about what this means, please do read the book by Mark Russinovich on Windows Internals . In a nutshell, it is an entry into the system service dispatcher table. Because we know that this is a function for USER objects, it must end up in the Windows subsystem (win32k.sys).

This already gives us enough information to find the kernel function. Before doing that, we’d need to make use of LiveKD - live kernel debugger which allows us to debug the kernel space without the need for serial connection.

Let’s see if we can find the function in Win32k.sys:

	0: kd> x win32k!*GuiResource*
	a1818e54 win32k!NtUserGetGuiResources

This is too easy. If you cannot find the function that easily, you still can look its exact address in the service table. The table can be found by looking at nt!KeServiceDescriptorTableShadow:

	82144b40  820b982c nt!KiServiceTable
	82144b44  00000000
	82144b48  00000187
	82144b4c  820b9e4c nt!KiArgumentTable
	82144b50  a18bc000 win32k!W32pServiceTable
	82144b54  00000000
	82144b58  00000304
	...

The table consists of two records to service tables, each record has 4 dwords. The third dword in each record is the number of entries in the actual table. There are two tables - kernel and win32k (Windows subsystem). Since we know that the service is for USER objects, it should be somewhere in the second table, win32k!W32pServiceTable. The original service index, 11A3h is 1000h + the index in this table, so the offset will be 68ch:

	0: kd> dds win32k!W32pServiceTable + 1a3h*4
	a18bc68c  a1818e54 win32k!NtUserGetGuiResources

You can get more verbose representation of the service table from this post: Defeating Kernel Native API Hookers by Direct KiServiceTable… where it looks like this:

	typedef struct ServiceDescriptorTable {
	SDE ServiceDescriptor[4];
	} SDT;
	typedef struct ServiceDescriptorEntry {
	PDWORD KiServiceTable;
	PDWORD CounterTableBase;
	DWORD ServiceLimit;
	PBYTE ArgumentTable;
	} SDE;

Now that we found the function, let’s see what it does:

	0: kd> uf /c win32k!NtUserGetGuiResources
	win32k!NtUserGetGuiResources (a1818e54)
	...
	win32k!NtUserGetGuiResources+0x29 (a1818e7d):
	call to nt!PsGetCurrentProcessWin32Process (820bcaf1)
	...
	win32k!NtUserGetGuiResources+0x8e:
	a1818ee2 8b7028          mov     esi,dword ptr [eax+28h]
	a1818ee5 eb08            jmp     win32k!NtUserGetGuiResources+0x9b (a1818eef)
	win32k!NtUserGetGuiResources+0x93:
	a1818ee7 8b7024          mov     esi,dword ptr [eax+24h]
	a1818eea eb03            jmp     win32k!NtUserGetGuiResources+0x9b (a1818eef)
	...

What this essentially does, is that the function retrieves _W32PROCESS object and then returns the value at either offset 24h or 28h, depending on arguments (one of them will be USER object, another GDI objects). Let’s confirm that:

	0: kd> !process 0 0 far.exe
	PROCESS 85f32710  SessionId: 1  Cid: 051c    Peb: 7ffdd000  ParentCid: 0e3c
	DirBase: bca75a40  ObjectTable: 9d004a60  HandleCount: 143.
	Image: Far.exe

PROCESS points to nt!_EPROCESS structure, which has _W32PROCESS in one of its fields:

	0: kd> dt nt!_EPROCESS 85f32710
	+0x000 Pcb              : _KPROCESS
	...
	+0x108 Win32Process     : 0xf7eb9858
	...

So, let’s see what is at those offsets:

	0: kd> dd 0xf7eb9858 + 24h
	f7eb987c  00000014 00000004 00000000 f7eb9888
	0: kd> dd 0xf7eb9858 + 28h
	f7eb9880  00000004 00000000 f7eb9888 00000000

Task Manager for far.exe shows similar numbers, with 4 being USER objects, 20 being GDI objects. Unfortunately, even though we found the structure which keeps the counters, this does not help much. There are no symbols for _W32PROCESS, but quick look at functions which supposedly create/destroy objects (such as win32k!HMDestroyObject, win32k!HMAllocObject, win32k!HMFreeObject) tells that they only update these counters. There does not seems to be a counter for each object type.

Well, if there are no counters, we may still be able to track objects if we were able to find where they are stored, and how. Let’s look at win32k!HMAllocObject. The first thing that you notice is how it handles one of its parameters:

	a17b618f 0fb67d10        movzx   edi,byte ptr [ebp+10h]
	win32k!HMAllocObject+0xf:
	a17b6193 8bf7            mov     esi,edi
	a17b6195 6bf60c          imul    esi,esi,0Ch
	a17b6198 8a9ef0b18aa1    mov     bl,byte ptr win32k!gahti+0x8 (a18ab1f0)[esi]

win32k!gahti seems to be a lookup table with records of size 12 (0ch). The parameter seems to be an index in this table. Let’s see what’s inside:

	0: kd> dds win32k!gahti
	a18ab1e8  00000000
	a18ab1ec  00000000
	a18ab1f0  00000000
	a18ab1f4  a178869f win32k!xx*DestroyWindow
	a18ab1f8  64777355
	a18ab1fc  00000039
	a18ab200  a1784903 win32k!_DestroyMenu
	a18ab204  00000000
	a18ab208  00000012
	a18ab20c  a175c581 win32k!DestroyUnlockedCursor
	a18ab210  75637355
	a18ab214  0000000e
	a18ab218  a176162f win32k!DestroySMWP
	a18ab21c  77737355
	a18ab220  00000009
	a18ab224  a1783d8a win32k!FreeHook
	a18ab228  00000000
	a18ab22c  00000011
	a18ab230  a1821a4f win32k!HMNullFnDestroy
	a18ab234  62637355
	a18ab238  00000000
	a18ab23c  a175a03e win32k!HMDestroyObject
	a18ab240  00000000
	a18ab244  00000012
	a18ab248  a175a03e win32k!HMDestroyObject
	a18ab24c  63617355
	a18ab250  0000000a
	a18ab254  a1821a4f win32k!HMNullFnDestroy
	a18ab258  39647355
	a18ab25c  00000009
	a18ab260  a17e9192 win32k!CleanupAndFreeDdeConv
	a18ab264  41647355
	0: kd> dds
	a18ab268  00000009
	a18ab26c  a17e97c1 win32k!FreeDdeXact
	a18ab270  42647355
	a18ab274  00000009
	a18ab278  a17d0314 win32k!DestroyMonitor
	a18ab27c  69647355
	a18ab280  00000040
	a18ab284  a17d2886 win32k!DestroyKL
	a18ab288  626b7355
	a18ab28c  00000000
	a18ab290  a17d2927 win32k!DestroyKF
	a18ab294  666b7355
	a18ab298  00000000
	a18ab29c  a1740ab9 win32k!DestroyEventHook
	a18ab2a0  65777355
	a18ab2a4  00000001
	a18ab2a8  a17b6564 win32k!FreeTimer
	a18ab2ac  6d747355
	a18ab2b0  00000002
	a18ab2b4  a1790323 win32k!FreeInputContext
	a18ab2b8  6d697355
	a18ab2bc  00000011
	a18ab2c0  a183756b win32k!FreeHidData
	a18ab2c4  61687355
	a18ab2c8  00000001
	a18ab2cc  a170ac0e win32k!FreeDeviceInfo

This looks like a bunch of records, each of them containing a method somehow related to destruction. It seems like it is a type table. The idea is confirmed by looking at win32k!HMFreeObject:

	win32k!HMFreeObject+0xd:
	a17b63c3 81e6ffff0000    and     esi,0FFFFh
	a17b63c9 6bf60c          imul    esi,esi,0Ch
	a17b63cc 0335441a8da1    add     esi,dword ptr [win32k!gSharedInfo+0x4 (a18d1a44)]
	a17b63d2 0fb64608        movzx   eax,byte ptr [esi+8]
	a17b63d6 6bc00c          imul    eax,eax,0Ch
	a17b63d9 8a98f0b18aa1    mov     bl,byte ptr win32k!gahti+0x8 (a18ab1f0)[eax]
	a17b63df f6c302          test    bl,2
	a17b63e2 7405            je      win32k!HMFreeObject+0x33 (a17b63e9)

If the input is a handle, then this look interesting. First, the handle is cut-off at 16 bit boundary. Then, multipled by 12 to get an offset in win32k!gSharedInfo which is unknown so far. Then, a byte at offset 8 is read from the record in gSharedInfo and it gives offset to a record in gahti table. This record is then used to make decision on how to free the object (test bl, 2) - this looks like a flag check. Searching through internet clarifies what gSharedInfo is:

	typedef struct _HANDLEENTRY {
	PHEAD   phead;                  /* pointer to the real object *///for example,  HOOK_INFO
	PVOID   pOwner;                 /* pointer to owning entity (thread or process) */
	BYTE    bType;                  /* type of object */
	BYTE    bFlags;                 /* flags - like destroy flag */
	WORD    wUniq;                  /* uniqueness count */
	} HANDLEENTRY, *PHE;
	typedef struct tagSHAREDINFO {
	struct tagSERVERINFO *psi;
	struct _HANDLEENTRY   *aheList;         // handle table pointer
	struct tagDISPLAYINFO *pDispInfo;       // global displayinfo
	ULONG                  ulSharedDelta;   // Delta between client AND kernel mapping of?
	// shared memory section. Only valid/used in client.
	/*
	* This field is used for DLL preloading to allow custom functinality
	* for all Windows processes.
	*
	LPWSTR pszDllList;
	WNDMSG awmControl [FNID_END - FNID_START + 1];
	WNDMSG DefWindowMsgs;
	WNDMSG DefWindowSpecMsgs;
	} SHAREDINFO and *PSHAREDINFO;
	extern SHAREDINFO gSharedInfo;

The important bits are that at offset 4 gSharedInfo contains a pointer to the handles table, each record being _HANDLEENTRY type (undeclared). The record in turn, contains its type at the offset 8. At the offset 0 it stores the pointer to the actual object (depends on type), and at offset 4 either the pointer to _W32THREAD (see dt win32k!_W32THRREAD) or _W32PROCESS (not declared), depending on the object type. Some objects are owned by the process, but some have thread affinity and this also depends on type. Small note however, is that the BYTE in the description above (copied from the internet) means 16-bit type.

This gives us all the information we need to start tracking objects. We know where to find objects (in the table referenced by gSharedInfo), we know how to get the type of object. We know how to translate the handle from user mode to kernel (though not vice versa yet). Just to clarify the last unknown, let’s loot at two handles. I decided to take two HWNDs using SpyXX, e603ea and 9e0214.

As we know, the index part is only first 16 bits, so the indexes are 3ea and 214. What are e6 and 9e, which are higher words? Let’s look at the handle entries:

	0: kd> dd win32k!gSharedInfo
	a18d1a40  ff9d0570 ff910000 ff9d1670 00000000
	...
	0: kd> dd ff910000h + 0ch*3eah
	ff912ef8  fe6ca788 f749dcf0 00e60001 fe0af808
	...
	0: kd> dd ff910000h + 0ch*214
	ff9118f0  fe69dd40 fe1a8660 009e0001 fd9d24b0

What we see is that both of them have this prefix in their 10th byte, where the flags are stored. Note also that the type of both objects (the value at offset 8) is 1, which is where xx*DestroyWindow in gahti table. So now we also know how to convert handles from kernel to user and vise versa.

How can we use this information for tracking leaks? In our case, all we wanted was to identify which object is actually leaked. So I’d do something like this:

• Start IE, navigate to the leaking page, then make a dump of all handles
• Repeat loading leaking page fixed number of times. Make a dump of all handles
• Compare dumps

Now, it is important to note that the handles table are global for the whole Windows subsystem. So if we want to isolate the handles for a particular process, we need to count only those which match the _W32PROCESS/_W32THREAD entries corresponding to the leaking process. I did not have a kernel driver which would allow me to easily correlate all this information. Instead, I wrote a small tool which allowed me to load a dump of all handles (from a text file produced manually using WinDBG/LiveKD using simply dd). The tool groups handles by owner handle (process or thread) and then by the handle type (since we now know types). If you know how many objects were leaked (and you probably would if you reproduce the problem fixed known number of times), and the number is quite big, it is easy to find the right group. That’s how the output of the tool looks like:

As a sanity check, I verified that the group is indeed right. I did this by taking the owner handle (process or thread) and using it to walk back towards the _EPROCESS structure. _EPROCESS has the image file name field which clearly states the name of the executable. If that matches with the leaking process - the original group is right. It works like this. If the handle 0xf7eb9858 is supposed to be “Far.exe” process, then:

	0: kd> dd 0xf7eb9858
	f7eb9858  85f32710 00000001 3004c033 ffffffff
	0: kd> dt nt!_EPROCESS 85f32710
	+0x000 Pcb              : _KPROCESS
	...
	+0x108 Win32Process     : 0xf7eb9858
	...
	+0x14c ImageFileName    : [16]  "Far.exe"

This confirms that 0xf7eb9858 is indeed the W32PROCESS of “Far.exe", and it is a _EPROCESS structure corresponding to the _W32PROCESS structure.

If you have a thread handle, it works like this. From the screenshot above, let’s look at what could be a Win32K thread f603c4b8:

	0: kd> dt win32k!_W32THREAD f603c4b8
	+0x000 pEThread         : 0x9723ed78 _ETHREAD
	...
	0: kd> dt nt!_KTHREAD 0x9723ed78
	+0x000 Header           : _DISPATCHER_HEADER
	...
	+0x144 Process          : 0x88d6d4b8 _KPROCESS
	...
	0: kd> dt nt!_EPROCESS 0x88d6d4b8
	+0x000 Pcb              : _KPROCESS
	...
	+0x108 Win32Process     : 0xfd889bc0
	...
	+0x14c ImageFileName    : [16]  "dexplore.exe"

A little bit long, and a kernel driver would help, but it is doable.

Once you’ve found a bucket of handles with the right number, you now know the type. This means that you know which functions create/destroy these objects. You can then use the !create_object2/!delete_object2 extension methods from my domdbg WinDBG extension (see Object tracking functions) to track handles that were created/destroyed. The tool also reports the handle value (as an index in the table), and we know how to convert it to the user mode value, so it’s easy then to map it to a stack using !object_stack extension method from the domdbg extension. For the details on how to use these commands, see the documentation of the domdbg extension. Also, by expanding the type node, you get the actual handle values. If you do some logging in your app you can match these handles with the log straight away.

In my case, however, that did not work. If you look at the gahti table above you would notice that some of the objects are quite generic - they have win32k!HMDestroyObject or win32k!HMNullFnDestroy as their destructor. This means that you can’t tell their type by just looking at the gahti table. That was my case - the object type was 7, and I still had no clue which object that is. As it often happens, this whole investigation was for nothing - it did not help a bit. I wrote an up which leaked pretty much all types of objects that I could imagine - and none of them was of type 7. But I decided to blog about it because this kind of information will help someone in some cases (for example, for windows, menus, icons, defer window position situations).

In my case, the help came from a good luck - I saw an error message in a log file at the same time as the leak was being generated. And I decided to comment out the piece of code (whole tooltips logic) which was responsible for the error. It helped the leak as well! It took a few hours of bisection to find the actual line responsible for the leak - it was TTF_SUBCLASS flag on a tooltip, a known Microsoft problem.

See more on the next page about the DisplayUserObjects tool.

About the DisplayUserObjects tool. You can download it from here (binary plus source code, Apache license). Use it at your own risk.

That’s how you would use it:

1. Start LiveKd with WinDbg (see instructions for LiveKd on how to do it)
2. (in Windbg) “dd /c 8 poi(win32k!gSharedInfo+4) l2000
3. Copy and paste the output into a text file. If you see lots of ???? symbols at the end (those are the markers of unreadable pages), delete them from the end until you have only numbers
4. Run “DisplayUserObjects [name of the file with the output]”

The second line produces the dump of the handle table. I don’t know how to get its size, but that’s not really necessary. Just pick a big number for the last argument ("l2000″) - l2000 or l5000, until you see lots of zeros at the end, or ??? marks. This means you are outside of the table. Zeros don’t matter much - they usually don’t affect the output much. If you want to be pedantic - scan backwards until you see a pattern of repeated triples of numbers (remember, the handle record is 12 bytes - 3 dwords, and they all look similar).