After my departure from the ReactOSÂ project and subsequent new work for David Solomon, it wasn’t clear how much research and development on Windows internals I would still be able to do on a daily basis. Thankfully, I haven’t given up my number one passion — innovating, pushing the boundaries of internals knowledge, and educating users through utilities and applications.
In this vein, I have been working during my spare time on various new utilities that use new undocumented APIs and expose the internals behind Windows Vista to discover more about how the operating system works, as well as to be able to provide useful information to administrators, developers, students, and anyone else in between. In this post, I want to introduce my latest tool, MemInfo. I’ll show you how MemInfo can help you find bad memory modules (RAM sticks) on your system, track down memory leaks and even assist in detecting rootkits!
One of the major new features present in Windows Vista is Superfetch. Mark Russinovich did an excellent writeup on this as part of his series on Windows Vista Kernel Changes in TechNet Magazine. Because Superfetch’s profiling and management does not occur at the kernel layer (but rather as a service, by design choice), there had to be a new system call to communicate with the parts of Superfetch that do live in the kernel, just like Windows XP’s prefetcher code, so that the user-mode service could request information as well as send commands on operations to be performed.
Here’s an image of Process Explorer identifying the Superfetch service inside one of the service hosting processes.
Because Superfetch goes much deeper than the simple file-based prefetching Windows XP and later offer, it requires knowledge of information such as the internal memory manager lists, page counts and usage of pages on the system, memory range information, and more. The new SuperfetchInformationClass added to NtQuery/SetInformationSystem provides this data, and much more.
MemInfo uses this API to query three kinds of information:
- a list of physical address ranges on the system, which describe the system memory available to Windows
- information about each page on the system (including its location on the memory manager lists, its usage, and owning process, if any)
- a list system/session-wide process information to correlate a process’ image name with its kernel-mode objectÂ
MemInfo ultimately provides this information to the user through a variety of command line options described when you run the utility. Some of its various uses include:
Seeing how exactly Windows is manipulating your memory, by looking at the page list summaries.
The Windows memory manager puts every page on the system on one of the many page lists that it manages (i.e. the standby list, the zeroed page list, etc). Windows Internals covers these lists and usage in detail, and MemInfo is capable of showing their sizes to you (including pages which are marked Active, meaning currently in-use and by the operating system and occupying physical memory (such as working sets) and not on any of the lists). This information can help you answer questions such as “Am I making good use of my memory?” or “Do I have damaged RAM modules installed?”.
For example, because Windows includes a bad page list where it stores pages that have failed internal consistency checks, MemInfo is an easy way (but not 100% foolproof, since Windows’ internal checks might not have detected the RAM is bad) to verify if any memory hardware on the system is misbehaving. Look for signs such as a highly elevated count of pages in the zeroed page list (after a day’s worth of computer use) to spot if Windows hasn’t been fully using your RAM to its potential (you may have too much!) or to detect a large memory deallocation by a process (which implies large allocations previously done).
Here’s MemInfo on my 32-bit Vista system, displaying summary page list information.Windows Vista also includes a new memory manager optimization called prioritized standby lists — the standby state is the state in which pages find themselves when they have been cached by Windows (various mechanisms are responsible for this of caching, including the cache manager and Superfetch) and are not currently active in memory. Mark covered these new lists in his excellent article as well.
To expose this information to system administrators, three new performance counters were added to Windows, displaying the size of the prioritized standby lists in groupings: priorities 0 through 3 are called Standby Cache Reserve, 4 and 5 are called Standby Cache Normal Priority, and finally, 6 and 7 are called Standby Cache Core. MemInfo on the other hand, which can also display these new lists, is an even better tool to identify memory in the standby state, since it is able to display the size of these lists individually.
While memory allocations on Windows XP (which could be part of application startup, the kernel-mode heap, or simple memory allocations coming from various processes) would consume pages from a single standby list and thus possibly steal away pages that more critical processes would’ve liked to have on standby, Windows Vista adds 8 prioritized lists, so that critical pages can be separated from less important pages and nearly useless pages. This way, when pages are needed for an allocation, the lower priority standby lists are used first (a process called repurposing). By making snapshots of MemInfo’s output over a period of time, you can easily see this behavior.
Here’s MemInfo output before, during, and after a large allocation of process private memory. Notice how initially, the bulk of my memory was cached on the standby lists. Most of the memory then became Active due to the ongoing large allocation, emptying the standby lists, starting by the lowest priority. Finally, after the memory was freed, most of the memory now went on the zero page list (meaning the system just had to zero 1GB+ of data).
Seeing to what use are your pages being put to by Windows
Apart from their location on one of the page lists, Windows also tracks the usage of each page on the system. The full list includes about a dozen usages, ranging from non-paged pool to private process allocations to kernel stacks. MemInfo shows you the partitioning of all your pages according to their usage, which can help pinpoint memory leaks. High page counts in the driver locked pages, non-paged pool pages and/or kernel stack pages could be indicative of abnormal system behavior.
The first two are critical resources on the system (much information is available on the Internet for tracking down pool leaks), while the latter is typically tightly maintained for each thread, so a large number may indicate leaked threads. Other usages should also expect to see a lower number of pages than ones like process private pages, which is usually the largest of the group.
At the time of this writing, here’s how Windows is using my 4GB of memory:
Looking at per-process memory usage, and detecting hidden processes
Internally, Windows associates private process pages with the kernel executive object that represents processes as managed by the process manager — the EPROCESS structure. When querying information about pages, the API mentioned earlier returns EPROCESS pointers — not something very usable from user-mode! However, another usage of this API is to query the internal list of processes that Superfetch’s kernel-mode component manages. This list not only allows to take a look at how much memory, exactly, belongs to each process on the system, but also to detect some forms of hidden processes!
Hidden processes are usually the cause of two things. The first is processes which have been terminated, but not yet fully cleaned up by the kernel, because of handles which are still open to them. Task Manager and Process Explorer will not show these processes, but MemInfo is the only tool apart from the kernel debugger which can (so you don’t have to reboot in debugging mode). See below on how MemInfo is showing a SndVol32.exe process, created by Windows Explorer when clicking on the speaker icon in the system tray — Explorer has a bug which leaks the handles, so the process is never fully deleted until Explorer is closed.
The second cause of why a process may be hidden is a rootkit that’s hooking various system calls and modifying the information returned to user-mode to hide a certain process name. More advanced rootkits will edit the actual system lists that the process manager maintains (as well as try to intercept any alternate methods that rootkit detection applications may use), but MemInfo adds a new twist, by using a totally new and undocumented Superfetch interface in Windows Vista. It’s likely that no rootkit in the wild currently knows about Superfetch’s own process database, so MemInfo may reveal previously hidden processes on your system. Unfortunately, as with all information, it’s only a matter of time until new rootkits adapt to this new API, so expect this to be obsolete in the next two years.
There’s many more uses for MemInfo that I’m sure you can find — including as a much faster replacement for !memusage 8 if you’ve used that command in WinDBG before. MemInfo is fully compatible with both 32-bit and 64-bit versions of Windows Vista (including SP1 and Windows Server 2008 RC1 – even though Server 2008 does not include Superfetch, because both client and server versions of the OS will use the same kernel as of Vista SP1, the API set is identical), but not any earlier version of Windows. Apart from these simple summary views, MemInfo is powerful enough to dump the entire database of pages on your system, with detailed information on each — valuable information for anyone that needs to deal with this kind of data. Furthermore, unlike using WinDBG to attach to the local kernel, it doesn’t require booting the system into debug mode.
You can download a .zip file containing both versions from this link. Make sure to run MemInfo in an elevated command prompt — since it does require administrative privileges. The documentation for MemInfo is located on the following page (this page is part of an upcoming website on which I plan to organize and offer help/links to my tools and articles).
375
到【灌水乐园】发言
[…] Alex Ionescu’s Blog wrote an interesting post today on MemInfo: Peer Inside Memory Manager Behavior on Windows Vista and Server 2008Here’s a quick excerpt After my departure from the ReactOS project and subsequent new work for David Solomon, it wasn’t clear how much research and development on Windows internals I would still be able to do on a daily basis. Thankfully, I haven’t given up my number one passion — innovating, pushing the boundaries of internals knowledge, and educating users through utilities and applications. In this vein, I have been working during my spare time on various new utilities that use new undocumented APIs and expose th […]
[…] Read the rest of this great post here […]
[…] Any reader who is interested in this topic is encouraged to refer to Andreas Schuster articles about the dmp file format, Mark Russinovich “Pushing the Limits of Windows: Physical Memory“, and Alex Ionescu’s excellent tool “meminfo“. […]
Hi
Awsome program. I have for a long time wanted a way to see what goes on in windows memory management.
I would like to write a monitor program that can log how the usage of physical memory changes over time as a result of program activity and system memory management.
To be able to do this I need to get the same kind of information from the system that meminfo gets.
Unfortunately the SuperfetchInformationClass is undocumented. I suppose you have used reverse engineering to find out how this works.
How can I write my own program that uses SuperfetchInformationClass ?
Could you help me somehow so I do not have to do my own full reverse engineering?
The best thing would be if I could get the full source for meminfo, that would be awesome.
Or, maybe you could provide me with your headerfile that just defines SuperfetchInformationClass ?
I promise to give you credit in any program I create based on this information.
Thanks in advance.
[…] Meminfo est intéressant également, il permet en complément de Sysinternal Process Explorer d’avoir […]
Thanks for this tool, and the clarified info in the 5th edition. Because of this article, I have updated my free GUI based tool that looks at memory usage over time to include the full standby lists (see blog at http://www.brianmadden.com/blogs/timmangan/archive/2009/08/18/updated-atm-tool.aspx ).
In using this tool to monitor on Windows 7, the ammount of memory in the lists, I sometimes see scenarios where it appears memory in a standby list become “reprioritized”. For example, after coming out of hybernation, I may see what looks like memory in standby list #7 (what I call the “system prefetch” list) become reprioritized to “normal” (#5).
I do not see such behavior described in WI 5th edition, although it clearly does not cover Windows 7 yet.
Hi!
I get the folowing error when I run ‘meminfo -u’ or ‘meminfo -s’ on Windows Server 2008 x64.
meminfo -u
MemInfo v2.10 – Show PFN database information
Copyright (C) 2007-2009 Alex Ionescu
http://www.alex-ionescu.com
Initializing PFN Database… Done
Failure initializing File Info connection: c0000034
I’ve downloaded the zip file and I’m using a 64 bit executable.
[…] Additional ressources Developping Debugger Extensions – PDC Conference The debugger extension – Several parts MemInfo: Peer Inside Memory Manager Behavior on Windows Vista and Server 2008 […]
[…] that meminfo gives you a way to investigate these page lists: Alex Ionescu provides further helpful commentary on the tool on his […]
[…] that meminfo gives you a way to investigate these page lists: Alex Ionescu provides further helpful commentary on the tool on his […]
[…] that meminfo gives you a way to investigate these page lists: Alex Ionescu provides further helpful commentary on the tool on his […]
It seems, some calculations of meminfo 32bit are truncated.
C:\Windows\system32>meminfo -r
MemInfo v2.10 – Show PFN database information
Copyright (C) 2007-2009 Alex Ionescu
http://www.alex-ionescu.com
Physical Memory Range: 00001000 to 0009D000 (156 pages, 624 KB)
Physical Memory Range: 00100000 to CAF67000 (831079 pages, 3324316 KB)
Physical Memory Range: CAF6E000 to CB3B9000 (1099 pages, 4396 KB)
Physical Memory Range: CB7D8000 to DD7DD000 (73733 pages, 294932 KB)
Physical Memory Range: DEFFF000 to DF000000 (1 pages, 4 KB)
Physical Memory Range: 00000000 to 1F000000 (1175552 pages, 507904 KB)
MmHighestPhysicalPage: 2224128
1175552 pages=4702208 kb = hex 47C000 kb
507904 kb = hex 7c000 kb, the 4 is truncated
C:\Windows\system32>meminfo -s
MemInfo v2.10 – Show PFN database information
Copyright (C) 2007-2009 Alex Ionescu
http://www.alex-ionescu.com
Initializing PFN Database… Done
PFN Database List Statistics
Zeroed:1099483 ( 203628 kb)
Free: 0 ( 0 kb)
Standby: 456521 (1826084 kb)
Modified: 84063 ( 336252 kb)
ModifiedNoWrite: 1 ( 4 kb)
Active/Valid: 440029 (1760116 kb)
Transition: 2 ( 8 kb)
Bad: 0 ( 0 kb)
Unknown: 1521 ( 6084 kb)
TOTAL:2081620 (4132176 kb)
2081620 pages = hex 7f0d50 kb
4132176 kb = hex 3f0d50 kb
it seems, truncation occurs after bit 22
[…] Indeed, as part of the undocumented Superfetch API that I first discovered and implemented in my meminfo tool, which has now been supplanted by the RAMMap utility from SysInternals, the memory manager will […]