Diagnosing symbol problems

Most problem when using windbg on my site is symbol issue, so log it.

 

 

Diagnosing symbol problems

If you’re having problems loading symbols or you think you’ve loaded incorrect symbols, please do the following before sending a bug to the Debugger TRIAGE alias (dbg). If you’ve followed these steps and you’ve still got problems, go to Section 2, You’ve Found a Bug.

First check your sympath:

 

kd> .sympath.
Symbol search path is: k:/sym/retail

If you sympath is wrong fix it, reload symbols (!reload module-name) and see if it works. If you are using the kernel debugger make sure your local %WINDIR% is not on you sympath.

If your sympath is OK, turn on "noisy" mode so you can see what symbol files dbghelp is loading and reload your module:

 

kd> !sym noisy
Noisy mode on.
kd> .reload nt
DBGHELP: LocatePDB-> Looking for ./ntoskrnl.pdb... file not found
DBGHELP: LocatePDB-> Looking for h:/release/i386/symbols/exe/ntoskrnl.pdb... file not found
DBGHELP: LocatePDB-> Looking for h:/release/i386/exe/ntoskrnl.pdb... file not found
DBGHELP: LocatePDB-> Looking for h:/release/i386/ntoskrnl.pdb... unmatched pdb
DBGHELP: LocatePDB-> Looking for k:/sym/retail/symbols/exe/ntoskrnl.pdb... file not found
DBGHELP: LocatePDB-> Looking for k:/sym/retail/exe/ntoskrnl.pdb... OK

The symbol handler first looks for an image that matches the module you are trying to load. The image itself is not always necessary, but if we find an incorrect one, the symbol handler will often fail. Line 4 shows that we found a pdb at h:/release/i386/ntoskrnl.pdb but the pdb didn’t match. Because the pdb didn’t match, we continue on. We then search for a DBG file and PDB file that match the loaded image, this is lines 5-6. If the symbol-search encountered a catastrophic failure, a message of the form:

ImgHlpFindDebugInfo(00000000, module.dll, c:/foo;c:/bar, 0823345, 0) failed

These failures include items such as file-system failures, network errors, and corrupt DBG files. This is usually not a bug – follow the procedures below and they will help you find the source of the problem.

To do verify symbols for a module do !lmi <mod_name>. This will display the status of symbols for the module.

kd> !lmi nt

Loaded Module Info: [nt]

         Module: ntoskrnl

   Base Address: 0x80400000

     Image Name: ntoskrnl.exe

   Machine Type: 332 (I386)

     Time Stamp: 387da733 Thu Jan 13 02:21:39 2000

       CheckSum: 29b38a

Characteristics: 10e

Debug Data Dirs:

           Type Size     VA  Pointer

       CODEVIEW   3e,  65bc,    65bc NB10, Sig: 387da732, Age: 1, Pdb: ntoskrnl.pdb

    Symbol Type: PDB,   Symbols loaded successfully

The Symbol Type row shows which type of symbols the image has and the error string if there were errors in loading the symbols. The possible symbol types are:

None	Image has no symbols
COFF	COFF symbols
CV	CODEVIEW Symbols
PDB	Image symbols taken from pdb
Export	Image symbols read from Image itself – this is done as last resort when other symbols cannot be found.
Deferred	Debugger hasn’t tried to load image symbols
SYM16	Sym files

Diagnosing symbol loading errors

When in noisy mode the debugger may print out error codes when it cannot load a symbol file.  The error codes for dbg files are listed in winerror.h.  The pdb error codes come from another source and the most common errors are printed in plain English text.  If the pdb error is a large number then a problem exists with your debugger installation and you should reinstall the debuggers using the method describe in http://dbg/download.  For example,

DBGHELP: LocatePDB-> Looking for //ntstress/symbols/x86/2106/symbols/exe/ntkrnlmp.pdb... pdb error 0xbcef678

Some common error codes for dbg files from winerror.h are

ERROR_BAD_FORMAT 0xb

ERROR_PATH_NOT_FOUND     0x3

ERROR_BAD_NETPATH            0x35

It’s possible that the symbol file cannot be loaded because of a networking error.  If you see ERROR_BAD_FORMAT or ERROR_BAD_NETPATH and you are loading symbols from another machine on the network, try copying the symbol file to your local machine and put its path in your symbol path.  Then try to reload the symbols.

If you get the pdb error 0xb, use //dbg/tools/x86/vc6pdbtst.exe to verify that you have a valid VC6 pdb.  The syntax for verifying a file is:

//symbols/tools/x86/vc6pdbtst.exe <pdbfilename>

 

If you are building with VC7 tools and debugging with our debuggers you must use the following switches in order to generate valid VC6 binaries:

• /Ztmp
• /tmp

Verifying your search path and symbols

Given a sympath of c:/foo;c:bar we will search in the following places for debug information:

In cases where the binary has been stripped of debug information, such as the standard NT builds, we first, look for a DBG file. For a DBG file we look in:

c:/foo/symbols/exe/ntoskrnl.dbg

c:/bar/symbols/exe/ntoskrnl.dbg

c:/foo/exe/ntoskrnl.dbg

c:/bar/exe/ntoskrnl.dbg

c:/foo/ntoskrnl.dbg

c:/bar/ntoskrnl.dbg

current-working-directory/ntoskrnl.dbg

Next look for a PDB file:

c:/foo/symbols/exe/ntoskrnl.pdb

c:/foo/exe/ntoskrnl.pdb

c:/foo/ntoskrnl.pdb

c:/bar/symbols/exe/ntoskrnl.pdb

c:/bar/exe/ntoskrnl.pdb

c:/bar/ntoskrnl.pdb

current-working-directory/ntoskrnl.pdb

Note that in the search for the DBG file we interleave searching through the foo and bar directories, but in the PDB search we do not.

Next, find your module in the module list. Note the starting address of the module you are interested in.  You can dump the module information using either the module name or starting address.

kd> lm

start    end        module name

78000000 78046000   msvcrt       (deferred)

80062000 80078480   hal          (deferred)

805d1000 80857f40   nt           (pdb symbols)              k:/sym/retail/exe/ntoskrnl.pdb

a0000000 a019a000   win32k       (deferred)

f5f96000 f5fa9000   ipsec        (deferred)

f6099000 f60bb000   fastfat      (deferred)

f61e3000 f61e6000   spud         (deferred)

f61fb000 f6234000   srv          (deferred)

 Module list truncated due to length.

In this case, we dump the module by start address.

kd> !lmi 805d1000

Loaded Module Info: [805d1000]

         Module: ntoskrnl

   Base Address: 0x805d1000

     Image Name: ntoskrnl.exe

   Machine Type: 332 (I386)

     Time Stamp: 39010216 Fri Apr 21 18:36:22 2000

       CheckSum: 28e5a7

Characteristics: 10e

Debug Data Dirs:

           Type Size     VA  Pointer

       CODEVIEW   3e,  6ffc,    6ffc NB10, Sig: 39010216, Age: 1, Pdb: ntoskrnl.pdb

    Symbol Type: PDB,   Symbols loaded succesfully

kd>

The highlighted time date stamp on the 6^th line is the first piece of information. This should match the time date stamp of the image you think you are loading and the DBG file you think you are loading. To find the time-date stamp from an image use link /dump /headers:

K:/bin>link -dump -headers ntoskrnl.exe

Microsoft (R) COFF Binary File Dumper Version 6.20.8755

Copyright (C) Microsoft Corp 1992-2000. All rights reserved.

Dump of file ntoskrnl.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES

             14C machine (i386)

              14 number of sections

        39010216 time date stamp Fri Apr 21 18:36:22 2000

               0 file pointer to symbol table

               0 number of symbols

              E0 size of optional header

             10E characteristics

                   Executable

                   Line numbers stripped

                   Symbols stripped

                   32 bit word machine

OPTIONAL HEADER VALUES

In this example, the time-date stamps match. If the symbols are mismatched – whatever strange behavior you are seeing, is not a debugger bug. Get the correct symbols and try again.

So the time-stamp matches and probably you've built with PDB information. If not, you should rebuild with PDB debug information because COFF is poorly supported. We need to find the Codeview NB10 record, which is a pointer to the PDB file. If you haven't split your symbols (no DBG file) this record will be in the Debug Directory of your executable image and will be of type "cv" = Codeview. If you have split symbols, this record will be located in the associated DBG file. Looking at the highlighted cv record in the !lmi output above, we see that the PDB signature is 39010216 (CODEVIEW    3e,  6ffc,    6ffc NB10, Sig: 39010216, Age: 1, Pdb: ntoskrnl.pdb). Now verify that the PDB you think should be being loaded has the same signature using the pdbdump program.

K:/>pdbdump k:/sym/retail/exe/ntoskrnl.pdb hdr
Header for MSPDB.dll = {
        intv =   19970116
        impv =   19970604
        }
Header for 'k:/sym/retail/exe/ntoskrnl.pdb' = {
        impv =   19970604
        age  = 0x1 (1)
        sig  = 0x39010216
        }

---

You’ve Found a Bug.

Send an email to the dbg alias with the following information:

• Name of image that is not loading and output of !lmi.
• The output of .sympath and .reload image-name
• The first couple of lines from the dump of your DBG file, if symbols are stripped, or your image if symbols have not been stripped. Use link /dump /headers to get these.
• The contents of the Debug Directory for your image or DBG, which contains the Codeview NB10 record. Do not send a debug directory that does not contain a NB10 record, as this is worthless to us. Use link /dump /headers to get this.
• The signature of the PDB file that you think should be being loaded. Use “pdbdump pdbname hdr” to get this.
• A remote session to the debugger that is failing.

An example of a good email is as follows:

 

The symbols for ntkrnlmp.exe are not loading.

kd> .sympath
Current Symbol Path is: d:/nt/bin/chk/i386/symbols/retail
kd> .reload nt
kd> !sym noisy
DBGHELP: FindExecutableImageEx-> Looking for D:/nt/bin/chk/i386/ntkrnlmp.exe...mismatched timestamp
DBGHELP: No image file available for ntkrnlmp.exe
DBGHELP: LocatePDB-> Looking for d:/nt/bin/chk/i386/symbols/retail/symbols/exe/ntkrnlmp.pdb...no file
DBGHELP: LocatePDB-> Looking for d:/nt/bin/chk/i386/symbols/retail/exe/ntkrnlmp.pdb...unmatched pdb OK
Kernel Version 2081 MP Checked
Kernel base = 0x80400000 PsLoadedModuleList = 0x80506fa0
DBGHELP: FindExecutableImageEx-> Looking for D:/nt/bin/chk/i386/ntkrnlmp.exe...mismatched timestamp
DBGHELP: No image file available for ntkrnlmp.exe
DBGHELP: LocatePDB-> Looking for d:/nt/bin/chk/i386/symbols/retail/symbols/exe/ntkrnlmp.pdb... no file
DBGHELP: LocatePDB-> Looking for d:/nt/bin/chk/i386/symbols/retail/exe/ntkrnlmp.pdb...unmatched pdb OK
kd> !lmi nt
Loaded Module Info: [nt]
         Module: ntoskrnl
   Base Address: 0x80400000
     Image Name: ntoskrnl.exe
   Machine Type: 332 (I386)
     Time Stamp: 37967462 time date stamp Wed Jul 21 18:31:14 1999        CheckSum: 29b38a
Characteristics: 10e
Debug Data Dirs:
           Type Size      VA  Pointer
       CODEVIEW 3e, 65bc, 65bc NB10, Sig: 387da732,Age: 1, Pdb: ntoskrnl.pdb
    Symbol Type: PDB, Symbols loaded succesfully
kd>

D:/nt/bin/chk/i386/symbols/retail/exe>pdbdump ntkrnlmp.pdb hdr
pdbdump ntkrnlmp.pdb hdr
Header for MSPDB.dll = {

        intv =   19970116

        impv =   19970604

        }

Header for 'ntkrnlmp.pdb' = {

        impv =   19960307

        age  = 0x1 (1)

        sig  = 0x387da732

        }

Remote /C MATH-X86 MATH-X86C

Note that since the checksum’s don’t match in the above example, you would get a not-very-nice reply saying your checksums don’t match, so of course your symbols don’t match.

3. Questions and Misconceptions

 

Q: What happens when the debugger is unable to read the image header?

A:  The following is the typical output of a .reload command

Loading Kernel Symbols
..........................................................................Unable to read image header for win32k.sys at 20000000`00000000 - status 0xD0000001
.Unable to read image header for ati2draa.dll at 20000000`00558000 - status 0xD0000001.........
Loading User Symbols
PPEB is NULL

The above output is actually expected in many scenarios. Each dot represents a module being loaded. The message about win32k indicates the debug information in the image for this module has been paged out by the OS. Because of this, the debugger will not be able to read and later match the timestamp / checksum / signature for the symbols, and using the symbol server will actually fail to find the correct symbols. But using a regular symbol path which points directly to the symbol tree will work fine and allow you to load the symbols.

The message about PPEB being NULL is expected for the system process. It simply means that we could not find the Process Environment Block which normally exists for each user-mode application. But the system process does not have any user mode address space, and therefore no PPEB.

Q: Suppose you do .reload lsasrv.dll and get the following (with !sym noisy):

SYMSRV: No ID tokens for lsasrv.dbg
SYMSRV: No ID tokens for lsasrv.dbg
DBGHELP: FindExecutableImageEx-> Looking for D:/windbg_beta/lsasrv.dll... no file
SYMSRV: No ID tokens for lsasrv.pdb
SYMSRV: No ID tokens for lsasrv.pdb
OK
DBGHELP:
SYMSRV: No ID tokens for lsasrv.dbg
SYMSRV: No ID tokens for lsasrv.dbg
DBGHELP: No base address for lsasrv.dll:  Please specify
SymLoadModule(F0F0F0F0, 00000000, "lsasrv.dll", "lsasrv", 00000000, 0) fails

A. Notice that SYMSRV has no ID tokens for lsasrv.dbg.   This means that the debugger has no way to accurately query the symbol server for the file.  The debugger is probably reporting this because the header for lsasrv.dll is not available (i.e., the dll is either not loaded, or the header is paged out) and the debugger requires this in order to know how to find the symbols.  DBGHELP then tries to go find lsasrv.dll and can't find it in the path.  This is the logical next step for the debugger because it can't make progress on loading symbols unless it has the header for the dll.

Q: I’ve successfully loaded symbols, but the stack seems to be wrong. Is the debugger broken?

A: Not necessarily. The most likely cause of your problem is that you’ve got incorrect symbols. Go through the steps outlined above to determine whether you’ve loaded valid symbols or not. Do not assume that because some things work that you have valid symbols.  Just because, for example, you can successfully dd nt!ntbuildnumber or you can u nt!KeInitializeProcess that your symbols are correct. Verify that they are correct using the procedures outlined above.

Q: Will the debugger still work with incorrect symbols?

A: Yes and no. Often times you can get away with symbols that don’t strictly match. For example, symbols from a previous day’s daily build will often work – depending on how much stuff has changed. If you have problems using non-matching symbols, though, you’re completely on your own. If the symbols do not match there is no guarantee whatsoever that you will or will not get correct results.

Q: I’m stopped in the kernel debugger and I want to view symbols for my user-mode process. Can I do it?

A: Mostly. The support for this scenario is poor because the kernel debugger doesn’t keep enough information around to track the module loads for each process, but there’s a reasonable workaround. To load symbols for a user-mode module do, .reload user-mode-module-name. For example, when debugging SMSS.EXE using the kernel debugger, it’s useful to be able to look up symbols from the user-mode process, SMSS.EXE. To be able to do this, I do: .reload smss.exe a then I have symbols.

Q: What does *** WARNING: symbols checksum and timestamp is wrong 0x0036d6bf 0x0036ab55 for ntkrnlmp.exe mean?

A: It means your symbols for ntkrnlmp.exe are wrong.

3. Reference

3.1 Useful KD Commands

lm – List modules and show information about debug information for a module.

!dh image-header-base – Dump header information for a loaded image beginning at image-header-base.

!lmi image-header-base – Dump important symbol information for a loaded image. It reads the image header at image-header-base.

!lmi image-name – Same as above, but uses the name of the loaded module.

.reload [image-name] – Reload symbols for the image image-name. If no image-name is specified, reload symbols for all images. It is necessary to reload symbols after sympath has been changed for changes to take affect.

!sym [noisy | quiet] – Specifying noisy turns on verbose mode for symbol loads. This is required to find anything useful out about the module loads.

.sympath [new-symbol-path] – If a new-symbol-path is specified, then set the symbol path to new-symbol-path. If ­new-symbol-path is null, return the current symbol path.

3.2 Useful Programs

link /dump /headers <image> - Dump the header of a PE image (an EXE, DLL or DBG file). Dumpbin is an alias for link /dump.

cvdump - Dump codeview information for a PDB file. Not generally useful, but will give you access to all of the codeview streams in their full glory. Using cvdump, you can check what type of debug information you image actually has.

pdbdump - Dumps pdb information for a PBD file. Though the interface for this program is horrible -- the program is useful for finding out the signature of a PDB file (pdbdump pdbfile hdr). For dumping out specific sections of the PDB data, use cvdump instead.

symchk <filename> /s <sympath> /v – Check if the specified image matches the DBG and the PDB on the current machine.