WinDbg basics HOWTO for QA engineers

Index

• Abstract
• Setup
  • Common settings, Target machine
  • Target machine settings for remote debugging (dual machine)
  • Host machine
• Debugging - how to get logs and (if necessary) crashdump
  • Single machine (local) debugging
  • Remote debugging (two machines)

Abstract

There is such debugger - WinDbg. It is developed by Microsoft and can be downloaded from this location:
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx.

In general there are 2 possible ways of debugging: with single machine and with 2 machines. When we have single machine we can store debug messages (logs) to file and if crash happens analyze automatically created memory dump (crashdump). After reboot, of course. When we use 2 machines, one of them (host) monitors another (target). The plus of two-machines way is that memory dump and latest logs can always be obtained against single machine method.

Setup

Common settings, Target machine

1. install latest version of DbgPrintLog with following command line:

   DbgPrintLog.exe --drv:inst B --drv:opt DoNotPassMessagesDown 1 --drv:opt BufferSize 16384
   

   This will allow to extract debug messages preceding system crash from memory dump and significantly reduce traffic over communication channel if dual machine debugging is used.
2. Setup crashdump:
   • Make sure that pagefile (virtual memory) is set to be greater than physical memory and is located on the same volume where running copy Windows is installed.
   • Make sure if there is enough disk space on the disk on the volume where running copy of Windows is installed. Memory dump includes snapshot of entire physical memory and some additional information.
   • My Computer -> Properties -> Advanced -> Startup and recovery
   • Enable memory dump
   • set dump type to Kernel Dump or Full Memory Dump.
     Attention: do not set Minidump/Small dump option, since it is absolutely useless for saving logs.
3. Setup verifier. It is Windows built-in driver validation tool, helps to detect problems early.
   • run verifier.exe. Under WinXP it is built-in, under 2000 it comes with DDK.
   • chose "Custom settings" - second option. Next
   • chose "Select from list" - bottom radio-button. Next
   • check everything, probably except "Low Resources Simulation". Next
   • chose "Select Driver from list" - last option. Next
   • check required driver(s). If target driver is not loaded at this moment, use "Add not loaded drivers to list" button under list. Drivers are usually located in WINDOWS/System32/drivers/ directory.
   • OK, do not re boot now
4. install WinDbg
5. put dbgprn.dll from latest version of DbgPrintLog to 'dbgext' subdirectory of WinDbg install directory. (e.g. C:/Program Files/WinDbg/dbgext)
6. reboot machine to make sure all settings are saved to disk before installing application and/or driver to be debugged.

When single machine debugging is used, no more preparation is required. You can continue reading from Debugging section.

Target machine settings for remote debugging (dual machine)

1. connect machines with IEEE1394 (FireWire) or NULL-modem cable.
   Note: IEEE1394 is supported since Windows XP only. Windows 95/98/ME, Windows NT4 and Windows 2000 cannot be used for remote debugging over IEEE1394.
2. do not forget to make common settings (see above)
3. open BOOT.INI file in the root directory of your boot volume.
4. find there record corresponding to your OS, smth. like this:

      multi(0)disk(0)rdisk(0)partition(1)/WINDOWS="Microsoft Windows XP Professional"
   

5. make a copy of this record in the next line
6. add to he copy the following:
   • if IEEE-1394 cable is used:

            /debug /debugport=1394 /CHANNEL=3
         

   • if NULL-modem cable connected to COM2 is used:

            /debug /debugport=2 /baudrate=115200
         

   final string shall look like this (without line break):

   multi(0)disk(0)rdisk(0)partition(1)/WINDOWS="Windows XP Pro" /noexecute=optin /fastdetect /debug
       /debugport=1394 /CHANNEL=3
   

7. make sure that TIMEOUT value in BOOT.INI has non-zero value
8. now BOOT.INI shall looks like this:

   [Boot Loader]
   Timeout=5
   Default=multi(0)disk(0)rdisk(0)partition(1)/WINDOWS
   [Operating Systems]
   multi(0)disk(0)rdisk(0)partition(1)/WINDOWS="Windows XP Pro" 
   multi(0)disk(0)rdisk(0)partition(1)/WINDOWS="Windows XP Pro" /noexecute=optin /fastdetect /debug
       /debugport=1394 /CHANNEL=3
   

9. disable FireWire (IEEE1394) card in Device Manager:
   My Computer -> Manage -> Device Manager

Host machine

1. do not forget to connect machines with cable!
2. install WinDbg
3. put dbgprn.dll from latest version of DbgPrintLog to 'dbgext' subdirectory of WinDbg install directory. (e.g. C:/Program Files/WinDbg/dbgext)
4. Run WinDbg
5. follow menu items: File -> Kernel Debug ->
   • 1394, chose channel 3, OK, Answer 'Save' if prompted for workspace saving. If you run WinDbg for the 1st time, error message "Unable to open file" may appear. This means, that system didn't detect remote FireWire port on target machine yet. In this case do the following:
     • keep WinDbg open
     • reboot target machine and chose OS item marked with [Debug] or [Debugger Enabled] (or smth. like this) in boot menu
     • host machine shall display New Hardware Found message and install drivers for Debug Port. You may be prompted for Windows installation disk.
     • in some cases you may need to reboot host machine and repeat procedure once again.
     • if properly connected and configured as described above machines still cannot establish debug connection, I don't know what else can help.
   • COM, just tell to what port of host machine NULL-modem cable is connected and set speed to 115200.

Debugging

Single machine (local) debugging

1. run DbgPrintLog for capturing logs:

   DbgPrintLog.exe --full --cpu -l 10
   

   This will start console, where help message and current log file name are displayed. Do not close it. Logs will be stored in files DbgPrintXXX.log in same directory. XXX in filename is a sequential number of file. The very first file has name DbgPrint.log. Previously created files are not overwritten and numbering is continued. -l 10 instructs logger to keep only 10 latest log files.
2. save debug info files .PDB from the build you are about to test to some separate location. If problem happens only with Release build, please configure your project to generate debug info for your binaries. If debug info is generated, .PDB files shall appear near compiled .EXE, .SYS, .DLL. When tuning Debug Info options, do not chose 'Separate Types' option.
3. mark saved .PDBs somehow. If system crashes, .PDBs generated for the binaries were installed would be required. Ideally, after each build binaries and their .PDBs should be stored together.
4. run dsync.exe from freboot package to synchronize system cache with on-disk data. This improves File System consistency if crash happens. You can download it from the following address:
   http://www2.alter.org.ua/soft/win/fastreboot
5. now install your drivers, services and application.
6. make your tests until something goes wrong:
   • Unexpected behavior
     1. switch to logger console and press 'Esc' to stop logging
     2. pack *.LOG files from the folder where DbgPrintLog.exe was run to archive.
     3. also, it would be nice to put this archive together with binaries and .PDBs used in given test session.
   • System crash, BSOD
     1. memory dump shall be created in the directory, specified during preparation. Usually it is created in WINDOWS directory and is named MEMORY.DMP.
     2. after reboot put copy of MEMORY.DMP together with binaries and .PDBs for possible future use.
        Note: verify creation time of this file. It may happen so, that you see dump of previous crash. In this case this file and further steps are useless.
     3. run WinDbg
     4. open MEMORY.DMP:
        File -> Open Crash Dump
     5. in the command prompt on WinDbg execute the following commands:

        !analyze -v
        !dbgprn.save -a --full --cpu -T R --file <full path to file>
        

        if !dbgprn.save says "Cannot find DbgPrnHk!DbgPrnHk_State export", try the following command:

        !dbgprn.lsig
        !dbgprn.save -a --full --cpu -T R --file <full path to file>
        

     6. Log will be saved to file <full path to file>. Pack obtained log-file to archive.
     7. also, it would be nice to put this archive together with binaries and .PDBs used in given test session.
     8. select entire text from log window of WinDbg and copy-paste it to some file, place this file together with copy of memory dump, archived logs and snapshot of binaries and .PDBs
7. send archive(s) with logs to developer and tell what did you do before problem appeared.

Remote debugging (two machines)

1. do not forget to connect machines with cable!
2. Run WinDbg on host machine
3. follow menu items: File -> Kernel Debug ->
   • 1394, chose channel 3, OK, Answer 'Save' if prompted for workspace saving.
   • COM, just tell to what port of host machine NULL-modem cable is connected.
4. reboot target machine and chose OS item marked with [Debug] or [Debugger Enabled] (or smth. like this) in boot menu.
5. WinDbg on host machine shall display information about astablishing debug connection and print some information about target system.
6. when target system is up, run there DbgPrintLog.exe for capturing logs:

   start DbgPrintLog.exe --full --cpu -l 10
   

   This will display console window, where help message and current log file name are displayed. Do not close it. Logs will be stored in files DbgPrintXXX.log in same directory. XXX in filename is a sequential number of file. The very first file has name DbgPrint.log. Previously created files are not overwritten and numbering is continued. -l 10 option instructs logger to keep only 10 latest log files.
7. save debug info files .PDB from the build you are about to test to some separate location. If problem happens only with Release build, please configure your project to generate debug info for your binaries. If debug info is generated succesfully, .PDB files shall appear near compiled .EXE, .SYS, .DLL. When tuning Debug Info options, do not chose 'Separate Types' option.
8. mark saved .PDBs somehow. If system crashes, .PDBs generated for the binaries were running would be required. Ideally, after each build binaries and their .PDBs should be stored together.
9. run dsync.exe from freboot package to synchonize system cache with on-disk data. This improves File System consistency if crash happens.
10. now install your drivers, services and application.
11. make your tests until something goes wrong:
    • Unexpected behavior
      1. switch to logger console and press 'Esc' to stop logging
      2. pack *.LOG files from the folder where DbgPrintLog.exe was run to archive.
      3. also, it would be nice to put this archive together with binaries and .PDBs used in given test session.
    • System crash, BSOD
      1. message about system stop shall appear in WinDbg window on host machine. Target machine will look frozen at this moment.
      2. in the command prompt on WinDbg execute the following commands:

         .dump /f <DumpFileName>
         !dbgprn.save -a --full --cpu -T R --file <full path to file>
         

         if !dbgprn.save says "Cannot find DbgPrnHk!DbgPrnHk_State export", try to close WinDbg, run it againg, open saved memory dump (with .dump command above) and run the following:

         !dbgprn.lsig
         !dbgprn.save -a --full --cpu -T R --file <full path to file>
         

         Log will be saved to file <full path to file>.
      3. reboot target machine, since it is unusable in curent state or execute in WinDbg console the following command:

         g
         

      4. pack obtained log-file to archive.
      5. also, it would be nice to put this archive together with binaries and .PDBs used in given test session.
      6. select entire text from log window of WinDbg and copy-paste it to some file, place this file together with copy of memory dump, archived logs and snapshot of binaries and .PDBs
      7. also put copy of memory dump together with binaries and .PDBs for possible future use.
12. send archive(s) with logs to developer and tell what did you do before problem appeared.