http://www.opferman.net/Text/ntsd.txt
Basic debugging & tips using NTSD By Toby Opferman (AKA Scrat/_Secret) http://www.opferman.com http;//www.opferman.net toby@opferman.com NOTE: This tutorial is without warrenty. Use at your own risk. I am not in any way responsible for any damages caused by use of this tutorial. NTSD is a debugger for WinNT & Win2K that can be used to debug application bugs and traps. To set NTSD to be launched when an application traps, you need to set the following registry key: HKEY_LOCAL_MACHINE/Software/Microsoft/WindowsNT/CurrentVersion/AeDebugger Set the value "Debugger" to "ntsd -p %ld -e %ld -g" or just "ntsd -p %ld -e %ld" or just "ntsd -p %ld" You can do ntsd /? and check out the options you like. The -g does is when it traps to the debugger, it does "auto go". Set the value "Auto" to 1 This is if you want to change to NTSD. If you have devstudio installed, this value may be set to devstudio, or it could be set to Dr. Watson. If it is set to Dr. Watson, I would suggest setting it to NTSD instead, since Dr. Watson just gives a stack trace and a small assembly listing. When an application traps, if the trap is not handled, NTSD should catch and break into the process where it trapped. To start using NTSD on an application, when it is set as your default debugger you can right click on the process in task manager and click "Debug Process". You can also get the PID of the process in Task Manager or by running qprocess and doing NTSD -P <PID>. Some of the basic commands that you would want to know are: kb - Stack Dump dd - Dump Dwords da - Dump ANSI String du - Dump UNICODE String ~* - List Threads ~<number><command> - Perform <command> on Thread <number> (Such as ~1kb Stack Dump Thread 1) u - Unassemble !reload - Reload Symbols g - Go p - Step 1 instruction, but do not trace into function calls t - Trace into, and step 1 instruction bp - Set Breakpoint bc - Clear Breakpoint Before we start on NTSD, let us look at symbols. Now, when using a debug build, symbols along with other debug information are embedded into the executable and the assembly is not optimized. This makes it easier to debug. However, when you want to release you would like to build a retail build. Synchronization, timing and memory may show up more in a retail build and you would have a hard time debugging them. What we want to be able to do, is debug our applications without recompiling and throwing in 100 print statements. On that note, if this bug is not that reproducable or the steps to reproduce it are long and not very easy, we would like to be able to find out what the problem is from it's crashed state. Let's take a look at something. I have a small source file that I wrote when I was working on an OS to remove the binary headers from executables. Compiling with optimizations and linking this source like so: cl /Ox /c rmhdr.c link /SUBSYSTEM:CONSOLE /OUT:rmhdr.exe rmhdr.obj Produces: RMHDR EXE 32,768 Now, there are no symbols or memory mappings of what the binary contains. So, if we compile with debug information, all we really need are the symbols. cl /Ox /Zi /c rmhdr.c link /SUBSYSTEM:CONSOLE /OUT:rmhdr.exe rmhdr.obj -pdb:rmhdr.pdb -debugtype:both -debug Produces: RMHDR EXE 53,506 53k Executable. Now, this file can be used to debug directly and you would have symbols for the function calls which make it easier to debug. However, Would you want to release a file with debug information? On the other hand, if you release a file without debug information and it crashes on someone's machine and you want to be able to debug this machine easily, you would want the debug information. So, what should we do? rebase -b 0x00400000 -x ./ -a rmhdr.exe When we run the rebase utility, we now have 2 files. RMHDR DBG 16,876 12-13-00 8:53p rmhdr.dbg RMHDR EXE 37,136 12-13-00 8:53p rmhdr.exe 37k is a little more than 32k, but you're already programming in C and not assembly (Or C++ if that's the case) so you're not too worried about size that much, at least not to the point to complain about this or else you'd be writing in assembly. Now, you say you have this DBG file, but what can you do with it? There's a few things you can do with it, let me provide an example. I purposely put a bug into this C program and will run it on Windows 98. And now you get the "This program has performed an illegal operation and will be shut down" "If this problem persists, contact the program vendor" It does ask you "Close", "Debug" or "Details" Well, you can debug, (The best debugger for Win98 would be like softice or even VC++), but I think that we can get enough information just by hitting "Details". This is what we get when we hit details: RMHDR caused an invalid page fault in module RMHDR.EXE at 0177:0040124b. Registers: EAX=3f3e3d3c CS=0177 EIP=0040124b EFLGS=00010246 EBX=00530000 SS=017f ESP=0063fcd4 EBP=0063fde4 ECX=004083c8 DS=017f ESI=81756014 FS=23f7 EDX=00000000 ES=017f EDI=00000000 GS=0000 Bytes at CS:EIP: c7 00 0a 00 00 00 8b 8d f8 fe ff ff 83 c1 04 89 Stack dump: 004083c8 004083e8 3f3e3d3c 00000000 47464544 4b4a4948 0063fd20 816e8050 816edcdc cddb6b80 0063fd20 bff7a0fe bff7b317 816e8000 00000000 00000014 Notice EIP = 0040124b, just using that number, we can narrow it down. There is a utility that comes with VC++ called dumpbin, which is very useful. dumpbin /symbols rmhdr.dbg > t.txt We now have a mapping of all the symbols and their addresses. We have to map 0040124b to the image. We usually will just take 124b part, or will have to adjust the number to match the symbols (i.e., it could really be 10124b in the map, depends on where the EXE was loaded into memory). One way is to go into t.txt now, and look at the very last entry. 235 00009F04 SECT3 notype External | __acmdln 236 0000A000 DEBUG notype External | end 0040124b Are the very last entries. Comparing the values, you can see that 0040124b is not an absolute 1-1 mapping to the symbols. Notice also that this is simple since we did not have to look at the stack and we know that EIP is still inside RMHDR.EXE (I made sure of that when I put the bug in). There are times when you may have to map the entire stack to find out the calling path. Well, let's try 124b. Now remeber, we can't search on this number. This number will not exist in the mapping. We need to find 2 functions where this number would lie between. 0F7 0000114F SECT1 notype () External | _RemoveHdr 0F8 00001295 SECT1 notype () External | _atol And here we found it. 124B is between _RemoveHdr and _atol in the source. This means, that the function that crashed is _RemoveHdr. Now, if we would have dropped into a debugger, we could look at the stack and even find out who called RemoveHdr, what the values on the parameters are, and etc. However, we didn't. I also ran this in Win98. We could generate a MAP file or COD file that would show the Asm with C with addresses and we'd be able now to go directly to RemoveHdr and find the spot where it crashed. (And I bet you never knew how useful those little GPF messages were!) However, this tutorial isn't just about debugging. It's about debugging with NTSD. So, let's take it to Win2k with NTSD set to trap as our default debugger. (Remeber, we set our default debugger as NTSD eariler). Now, I run the program and it traps into NTSD. Microsoft(R) Windows 2000 Debugger Version 5.00.2184.1 Copyright (C) Microsoft Corp. 1981-1999 CommandLine: *** wait for debug event Symbol search path is: C:/WINDOWS NTSD ModLoad: 00400000 0040a000 rmhdr.exe NTSD ModLoad: 77f80000 77ff9000 ntdll.dll NTSD ModLoad: 77e80000 77f36000 KERNEL32.dll NTSD: access violation NTSD: !!! second chance !!! eax=00000018 ebx=7ffdf000 ecx=004083c8 edx=00000000 esi=77e84ed2 edi=00008000 eip=0040124b esp=0012fe5c ebp=0012ff6c iopl=0 nv up ei pl zr na po nc cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246 *** ERROR: Module load completed but symbols could not be loaded for rmhdr.exe rmhdr+124b: 0040124b c7000a000000 mov dword ptr [eax],0xa ds:0023:00000018=???????? 0:000> It traps into rmhdr. Look at the line that says "symbols could not be loaded" We need symbols! 0:000> u rmhdr+124b: 0040124b c7000a000000 mov dword ptr [eax],0xa 00401251 8b8df8feffff mov ecx,[ebp-0x108] 00401257 83c104 add ecx,0x4 0040125a 898df8feffff mov [ebp-0x108],ecx 00401260 83bdfcfeffff00 cmp dword ptr [ebp-0x104],0x0 00401267 7591 jnz rmhdr+0x11fa (004011fa) 00401269 8b95f0feffff mov edx,[ebp-0x110] 0040126f 52 push edx 0:000> kb ChildEBP RetAddr Args to Child 0012ff6c 0040107c 00300eca 00300ed6 0007a120 rmhdr+0x124b 0012ff80 004018de 00000004 00300eb0 00300e00 rmhdr+0x107c 0012ffc0 77e87903 00008000 77e84ed2 7ffdf000 rmhdr+0x18de 0012fff0 00000000 0040182a 00000000 000000c8 KERNEL32!SetUnhandledExceptionFilte r+0x5c 0:000> Nice, we have no idea how this function was called. First, we have a small buffer, so we want to go to the top left corner of the window, go to "Properties -> Layout Tab" then go to Buffer Height. It is probably 300 on Win2k and 80 on NT. Set it to 9999. Now, we have room to work. Looking at that, we could be anywhere. Symbols are usually put into C:/WINNT/Symbols/EXE & C:/WINNT/Symbols/Dll So, we copy the rmhdr.dbg file into C:/WINNT/SYMBOLS/EXE if you have the symbols environment variable set. Or, to be simpler, you may copy it to your environment path and NTSD will pick it up. 0:000> !reload 0:000> kb ChildEBP RetAddr Args to Child 0012ff6c 0040107c 00300eca 00300ed6 0007a120 rmhdr!RemoveHdr+0xfc 0012ff80 004018de 00000004 00300eb0 00300e00 rmhdr!main+0x7c 0012ffc0 77e87903 00008000 77e84ed2 7ffdf000 rmhdr!mainCRTStartup+0xb4 0012fff0 00000000 0040182a 00000000 000000c8 KERNEL32!SetUnhandledExceptionFi r+0x5c 0:000> r eax=00000018 ebx=7ffdf000 ecx=004083c8 edx=00000000 esi=77e84ed2 edi=00008000 eip=0040124b esp=0012fe5c ebp=0012ff6c iopl=0 nv up ei pl zr na po nc cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246 rmhdr!RemoveHdr+fc: 0040124b c7000a000000 mov dword ptr [eax],0xa ds:0023:00000018=???????? 0:000> u rmhdr!RemoveHdr+fc: 0040124b c7000a000000 mov dword ptr [eax],0xa 00401251 8b8df8feffff mov ecx,[ebp-0x108] 00401257 83c104 add ecx,0x4 0040125a 898df8feffff mov [ebp-0x108],ecx 00401260 83bdfcfeffff00 cmp dword ptr [ebp-0x104],0x0 00401267 7591 jnz rmhdr!RemoveHdr+0xab (004011fa) 00401269 8b95f0feffff mov edx,[ebp-0x110] 0040126f 52 push edx 0:000> We now have symbols! We see that we are moving 10 into EAX, which is set to 18h. Well, let me show you some features of NTSD before we figure out what the problem is. If you look at the first "Return Address" on the stack, we can dissassemble it and look at the calling code. 0:000> u 40107c rmhdr!main+7c: 0040107c 83c40c add esp,0xc 0040107f 33c0 xor eax,eax 00401081 5d pop ebp 00401082 c3 ret rmhdr!DispatchMsg: 00401083 55 push ebp 00401084 8bec mov ebp,esp 00401086 51 push ecx 00401087 8b4508 mov eax,[ebp+0x8] 0:000> Doing this requires at least some assembly knowledge however. 0:000> ~* 0 id: 100.228 Suspend: 0 Teb 7ffde000 Unfrozen 1 id: 100.310 Suspend: 0 Teb 7ffdd000 Unfrozen 0:000> We see there are two threads. 0:000> ~1kb ChildEBP RetAddr Args to Child 0045ffb4 77e92ca8 00000000 00000200 0040fcbc ntdll!DbgBreakPoint+0x1 0045ffec 00000000 77eabe5a 00000000 00000000 KERNEL32!CreateFileA+0x11b 0:000> ~0kb ChildEBP RetAddr Args to Child 0012ff6c 0040107c 00300eca 00300ed6 0007a120 rmhdr!RemoveHdr+0xfc 0012ff80 004018de 00000004 00300eb0 00300e00 rmhdr!main+0x7c 0012ffc0 77e87903 00008000 77e84ed2 7ffdf000 rmhdr!mainCRTStartup+0xb4 0012fff0 00000000 0040182a 00000000 000000c8 KERNEL32!SetUnhandledExceptionFilte r+0x5c 0:000> However, this program is single threaded. One of the threads could have been created by an API we called (A lot of APIs create seperate threads). However, "DbgBreakPoint" kind of makes it look to be a break point thread created by NTSD to break into the thread. We are only concerned with thread 0. ChildEBP RetAddr Args to Child 0012ff6c 0040107c 00300eca 00300ed6 0007a120 rmhdr!RemoveHdr+0xfc Now, we know that the prototype of RemoveHdr is void RemoveHdr(char *InFileName, char *OutFileName, int Size) Let's take a look at what is stored in the third parameter. We also know we called the function as so: C:/>rmhdr tdconfig.td happy 500000 Stripping 500000 Bytes And 50000 should have been passed to rmhdr as an integer. 0007a120 is what is listed as the third parameter. Being passed by value, this is the value in decimal 500000. So, we know we're passing a correct parameter. Next, we notice the first two parameters are ANSI strings. (Notice that just because it lists 3 parameters on all the functions, this is not nessecarily how many parameters the function has.) 0:000> da 300eca 00300eca "tdconfig.td" 0:000> da 300ed6 00300ed6 "happy" 0:000> (Notice, for Unicode strings, type "du <address>"). We notice we are passing down the correct parameters. Parameter 1 is the inputname we gave, and parameter 2 is the output name we gave. We can do something else now. We can find the address of RemoveHdr. 0:000> x rmhdr!RemoveHdr 0040114f rmhdr!RemoveHdr 0:000> Wild cards can be used as well, example: 0:000> x rmhdr!* ; would list all the symbols for rmdhr. Well, we now have the address. But, we actually don't need the address. Because, you can referr to the function by its symbol. Let's unassemble. 0:000> u rmhdr!RemoveHdr rmhdr!RemoveHdr: 0040114f 55 push ebp 00401150 8bec mov ebp,esp 00401152 81ec10010000 sub esp,0x110 00401158 681c814000 push 0x40811c 0040115d 8b4508 mov eax,[ebp+0x8] 00401160 50 push eax 00401161 e826030000 call rmhdr!fopen (0040148c) 00401166 83c408 add esp,0x8 0:000> u rmhdr!RemoveHdr+1a: 00401169 8985f0feffff mov [ebp-0x110],eax 0040116f 83bdf0feffff00 cmp dword ptr [ebp-0x110],0x0 00401176 750f jnz rmhdr!RemoveHdr+0x38 (00401187) 00401178 6a01 push 0x1 0040117a e804ffffff call rmhdr!DispatchMsg (00401083) 0040117f 83c404 add esp,0x4 00401182 e90a010000 jmp rmhdr!RemoveHdr+0x142 (00401291) 00401187 6820814000 push 0x408120 0:000> u rmhdr!RemoveHdr+3d: 0040118c 8b4d0c mov ecx,[ebp+0xc] 0040118f 51 push ecx 00401190 e8f7020000 call rmhdr!fopen (0040148c) 00401195 83c408 add esp,0x8 00401198 8985f4feffff mov [ebp-0x10c],eax 0040119e 83bdf4feffff00 cmp dword ptr [ebp-0x10c],0x0 004011a5 751e jnz rmhdr!RemoveHdr+0x76 (004011c5) 004011a7 8b95f0feffff mov edx,[ebp-0x110] 0:000> u rmhdr!RemoveHdr+5e: 004011ad 52 push edx 004011ae e863020000 call rmhdr!fclose (00401416) 004011b3 83c404 add esp,0x4 004011b6 6a02 push 0x2 004011b8 e8c6feffff call rmhdr!DispatchMsg (00401083) 004011bd 83c404 add esp,0x4 004011c0 e9cc000000 jmp rmhdr!RemoveHdr+0x142 (00401291) 004011c5 6a00 push 0x0 0:000> r eip eip=0040124b ..... we keep unassembling until we get to the crashed area. 0:000> u rmhdr!RemoveHdr+ed: 0040123c 52 push edx 0040123d e85d020000 call rmhdr!fwrite (0040149f) 00401242 83c410 add esp,0x10 00401245 8b85f8feffff mov eax,[ebp-0x108] 0040124b c7000a000000 mov dword ptr [eax],0xa 00401251 8b8df8feffff mov ecx,[ebp-0x108] 00401257 83c104 add ecx,0x4 0040125a 898df8feffff mov [ebp-0x108],ecx 0:000> Now, we call fwrite before we move 10 into this variable. Notice EAX's address comes from EBP-0x108, which means it's a local variable. We are assigning 10 to a local variable, infact, a pointer, since we reference this address after we get it from the stack. Right after we call fwrite(). Should be easy to figure out. (Also Note: 0:000> r eax=00000018 ebx=7ffdf000 ecx=004083c8 edx=00000000 esi=77e84ed2 edi=00008000 eip=0040124b esp=0012fe5c ebp=0012ff6c iopl=0 nv up ei pl zr na po nc cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246 rmhdr!RemoveHdr+fc: 0040124b c7000a000000 mov dword ptr [eax],0xa ds:0023:00000018=???????? 0:000> Hitting r will display all registers and take you to the spot where it crashed or asserted again. Using u from here will start dissassembling after the point where it faulted. ) We lucked out. There is only 1 fwrite in the function, so we don't even have to try to see how many calls happened or who called what before when. if(BlockSize) fwrite(Buffer, 1, BlockSize, OutFile); *x = 10; x++; What do we see here! A pointer being assigned the value 10. Now, why would this crash? As we look further we find: int BlockSize, *x; And no other reference to *x in the entire function. x has not bee allocated, it's tring to write randomly to memory. We have solved the problem. (Also, thank god for small functions, please do not write 5000000000 line functions! They are hard to debug!). We found the problem without rerunning the program. Once it trapped into NTSD, we were able to use NTSD to determine where the program had crashed without tring to put print statements in and recompiling (Which can sometimes change a binary in a way to where the bug doesn't seem to show up anymore!). We also got it without rerunning and tring to step through the code. This can save you time, depending on the problem, size of the program and where it crashes, rerunning or recompiling may not be an option. However, this was a very simple case. You may find yourself with traps that could have corrupted the stack, wrong version of a dbg so you get bad symbols! or heap corruption which make debugging all the more nasty. For those who do not know assembly that well, you can generate assembly map files, which shows the C source with the generated assembly below the lines. This may help some who are not so good with assembly to debug traps. Now I'm going to break into a program and show some of the things you can do with NTSD. When you break into a process, or hit "CONTROL + C" when in NTSD while it's in a process, you end up stopping the program. When you break in, however, you will be in a new thread created by NTSD just to break in. These threads won't show, however, if you set a break point in the program or if you trap into the program. This only happens when you force NTSD to break into the program. Now, let's break into CuteFTP. I run CuteFTP and look for the process ID on task manager. It's 740, so I break into it. C:/>ntsd -p 740 Microsoft(R) Windows 2000 Debugger Version 5.00.2184.1 Copyright (C) Microsoft Corp. 1981-1999 CommandLine: *** wait for debug event Symbol search path is: C:/WINDOWS NTSD ModLoad: 00400000 004da000 CUTEFTP.EXE NTSD ModLoad: 77f80000 77ff9000 ntdll.dll NTSD ModLoad: 77570000 775a0000 WINMM.dll NTSD ModLoad: 77e10000 77e75000 USER32.dll NTSD ModLoad: 77e80000 77f36000 KERNEL32.dll NTSD ModLoad: 77f40000 77f7c000 GDI32.dll NTSD ModLoad: 77db0000 77e0a000 ADVAPI32.dll NTSD ModLoad: 77d40000 77daf000 RPCRT4.dll NTSD ModLoad: 75050000 75058000 WSOCK32.dll NTSD ModLoad: 75030000 75044000 WS2_32.dll NTSD ModLoad: 78000000 78046000 MSVCRT.dll NTSD ModLoad: 75020000 75028000 WS2HELP.dll NTSD ModLoad: 76b30000 76b6e000 comdlg32.dll NTSD ModLoad: 77c70000 77cba000 SHLWAPI.dll NTSD ModLoad: 77b50000 77bda000 COMCTL32.dll NTSD ModLoad: 775a0000 777e0000 SHELL32.dll NTSD ModLoad: 77800000 7781d000 WINSPOOL.DRV NTSD ModLoad: 752f0000 7530f000 oledlg.dll NTSD ModLoad: 77a50000 77b45000 ole32.dll eax=00000000 ebx=00000000 ecx=00000101 edx=ffffffff esi=00000000 edi=00000200 eip=77f9f9df esp=00ddffa8 ebp=00ddffb4 iopl=0 nv up ei ng nz na po nc cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000286 ntdll!DbgBreakPoint: 77f9f9df cc int 3 0:002> ~* 0 id: 2e4.210 Suspend: 0 Teb 7ffde000 Unfrozen 1 id: 2e4.2a8 Suspend: 0 Teb 7ffdd000 Unfrozen 2 id: 2e4.228 Suspend: 0 Teb 7ffdc000 Unfrozen 0:002> ~0kb *** ERROR: Module load completed but symbols could not be loaded for cuteftp.exe ChildEBP RetAddr Args to Child 0006fbbc 0044fad5 004939e0 00000000 00000000 USER32!DispatchMessageW+0x24b 004939e0 0000000f 00000000 00000000 00328f9c CUTEFTP+0x4fad5 0:002> ~1kb ChildEBP RetAddr Args to Child 00d8ff74 77d4b407 77d4b7bf 000b7658 00000000 ntdll!ZwReplyWaitReceivePortEx+0xb 00d8ffa8 77d4b771 000b65e8 00d8ffec 77e92ca8 RPCRT4!RpcBindingSetOption+0x18e 00d8ffb4 77e92ca8 000b77a0 00000000 400b7154 RPCRT4!RpcBindingSetOption+0x4f8 00d8ffec 00000000 77d4b759 000b77a0 00000000 KERNEL32!CreateFileA+0x11b 0:002> ~2kb ChildEBP RetAddr Args to Child 00ddffb4 77e92ca8 00000000 00000200 0040fcbc ntdll!DbgBreakPoint 00ddffec 00000000 77eabe5a 00000000 00000000 KERNEL32!CreateFileA+0x11b 0:002> One of the threads is the breakpoint created by NTSD, that's thread #3, the one we are in now. Notice, CuteFTP has no symbols. But, lets see some things we can do with NTSD. First, let's set a break point. 0:002> ~0kb ChildEBP RetAddr Args to Child 0006fbbc 0044fad5 004939e0 00000000 00000000 USER32!DispatchMessageW+0x24b 004939e0 0000000f 00000000 00000000 00328f9c CUTEFTP+0x4fad5 0:002> ~0r eax=0000000b ebx=77e1a57c ecx=0006f824 edx=00000000 esi=004939e0 edi=004939e0 eip=77e1414f esp=0006fb9c ebp=0006fbbc iopl=0 nv up ei pl zr na po nc cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246 USER32!DispatchMessageW+24b: 77e1414f c21000 ret 0x10 0:002> bp 44fad5 0:002> g eax=00000001 ebx=77e1a57c ecx=00000007 edx=ffffffff esi=004939b0 edi=004939e0 eip=0044fad5 esp=0006fbd4 ebp=004939e0 iopl=0 nv up ei pl zr na po nc cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246 CUTEFTP+4fad5: 0044fad5 85c0 test eax,eax 0:000> We set a breakpoint on the return address of thread 0. We hit 'g' which tells the process to continue. However, it will stop if it hits an NTSD set break point or a hard coded int 3 breakpoint or if the application traps. However, this time, it hit our NTSD defined break point. This is obviously the message loop (DispatchMessageW, Gives that away) We know the parameter to DispatchMessage is an address to an MSG structure. We can dump this. 0:000> dd 4939e0 004939e0 000101a2 00000219 00000007 00000000 004939f0 0032cd1b 0000010b 00000198 00000000 00493a00 00000000 00464eb7 008b0a40 000001cb 00493a10 000001e9 00000200 00400000 00000000 00493a20 00082788 ffffffff 008b09e0 00000000 00493a30 008b1e20 00000000 008b09d0 008b0ae0 00493a40 008b0ab0 00000000 00000000 00000000 00493a50 00000000 00000000 00000000 00000000 0:000> ~* 0 id: 2e4.210 Suspend: 0 Teb 7ffde000 Unfrozen 1 id: 2e4.2a8 Suspend: 0 Teb 7ffdd000 Unfrozen 0:000> Notice there are only 2 threads now. These are both CuteFTP threads, not NTSD breakpoint threads. Now, I am going to teach you a trick. Every thread has an Information Block that stores information about itself. You can look it up on MSDN, "TIB". It's pretty much the same structure for all Win32 platforms, as application programs sometimes hard code using this data. To get the locations of what most of the structure means, search MSDN. I will show you one part of the TIB. The TIB is pointed to by FS. Let's say you have this situtation. You break into a program you have, you set a breakpoint on a particular API you know is failing. Then you set a break point on it's return. Now, you see it failed, but you would need to call GetLastError() to get the error number! You would need to recompile this program and put in a print message just to find one number! What to do? 0:000> x KERNEL32!GetLastError 77e8668c KERNEL32!GetLastError 0:000> u KERNEL32!GetLastError KERNEL32!GetLastError: 77e8668c 64a118000000 mov eax,fs:[00000018] 77e86692 8b4034 mov eax,[eax+0x34] 77e86695 c3 ret 77e86696 85ff test edi,edi 77e86698 0f84053c0400 je KERNEL32!VerLanguageNameA+0xef (77eca2a3) 77e8669e 8d8d58ffffff lea ecx,[ebp-0xa8] 77e866a4 6a00 push 0x0 77e866a6 51 push ecx 0:000> Well, what do you know? GetLastError() is pretty simple. Let's find out what the Last Error was! 0:000> dd fs:18 0038:00000018 7ffde000 00000000 000002e4 00000210 0038:00000028 00000000 00000000 7ffdf000 00000000 0038:00000038 00000000 00000000 e1fe9228 00000000 0038:00000048 00000000 00000000 00000000 00000000 0038:00000058 00000000 00000000 00000000 00000000 0038:00000068 00000000 00000000 00000000 00000000 0038:00000078 00000000 00000000 00000000 00000000 0038:00000088 00000000 00000000 00000000 00000000 0:000> dd 7ffde000+34 7ffde034 00000000 00000000 00000000 e1fe9228 7ffde044 00000000 00000000 00000000 00000000 7ffde054 00000000 00000000 00000000 00000000 7ffde064 00000000 00000000 00000000 00000000 7ffde074 00000000 00000000 00000000 00000000 7ffde084 00000000 00000000 00000000 00000000 7ffde094 00000000 00000000 00000000 00000000 7ffde0a4 00000000 00000000 00000000 00000000 0:000> As you notice, the first DWORD listed is 0. Now, when I did this, there was no error. But, you can see how this can be useful. Remeber you can always clear and set breakpoints. Use "T" and "P" to step through and execute the code. That is all for now. Hopefully, you will practice real debugging and improve your debugging skills after reading this basic debuggin tutorial.