【转帖】关于WinDbg的一些文章

电玩女神 2022-08-23 15:56 747阅读 0赞

[A word for WinDbg][]：不错的入门文章，介绍了WinDbg的安装、Symbols、常用命令和一个调试示例。  
[A word for WinDbg (2)][A word for WinDbg _2]  
[More on debugging with SOS.DLL - enter Visual Studio][]  
[Troubleshooting ASP.NET using WinDbg and the SOS extension][]

、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、

### A word for WinDbg ###

Before I came to work at Microsoft I worked as a professional developer on the Microsoft platform and I used to work with the Visual Studio (6 latterly) debugger which I thought was a pretty cool debugger.

When I joined Microsoft I found that a lot of people worked with WinDbg as their primary debugger and, for a while, I just didn't get it as it seemed like an unnecessarily grungy tool for achieving things that Visual Studio could already do.

However, over a period of time I came to see the light and I'm now of the view that WinDbg is about the best debugger I've ever used (including those ones I used to work with on Unix platforms). WinDbg is just the most powerful debugger out there for user-mode debugging and I don’t do any other kind of debugging J

So, there is a lot of information out there already on WinDbg but I just wanted to try and highlight it here and provide a single-pager on it as it seems to be grossly overlooked as a debugger for managed code and, yet, it’s capable of a whole tonne of stuff that no single other debugger can do for you right now.

People are often put off WinDbg because of its interface and the vast array of commands so in this little series of postings I’ll try and demystify a little bit and show how you can debug a few common scenarios in managed code with it.

In this post, let’s just get to the point where we can use WinDbg to debug a regular process and do some work with it then in the next post(s) I’ll look at more stuff that you can do.

# Installation #

So, first things first, get hold of a copy of WinDbg from [here][] and install it. If you want to follow along with this post then you can go and download it right now - the install takes about 2 minutes once you've got the 10MB download down from the site.

The first thing you'll realise is how lightweight an installation WinDbg is and that (sometimes) means that you can install it into places that you'd never install a copy of Visual Studio or the Visual Studio remote debug set up. Indeed, often if you raise a support call with Microsoft you'll get asked to install WinDbg and it's automated helper (the "autodumper") in order to get a crash dump of your application to send back to Microsoft.

Before we get into debugging something I need to have a quick word or two about Symbols and Commands.

# Symbols #

The first issue that we need to address in order to progress is that of **symbols**. If you want to get decent stack traces and dumps of variable values and so on out of the debugger then you need **symbols** for the modules that you’re debugging whether those modules are yours or someone else’s and whether those modules are managed or unmanaged (although there’s quite a lot more that you can do with managed code without symbols).

Symbols are typically either **private** symbols (include variable information), **retail** symbols (function information but not variable information) or **export** symbols (generally not so useful for debugging purposes). If you’re debugging with “export” symbols it’s not usually enough to actually work out what’s going on.

For your own code you’re responsible for building the symbol files (usually .PDB program databases) by setting the right flags on the compiler or choosing the right configuration in a project (this is true for VC++, VB6 and all the .NET languages). This will give you full private symbols.

When the debugger (and lots of other tools) want to find symbols they typically check the contents of an environment variable named **\_NT\_SYMBOL\_PATH** which is used (just like **PATH** – i.e. a list of folder separated by semi-colons) to find symbol files. You can set **\_NT\_SYMBOL\_PATH** prior to running WinDbg or you can use the debugger’s own symbol path settings to find symbol files. These settings are controlled via the File->Symbol File Path menu option or using the **.sympath** command.

For other people’s code you need to get symbols from them. For Microsoft code you’re in luck because the symbols are available from a public symbol server on the internet. In order to make use of the symbol server you should set your symbol path to be something like;

**SRV\*C:/MyLocalSymbols\*http://msdl.microsoft.com/download/symbols**

What this says is that the debugger should first check a local cache named **C:/MyLocalSymbols** and if symbols are not found there then go out to the Microsoft symbol server for symbols and, if found, download them and cache them in that folder. Note that symbols are downloaded based upon name, version and checksum information from the modules being debugged so the debugger usually knows if the symbols match the code properly.

If you wanted to combine this with a local location for your own code then you’d use something like;

**C:/MyCodesSymbols; SRV\*C:/MyLocalSymbols\*http://msdl.microsoft.com/download/symbols**

so that the debugger would check your folder first and would then proceed to look in the cache and finally to the symbol server for symbols.

# Commands #

A core set of commands would be as below. I rarely venture beyond these as I’m not an advanced user of WinDbg by any means and this gets me by.

**.hh** \- bring up the help file :-) The help file is fantastic so don't skip on it - everything's documented in there.

**g** \- "GO!". That is, continue running.

**bp** (Address) - set a break point. Note that (address) can take many formats here including just a memory location but the most common format specifier here is to use syntax such as;

bp KERNEL32!CreateFileA

(remember that most Windows functions have an ASCII and a Unicode variant so we have KERNEL32!CreateFileA and KERNEL32!CreateFileW).

**bl** \- list all breakpoints. Each breakpoint listed has a number in the list which you need for...

**bc** (number), **be** (number), **bd** (number) - these respectively clear, enable and disable breakpoints from the list.

**kb**, **kp**, **kd** etc. The commands beginning with "k" show the stack for the thread that the debugger is looking at. There are a few variants so check out the help for those. The simplest variant that I use is;

kb 200

(200 is the maximum depth of stack trace that I'm looking for)

**dw**, **db**, **ds**, etc. The commands beginning with "d" show the contents of memory. One of the most common is "dt" which shows you the contents of memory laid out according to the specification of a type as long as the debugger can see the definition of a type.

**sxe**, **sxd**, **sxi**, **sxn**. The commands beginning with "sx" set the behaviour for what the debugger should do when an exception occurs. A good example here would be;

sxe 0xc0000005

which is saying "I want to break into the debugger if there's an access violation".

**lm** \- lists the modules loaded by the program and what kind of symbols are loaded for the modules. Note that export symbols are not really symbols at all but are really the debugger guessing which function you're in based upon the export table of the DLL. For COM servers in particular (which have only 4 exported functions or so) you'll often find yourself appearing in DllRegisterServer if you only have export symbols. Good examples here would be;

lm v (verbose mode)

lm v mUSER\* (verbose mode, matc any modules that begin with USER\*)

**x** \- Examine symbols. This is very, very useful as it shows you the set of symbols loaded for particular modules. A good example here would be;

x USER32!\* (show me all the symbols loaded from the USER32 module)

x USER32!Create\* (show me all the symbols loaded from the USER32 module that begin with "Create")

**.cls** – clear the screen. You’ll be needing this one!

**.reload** – this causes the symbol information to be reloaded for a particular module. The most common form of this would be to do something like;

.reload /f user32.dll

Where the /f overrides the debugger’s naturally lazy mode of working whereby it wouldn’t actually do the reload right there and then. Normally, this is a useful parameter to use.

**~** \- this is the tilde character and it lists all the threads in the process and shows their status. This can be combined with wildcards in order to execute particular commands on all threads in the process. The single most common one is probably to combine it with a “stack” command to do something like;

~\* kb 200

which will show you the stack frames for all the threads in the process.

**~N s** – this changes the debugger’s focus to another thread. For example;

~7 s

Will switch the debugger so that it’s focused on thread 7. Note that 7 here is the debugger’s thread number rather than the real thread ID and it comes from the list given by the ~ command.

Along with the core set of commands, WinDbg is capable of loading up **debugger extensions** to extend the core functionality of the debugger. These are “bang commands” in that they begin with an exclamation mark. You can manipulate the extensions that the debugger has loaded using the following commands;

**.chain** – this shows the extensions that the debugger has loaded

**.load** – this loads an extension DLL (e.g. **.load SOS.DLL)**

.**unload** – unloads an extension DLL

**.unloadall**

**.setdll** – this sets the “default” extension DLL.

A quick word about **.setdll** and how command processing works here. If you have an extension named **UEXT.DLL** which contains a command named **help** (and most extensions will have a **help** command) then you can run that command using;

**!UEXT.help**

And that works fine. If you find yourself using UEXT more than any other extension then you can use;

**.setdll UEXT.DLL**

To make UEXT.DLL the default extension and then you can just use;

**!help**

And that will now default to mean **!UEXT.help** because of the default extension setting.

# A First Debugging Session #

So, let’s finish up this post with a quick debugging session and next time around I’ll talk about debugging managed code rather than just any old process.

1.  Run up WinDbg and also run up a copy of NotePad.
2.  Within WinDbg attach to the NotePad process that you just ran by either using the File->Attach to Process menu or the F6 shortcut. This will give you a dialog where you can select a process. Note that the “Non Invasive” option would allow you to detach from the debugged process without killing it (if you’re on Windows XP or 2003) but it limits the commands that you can use so leave it unchecked.
3.  Don’t worry about the workspace information, let it go.
4.  The debugger should present some diagnostics as it loads information and will then stop with the command prompt. The debuggee (NotePad) is now halted.
5.  Let’s take a look around. Firstly, let’s look at the modules that are loaded by notepad. Execute a “**lm v”** to list all the modules and follow it up with a simple **“lm”**. On my system I get this;

0:001> lm

start end module name

01000000 01014000 notepad (no symbols)

4ce00000 4cf02000 COMCTL32 (export symbols) C:/WINDOWS/WinSxS/x86\_Microsoft.Windows.Common-Controls\_6595b64144ccf1df\_6.0.2600.2149\_x-ww\_a84b1f06/COMCTL32.dll

5ad70000 5ada7000 uxtheme (export symbols) C:/WINDOWS/system32/uxtheme.dll

605d0000 605d9000 mslbui (export symbols) C:/WINDOWS/system32/mslbui.dll

61220000 61232000 MSH\_ZWF (export symbols) C:/Program Files/Microsoft Hardware/Mouse/MSH\_ZWF.dll

73000000 73026000 WINSPOOL (export symbols) C:/WINDOWS/system32/WINSPOOL.DRV

74720000 7476b000 MSCTF (export symbols) C:/WINDOWS/system32/MSCTF.dll

763b0000 763f9000 comdlg32 (export symbols) C:/WINDOWS/system32/comdlg32.dll

77120000 771ac000 OLEAUT32 (export symbols) C:/WINDOWS/system32/OLEAUT32.DLL

771b0000 779c4000 SHELL32 (export symbols) C:/WINDOWS/system32/SHELL32.dll

779d0000 77a46000 SHLWAPI (export symbols) C:/WINDOWS/system32/SHLWAPI.dll

77a50000 77b8d000 ole32 (export symbols) C:/WINDOWS/system32/ole32.dll

77c10000 77c68000 msvcrt (export symbols) C:/WINDOWS/system32/msvcrt.dll

77cc0000 77d5b000 ADVAPI32 (export symbols) C:/WINDOWS/system32/ADVAPI32.dll

77d60000 77df1000 USER32 (export symbols) C:/WINDOWS/system32/USER32.dll

77e00000 77e93000 RPCRT4 (export symbols) C:/WINDOWS/system32/RPCRT4.dll

77ea0000 77ee5000 GDI32 (export symbols) C:/WINDOWS/system32/GDI32.dll

7c800000 7c8f2000 kernel32 (export symbols) C:/WINDOWS/system32/kernel32.dll

7c900000 7c9ae000 ntdll (export symbols) C:/WINDOWS/system32/ntdll.dll

1.  Let’s get some symbols because we can’t really work with export symbols. Set up your symbol path (either through the File->Symbol Path menu or through the **.sympath** command) so look something like this;

**SRV\*e:/LocalSymbolCache\*http://msdl.microsoft.com/download/symbols**

1.  Reload symbols for the loaded modules by using the “**.reload /f”** command – this may well take a little time as the symbols for all those modules that NotePad loaded trickle down from the internet to your local cache. This gets better with time as you pull down common symbols from the net.
2.  After the reload completes do another **lm** and make sure you got some symbols. My session now gives me;

0:001> lm

start end module name

01000000 01014000 notepad (pdb symbols) e:/LocalSymbols/notepad.pdb/15800B8231AF4FDE85232D42B267D3E51/notepad.pdb

4ce00000 4cf02000 COMCTL32 (pdb symbols) e:/LocalSymbols/MicrosoftWindowsCommon-Controls-6.0.2600.2149-comctl32.pdb/4133208CAF9A4B93982A1F873FBC79261/MicrosoftWindowsCommon-Controls-6.0.2600.2149-comctl32.pdb

5ad70000 5ada7000 uxtheme (pdb symbols) e:/LocalSymbols/uxtheme.pdb/3A4D37B97BA34837AC589BDB15CC265D2/uxtheme.pdb

605d0000 605d9000 mslbui (pdb symbols) e:/LocalSymbols/MSLBUI.pdb/ED5CCEFBD83C4F929AAB9A1029C54DDD1/MSLBUI.pdb

61220000 61232000 MSH\_ZWF (export symbols) C:/Program Files/Microsoft Hardware/Mouse/MSH\_ZWF.dll

73000000 73026000 WINSPOOL (pdb symbols) e:/LocalSymbols/winspool.pdb/20EE1501116B4C4180ABE40B209DEC962/winspool.pdb

74720000 7476b000 MSCTF (pdb symbols) e:/LocalSymbols/msctf.pdb/F8F1C6E0E1CC4D1CB4936C12A6A6D4C92/msctf.pdb

763b0000 763f9000 comdlg32 (pdb symbols) e:/LocalSymbols/comdlg32.pdb/2BC57F8BE63A429FB4B439EA2854387A2/comdlg32.pdb

77120000 771ac000 OLEAUT32 (pdb symbols) e:/LocalSymbols/oleaut32.pdb/96D8A9A6CB704201A365F345B2A9F72D2/oleaut32.pdb

771b0000 779c4000 SHELL32 (pdb symbols) e:/LocalSymbols/shell32.pdb/FE0DC899DD78422AB29561FE8255E1C92/shell32.pdb

779d0000 77a46000 SHLWAPI (pdb symbols) e:/LocalSymbols/shlwapi.pdb/231AB6F0A19F4DBB9837440BCC167BEC2/shlwapi.pdb

77a50000 77b8d000 ole32 (pdb symbols) e:/LocalSymbols/ole32.pdb/656B303C367B4E679CA249E32CE572D92/ole32.pdb

77c10000 77c68000 msvcrt (pdb symbols) e:/LocalSymbols/msvcrt.pdb/8E17C9B9A55C4118A7D35025C5BE51871/msvcrt.pdb

77cc0000 77d5b000 ADVAPI32 (pdb symbols) e:/LocalSymbols/advapi32.pdb/57E2A801C610451AA66B7CD94F0910F72/advapi32.pdb

77d60000 77df1000 USER32 (pdb symbols) e:/LocalSymbols/user32.pdb/35948255FAC04CA99F5CF61959B45DC42/user32.pdb

77e00000 77e93000 RPCRT4 (pdb symbols) e:/LocalSymbols/rpcrt4.pdb/79C7047CB77B435DB01EB365C3AF17152/rpcrt4.pdb

77ea0000 77ee5000 GDI32 (pdb symbols) e:/LocalSymbols/gdi32.pdb/E7D5B033EFE5466E97DB048BE4C88FA62/gdi32.pdb

7c800000 7c8f2000 kernel32 (pdb symbols) e:/LocalSymbols/kernel32.pdb/5091FA1742544F2EA3D425642634F78E2/kernel32.pdb

7c900000 7c9ae000 ntdll (pdb symbols) e:/LocalSymbols/ntdll.pdb/7B2A2EF0E63F4848A8D309E84E47322C2/ntdll.pdb

1.  So, that’s modules and symbols. Let’s see if we can set some breakpoints. Suppose that I want to hit a breakpoint anytime that NotePad opens up a file. Let’s do that. Firstly, take a look at symbols in KERNEL32 to see if we can find the CreateFile function that NotePad uses.
2.  Issue a **“x KERNEL32!CreateF\*”** and see what results we get back you should find the ASCII **CreateFileA** and the Unicode **CreateFileW** functions in there.
3.  Set a breakpoint on a function. Issue a “**bp kernel32!CreateFileW**” to set a breakpoint on that function.
4.  Issue a “GO” command **“g”** to continue the debuggee running.
5.  Note that when the debuggee is running you can always get back to the debugger by issuing a **CTRL+BREAK** to the debugger window when it will try and halt the debuggee for you and drop you back to the command prompt.
6.  Open a file with notepad. You’ll spot NotePad loading some more modules in order to do this operation and get that dialog onto the screen. Your breakpoint will more than likely hit before the file dialog comes up.
7.  Check-out where you are with your breakpoint – i.e. what the stack frame looks like. Issue a **“kb 200”** command. You should get results as below;

0:000> kb

ChildEBP RetAddr Args to Child

0006be18 76671b2a 76671b74 00000020 00000003 kernel32!CreateFileW

0006be94 766724e2 000a6b40 0006bf4c 00000000 cscui!IsCSCEnabled+0x38

0006bea8 77a68b49 000a7084 77a51a60 0006bf44 cscui!DllGetClassObject+0x72

0006bec4 77a80f5e 000a7084 77a51a60 0006bf44 ole32!CClassCache::CDllPathEntry::DllGetClassObject+0x2d

0006bedc 77a80e9a 0006bef0 77a51a60 0006bf44 ole32!CClassCache::CDllFnPtrMoniker::BindToObjectNoSwitch+0x1f

0006bf08 77a81cc6 0006bf4c 00000000 0006c540 ole32!CClassCache::GetClassObject+0x38

0006bf84 77a806aa 77b76ca4 00000000 0006c540 ole32!CServerContextActivator::CreateInstance+0x106

0006bfc4 77a81e19 0006c540 00000000 0006ca8c ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7

0006c018 77a81d90 77b76ca8 00000000 0006c540 ole32!CApartmentActivator::CreateInstance+0x110

0006c038 77a8101e 77b76ca8 00000001 00000000 ole32!CProcessActivator::CCICallback+0x6d

0006c058 77a80fd5 77b76ca0 0006c39c 00000000 ole32!CProcessActivator::AttemptActivation+0x2c

0006c090 77a81e7a 77b76ca0 0006c39c 00000000 ole32!CProcessActivator::ActivateByContext+0x42

0006c0b8 77a806aa 77b76ca0 00000000 0006c540 ole32!CProcessActivator::CreateInstance+0x49

0006c0f8 77a81bc4 0006c540 00000000 0006ca8c ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7

0006c348 77a806aa 77b765d4 00000000 0006c540 ole32!CClientContextActivator::CreateInstance+0x8f

0006c388 77a805dc 0006c540 00000000 0006ca8c ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7

0006cb38 77a64eb1 000a2f08 00000000 00000001 ole32!ICoCreateInstanceEx+0x3c9

0006cb60 77a64e80 000a2f08 00000000 00000001 ole32!CComActivator::DoCreateInstance+0x28

0006cb84 77a65102 000a2f08 00000000 00000001 ole32!CoCreateInstanceEx+0x1e

0006cbb4 779d69a5 000a2f08 00000000 00000001 ole32!CoCreateInstance+0x37

1.  So, we can see the direct call-stack that led to our call to **CreateFileW**. If we had full symbols for these modules we could use a **KP** rather than a **KB** command and we’d get a list of parameters to the functions. As it is with what we have here we’d need to resort to disassembly to get the parameter information here and it’s beyond this posting.
2.  Continue the debuggee by issuing a **“g”** command and continue issuing **“g”** commands until NotePad finally puts its dialog on the screen – note how many times we hit this function here J If you’re interested, issue a few KB’s on some of these call chains and see what exactly it is that Notepad is doing.
3.  Break the debuggee by hitting CTRL+BREAK on the WinDbg window.
4.  Issue a “~” command to get a picture of the threads in the process. In my session I get 3 as below;

0:002> ~

0 Id: 9bc.b58 Suspend: 1 Teb: 7ffdf000 Unfrozen

1 Id: 9bc.8a4 Suspend: 1 Teb: 7ffde000 Unfrozen

. 2 Id: 9bc.d4 Suspend: 1 Teb: 7ffdd000 Unfrozen

1.  Note that this display indicates that the debugger is focused on thread “2” which is really OS thread 8A4 in process 9BC (all hex I’m afraid).
2.  Check out where all the threads are by issuing a “~\* kb 200”. You might not get anything too exciting here but it’s illustrative of how to use the technique.
3.  Finally, close off this session by having a look at some extension commands.
    
    1.  List the set of extensions that you’ve got by issuing the “**.chain**” command.
    2.  Have a look at the commands available in these extensions by issuing: **!exts.help, !uext.help, !ntdsexts.help, !ext.help** and see what commands are available to you.
    3.  Try an extension command. For example, **“!exts.cs”** will display all the Critical Sections that the process has and their status as to whether they are locked or not. This can be a useful command in many server-side scenarios.
4.  End the debugging session with a **“q”** command.

That’s it. I’ll post again in the near future around how you can build on what I’ve just walked through here in order to debug managed code with WinDbg and take advantage of features within the debugger that don’t (yet) appear elsewhere.

、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、

### A word for WinDbg (2) ###

Continuing on from the [previous post][] on using WinDbg let’s take what we learnt in that previous post and apply it to managed code.

WinDbg supports the debugging of managed code through an extension named **SOS.DLL**. This is named for esoteric reasons that I’ll not get into here but you can find out where that name came from by Googling the web for it.

This DLL is shipped with the latest debugger so you’ll find a version of it which matches V1.1 of the .NET framework in the debugger’s installation folder and you’ll also find a sub-folder labelled CLR10 which contains the previous version of the DLL.

You interact with the SOS.DLL extension by loading it up into WinDbg as you load any other extension. So, if I want to make use of SOS.DLL I issue;

**.load SOS.DLL**

And the debugger loads the extension for me and I can see it in the results of a;

**.chain**

command and I can set it as my default extension by using;

**.setdll SOS.DLL**

Before we go into SOS.DLL in more depth I’ll add a quick word about the symbols for the .NET framework. If you have the .NET framework SDK installed on your machine then you should have symbols for DLLs such as **mscorwks.dll** and **mscoree.dll** which make up the CLR installed onto your machine. On my machine (with VS.NET 2003 installed) these symbol files exist in a folder named;

**C:/program files/Microsoft Visual Studio.NET 2003/SDK/v1.1/Symbols**

So, this means that you can add this folder to your symbol path (see the previous post) and have WinDbg pick up symbols for the CLR bits for you locally. Alternatively, these symbols will come down from the symbol server anyway but it might save you some download time.

So, let’s consider that straight away what that adds. If I have a piece of managed code such as the following;

using System;

using System.Threading;

namespace ConsoleApplication12

class BadClass

public BadClass()

~BadClass()

Thread.Sleep(Timeout.Infinite);

class EntryPoint

static void Main(string\[\] args)

BadClass c1 = new BadClass();

BadClass c2 = new BadClass();

System.GC.Collect();

System.GC.WaitForPendingFinalizers();

Console.WriteLine("Done");

Console.ReadLine();

Then I’ve created a bit of a monster in that what you should find is that this piece of code probably runs to completion if you run it in **debug** mode but it’ll never complete if you run it in **release** mode. This is because in release mode we made **c1** and **c2** eligible for garbage collection at an earlier point and that allows this code to run the finalizer for **BadClass** and that finalizer blocks the finalizer thread.

How to debug this with WinDbg? Run the process in release mode, attach WinDbg and have a look at the stack traces with a;

**~\* kb 200**

command. I can see straight away from this that one of my threads looks like this;

2 Id: 7b8.a08 Suspend: 1 Teb: 7ffdd000 Unfrozen

ChildEBP RetAddr Args to Child

06bffd50 7c90fb74 7c8023ae 00000001 06bffd84 ntdll!KiFastSystemCallRet

06bffd54 7c8023ae 00000001 06bffd84 00000000 ntdll!NtDelayExecution+0xc

**06bffdac 7929703b ffffffff 00000001 ffffffff KERNEL32!SleepEx+0x61**

06bffdcc 792e42d5 ffffffff 06bffe04 06bffe34 mscorwks!Thread::UserSleep+0x93

06bffddc 0095a492 06bffde8 ffffffff 06bffe3c mscorwks!ThreadNative::Sleep+0x30

WARNING: Frame IP not in any known module. Following frames may be wrong.

06bffe34 791f9118 06bffeb0 7920fc3f ffffffff 0x95a492

06bffec0 791f91dd 04a619a4 00158010 00147880 mscorwks!MethodTable::CallFinalizer+0xee

06bffed4 791e1f08 04a619a4 7c8099f2 00000000 mscorwks!CallFinalizer+0x84

06bfff6c 791bbed0 04a619a4 00000000 0012fb9c mscorwks!CallFinalizer+0x245

**06bfffb4 7c812ccb 00000000 0012fb9c 003e0000 mscorwks!GCHeap::FinalizerThreadStart+0xc2**

06bfffec 00000000 791d0020 00000000 00000000 KERNEL32!BaseThreadStart+0x37

So, I can see that this is my **finalizer** thread and somewhere in the call chain I can see that this thread is calling **Sleep** which sounds like a bad idea to me for a finalizer method.

Instantly, you get a good picture of what might be going on here. As an aside, it’s worth taking a look around the **mscorwks**, **mscoree** and friends DLLs to have a look at places where you can set breakpoints. You can do this with;

**X mscoree!\***

**X mscorwks!\***

So, our example so far has been nice but it doesn’t directly interact with managed code. If we want to do that then we need to look to the **SOS.DLL** and the functionality that it offers. We can see this with **!SOS.help** which gives;

0:003> !sos.help

SOS : Help

COMState | List COM state for each thread

ClrStack | Provides true managed stack trace, source and line numbers.

Additional parameters: -p\[arams\] -l\[ocals\] -r\[egs\] -a\[ll\].

DumpClass <addr> | Dump EEClass info

DumpDomain \[<addr>\] | List assemblies and modules in a domain

DumpHeap \[-stat\] \[-min 100\] \[-max 2000\] \[-mt 0x3000000\] \[-type <partial type name>\] \[-fix\] \[start \[end\]\] | Dump GC heap contents

DumpMD <addr> | Dump MethodDesc info

DumpMT \[-MD\] <addr> | Dump MethodTable info

DumpModule <addr> | Dump EE Module info

DumpObj <addr> | Dump an object on GC heap

DumpStack \[-EE\] \[-smart\] \[top stack \[bottom stack\] | -EE only shows managed stack items.

DumpStackObjects \[top stack \[bottom stack\]

DumpVC <mt> <addr> | Dump a value class object

EEHeap \[-gc\] \[-win32\] \[-loader\] | List GC/Loader heap info

EEStack \[-short\] \[-EE\] | List all stacks EE knows

EEVersion | List mscoree.dll version

FinalizeQueue \[-detail\] | Work queue for finalize thread

GCInfo \[<MD>\] \[IP\] | Dump GC encoding info for a managed method

GCRoot <addr> | Find roots on stack/handle for object

IP2MD <addr> | Find MethodDesc from IP

Name2EE <module name> <item name> | Find memory address of EE data given a class/method name

ObjSize \[<addr>\] | Find number of bytes that a root or all roots keep alive on GC heap.

ProcInfo \[-env\] \[-time\] \[-mem\] | Display the process info

RWLock \[-all\] <addr> | List info for a Read/Write lock

SyncBlk \[-all|\#\] | List syncblock

ThreadPool | Display CLR threadpool state

Threads | List managed threads

Token2EE <module name> <mdToken> | Find memory address of EE data for metadata token

u \[<MD>\] \[IP\] | Unassembly a managed code

# Looking at “Global State” #

So, there are some simpler “global-state” commands in here which show us information about the CLR. These would be things such as;

**!ProcInfo**

0:003> !ProcInfo

\---------------------------------------

Process Times

Process Started at: 2004 Aug 3 12:5:20.76

Kernel CPU time : 0 days 00:00:00.03

User CPU time : 0 days 00:00:00.04

Total CPU time : 0 days 00:00:00.07

\---------------------------------------

Process Memory

WorkingSetSize: 7476 KB PeakWorkingSetSize: 7476 KB

VirtualSize: 142832 KB PeakVirtualSize: 142832 KB

PagefileUsage: 2916 KB PeakPagefileUsage: 2916 KB

\---------------------------------------

32 percent of memory is in use.

Memory Availability (Numbers in MB)

Total Avail

Physical Memory 2046 1372

Page File 3944 3350

Virtual Memory 2047 1907

**!Threads**

0:003> !Threads

ThreadCount: 2

UnstartedThread: 0

BackgroundThread: 1

PendingThread: 0

DeadThread: 0

PreEmptive GC Alloc Lock

ID ThreadOBJ State GC Context Domain Count APT Exception

0 cd0 0014cf38 2000020 Enabled 00000000:00000000 00147880 0 Ukn

2 a08 00158010 2001220 Enabled 04a61d94:04a61ff4 00147880 0 Ukn (Finalizer)

**!ThreadPool**

0:003> !ThreadPool

Worker Thread: Total: 0 Running: 0 Idle: 0 MaxLimit: 0 MinLimit: 0

Work Request in Queue: 0

\--------------------------------------

Number of Timers: 0

\--------------------------------------

Completion Port Thread: Total: 0 Free: 0 MaxFree: 0 CurrentLimit: 0 MaxLimit: 1000 MinLimit: 0

**!COMState**

0:003> !COMState

ID TEB APT APTId CallerTID Context

0 cd0 7ffdf000 Ukn

1 e3c 7ffde000 Ukn

2 a08 7ffdd000 Ukn

3 d88 7ffdc000 Ukn

The last one here, COMState, is useful for seeing how your .NET threads have initilaised themselves to COM (i.e. are they STA threads or MTA threads and more detailed info on where they are with respect to COM apartments, contexts, etc).

**!FinalizeQueue**

0:003> !FinalizeQueue

SyncBlock to be cleaned up: 0

\----------------------------------

generation 0 has 0 finalizable objects (0014d330->0014d330)

generation 1 has 0 finalizable objects (0014d330->0014d330)

generation 2 has 0 finalizable objects (0014d330->0014d330)

Ready for finalization 1 objects (0014d330->0014d334)

Statistics:

MT Count TotalSize Class Name

935108 2 24 ConsoleApplication12.BadClass

Total 2 objects

In our scenario you can get a clear picture of what’s present on the queue for finalization in that we have 2 instances of **BadClass** resident in there.

**!EEVersion**

0:003> !EEVersion

1.1.4322.573 retail

Workstation build

# Looking at Managed Memory #

We can get a picture of the “global” state of memory within the application through the **dumpheap** command.

The simplest way to use **dumpheap** is to provide the **–stat** option which will give a complete list of what’s allocated on the managed heap. Note that for a big process you might need to go and get a cup of tea whilst this list is being created. In our small example **dumpheap –stat** gives us;

total 57 objects

Statistics:

MT Count TotalSize Class Name

79c077b4 1 20 System.Collections.ArrayList

79bff0a4 1 20 System.AppDomainSetup

79c19424 1 24 System.LocalDataStoreMgr

935108 2 24 ConsoleApplication12.BadClass

79bfcb44 1 32 System.SharedStatics

79c0d0cc 1 52 System.Collections.Hashtable

95236c 1 56 System.Char\[\]

79bfc414 1 64 System.ExecutionEngineException

79bfc2dc 1 64 System.StackOverflowException

79bfc1a4 1 64 System.OutOfMemoryException

9526b0 1 76 System.Byte\[\]

79bfdbec 1 80 System.AppDomain

952c28 1 144 System.Collections.Hashtable/bucket\[\]

952964 1 292 System.Int32\[\]

79bf9af8 26 1448 System.String

14d0f0 11 2800 Free

95209c 5 6328 System.Object\[\]

Total 57 objects

So we can see that we have 57 objects here. If you examine the line for our managed type **BadClass** you’ll see that it has a MT column for its method table. This effectively describes the type and we can use it to restrict the output of **dumpheap** by doing;

**!dumpheap –mt 935108**

0:003> !dumpheap -mt 935108

Address MT Size

04a61998 00935108 12

04a619a4 00935108 12

total 2 objects

Statistics:

MT Count TotalSize Class Name

935108 2 24 ConsoleApplication12.BadClass

Total 2 objects

Note that this actually gives us *addresses* of the instances of our type (2 in our case) which we’ll come back to in a second but, first, how would we get the MethodTable for a particular type without sitting through !**dumpheap –stat** first? We can use the **Name2EE** function as below;

**!Name2EE ConsoleApplication12.exe ConsoleApplication12.BadClass**

0:003> !Name2EE ConsoleApplication12.exe ConsoleApplication12.BadClass

\--------------------------------------

MethodTable: 00935108

EEClass: 06c2334c

Name: ConsoleApplication12.BadClass

And now we know the MethodTable address for our type which is nice J Now, coming back to those addresses. These allow us to dump out the objects themselves using the !**dumpobj** command;

**!dumpobj 04a61998**

0:003> !dumpobj 04a61998

Name: ConsoleApplication12.BadClass

MethodTable 0x00935108

EEClass 0x06c2334c

Size 12(0xc) bytes

mdToken: 02000002 (t:/Temp/ConsoleApplication12/bin/Release/ConsoleApplication12.exe)

This object isn’t so interesting so I’ve gone back and dumped out a hashtable so you can see what a real object looks like;

0:003> !dumpobj 04a61a90

Name: System.Collections.Hashtable

MethodTable 0x79c0d0cc

EEClass 0x79c0d20c

Size 52(0x34) bytes

mdToken: 020000f9 (c:/windows/microsoft.net/framework/v1.1.4322/mscorlib.dll)

FieldDesc\*: 79c0d270

MT Field Offset Type Attr Value Name

79c0d0cc 4000395 4 CLASS instance 04a61d04 buckets

79c0d0cc 4000396 1c System.Int32 instance 0 count

79c0d0cc 4000397 20 System.Int32 instance 0 occupancy

79c0d0cc 4000398 24 System.Int32 instance 7 loadsize

79c0d0cc 4000399 28 System.Single instance 0.720000 loadFactor

79c0d0cc 400039a 2c System.Int32 instance 0 version

79c0d0cc 400039b 8 CLASS instance 00000000 keys

79c0d0cc 400039c c CLASS instance 00000000 values

79c0d0cc 400039d 10 CLASS instance 00000000 \_hcp

79c0d0cc 400039e 14 CLASS instance 00000000 \_comparer

79c0d0cc 400039f 18 CLASS instance 00000000 m\_siInfo

79c0d0cc 4000394 0 CLASS shared static primes

>> Domain:Value 00147880:04a61ac4 <<

So, in essence, we can traverse the entire managed heap from within WinDbg here and we can see the details of every single instance that is stored on that managed heap and we can track down all the members of those instances ad infinitum. This is really, really powerful stuff.

# Looking at Stack Frames #

What about stack frames? Well, if we look at the original stack trace that I had when I was trying to diagnose my “hung application” problem we can see the frames look like this;

2 Id: 7b8.a08 Suspend: 1 Teb: 7ffdd000 Unfrozen

ChildEBP RetAddr Args to Child

06bffd50 7c90fb74 7c8023ae 00000001 06bffd84 ntdll!KiFastSystemCallRet

06bffd54 7c8023ae 00000001 06bffd84 00000000 ntdll!NtDelayExecution+0xc

06bffdac 7929703b ffffffff 00000001 ffffffff KERNEL32!SleepEx+0x61

06bffdcc 792e42d5 ffffffff 06bffe04 06bffe34 mscorwks!Thread::UserSleep+0x93

06bffddc 0095a492 06bffde8 ffffffff 06bffe3c mscorwks!ThreadNative::Sleep+0x30

**WARNING: Frame IP not in any known module. Following frames may be wrong.**

06bffe34 791f9118 06bffeb0 7920fc3f ffffffff 0x95a492

06bffec0 791f91dd 04a619a4 00158010 00147880 mscorwks!MethodTable::CallFinalizer+0xee

06bffed4 791e1f08 04a619a4 7c8099f2 00000000 mscorwks!CallFinalizer+0x84

06bfff6c 791bbed0 04a619a4 00000000 0012fb9c mscorwks!CallFinalizer+0x245

06bfffb4 7c812ccb 00000000 0012fb9c 003e0000 mscorwks!GCHeap::FinalizerThreadStart+0xc2

06bfffec 00000000 791d0020 00000000 00000000 KERNEL32!BaseThreadStart+0x37

So, what’s the bit that’s highlighted there? How come WinDbg can’t fathom this stack frame? Essentially what this points to is that in between the **mscorwks!MethodTable::CallFinalizer** function and **mscorwks!ThreadNative::Sleep** function we have managed code. If we want to see the managed stack frames then we can issue;

**!ClrStack**

0:002> !ClrStack

Thread 2

ESP EIP

06bffe04 7c911444 \[FRAME: ECallMethodFrame\] \[DEFAULT\] Void System.Threading.Thread.Sleep(I4)

06bffe14 06d4010e \[DEFAULT\] \[hasThis\] Void ConsoleApplication12.BadClass.Finalize()

at \[+0x1e\] \[+0x6\]

which works on the current thread (i.e. the one that the debugger is “focused” on) and dumps out the managed stack – you can see that we now could guess that we have a problem in **BadClass.Finalize** which we kind of knew all along ;-)

We can also do **!ClrStack –params –locals –regs –all** to include parameter, local and register information which is going to be really helpful for real-world debugging. For managed objects that we find we can then go and do **!dumpobj** to take a look at those objects.

# Setting BreakPoints #

Setting breakpoints isn’t quite as easy for managed code in WinDbg as it might be but it’s far from impossible. What we have to work out to set a breakpoint on a manged function is the *address* of the code for that function.

For our type, **BadClass** let’s say that we want to set a breakpoint in its constructor. What we do is;

1)Get the MethodTable address for the type by issuing our **!Name2EE** command.

a.**!Name2EE ConsoleApplication12.exe ConsoleApplication12.BadClass**

0:002> !Name2EE ConsoleApplication12.exe ConsoleApplication12.BadClass

\--------------------------------------

MethodTable: 00935108

EEClass: 06c2334c

Name: ConsoleApplication12.BadClass

2)We can now get hold of the method table for this class by using the **!dumpmt** command as below;

a.**!dumpmt –md 00935108**

0:002> !dumpmt -md 00935108

EEClass : 06c2334c

Module : 0015b610

Name: ConsoleApplication12.BadClass

mdToken: 02000002 (t:/Temp/ConsoleApplication12/bin/Release/ConsoleApplication12.exe)

MethodTable Flags : 180000

Number of IFaces in IFaceMap : 0

Interface Map : 00935148

Slots in VTable : 5

\--------------------------------------

MethodDesc Table

Entry MethodDesc JIT Name

79bf84eb 79bf84f0 None \[DEFAULT\] \[hasThis\] String System.Object.ToString()

79bf8473 79bf8478 None \[DEFAULT\] \[hasThis\] Boolean System.Object.Equals(Object)

79bf848b 79bf8490 None \[DEFAULT\] \[hasThis\] I4 System.Object.GetHashCode()

009350fb 00935100 None \[DEFAULT\] \[hasThis\] Void ConsoleApplication12.BadClass.Finalize()

009350eb 009350f0 None \[DEFAULT\] \[hasThis\] Void ConsoleApplication12.BadClass..ctor()

3)We can see that the address for our constructor here is 009350eb. We can provide this as a breakpoint to WinDbg

a.**bp 009350eb**

b.and when that breakpoint hits we can issue a step command (“**t**”) and we’ll find that we’re inside the function that we wanted to break in.

# Determining Roots #

Last topic for this post – how to determine what is holding on to your CLR memory. I talked about how we can use **!dumpheap** to take a look at what’s on the managed heap but we can also use the fantastic **!gcroot** command in order to determine for any managed object what “set of roots” are actually causing that object to remain “alive” rather than be garbage collected by the GC.

This is a particularly useful command if you suspect you’ve got a memory leak somewhere in the sense that something is holding on to a managed object longer than it should be – you can break into the process with WinDbg and take a look at what “roots” are present for your particular object and, hopefully, that’d move you forward in determining what’s going on.

An example of the output here on **!gcroot** is as below let’s use the following piece of code;

class EntryPoint

private static ArrayList al = new ArrayList();

static void Main(string\[\] args)

for (int i = 0; i < 10; ++i)

switch (i % 3)

case 0:

al.Add("Hello");

break;

case 1:

al.Add(new Exception(""));

break;

case 2:

al.Add(new object());

break;

Console.ReadLine();

So, if I want to know what objects are around at the point of the Console.ReadLine()call then I can use **!dumpheap –stat** to show me;

total 132 objects

Statistics:

MT Count TotalSize Class Name

79c53524 1 12 System.IO.TextReader/NullTextReader

79c0c46c 1 12 System.\_\_Filters

79c0bfa8 1 12 System.Reflection.Missing

79c03458 1 12 System.IO.Stream/NullStream

79c8d504 1 16 System.IO.TextReader/SyncTextReader

79c58114 1 20 System.Text.CodePageEncoding/Decoder

79c3667c 1 20 System.Text.UTF8Encoding/UTF8Encoder

79c363c4 1 20 System.IO.TextWriter/NullTextWriter

79c077b4 1 20 System.Collections.ArrayList

79c03efc 1 20 System.Text.UTF8Encoding

79bff0a4 1 20 System.AppDomainSetup

79c61b94 2 24 System.Text.Encoding/DefaultEncoder

79bfcb44 1 32 System.SharedStatics

79bf8364 3 36 System.Object

79c36114 2 40 System.Text.CodePageEncoding

79bfb4f4 2 40 System.Text.StringBuilder

79c61cac 2 48 System.IO.TextWriter/SyncTextWriter

79c57f8c 1 56 System.IO.StreamReader/NullStreamReader

79c53fac 1 56 System.IO.StreamReader

79c618d4 3 60 System.IO.\_\_ConsoleStream

79c0b37c 4 64 System.RuntimeType

79bfc414 1 64 System.ExecutionEngineException

79bfc2dc 1 64 System.StackOverflowException

79bfc1a4 1 64 System.OutOfMemoryException

79bfdbec 1 80 System.AppDomain

79c0c584 3 84 System.Reflection.MemberFilter

79c35d24 3 156 System.IO.StreamWriter

79bfbcdc 3 192 System.Exception

9526b0 6 1356 System.Byte\[\]

95236c 12 2008 System.Char\[\]

79bf9af8 56 3280 System.String

14d0f0 6 10220 Free

95209c 7 10440 System.Object\[\]

Total 132 objects

And if I now want to know what instances of **System.Exception** are around I can do **!dumpheap –mt 79bfbcdc** and get;

0:003> !dumpheap -mt 79bfbcdc

Address MT Size

04a619fc 79bfbcdc 64

04a61a48 79bfbcdc 64

04a61a94 79bfbcdc 64

total 3 objects

Statistics:

MT Count TotalSize Class Name

79bfbcdc 3 192 System.Exception

Total 3 objects

And if I now want to know *why* the first of these instances is still around I can use **!gcroot** on it to give me;

0:003> !gcroot 04a619fc

Scan Thread 0 (ec4)

ESP:12f644:Root:04a6197c(System.Collections.ArrayList)->04a61990(System.Object\[\])->04a619fc(System.Exception)

ESP:12f658:Root:04a6197c(System.Collections.ArrayList)->04a61990(System.Object\[\])

ESP:12f668:Root:04a6197c(System.Collections.ArrayList)->04a61990(System.Object\[\])

Scan Thread 2 (eb8)

Scan HandleTable 15f008

Scan HandleTable 149c88

HANDLE(Strong):9411c0:Root:05a62000(System.Object\[\])->04a6197c(System.Collections.ArrayList)

I can see that my System.Exception instance at 04a619fc is being referenced (rooted) by an array of objects at 04a61990 and that is rooted by an ArrayList at 04a619c.

So, we can chase back object references as far as we need to work out what’s going on with our managed memory and why particular instances are not yet eligible for GC.

# Working with Exceptions #

The WinDbg debugger does not know anything about managed exceptions. Whenever managed code throws an exception it throws a native exception of type **0xe0434f4d**

If you want to stop on managed exceptions then you can issue to the debugger;

**sxe 0xe0434f4d**

**or**

**sxe clr (much easier!)**

and it’ll break at the point where the exception is thrown. You can grab the exception record from an unmanaged point of view by issuing a display exception record command with a minus 1 as;

**.exr -1**

And this will give you a result such as;

0:000> .exr -1

ExceptionAddress: 7c82dad4 (KERNEL32!RaiseException+0x00000038)

ExceptionCode: e0434f4d (CLR exception)

ExceptionFlags: 00000001

NumberParameters: 0

I think there’s a nice way of turning this into a managed exception object but, at the time of writing, I can’t work it out. What you can do is to execute a **!dumpstackobjects** and the usual case would be to should find your exception living within the set of objects that you get back there.

# Wrap-up #

That’s it for this posting – if you want a more info on debugging with SOS.DLL check out [this big guide][] that’s over on MSDN which will give you more info and also have a look at resources such as;

[http://blogs.msdn.com/mvstanton/archive/2004/04/05/108023.aspx][http_blogs.msdn.com_mvstanton_archive_2004_04_05_108023.aspx]

[http://msdn.microsoft.com/msdnmag/issues/03/06/Bugslayer/default.aspx][http_msdn.microsoft.com_msdnmag_issues_03_06_Bugslayer_default.aspx]

Enjoy!

、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、、

### More on debugging with SOS.DLL - enter Visual Studio ###

I can’t get off this debugging topic at the moment.

To follow up briefly on the postings that I put up about [WinDbg and the SOS.DLL][] I wanted to add some information about Visual Studio because I think I goofed really with the last couple of postings on that topic.

You learn something new every day and I just learnt some stuff about Visual Studio. It turns out that Visual Studio can load up the SOS.DLL extension with consummate ease. I'd thought that this was a 2005 feature but it turns out that it’s in Visual Studio 2003 as well so that's a major bonus.

So, if you're keen to play with SOS.DLL you can work happily with it inside of Visual Studio and you don't have to worry so much about going and getting WinDbg unless you want some of its more advanced features.

How do you make this work?

Firstly, you need to make sure that the debugging project that you’re working in has support for **unmanaged** debugging. You can do this in your project properties (Configuration Properties->Debugging->Enable Unmanaged Debugging) and you can also explicitly set what type of debugging you want to do if you attach the debugger to a process through the **Tools->Debug Processes** menu option.

Note that mixing managed/unmanaged debugging does seem to slow the debugger’s operation down quite a bit.

Once you’ve got unmanaged debugging switched on you can work with the debugger exactly as you usually do and set breakpoints and so on.

At the point where you want some SOS.DLL functionality, switch to your **Immediate** window (**Debug->Windows->Immediate**).

Put the **Immediate** window into **immediate** mode rather than command mode. You do this by typing;

**immed**

at the command prompt (you revert back to command mode by typing “**>cmd”**). You can spot command mode because it gives you a “>” prompt.

When in immediate mode type in;

**.load sos**

And there you go. Type in **!EEVersion** and a quick **!ProcInfo** so that you feel at home and you’re off on your extended debugging session.

Note that within Visual Studio, commands such as **!ClrStack** work on the thread that you have selected within the **Threads** window – i.e. exactly as you’d expect.

The beta version of Visual Studio 2005 also loads up SOS.DLL in the same way.

In V2.0 it looks like some new commands have been added and a whole bunch of addition help features are there. Doing a !Help on V2.0 gives a command list;

*  DumpObj (do) 
 *  DumpStackObjects (dso) 
 *  DumpHeap 
 *  DumpVC 
 *  GCRoot
 *  ObjSize
 *  FinalizeQueue
 *  PrintException (pe)
 *  TraverseHeap
 *  Threads
 *  CLRStack
 *  IP2MD
 *  U
 *  DumpStack
 *  EEStack
 *  GCInfo
 *  COMState
 *  DumpDomain
 *  EEHeap
 *  Name2EE
 *  SyncBlk
 *  DumpMT
 *  DumpClass
 *  DumpMD
 *  Token2EE
 *  EEVersion
 *  DumpModule
 *  ThreadPool
 *  DumpAssembly
 *  DumpMethodSig
 *  DumpRuntimeTypes
 *  DumpSig
 *  VerifyHeap
 *  DumpLog
 *  FindAppDomain
 *  SaveModule
 *  GCHandles
 *  GCHandleLeaks
 *  VMMap
 *  VMStat
 *  ProcInfo
 *  FAQ

You can get a (much improved) set of help information about these commands by executing;

**!Help <command>**

and you even get examples of usage and output.

Some of these commands are carried over from SOS V1.1 but a lot are new. It appears that some of the commands do not work inside of Visual Studio – the 2 that I found are as below;

**VMStat, VMMap**

Enjoy.

[A word for WinDbg]: http://mtaulty.com/blog/%28jawrmi45unnjdl2wzzpagbjs%29/archive/2004/08/03/608.aspx
[A word for WinDbg _2]: http://mtaulty.com/blog/%2802mh0cmzf3g30ynyniegwfrb%29/archive/2004/08/03/609.aspx
[More on debugging with SOS.DLL - enter Visual Studio]: http://mtaulty.com/blog/%28pbhrwj55l1xmho55xjrlfxbf%29/archive/2004/08/06/628.aspx
[Troubleshooting ASP.NET using WinDbg and the SOS extension]: http://support.microsoft.com/kb/892277/
[here]: http://www.microsoft.com/whdc/devtools/debugging/default.mspx
[previous post]: http://mtaulty.com/blog/archive/2004/08/03/608.aspx
[this big guide]: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/DBGch01.asp
[http_blogs.msdn.com_mvstanton_archive_2004_04_05_108023.aspx]: http://blogs.msdn.com/mvstanton/archive/2004/04/05/108023.aspx
[http_msdn.microsoft.com_msdnmag_issues_03_06_Bugslayer_default.aspx]: http://msdn.microsoft.com/msdnmag/issues/03/06/Bugslayer/default.aspx
[WinDbg and the SOS.DLL]: http://mtaulty.com/blog/archive/2004/08/03/609.aspx