Black Magic 101, or getting WinDBG to .Net city

The legendary WinDBG. An ancient, slow and enigmatic tool that is also… indispensable for post-mortem analysis.
When there is a need to analyze a .Net dump, WinDBG requires additional configuration.

While not particularly complicated, the configuration process at first can seem like dark magic.

configuring_windbg_is_easy.jpg

The following is what works for me, and I think should work for most, if not all needs.

1) Bitness of the dump

This might not be apparent to developers who work exclusively with managed code, so this is worth mentioning. When we are talking about WinDBG and dumps, bitness matters. Alot. There are 32 and 64 bit versions of WinDBG and the bitness must match the bitness of the dump. Otherwise, you might see in WinDBG something like this:

1
2
3
4
5
 0:000> .loadby sos clr
The call to LoadLibrary(C:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos) failed, Win32 error 0n193
"%1 is not a valid Win32 application."
Please check your debugger configuration and/or network access.

2) Symbols

Stored in files with pdb extension, symbols are necessary for some of WinDBG commands. Without symbols, it is not possible to see type and method names in stack traces and objects stored in managed heaps.
The easiest way to do this, is to press Ctrl+S or go to File->Symbol File Path, and then enter this string

1
2
3
4
cache*c:\symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols;srv*http://srv.symbolsource.org/pdb/Public
// **http://msdl.microsoft.com/download/symbols** is the address of Microsoft symbol server
// **http://srv.symbolsource.org/pdb/Public** is address of NuGet symbols server


Then, after pressing ok, execute .reload command. What this does, is configures WinDBG to look for symbols first in c:\symbols folder, then try to go to Microsoft symbol server, and if not found yet, try NuGet symbol server. All symbols downloaded will be cached in c:\symbols folder, so they won’t be downloaded next time they are needed.


The next step then is to execute the following commands and watch the output carefully.
1
2
3
4
5
6
7
8
9
10
 0:000> !sym noisy on
noisy mode - symbol prompts on
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
//the rest of !analyze -v command is omitted for clarity


What happens here is that !sym noisy on command turns on verbose output that is related to symbols. The command !analyze -v has a very nice and undocumented side-effect, it downloads all the needed symbols and also relevant SOS Extension version (more about SOS in the next step). At this stage, browse through output, to see if there are any symbols that failed to load. If some symbols failed to load, it might be necessary to get them throuh other means, such as compiling the app/library in debug mode if source code is available, or getting the Pdb files from the machine where the code has been running.


Symbols related output looks like this
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
  INFO:  HTTP_STATUS_OK
SYMSRV: C:\ProgramData\dbg\sym\esent.pdb\B08C5A22A85340C58BB25F83B7E5A08E1\esent.pdb - file not found
SYMSRV: esent.pdb from http://msdl.microsoft.com/download/symbols: 509434 bytes - 0 percentSYMSRV: Waiting for the server to respond to a request.

SYMSRV: Successfully received a response from 0 percentSYMSRV: Waiting for the server to respond to a request.
SYMSRV: Successfully received a response from 1 percentSYMSRV: Waiting for the server to respond to a request.
SYMSRV: Successfully received a response from the server.
SYMSRV: Waiting for the server to respond to a request.
SYMSRV: Successfully received a response from the server.
SYMSRV: Waiting for the server to respond to a request.

... some lines cut for clarity

SYMSRV: Successfully received a response from the server.
SYMSRV: Waiting for the server to respond to a request.
SYMSRV: Successfully received a response from 21 percentSYMSRV: Waiting for the server to respond to a request.
SYMSRV: Successfully received a response from the server.

... some more lines cut for clarity

SYMSRV: Waiting for the server to respond to a request.
SYMSRV: Successfully received a response from the server.
SYMSRV: Waiting for the server to respond to a request.
SYMSRV: Successfully received a response from copied
SYMSRV: c:\symbols\esent.pdb\B08C5A22A85340C58BB25F83B7E5A08E1\esent.pdb - file not found
SYMSRV: c:\symbols\esent.pdb\B08C5A22A85340C58BB25F83B7E5A08E1\esent.pdb - file not found
DBGHELP: C:\ProgramData\dbg\sym\esent.pdb\B08C5A22A85340C58BB25F83B7E5A08E1\esent.pdb cached to c:\symbols\esent.pdb\B08C5A22A85340C58BB25F83B7E5A08E1\esent.pdb
DBGHELP: C:\ProgramData\dbg\sym\esent.pdb\B08C5A22A85340C58BB25F83B7E5A08E1\esent.pdb cached to c:\symbols\esent.pdb\B08C5A22A85340C58BB25F83B7E5A08E1\esent.pdb

DBGHELP: esent - public symbols
c:\symbols\esent.pdb\B08C5A22A85340C58BB25F83B7E5A08E1\esent.pdb


3) Load SOS Debugging Extension

SOS is an extension for WinDBG for managed dumps (read -> additional commands to analyze .Net process dumps), without which is it extremely hard to do something useful regarding managed parts of a .Net process dump. Loading this extension is easy with simple command : .loadby sos clr. This should work if the dump was taken on a machine with the same version of .Net as the current workstation, or if there is a proper version of SOS extension in the local cache. !analyze -v command should have grabbed the correct SOS Extension version from the symbols server and downloaded that in the cache in the previous step. But, in some cases, the versions are not properly tagged and are not downloaded by !analyze -v. In such cases, something like the following output can be seen when executing any SOS command.

1
2
3
4
5
The version of SOS does not match the version of CLR you are debugging.  Please load the matching version of SOS for the version of CLR you are debugging.
CLR Version: 4.0.30319.1
SOS Version: 4.0.30319.235
CLRDLL: C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\mscordacwks.dll:4.0.30319.235 f:8 doesn't match desired version 4.0.30319.01 f:8


So, what now?
The easiest (and not always available) way is to copy those files from the machine where the dump was taken:

  • clr.dll
  • mscordacwks.dll
  • SOS.dll
    Copy them from the folder [Windows folder]\Microsoft.NET\Framework64[Version of .Net Framework]
    Where version of .Net Framework can be either v4.0.30319 or v2.0.50727 (I truly doubt that there is someone using .Net 1.0 or 1.1 so…). Also notice the Framework64 part, for 32bit dump, the DLLs should be copied from Framework folder instead.
    Then, execute the following command: .cordll -u -lp [full path where the dumps are located]. Extremely important not to have whitespaces in the path, as WinDBG simply cannot handle paths with whitespaces (ancient tool, yes!)
    Alternatively, if there is no access to machine where the dump was taken, there are two places where the relevant DLLs can be located; since in each .Net update there is a new SOS extension version, it should be possible to get the DLLs from the two version history lists here:

4) Installing SOSEX extension

While strictly not necessary for analysis of managed dumps, SOSEX is a WinDBG extension that provides alot of useful commands, some are better version of SOS commands, and some are unique to SOSEX, such as deadlock detection or building heap index for faster traversal.
This one is pretty simple. First step: download the extension from here. Second step: execute command .load [full path to SOSEX dll]\sosex. Here also it is very important to have SOSEX dll in path without whitespaces, otherwise WinDBG won’t be able to handle it.
*Also, note that bitness matters here, so get SOSEX bitness matching bitness of the dump you are about to analyze.
So, what kind of commands does it provide? Well, this… this is a story for another day and another blog post.