Diagnosing Mutex Deadlocks: Identifying the Owner Thread via WinDbg and KD
Two primary methods exist for analyzing thread blocks based on WaitHandles: a non-deterministic approach compatible with crash dumps (limited to named synchronization objects), and a deterministic approach requiring a live system with a kernel debugger.
User-Mode Crash Dump Analysis
This method relies on !DumpStackObjects to find WaitHandle references on a callstack, acting as an automated trial-and-error search for named synchronization object identifiers. It functions on hang dumps generated by ADPlus and live systems.
Assume a process where multiple threads experience unexpected blocking. After generating a hang dump, load SOS and inspect the managed threads:
0:005> .loadby sos mscorwks
0:005> !threads
ThreadCount: 12
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
0 1 a10 00210fd0 200a020 Enabled 00000000:00000000 0015c100 0 MTA
2 2 2c40 00192480 b220 Enabled 00000000:00000000 0015c100 0 MTA (Finalizer)
3 3 1ba0 0019a1c0 200b020 Enabled 00000000:00000000 0015c100 0 MTA
4 4 1f9c 0019c138 200b020 Enabled 00000000:00000000 0015c100 0 MTA
5 5 d40 0021cd18 200b020 Enabled 00000000:00000000 0015c100 0 MTA
...
Threads with state 200a020 are waiting. To determine if they are blocking on a WaitHandle rather than Thread.Sleep(), inspect the top managed stack frame using the native thread ID (first column):
0:005> ~5e !clrstack
OS Thread Id: 0xd40 (5)
ESP EIP
022cf588 7c90eb94 [HelperMethodFrame_1OBJ: 022cf588] System.Threading.WaitHandle.WaitOneNative(...)
022cf634 793d424e System.Threading.WaitHandle.WaitOne(Int64, Boolean)
022cf64c 793d4193 System.Threading.WaitHandle.WaitOne(Int32, Boolean)
022cf65c 793d420e System.Threading.WaitHandle.WaitOne()
022cf660 00cb0250 DeadlockApp.Worker.Run()
...
The presence of WaitOneNative() confirms the thread is waiting for a synchronization object. Next, examine the unmanaged callstack to retrieve the handle:
0:005> ~5 s
eax=00000000 ebx=022cf350 ecx=79690098 edx=79690048 esi=00000000 edi=7ffde000
0:005> kb 5
ChildEBP RetAddr Args to Child
022cf324 7c90e9ab 7c8094f2 00000001 022cf350 ntdll!KiFastSystemCallRet
022cf328 7c8094f2 00000001 022cf614 00000000 ntdll!ZwWaitForMultipleObjects+0xc
022cf3c4 79f8ead4 00000001 022cf614 00000001 KERNEL32!WaitForMultipleObjectsEx+0x12c
...
The WaitForMultipleObjectsEx API signature shows nCount as the first argument (1) and lpHandles as the second (022cf614). Dumping this memory address reveals the underlying handle:
0:005> dd 022cf614 L1
022cf614 000009b0
Query the handle details to identify the synchronization object:
0:005> !handle 9b0 f
Handle 9b0
Type Mutant
GrantedAccess 0x1f0001:
Name \BaseNamedObjects\{A1B2C3D4-E5F6-7890-ABCD-EF1234567890}
Object Specific Information
Mutex is Owned
The thread is waiting for a Mutex named {A1B2C3D4-E5F6-7890-ABCD-EF1234567890}. To find the owner thread in a crash dump, iterate through other threads using !DumpStackObjects (!dso) looking for a System.Threading.Mutex reference:
0:005> ~2e !dso
OS Thread Id: 0x1b50 (2)
ESP/REG Object Name
00ecf748 01273b30 System.String Worker 2 sleeping
00ecf880 0127383c System.Threading.Mutex
...
Verify this method acquires the mutex by correlating the stack address (00ecf880) with the managed stack, confirming it falls within DeadlockApp.Worker.Run(). Use !IP2MD and !DumpIL to confirm the IL contains Mutex::WaitOne().
Inspect the managed Mutex object to retrieve its underlying wait handle:
0:002> !do 0127383c
Name: System.Threading.Mutex
Fields:
MT Field Offset Type VT Attr Value Name
790fe160 40005a6 c System.IntPtr 0 instance 2384 waitHandle
...
Converting the decimal waitHandle value (2384) to hexadecimal yields 950. Checking this handle confirms it points to the exact same named Mutex:
0:002> !handle 950 f
Handle 950
Type Mutant
Name \BaseNamedObjects\{A1B2C3D4-E5F6-7890-ABCD-EF1234567890}
Object Specific Information
Mutex is Owned
Deterministic Kernel Debugging
A kernel debugger provides deterministic owner identification on live systems without trial-and-error. Ensure debug symbols are configured correctly, then launch local kernel debugging:
C:\debug>set _NT_SYMBOL_PATH=SRV*C:\symbols*http://msdl.microsoft.com/download/symbols
C:\debug>kd /KL
Microsoft (R) Windows Debugger Version 6.6.0007.5
...
lkd>
Locate the target process:
lkd> !process 0 0 DeadlockApp.exe
PROCESS 8a10e020 SessionId: 0 Cid: 2f01 Peb: 7ffde000 ParentCid: 12b0
DirBase: 0cb83000 ObjectTable: e72125c8 HandleCount: 122.
Image: DeadlockApp.exe
Pass the process address and flag 2 to !process to display thread synchronization details. Ensure the user-mode debugger is in run mode (F5) before executing this:
lkd> !process 8a10e020 2
PROCESS 8a10e020 SessionId: 0 Cid: 2f01 Peb: 7ffde000 ParentCid: 12b0
DirBase: 0cb83000 ObjectTable: e72125c8 HandleCount: 122.
Image: DeadlockApp.exe
THREAD 8a96da8 Cid 2f01.1b50 Teb: 7ffda000 Win32Thread: 00000000 WAIT: (DelayExecution) UserMode Alertable
8a96e98 NotificationTimer
THREAD 8b285020 Cid 2f01.0d40 Teb: 7ffd6000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Alertable
8c148fc0 Mutant - owning thread 8a96da8
THREAD 8c927360 Cid 2f01.2e10 Teb: 7ffad000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Alertable
8c148fc0 Mutant - owning thread 8a96da8
The output directly reveals the waiting threads (0d40 and 2e10) and the owner thread (8a96da8). The owning thread's CID is 2f01.1b50, where 1b50 is the OS Thread ID. Switching back to the user-mode debugger and checking !threads maps OSID 1b50 directly to the blocking thread, matching the crash dump findings deterministically.