Fading Coder

One Final Commit for the Last Sprint

Home > Tech > Content

Diagnosing Mutex Deadlocks: Identifying the Owner Thread via WinDbg and KD

Tech 1

Two primary methods exist for analyzing thread blocks based on WaitHandles: a non-deterministic approach compatible with crash dumps (limited to named synchronization objects), and a deterministic approach requiring a live system with a kernel debugger.

User-Mode Crash Dump Analysis

This method relies on !DumpStackObjects to find WaitHandle references on a callstack, acting as an automated trial-and-error search for named synchronization object identifiers. It functions on hang dumps generated by ADPlus and live systems.

Assume a process where multiple threads experience unexpected blocking. After generating a hang dump, load SOS and inspect the managed threads:

0:005> .loadby sos mscorwks
0:005> !threads
ThreadCount: 12
                                      PreEmptive   GC Alloc           Lock
       ID OSID ThreadOBJ   State     GC       Context       Domain   Count APT Exception
   0    1  a10 00210fd0   200a020 Enabled  00000000:00000000 0015c100     0 MTA
   2    2 2c40 00192480      b220 Enabled  00000000:00000000 0015c100     0 MTA (Finalizer)
   3    3 1ba0 0019a1c0   200b020 Enabled  00000000:00000000 0015c100     0 MTA
   4    4 1f9c 0019c138   200b020 Enabled  00000000:00000000 0015c100     0 MTA
   5    5  d40 0021cd18   200b020 Enabled  00000000:00000000 0015c100     0 MTA
   ...

Threads with state 200a020 are waiting. To determine if they are blocking on a WaitHandle rather than Thread.Sleep(), inspect the top managed stack frame using the native thread ID (first column):

0:005> ~5e !clrstack
OS Thread Id: 0xd40 (5)
ESP       EIP
022cf588 7c90eb94 [HelperMethodFrame_1OBJ: 022cf588] System.Threading.WaitHandle.WaitOneNative(...)
022cf634 793d424e System.Threading.WaitHandle.WaitOne(Int64, Boolean)
022cf64c 793d4193 System.Threading.WaitHandle.WaitOne(Int32, Boolean)
022cf65c 793d420e System.Threading.WaitHandle.WaitOne()
022cf660 00cb0250 DeadlockApp.Worker.Run()
...

The presence of WaitOneNative() confirms the thread is waiting for a synchronization object. Next, examine the unmanaged callstack to retrieve the handle:

0:005> ~5 s
eax=00000000 ebx=022cf350 ecx=79690098 edx=79690048 esi=00000000 edi=7ffde000
0:005> kb 5
ChildEBP RetAddr  Args to Child
022cf324 7c90e9ab 7c8094f2 00000001 022cf350 ntdll!KiFastSystemCallRet
022cf328 7c8094f2 00000001 022cf614 00000000 ntdll!ZwWaitForMultipleObjects+0xc
022cf3c4 79f8ead4 00000001 022cf614 00000001 KERNEL32!WaitForMultipleObjectsEx+0x12c
...

The WaitForMultipleObjectsEx API signature shows nCount as the first argument (1) and lpHandles as the second (022cf614). Dumping this memory address reveals the underlying handle:

0:005> dd 022cf614 L1
022cf614  000009b0

Query the handle details to identify the synchronization object:

0:005> !handle 9b0 f
Handle 9b0
  Type          Mutant
  GrantedAccess 0x1f0001:
  Name         \BaseNamedObjects\{A1B2C3D4-E5F6-7890-ABCD-EF1234567890}
  Object Specific Information
    Mutex is Owned

The thread is waiting for a Mutex named {A1B2C3D4-E5F6-7890-ABCD-EF1234567890}. To find the owner thread in a crash dump, iterate through other threads using !DumpStackObjects (!dso) looking for a System.Threading.Mutex reference:

0:005> ~2e !dso
OS Thread Id: 0x1b50 (2)
ESP/REG  Object   Name
00ecf748 01273b30 System.String    Worker 2 sleeping
00ecf880 0127383c System.Threading.Mutex
...

Verify this method acquires the mutex by correlating the stack address (00ecf880) with the managed stack, confirming it falls within DeadlockApp.Worker.Run(). Use !IP2MD and !DumpIL to confirm the IL contains Mutex::WaitOne().

Inspect the managed Mutex object to retrieve its underlying wait handle:

0:002> !do 0127383c
Name: System.Threading.Mutex
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
790fe160  40005a6        c        System.IntPtr  0 instance     2384 waitHandle
...

Converting the decimal waitHandle value (2384) to hexadecimal yields 950. Checking this handle confirms it points to the exact same named Mutex:

0:002> !handle 950 f
Handle 950
  Type          Mutant
  Name         \BaseNamedObjects\{A1B2C3D4-E5F6-7890-ABCD-EF1234567890}
  Object Specific Information
    Mutex is Owned

Deterministic Kernel Debugging

A kernel debugger provides deterministic owner identification on live systems without trial-and-error. Ensure debug symbols are configured correctly, then launch local kernel debugging:

C:\debug>set _NT_SYMBOL_PATH=SRV*C:\symbols*http://msdl.microsoft.com/download/symbols 
C:\debug>kd /KL

Microsoft (R) Windows Debugger Version 6.6.0007.5
...
lkd>

Locate the target process:

lkd> !process 0 0 DeadlockApp.exe
PROCESS 8a10e020  SessionId: 0 Cid: 2f01    Peb: 7ffde000  ParentCid: 12b0
    DirBase: 0cb83000  ObjectTable: e72125c8  HandleCount: 122.
    Image: DeadlockApp.exe

Pass the process address and flag 2 to !process to display thread synchronization details. Ensure the user-mode debugger is in run mode (F5) before executing this:

lkd> !process 8a10e020 2
PROCESS 8a10e020  SessionId: 0 Cid: 2f01    Peb: 7ffde000  ParentCid: 12b0
    DirBase: 0cb83000 ObjectTable: e72125c8  HandleCount: 122.
    Image: DeadlockApp.exe
 
        THREAD 8a96da8  Cid 2f01.1b50  Teb: 7ffda000 Win32Thread: 00000000 WAIT: (DelayExecution) UserMode Alertable
            8a96e98 NotificationTimer
 
        THREAD 8b285020  Cid 2f01.0d40  Teb: 7ffd6000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Alertable
            8c148fc0  Mutant - owning thread 8a96da8
 
        THREAD 8c927360  Cid 2f01.2e10  Teb: 7ffad000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Alertable
            8c148fc0  Mutant - owning thread 8a96da8

The output directly reveals the waiting threads (0d40 and 2e10) and the owner thread (8a96da8). The owning thread's CID is 2f01.1b50, where 1b50 is the OS Thread ID. Switching back to the user-mode debugger and checking !threads maps OSID 1b50 directly to the blocking thread, matching the crash dump findings deterministically.

Tags: WinDbg

Related Articles

Understanding Strong and Weak References in Java

Strong References Strong reference are the most prevalent type of object referencing in Java. When an object has a strong reference pointing to it, the garbage collector will not reclaim its memory. F...

Comprehensive Guide to SSTI Explained with Payload Bypass Techniques

Introduction Server-Side Template Injection (SSTI) is a vulnerability in web applications where user input is improper handled within the template engine and executed on the server. This exploit can r...

Implement Image Upload Functionality for Django Integrated TinyMCE Editor

Django’s Admin panel is highly user-friendly, and pairing it with TinyMCE, an effective rich text editor, simplifies content management significantly. Combining the two is particular useful for bloggi...

Leave a Comment

Anonymous

◎Feel free to join the discussion and share your thoughts.