Windows - WaitForSingleObject never signalled when unloading tracy.dll while application remains active

[robertblaketaylor]

Running into an issue with tracy as a dll when unloading the dll from an active process (outside of exit). I see the worker thread is still alive and looping up to the point where the thread is destructed here:

Profiler::~Profiler()
{
    m_shutdown.store( true, std::memory_order_relaxed );

#ifdef TRACY_HAS_SYSTEM_TRACING
    if( s_sysTraceThread )
    {
        SysTraceStop();
        s_sysTraceThread->~Thread();
        tracy_free( s_sysTraceThread );
    }
#endif

#ifndef TRACY_NO_FRAME_IMAGE
    s_compressThread->~Thread();
    tracy_free( s_compressThread );
#endif

    s_thread->~Thread(); // <---- Locks up here
    tracy_free( s_thread );

    tracy_free( m_lz4Buf );
    tracy_free( m_buffer );
    LZ4_freeStream( (LZ4_stream_t*)m_stream );

    if( m_sock )
    {
        m_sock->~Socket();
        tracy_free( m_sock );
    }

    if( m_broadcast )
    {
        m_broadcast->~UdpBroadcast();
        tracy_free( m_broadcast );
    }

    assert( s_instance );
    s_instance = nullptr;
}

I see that WaitForSingleObject in the Thread class locks up the process waiting on m_hnd to signal here:

class Thread
{
public:
    Thread( void(*func)( void* ptr ), void* ptr )
        : m_func( func )
        , m_ptr( ptr )
        , m_hnd( CreateThread( nullptr, 0, Launch, this, 0, nullptr ) )
    {}

    ~Thread()
    {
        WaitForSingleObject( m_hnd, INFINITE ); // <---- hung here waiting for a signal
        CloseHandle( m_hnd );
    }

    HANDLE Handle() const { return m_hnd; }

private:
    static DWORD WINAPI Launch( void* ptr ) { ((Thread*)ptr)->m_func( ((Thread*)ptr)->m_ptr ); return 0; }

    void(*m_func)( void* ptr );
    void* m_ptr;
    HANDLE m_hnd;
};

Any thoughts as to why that is happening?

Here is the callstack when it hangs:

ntdll.dll!00007ffe5700cdf4() (Unknown Source:0)
KernelBase.dll!00007ffe54931a5e() (Unknown Source:0)
tracy.dll!tracy::Thread::~Thread() Line 41 (c:\open\fbsource\arvr\projects\socialvr\third-party\tracy\source\tracy\client\TracyThread.hpp:41)
tracy.dll!tracy::Thread::`scalar deleting destructor'(unsigned int) (Unknown Source:0)
tracy.dll!tracy::Profiler::~Profiler() Line 1321 (c:\open\fbsource\arvr\projects\socialvr\third-party\tracy\source\tracy\client\TracyProfiler.cpp:1321)
tracy.dll!tracy::`dynamic atexit destructor for 's_profiler''() (Unknown Source:0)
ucrtbased.dll!00007ffe06e5a9d7() (Unknown Source:0)
ucrtbased.dll!00007ffe06e5a385() (Unknown Source:0)
ucrtbased.dll!00007ffe06e5a4ba() (Unknown Source:0)
ucrtbased.dll!00007ffe06e5abc1() (Unknown Source:0)
tracy.dll!__scrt_dllmain_uninitialize_c() Line 400 (d:\agent\_work\1\s\src\vctools\crt\vcstartup\src\utility\utility.cpp:400)
tracy.dll!dllmain_crt_process_detach(const bool is_terminating) Line 108 (d:\agent\_work\1\s\src\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp:108)
tracy.dll!dllmain_crt_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 139 (d:\agent\_work\1\s\src\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp:139)
tracy.dll!dllmain_dispatch(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 212 (d:\agent\_work\1\s\src\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp:212)
tracy.dll!_DllMainCRTStartup(HINSTANCE__ * const instance, const unsigned long reason, void * const reserved) Line 254 (d:\agent\_work\1\s\src\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp:254)
ntdll.dll!00007ffe56f89a1d() (Unknown Source:0)
ntdll.dll!00007ffe56fda9cb() (Unknown Source:0)
ntdll.dll!00007ffe56fda427() (Unknown Source:0)
ntdll.dll!00007ffe56fda678() (Unknown Source:0)
ntdll.dll!00007ffe56f7fd0a() (Unknown Source:0)

[wolfpld]

The wait is for the m_hnd thread to join. For some reason it is not able to.

[robertblaketaylor]

We'll need to use the manual lifetime functions for our use case. The DLL is being unloaded and reloaded at runtime as a dep of another dll that is instrumented for tracing. Upon calling FreeLibrary the loader lock is obtained such that neither the compress thread or the worker thread can exit because they also wish to obtain a lock to shoot out the thread specific signal. If someone has found a more robust solution at the DLL level (vs. the manual approach), would love to learn more.

[robertblaketaylor]

Resolved by switching to manual lifetime.

[ivan-flink]

I also faced with this issue and got exactly the same call stack:

ntdll.dll!00007ffe7b601b14()
KernelBase.dll!00007ffe78e7b1bf() 
MyDll.dll!tracy::Thread::~Thread() Line 47 client\TracyThread.hpp:47 <---------- Here we are waiting for s_symbolThread inside DLLMain which is prohibited
MyDll.dll!tracy::Profiler::~Profiler()
MyDll.dll!tracy::`dynamic atexit destructor for 's_profiler''() (Unknown Source:0) <---------- calling dtor for static object Profiler
MyDll.dll!_execute_onexit_table::__l2::<lambda>()
MyDll.dll!__crt_seh_guarded_call<int>::operator()
MyDll.dll!__acrt_lock_and_call<int <lambda>(void)>
MyDll.dll!_execute_onexit_table
MyDll.dll!common_exit::__l2::<lambda>()
MyDll.dll!__crt_seh_guarded_call<void>::operator()
MyDll.dll!__acrt_lock_and_call<void <lambda>(void)>
MyDll.dll!common_exit
MyDll.dll!_cexit()
MyDll.dll!__scrt_dllmain_uninitialize_c()
MyDll.dll!dllmain_crt_process_detach
MyDll.dll!dllmain_crt_dispatch
MyDll.dll!dllmain_dispatch <---------------------------------- We are running inside DLLMain
MyDll.dll!_DllMainCRTStartup
ntdll.dll!00007ffe7b5ff73a()
ntdll.dll!00007ffe7b4abc33()

According to Microsoft, synchronisation with other threads is prohibited, because it "can cause a deadlock"
https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-best-practices