something Close the thread in DLL

Cleaning up threads in a DLL: _endthreadex() vs TerminateThread()

Because of the restrictions on DllMain (and I understood that the same applies to global&static object constructors&destructors in a DLL), such a simple thing as a singleton logger with asynchronous file writing/flushing thread becomes too tricky. The singleton logger is in a DLL, and I have limited influence on the executable loading and unloading this DLL time to time. I can force that executable to call my DLL initialization function before any use, so in the initialization function I can use a critical section to guard a variable telling whether the DLL has been already initialized or it needs init this time. This way initialization from DllMain is avoided, which would lead to a deadlock because I need to launch threads from initialization, and threads call DllMain with DLL_THREAD_ATTACH reason, and that obtains the same loader lock as the one already obtained when we are initializing in DllMain on DLL_PROCESS_ATTACH event.

C++11 thread cannot be used because of this bug (not fixed in MSVC++2013). So I'm using _beginthreadex(), because CreateThread documentation says:

A thread in an executable that calls the C run-time library (CRT) should use the _beginthreadex and _endthreadex functions for thread management rather than CreateThread and ExitThread; this requires the use of the multithreaded version of the CRT. If a thread created using CreateThread calls the CRT, the CRT may terminate the process in low-memory conditions.

But I have no control over the executable to ensure that some deinitialization function from the DLL is called before unloading the DLL. So the only options for cleanup are DllMain's DLL_PROCESS_DETACHand destructors of global/static variables. The problem is that they are called with loader lock obtained, so I cannot make there the DLL threads to exit gracefully because those threads upon a normal exit would attempt to call DllMain with DLL_THREAD_DETACH, which would result in a deadlock (loader lock again). MSDN suggests to use TerminateThread() to handle this:

DLL A gets a DLL_PROCESS_DETACH message in its DllMain and sets an event for thread T, signaling it to exit. Thread T finishes its current task, brings itself to a consistent state, signals DLL A, and waits infinitely. Note that the consistency-checking routines should follow the same restrictions as DllMain to avoid deadlocking. DLL A terminates T, knowing that it is in a consistent state.

So I'm afraid of using _beginthreadex() + TerminateThread() pair, instead of the designed _endthreadex() (the latter would be called by the thread itself if the thread returned normally).

tl;dr Consider a thread which returns from its entry function vs. a thread which does something like Sleep(INFINITE) in the end of its function waiting to get terminated (i.e. after it has got the resources consistent and signalled to the terminating thread that it's ready). Do some CRT or C++11 resources (like thread_local) etc. leak or get corrupted etc. if _endthreadex() is not called, but TerminatThread() is called instead?

c++ multithreading c++11 winapi dll

shareimprove this question

edited Aug 31 '16 at 6:17

asked Aug 31 '16 at 5:27

转存失败重新上传取消

Serge Rogatch

6,89222 gold badges3737 silver badges7171 bronze badges

• 2

  You can use CreateThread perfectly well. The docs advising to use _behindthreadex are out of date. If you can't use CRT in a thread created by CreateThread then the system threadpool would be off limits. It isn't. – David Heffernan Aug 31 '16 at 6:00

• 1

  stackoverflow.com/a/2101036 – David Heffernan Aug 31 '16 at 6:47

• "I need to launch threads from initialization" - absolute safe and no problem create threads from DllMain - here no any deadlock. you cannot WAIT for this thread start/do some job/exit - this (wait) already cause deadlock – RbMm Aug 31 '16 at 7:06

• That won't work. The FlsCallback that is set by the CRT is never invoked when TerminateThread is called. So you will leak thread-local data. – rustyx Aug 31 '16 at 10:27

• 1

  @RemyLebeau: you're thinking of the "process exit" case. DLL_PROCESS_DETACH is also called if the DLL is explicitly unloaded, and in that case the threads may still be running. – Harry Johnston Aug 31 '16 at 21:09

show 2 more comments

1 Answer

activeoldestvotes

6

 

OK. First, let's cover a few minor points:

• As David mentions in the comments, you don't need to use _beginthreadex() rather than CreateThread(). Similarly, it is fine to use ExitThread() or similar instead of _endthreadex() on any currently supported version of Visual Studio and Windows.

• Despite what that MSDN article says, the accepted wisdom is that it is never OK to use TerminateThread().

• It is also generally accepted that, provided you understand the restrictions implied by the loader lock, it is OK to use CreateThread() in DllMain's DLL_PROCESS_ATTACH processing. However, if you are able to use a proper initialization routine rather than DllMain, as in your case, so much the better.

So, if I've understood your situation correctly, it can be summarized as follows:

• Your DLL needs one or more background threads.

• The executable unloads your DLL without warning.

That's kind of stupid, but that's not your fault. Luckily it isn't impossible to deal with.

If it is acceptable for the thread(s) to continue running after the executable thinks it has unloaded your DLL, you can use the FreeLibraryAndExitThread() pattern. In your initialization function, and wherever else you create a thread, call GetModuleHandleEx() to increment the DLL reference count. That way, when the executable calls FreeLibrary() the DLL won't actually be unloaded if any of the threads are still running. The threads exit by calling FreeLibraryAndExitThread(), maintaining the reference count.

This approach might not directly meet your needs, however, because it doesn't allow you to detect when the executable has unloaded the library so that you can signal the thread(s) to terminate.

There may be more clever solutions, but I'd suggest using a helper DLL. The idea is that the helper DLL rather than your primary DLL keeps track of the thread reference count, i.e., you load the helper DLL each time you create a background thread, and unload it each time a background thread exits. The helper DLL need only contain a single function which calls SetEvent() and then FreeLibraryAndExitThread().

When a background thread is notified that the DLL is being unloaded, it cleans up, and then calls the helper DLL to set an event and exit the thread. Once the event is set, your main DLL's detach routine knows that the thread is no longer running code from the main DLL. Once every background thread has finished cleanup, it is safe for the main DLL to unload - it doesn't matter that the threads are still running, because they are running code from the helper DLL, not the main DLL. The helper DLL in turn will unload automatically once the last thread calls FreeLibraryAndExitThread().

---

Looking at this again, a year or so later, it would probably be safer to reverse that: have the main DLL contain nothing but the initialization function and whatever other functions the program is calling, plus a DllMain that signals the background threads to exit, and have a secondary DLL that contains everything else.

In particular, if the secondary DLL contains all the code your background threads need, then it is safe for the primary DLL to unload while the background threads are still running.

The advantage of this variant is that when your background threads see the signal to exit, it doesn't matter if the primary DLL has already unloaded, so your DllMain function doesn't have to wait while holding the loader lock. That way, the process won't deadlock if one of the background threads inadvertently does something that requires the loader lock.

---

As a variant on the same idea, if you really really don't want to use FreeLibraryAndExitThread() on your CRT threads, you could instead have an extra thread living in the secondary DLL that would coordinate the unloading. This thread would be started with CreateThread() and wouldn't use any CRT functions, so that it is unquestionably safe for it to exit via FreeLibraryAndExitThread(). Its only responsibility would be to wait for all the other threads to exit before unloading the secondary DLL.

It isn't really necessary to distinguish between CRT and non-CRT threads any more, but if you want to strictly follow the rules-as-documented this would be one way of doing it.

 

WaitForSingleObject for thread object does not work in DLL unload

0

 

I've stumbled upon an unexpected behavior of Windows thread mechanism when DLL is unloaded. A have a pack of worker thread objects and I'm trying to finish them graciously when DLL is unloaded (via DllMain DLL_PROCESS_DETACH). The code is very simple (I do send an event to finish the thread's wait loop):

WaitForSingleObject( ThrHandle, INFINITE );
CloseHandle( ThrHandle );

Yet the WaitForSingleObject hangs the whole thing. It works fine if I perform it before DLL is unloaded. How this behavior can be fixed?

c++ multithreading winapi dll

shareimprove this question

asked Mar 6 '16 at 14:22

转存失败重新上传取消

aleksv

4766 bronze badges

• 3

  Deadlock inside DllMain() is a very common problem. If this code you are waiting on does anything to get another DLL unloaded then the loader lock will always hang your program. Use the debugger to diagnose this. Something you should already have done to document this question, like posting the call stack of that thread. The long and short of it is that waiting for anything in DllMain() is dangerous and should always be avoided. – Hans Passant Mar 6 '16 at 14:28 

• No, deadlock is impossible. All the worker thread does is wait for an event. WaitForSingleObject( Event, INFINITE ); Which I send before calling the WaitForSingleObject( ThrHandle, INFINITE ) – aleksv Mar 6 '16 at 14:32

• 2

  Sounds like you don't know what "loader lock" means, you are waiting for two sync objects. If they don't get signaled in the right order then you deadlock. Googles well. – Hans Passant Mar 6 '16 at 14:33 

• 1

  Well, of course not, the loader lock is not held. Insisting that you don't have a problem doesn't help us help you, post that stack trace. – Hans Passant Mar 6 '16 at 14:42

• 6

  DllMain calls are serialized. When the worker thread exits, it tries to send the DLL_THREAD_DETACH notification, but it can't do that until your DLL_PROCESS_DETACH completes. Hence the deadlock. You need to invent a function in your DLL for the app to call before it unloads the DLL. That function can shut down the worker threads. – Raymond Chen Mar 6 '16 at 16:52

show 5 more comments

 

1 Answer

activeoldestvotes

11

 

You can't wait for a thread to exit in DllMain(). Unless the thread had already exited by the time the DLL_PROCESS_DETACH was received, doing so will always deadlock. This is the expected behaviour.

The reason for this is that calls to DllMain() are serialized, via the loader lock. When ExitThread() is called, it claims the loader lock so that it can call DllMain() with DLL_THREAD_DETACH. Until that call has finished, the thread is still running.

So DllMain is waiting for the thread to exit, and the thread is waiting for DllMain to exit, a classic deadlock situation.

See also Dynamic-Link Library Best Practices on MSDN.

The solution is to add a new function to your DLL for the application to call before unloading the DLL. As you have noted, your code already works perfectly well when called explicitly.

---

In the case where backwards compatibility requirements make adding such a function impossible, and if you must have the worker threads, consider splitting your DLL into two parts, one of which is dynamically loaded by the other. The dynamically loaded part would contain (at a minimum) all of the code needed by the worker threads.

When the DLL that was loaded by the application itself receives DLL_PROCESS_DETACH, you just set the event to signal the threads to exit and then return immediately. One of the threads would have to be designated to wait for all the others and then free the second DLL, which you can do safely using FreeLibraryAndExitThread().

(Depending on the circumstances, and in particular if worker threads are exiting and/or new ones being created as part of regular operations, you may need to be very careful to avoid race conditions and/or deadlocks; this would likely be simpler if you used a thread pool and callbacks rather than creating worker threads manually.)

---

In the special case where the threads do not need to use any but the very simplest Windows APIs, it might be possible to use a thread pool and work callbacks to avoid the need for a second DLL. Once the callbacks have exited, which you can check using WaitForThreadpoolWorkCallbacks(), it is safe for the library to be unloaded - you do not need to wait for the threads themselves to exit.

The catch here is that the callbacks must avoid any Windows APIs that might take the loader lock. It is not documented which API calls are safe in this respect, and it varies between different versions of Windows. If you are calling anything more complicated than SetEvent or WriteFile, say, or if you are using a library rather than native Windows API functions, you must not use this approach.

 

What is the point of FreeLibraryAndExitThread?

The Free­Library­And­Exit­Thread
function seems pointless.
I mean, all the function does is

DECLSPEC_NORETURN
void WINAPI FreeLibraryAndExitThread(
    HMODULE hLibModule,
    DWORD dwExitCode)
{
    FreeLibrary(hLibModule);
    ExitThread(dwExitCode);
}

Who needs such a trivial function?
If I wanted to do that, I could just write it myself.

DWORD CALLBACK MyThreadProc(void *lpParameter)
{
    ... blah blah blah ...
    // FreeLibraryAndExitThread(g_hinstSelf, 0);
    FreeLibrary(g_hinstSelf);
    ExitThread(0);
}

And then you discover that occasionally your program crashes.
What’s going on?

Let’s rewind and look at the original problem.

Originally, you had code that did something like this:

DWORD CALLBACK SomethingThreadProc(void *lpParameter)
{
 ... do something ...
 return 0;
}
void DoSomethingInTheBackground()
{
 DWORD dwThreadId;
 HANDLE hThread = CreateThread(nullptr, 0, SomethingThreadProc,
                  nullptr, 0, &dwThreadId);
 if (hThread) CloseHandle(hThread);
}

This worked great, until somebody did this to your DLL:

HMODULE hmodDll = LoadLibrary(TEXT("awesome.dll"));
if (hmodDll) {
 auto pfn = reinterpret_cast<decltype(DoSomethingInTheBackground)*>
            (GetProcAddress(hmodDll, "DoSomethingInTheBackground"));
 if (pfn) pfn();
 FreeLibrary(hmodDll);
}

This code fragment calls your
Do­Something­In­The­Background
function and then immediately unloads the DLL,
presumably because all they wanted to do was call that one function.

Now you have a problem:
That
Free­Library
call frees your DLL,
while your
Something­Thread­Proc is still running!
Result:
A crash at an address where there is no code.
Older debuggers reported this as a crash in 〈unknown〉;
newer ones can dig into the recently-unloaded modules list
and report it as a crash in
awesome_unloaded.

This is a very common class of error.
When I helped out the application compatibility team

by looking at crashes in third-party code,
the majority of the crashes I looked at in Internet Explorer
were of this sort,
where a plug-in got unloaded while it still had a running thread.

How do you prevent your DLL from being unloaded while you still
have code running (or have registered callbacks)?
You perform a bonus Load­Library on yourself,
thereby bumping your DLL reference count by one.

If you don’t need to support Windows 2000,
you can use the new Get­Module­Handle­Ex function,
which is much more convenient and probably a lot faster, too.

BOOL IncrementDLLReferenceCount(HINSTANCE hinst)
{
 HMODULE hmod;
 return GetModuleHandleEx(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS,
                          reinterpret_cast<LPCTSTR>(hinst),
                          &hmod);
}

Bumping the DLL reference count means that when the original person
who called Load­Library finally calls
Free­Library,
your DLL will still remain in memory because the reference count
has not yet dropped all the way to zero because you have taken
a reference to the DLL yourself.

When you unregister your callback or your background thread finishes,
you call
Free­Library to release your reference to the DLL,
and if that’s the last reference, then the DLL will be unloaded.

But wait, now we have a problem.
When you call
Free­Library to release your reference to the DLL,
that call might end up unloading the code that is making the call.
When the call returns, there is no more code there.
This most commonly happens when you are calling
Free­Library on yourself and that was the last reference.
In rarer circumstances, it happens indirectly through a
chain of final references.

Let’s walk through that scenario again, since understanding it is central
to solving the problem.

1. Some application calls Load­Library on your DLL.
   The reference count on your DLL is now 1.

    

2. The application calls a function in your DLL that uses a background
   thread.

    

3. Your DLL prepares for the background thread by doing a
   Get­Module­Handle­Ex on itself,
   to avoid a premature unload.
   The reference count on your DLL is now 2.

    

4. Your DLL starts the background thread.
5. The application decides that it doesn’t need your DLL any more,
   so it calls Free­Library.
   The reference count on your DLL is now 1.

    

6. Your DLL background thread finishes its main work.
   The thread procedure ends with the lines

    

       FreeLibrary(g_hinstSelf);
       return 0;
   

7. The thread procedure calls
   Free­Library(g_hinst­Self)
   to drop its reference count.

    

8. The
   Free­Library function frees your DLL.

    

9. The
   Free­Library function returns to its caller,
   namely your thread procedure.

    

10. Crash, because your thread procedure was unloaded!

This is why you need
Free­Library­And­Exit­Thread:
So that the return address of the Free­Library
is not in code that’s being unloaded by the
Free­Library itself.

Change the last two lines of the thread procedure to
Free­Library­AndExit­Thread(g_hinstSelf, 0);
and watch what happens.
The first five steps are the same, and then we take a turn:

1. Your DLL background thread finishes its main work.
   The thread procedure ends with a call to

    

       FreeLibraryAndExitThread(g_hinstSelf, 0);
   

2. The
   Free­Library­And­Exit­Thread
   function calls
   Free­Library(g_hinst­Self).

    

3. The
   Free­Library function frees your DLL.

    

4. The
   Free­Library function returns to its caller,
   which is not your thread procedure but rather the
   Free­Library­And­Exit­Thread
   function,
   which was not unloaded.

    

5. The
   Free­Library­And­Exit­Thread
   function calls Exit­Thread(0).

    

6. The thread exits and no further code is executed.

That’s why the
Free­Library­And­Exit­Thread
function exists:
So you don’t pull the rug out from underneath yourself.
Instead, you have somebody else pull the rug for you.

This issue of keeping your DLL from unloading prematurely
rears its head in several ways.
We’ll look at some of them in the next few days.

Bonus chatter:
The thread pool version of
Free­Library­And­Exit­Thread
is
Free­Library­When­Callback­Returns.