Detours vs. Mhook
Detours is available for free with a noncommercial license but it only supports the x86 platform. Detours can also be licensed for commercial use which also gives you full x64 support, but you only get to see the licensing conditions after signing an NDA.
Mhook is freely distributed under an MIT license with support for x86 and x64.
Detours shies away from officially supporting the attachment of hooks to a running application. Of course, you are free to do it - but if you end up causing a random crash here or there, you can only blame yourself.
Mhook was meant to be able to set and remove hooks in running applications – after all, that’s what you need it for in the real world. It does its best to avoid overwriting code that might be under execution by another thread.
Detours supports transactional hooking and unhooking; that is, setting a bunch of hooks at the same time with an all-or-nothing approach. Hooks will only be set if all of them can be set, otherwise the library will roll back any changes made. Mhook does not do this.
Detours has a built-in x86 (and, when paid for, x64) disassembler so it can automatically hook an API. This is the fundamental difference between Detours and Mhook, and probably the only one that really needs improvement: Mhook has no disassembler so the user must first, by hand, examnine the first few bytes of the target API and make the resulting information available to Mhook. This also means that Mhook will not function on an OS where the disassembly of the target function’s first few bytes is different from what has been anticipated. It is possible to give Mhook information on several possible disassemblies at once, thereby supporting multiple operating systems, but this is a bit inconvenient. On the other hand, the lack of a disassembler allows the library to remain very lightweight.
Finally, Mhook is pretty wasteful when it comes to allocating memory for the trampolines it uses. Detours allocates blocks of memory as needed, and uses the resulting data area to store as many trampolines within as will fit. Mhook, on the other hand, uses one call to VirtualAlloc per hook being set. Every hook needs less than 100 bytes of storage so this is very wasteful, since VirtualAlloc ends up grabbing 64K from the process' virtual address space every time Mhook calls it. (Actual allocated memory will be a single page which is also quite wasteful.) In the end though, this probably does not really matter, unless you are setting a very large number of hooks in an application. Also, this is very easy to fix.
With that out of the way, if you’re still here, let’s delve into it.