something talk about lock from FreeBSD mail list

On Sat, 11 Apr 2009, Andrew Brampton wrote:
I'm having a problem with memguard(9) on FreeBSD 7.1 but before I ask about that I just need to check my facts about malloc.
When in interrupt context malloc must be called with M_NOWAIT, this is because I can't sleep inside a interrupt. Now when I hold a spinlock (MTX_SPIN) I am also not allowed to sleep or obtain a sleepable mutex (such as MTX_DEF). So I assume while holding a spin lock any mallocs I do must have the M_NOWAIT flag? This was not clear from the manual pages, but at least makes sense to me.
So my problem with memguard stems from the fact that I am locking a spinlock, and then I'm calling malloc with M_NOWAIT. But inside memguard_alloc a MTX_DEF is acquired causing WITNESS to panic.
So I think fundamental memguard is flawed and should be using MTX_SPIN instead of MTX_DEF otherwise it can't be called from inside a interrupt or when a spin lock is held. But maybe I'm missing something?
Also on a related note, I see that MTX_SPIN disables interrupts, making it a rather "heavy" spinlock. Is there a lighter spin lock that literally just spins? I read that MTX_DEF are far quicker to acquire , but surely a light spinlock would be easier to acquire than sleeping?
I think for the moment I will fix my code by not using a MTX_SPIN (since the code is not in a interrupt), however, I think memguard should change its lock.


Your understanding is mostly right. The missing bit is this: there are two kinds of interrupt contexts -- fast/filter interrupt handlers, which borrow the stack and execution context of the kernel thread they preempt, and interrupt threads, which get their own complete thread context.
Fast interrupt handlers are allowed unlock to acquire spinlocks so as to avoid deadlock because of the borrowed context. This means they can't perform any sort of sleep, or acquire any locks that might sleep, since the thread they've preempted may hold conflicting locks, or be the one that would have woken up the sleep that the handler performed. Almost no code will run in fast handlers -- perhaps checking some device registers, doing work on a lockless or spinlock-protected queue, and waking up a worker thread.
This is why, BTW, spin locks disable interrupt: they need to control preemption by other interrupt handlers to avoid deadlock, but they are not intended for use except when either in the scheduler, in a few related IPI contexts, or when synchronizing between normal kernel code and a fast handler.
Full interrupt thread contexts are permitted to perform short lock sleeps, such as those performed when contending default mutexes, rwlocks, and rmlocks. They are permitted to invoke kernel services such as malloc(9), UMA(9), the network stack, etc, as long as they use M_NOWAIT and don't invoke msleep(9) or similar unbounded sleeps -- again to avoid the possibility of deadlocks, since you don't want an interrupt thread sleeping waiting for an event that only it can satisfy.
So the first question, really, is whether you are or mean to be using fast/filter interrupt handler. Device drivers will never call memory allocation, free, etc, from there, but will defer it to an ithread using the filter mechanism in 8.x, or to a task queue or other worker in 7.x and earlier. If you're using a regular INTR_MPSAFE ithread, you should be able to use only default mutexes (a single atomic operation if uncontended) without disabling interrupts, etc.


Robert N M WatsonComputer LaboratoryUniversity of Cambridge


On Sat, 11 Apr 2009, Andrew Brampton wrote:
Thanks very much for your detailed reply. I'm slowly understanding how everything inFreeBSD fits together, and I appreciate your help.
I've been given a project to take over, and all of the design decisions were made before I started working on it, thus I'm playing catch up. One of the decisions was to implement their own version of a spin lock, which literally looks something like this:
lock_aquire() { critical_enter(); while (! lockHeld ) {} lockHeld++;}
This was actually the code tripping up MemGuard, as it is inside a critical section, which MemGuard is unable to sleep within. This is all running inside a kthread_create thread (I'm unsure of the proper name of this type of thread).
Anyway, that is why I also asked about a lighter weight spin lock (perhaps similar to this one). I tempted to replace this custom spinlock with the standard MTX_DEF, however I'm unsure of its impact. The custom spin lock seems quick and light to acquire, and it does not concern me that a interrupt can potentially interrupt the code.
On a related note, if you change the lock in memguard to a MTX_SPIN, it panics the kernel during boot. So that is not an option :) I was only using memguard because I suspected memory being used after it was freed. However, I think I will either change my locks to MTX_DEF or live without memguard.
I realise I've not really asked any questions, but I would be grateful for any insights anyone may have. Andrew


My advice, unless you're definitely executing code in fast interrupt contexts, is to simply use theFreeBSD default mutex primitive instead of either a custom-build spinlock or aFreeBSD MTX_SPIN mutex. The default mutex is adaptive, and will spin when contending the lock unless the thread holding the lock isn't executing, in which case it will fall back on a context switch. Our mutexes also make correct use of memory barriers, which the above example code doesn't appear to, so will work on systems that have weaker memory ordering properties. Using the default mutex scheme also allows you to take advantage of WITNESS, our lock order verifier, which proves a really useful tool when a lot of locks are in flight.
The critical sections you're using above may not have the effect you intend: they prevent preemption by another thread, and they prevent migration to another CPU, but they don't prevent fast interrupt handlers from executing. Any synchronization with a fast interrupt handler needs to be done either using spinlocks, or other atomic operations.
Robert N M WatsonComputer LaboratoryUniversity of Cambridge