call_rcu and synchronize_rcu with unloadable modules

转载地址：http://lwn.net/Articles/217484/

RCU (read-copy update) is a synchronization mechanism that can be thoughtof as a replacement for read-writer locking (among other things),but with very low-overhead readers that are immune to deadlock,priority inversion, and unbounded latency.RCU read-side critical sections are delimited by rcu_read_lock()and rcu_read_unlock(), which, in non-CONFIG_PREEMPTkernels, generate no code whatsoever.

This means that RCU writers are unaware of the presence of concurrentreaders, so that RCU updates to shared data must be undertaken quitecarefully, leaving an old version of the data structure in placeuntil all pre-existing readers have finished.These old versions are needed becausesuch readers might hold a reference to them.RCU updates can therefore be rather expensive, and RCU is thus bestsuited for read-mostly situations.

How can an RCU writer possibly determine when all readers are finished, giventhat readers might well leave absolutely no trace of their presence?There is a synchronize_rcu() primitive that blocks untilall pre-existing readers have completed.An updater wishing to delete an element p from a linkedlist might do the following, while holding an appropriate lock,of course:

list_del_rcu(p);synchronize_rcu();kfree(p); But the above code cannot be used in IRQ context -- the call_rcu() primitive must be used instead.This primitive takes a pointer to an rcu_head struct placedwithin the RCU-protected data structure and another pointerto a function that may be invoked later to free that structure.Code to delete an element p from the linked list from IRQcontext might then be as follows: list_del_rcu(p);call_rcu(&p->rcu, p_callback); Since call_rcu() never blocks, this code can safely be usedfrom within IRQ context.The function p_callback() might be defined as follows: static void p_callback(struct rcu_head *rp){struct pstruct *p = container_of(rp, struct pstruct, rcu);kfree(p);}

Unloading Modules That Use call_rcu()

But what if p_callback is defined in an unloadable module?

If we unload the module while some RCU callbacks are pending, the CPUsexecuting thesecallbacks are going to be severely disappointed when they are later invoked,as fancifully depicted on the right.

We could try placing a synchronize_rcu() in the module-exitcode path, but this is not sufficient.Although synchronize_rcu() does wait for a grace periodto elapse, it does not wait for the callbacks to complete.

One might be tempted to try several back-to-backsynchronize_rcu() calls, but this is still not guaranteed to work.If there is a very heavy RCU-callback load, then some of the callbacks mightbe deferred in order to allow other processing to proceed.Such deferral is required in realtime kernels in order to avoid excessivescheduling latencies.

rcu_barrier()

We instead need the rcu_barrier() primitive.This primitive is similar to synchronize_rcu(), but instead ofwaiting solely for a grace period to elapse, it also waits forall outstanding RCU callbacks to complete.Pseudo-code using rcu_barrier() is as follows:

1. Prevent any new RCU callbacks from being posted.
2. Execute rcu_barrier().
3. Allow the module to be unloaded.

Quick Quiz #1: Why is there no srcu_barrier()?

Quick Quiz #2: Why is there no rcu_barrier_bh()?

The rcutorture module makes use of rcu_barrier in its exit functionas follows:

1 static void 2 rcu_torture_cleanup(void) 3 { 4 int i; 5 6 fullstop = 1; 7 if (shuffler_task != NULL) { 8 VERBOSE_PRINTK_STRING("Stopping rcu_torture_shuffle task"); 9 kthread_stop(shuffler_task); 10 } 11 shuffler_task = NULL; 12 13 if (writer_task != NULL) { 14 VERBOSE_PRINTK_STRING("Stopping rcu_torture_writer task"); 15 kthread_stop(writer_task); 16 } 17 writer_task = NULL; 18 19 if (reader_tasks != NULL) { 20 for (i = 0; i < nrealreaders; i++) { 21 if (reader_tasks[i] != NULL) { 22 VERBOSE_PRINTK_STRING( 23 "Stopping rcu_torture_reader task"); 24 kthread_stop(reader_tasks[i]); 25 } 26 reader_tasks[i] = NULL; 27 } 28 kfree(reader_tasks); 29 reader_tasks = NULL; 30 } 31 rcu_torture_current = NULL; 32 33 if (fakewriter_tasks != NULL) { 34 for (i = 0; i < nfakewriters; i++) { 35 if (fakewriter_tasks[i] != NULL) { 36 VERBOSE_PRINTK_STRING( 37 "Stopping rcu_torture_fakewriter task"); 38 kthread_stop(fakewriter_tasks[i]); 39 } 40 fakewriter_tasks[i] = NULL; 41 } 42 kfree(fakewriter_tasks); 43 fakewriter_tasks = NULL; 44 } 45 46 if (stats_task != NULL) { 47 VERBOSE_PRINTK_STRING("Stopping rcu_torture_stats task"); 48 kthread_stop(stats_task); 49 } 50 stats_task = NULL; 51 52 /* Wait for all RCU callbacks to fire. */ 53 rcu_barrier(); 54 55 rcu_torture_stats_print(); /* -After- the stats thread is stopped! */ 56 57 if (cur_ops->cleanup != NULL) 58 cur_ops->cleanup(); 59 if (atomic_read(&n_rcu_torture_error)) 60 rcu_torture_print_module_parms("End of test: FAILURE"); 61 else 62 rcu_torture_print_module_parms("End of test: SUCCESS"); 63 } Line 6 sets a global variable that prevents any RCU callbacks fromre-posting themselves.This will not be necessary in most cases, since RCU callbacks rarelyinclude calls to call_rcu().However, the rcutorture module is an exception to this rule, andtherefore needs to set this global variable.

Lines 7-50 stop all the kernel tasks associated with thercutorture module.Therefore, once execution reaches line 53, no more rcutortureRCU callbacks will be posted.The rcu_barrier() call on line 53 waits for any pre-existingcallbacks to complete.

Then lines 55-62 print status and do operation-specific cleanup,and then return, permitting the module-unload operation to be completed.

Quick Quiz #3: Is there any other situation wherercu_barrier() might be required?

Your module might have additional complications.For example, if your module invokes call_rcu() fromtimers, you will need to first cancel all the timers, and onlythen invoke rcu_barrier() to wait for any remainingRCU callbacks to complete.

Implementing rcu_barrier()

Dipankar Sarma's implementation of rcu_barrier()makes use of the fact that RCU callbacks are never reordered oncequeued on one of the per-CPU queues.His implementation queues an RCU callback on each ofthe per-CPU callback queues, and then waits until they have allstarted executing, at which point, all earlier RCU callbacks areguaranteed to have completed.

The code for rcu_barrier() is as follows:

1 void rcu_barrier(void) 2 { 3 BUG_ON(in_interrupt()); 4 /* Take cpucontrol mutex to protect against CPU hotplug */ 5 mutex_lock(&rcu_barrier_mutex); 6 init_completion(&rcu_barrier_completion); 7 atomic_set(&rcu_barrier_cpu_count, 0); 8 on_each_cpu(rcu_barrier_func, NULL, 0, 1); 9 wait_for_completion(&rcu_barrier_completion); 10 mutex_unlock(&rcu_barrier_mutex); 11 } Line 3 verifies that the caller is in process context, andlines 5 and 10 use rcu_barrier_mutex to ensure thatonly one rcu_barrier() is using the global completionand counters at a time, which are initialized on lines 6 and 7.Line 8 causes each CPU to invoke rcu_barrier_func(),which is shown below.Note that the final "1" in on_each_cpu()'s argument listensures that all the calls to rcu_barrier_func() willhave completed before on_each_cpu() returns.Line 9 then waits for the completion.

The rcu_barrier_func() runs on each CPU, where it invokescall_rcu() to post an RCU callback, as follows:

1 static void rcu_barrier_func(void *notused) 2 { 3 int cpu = smp_processor_id(); 4 struct rcu_data *rdp = &per_cpu(rcu_data, cpu); 5 struct rcu_head *head; 6 7 head = &rdp->barrier; 8 atomic_inc(&rcu_barrier_cpu_count); 9 call_rcu(head, rcu_barrier_callback); 10 } Lines 3 and 4 locate RCU's internal per-CPU rcu_data structure,which contains the struct rcu_head that needed for the latercall to call_rcu().Line 7 picks up a pointer to this struct rcu_head, and line 8increments a global counter.This counter will later be decremented by the callback.Line 9 then registers the rcu_barrier_callback()on the current CPU's queue.

The rcu_barrier_callback() function simply atomicallydecrements the rcu_barrier_cpu_count variable and finalizesthe completion when it reaches zero, as follows:

1 static void rcu_barrier_callback(struct rcu_head *notused) 2 { 3 if (atomic_dec_and_test(&rcu_barrier_cpu_count)) 4 complete(&rcu_barrier_completion); 5 }

Quick Quiz #4: What happens if CPU 0's rcu_barrier_func()executes immediately (thus incrementing rcu_barrier_cpu_countto the value one), but the other CPU's rcu_barrier_func()invocations are delayed for a full grace period?Couldn't this result in rcu_barrier() returning prematurely?

rcu_barrier() Summary

The rcu_barrier() primitive has seen relatively little use,since most code using RCU is in the core kernel rather than in modules.However, if you are using RCU from an unloadable module, you need to usercu_barrier() so that your module may be safely unloaded.

Answers to Quick Quizzes

Quick Quiz #1: Why is there no srcu_barrier()?

Since there is no call_srcu(), there can be no outstandingSRCU callbacks.Therefore, there is no need to wait for them.

Quick Quiz #2: Why is there no rcu_barrier_bh()?

Because no one has needed it yet.As soon as someone needs to use call_rcu_bh() from within anunloadable module, they will need an rcu_barrier_bh().

Quick Quiz #3: Is there any other situation wherercu_barrier() might be required?

Interestingly enough, rcu_barrier() was not originallyimplemented for module unloading.Nikita Danilov was using RCU in a filesystem, which resulted in asimilar situation at filesystem-unmount time.Dipankar Sarma coded up rcu_barrier() in response, so thatNikita could invoke it during the filesystem-unmount process.

Much later, yours truly hit the RCU module-unload problemwhen implementing rcutorture, and found that rcu_barrier()solves this problem as well.

Quick Quiz #4: What happens if CPU 0's rcu_barrier_func()executes immediately (thus incrementing rcu_barrier_cpu_countto the value one), but the other CPU's rcu_barrier_func()invocations are delayed for a full grace period?Couldn't this result in rcu_barrier() returning prematurely?

This cannot happen.The reason is that on_each_cpu() has its last argument,the wait flag, set to "1".This flag is passed through to smp_call_function() andfurther to smp_call_function_on_cpu(), causing this latterto spin until the cross-CPU invocation of rcu_barrier_func() has completed.This by itself would prevent a grace period from completing onnon-CONFIG_PREEMPT kernels, since each CPU must undergo a context switch(or other quiescent state) before the grace period can complete.However, this is of no use in CONFIG_PREEMPT kernels.

Therefore, on_each_cpu() disables preemption across itscall to smp_call_function() and also across the localcall to rcu_barrier_func().This prevents the local CPU from context switching, again preventinggrace periods from completing.This means that all CPUs have executed rcu_barrier_func()before the first rcu_barrier_callback() can possibly execute,in turn preventing rcu_barrier_cpu_count from prematurelyreaching zero.

Currently, -rt implementations of RCU keep but a single global queuefor RCU callbacks, and thus do not suffer from this problem.However, when the -rt RCU eventually does have per-CPU callbackqueues, things will have to change.One simple change is to add an rcu_read_lock() before line 8of rcu_barrier() and an rcu_read_unlock() afterline 8 of this same function.If you can think of a better change, please let me know!