Xenstore的监视(watch)功能很实用,在xenstore监视目标文件夹里发生的任何修改,都会通知watch的注册者。xen虚拟机的后端驱动程序,就是通过watch来检测前端设备的改变。
需要注意的:
(1)注册watch不需要开始一个transaction,只要用xs_open打开连接就行了。内核可以直接调用register_xenbus_watch
(2)watch在注册时,xenstored会马上产生一个事件。它就是这么设计的。如果不想处理这个时间,需要设法把它忽略掉。
watch的用法:
内核空间:(转自http://wiki.xen.org/xenwiki/XenBus)
static struct xenbus_watch xb_watch = {
.node = "memory",
.callback = watch_target;
};
ret = register_xenbus_watch(&xb_watch);
if(IS_ERR(ret)) {
IPRINTK("Failed to initialize balloon watcher\n");
} else {
IPRINTK("Balloon xenbus watcher initialized\n");
}
用户空间:用户空间和内核稍有不同。用户空间可以用xs_watch函数定义一个监视,但是不能指定回调函数,而是需要调用xs_read_watch进行阻塞式的读取,像这样:
xs_handle *xh=xs_open(0);
assert(xs_watch(xh, path, token));
char **result = 0;
result = xs_read_watch(xh, &num); //会阻塞
free(result);
xs_unwatch(xh, path, token);
如果不想阻塞,就得自己开一个线程用来监听。但是这里有一个bug(4.1.3版本xen):xs_read_watch不是线程安全的。如果在xs_read_watch阻塞的时候,使用pthread_cancel来终止线程,xs_read_watch不会释放所有的资源(锁)。而主线程如果调用xs_close关闭与xenstore的连接,则会因为不能得到这些资源而死锁。具体分析如下:
tools/xenstore/xs.c
char **xs_read_watch(struct xs_handle *h, unsigned int *num)
{
struct xs_stored_msg *msg;
char **ret, *strings, c = 0;
unsigned int num_strings, i;
mutex_lock(&h->watch_mutex); //------------------>加锁
//此处应该加一个pthread_cleanup_push(pthread_mutex_unlock, &h->watch_mutex)
#ifdef USE_PTHREAD
/* Wait on the condition variable for a watch to fire.
* If the reader thread doesn't exist yet, then that's because
* we haven't called xs_watch. Presumably the application
* will do so later; in the meantime we just block.
*/
while (list_empty(&h->watch_list) && h->fd != -1)
condvar_wait(&h->watch_condvar, &h->watch_mutex);
#else /* !defined(USE_PTHREAD) */
/* Read from comms channel ourselves if there are no threads
* and therefore no reader thread. */
assert(!read_thread_exists(h)); /* not threadsafe but worth a check */
if ((read_message(h) == -1)) //----------------------->没有消息时阻塞在这里
return NULL;
xs_read_watch一开始加锁,紧接着调用read_message。read_message会阻塞调用read函数。如果这时主线程调用pthread_cancel来取消运行xs_read_watch的线程,h->watch_mutex就不会被释放,会造成后面的死锁。
很挫的解决办法:在调用xs_unwatch和xs_close之前,检查xs_handle内部锁的情况,强行解锁资源。
xs_handle这个结构体在头文件里只有类型声明,具体的定义在xs.c文件里,仅供xenstore内部使用。要调用xs_handle内部的对象,首先得把xs_handle的详细声明提取出来:
//used to extract (>_<) xs_handle internal members.
typedef struct list_head {
struct list_head *next, *prev;
}list_head_struct;
typedef struct
{
int fd;
pthread_t read_thr;
int read_thr_exists;
struct list_head watch_list;
pthread_mutex_t watch_mutex;//监视信号量,可能死锁
pthread_cond_t watch_condvar;
int watch_pipe[2];
struct list_head reply_list;
pthread_mutex_t reply_mutex;//回复信号量
pthread_cond_t reply_condvar;
pthread_mutex_t request_mutex;//请求信号量
}my_xs_handle;
#if __GNUC__ > 3
#define offsetof(a,b) __builtin_offsetof(a,b)//提取偏移量的宏。
#else
#define offsetof(a,b) ((unsigned long)&(((a *)0)->b))
#endif
然后像这样提取xh内部成员:
pthread_mutex_t *pm_watch = (pthread_mutex_t *)(((void *)xh) + offsetof(my_xs_handle, watch_mutex));
解锁的宏。其实写成个函数也行
#define check_xh_lock(x) do {\
pthread_mutex_t *pm = (pthread_mutex_t *)(((void *)pthis->xh) + offsetof(my_xs_handle, x)); \
if (pthread_mutex_trylock(pm) == EBUSY){ \ //pthread_mutex_trylock 如果没上锁,则加锁;否则返回EBUSY
cout << "thread_cleanup -> " #x " is already locked!" << endl; \
if (0 != pthread_mutex_unlock(pm)) \
cout << "thread_cleanup -> error unlocking!" << endl; \
else cout << "thread_cleanup -> unlocking " #x << endl; \
} else assert(pthread_mutex_unlock(pm)==0); \
} while (0)
check_xh_lock(watch_mutex);
check_xh_lock(request_mutex);
check_xh_lock(reply_mutex);
cout << "----- unwatch -----" << endl;
xs_unwatch(pthis->xh, pthis->path.c_str(), map_path("watch", pthis->path).c_str());
check_xh_lock(watch_mutex);
check_xh_lock(request_mutex);
check_xh_lock(reply_mutex);
cout << "----- close -----" << endl;
xs_close(pthis->xh);
pthis->xh=0;
测试如下:
viktor@buxiang-OptiPlex-330:~/proj/xc_map$ sudo ./domu domu.cpp
watch -> add watch on path mmap/domu-cpp callback func 0x8049aad //主线程打开监视线程
thread_func -> begin watch thread path= mmap/domu-cpp //监视线程注册watch
thread_func -> watch event path= mmap/domu-cpp token= watch/mmap/domu-cpp //注册时会发生一次事件,忽略之
watch_callback -> entering
//(在这里等待,此时按下Ctrl+C)
^Cmain received signal Interrupt
unwatch -> **stopping work thread. waiting here...work thread already stopped. //主线程用pthread_cancel关闭监视线程
thread_cleanup -> path= mmap/domu-cpp thread=b6d6eb70 //监视线程退出,主线程开始清理
----- unwatch -----
map_path -> watch/mmap/domu-cpp
thread_cleanup -> watch_mutex is already locked! //此时watch_mutex上锁了,说明xs_read_watch没有释放资源
thread_cleanup -> unlocking watch_mutex //强行解锁
----- close ----- //调用xs_unwatch和xs_close关闭连接。