xen块设备体系结构(6)

blktap 续

blktap_device

blktap_device的结构很简单：

struct blktap_device {
spinlock_t                     lock;
struct gendisk                *gd;
};

其中struct gendisk结构是内核块设备结构block_device用到的通用disk结构

blktap_device_open

从内核的通用结构 block_device -> bd_disk 中得到硬盘数据结构 struct gendisk 。 从gendisk->private_data中得到 blktap_device

我们/dev/xen/blktap-2/tapdiskXXX 这些块设备用得就是 blktap_device 结构

blktap_device_release

通过传入的gendisk结构，得到blktap_device,  block_device,  blktap 等结构， 调用blktap_device_release之后，最后把blktap结构的 dev_inuse 位设置为BLKTAP_DEVICE_CLOSED, 调用 blktap_ring_kick_user ，wake_up blktap->ring设备里的 poll_wait 信号。

blktap_device_getgeo

返回struct hd_geometry，包含块设备head, cylinder, sector等信息

blktap_device_create

blktap 环设备 blktapXXX，当调用ioctl 并传入cmd为BLKTAP2_IOCTL_CREATE_DEVICE时，会调用 blktap_device_create 来创建tapdevXXX设备。

if (test_bit(BLKTAP_DEVICE, &tap->dev_inuse))

return -EEXIST;

if (blktap_device_validate_params(tap, params))
return -EINVAL;

gd = alloc_disk(1);
if (!gd) {
err = -ENOMEM;
goto fail;
}

if (minor < 26) {
sprintf(gd->disk_name, "td%c", 'a' + minor % 26);
} else if (minor < (26 + 1) * 26) {
sprintf(gd->disk_name, "td%c%c",
'a' + minor / 26 - 1,'a' + minor % 26);
} else {
const unsigned int m1 = (minor / 26 - 1) / 26 - 1;
const unsigned int m2 = (minor / 26 - 1) % 26;
const unsigned int m3 =  minor % 26;
sprintf(gd->disk_name, "td%c%c%c",
'a' + m1, 'a' + m2, 'a' + m3);
}

gd->major = blktap_device_major;
gd->first_minor = minor;
gd->fops = &blktap_device_file_operations;
gd->private_data = tapdev;

spin_lock_init(&tapdev->lock);
rq = blk_init_queue(blktap_device_do_request, &tapdev->lock);
if (!rq) {
err = -ENOMEM;
goto fail;
}
elevator_init(rq, "noop");

gd->queue     = rq;
rq->queuedata = tapdev;
tapdev->gd    = gd;

blktap_device_configure(tap, params);
add_disk(gd);

if (params->name[0])
strncpy(tap->name, params->name, sizeof(tap->name)-1);

set_bit(BLKTAP_DEVICE, &tap->dev_inuse);

dev_info(disk_to_dev(gd), "sector-size: %u capacity: %llu\n",
queue_logical_block_size(rq),
(unsigned long long)get_capacity(gd));

return 0;

test_bit 检查 tap设备是否在使用，如果已被使用报错退出。blktap_device_validate_params 检查blktap_params参数。比如sector size 不能 < 512 or > 4096，disk的capacity是否超过最大值 等。调用alloc_disk 创建一个gendisk结构，然后初始化这个gendisk结构，如下：

gd->major = blktap_device_major;
gd->first_minor = minor;
gd->fops = &blktap_device_file_operations;
gd->private_data = tapdev;

调用 blk_init_queue 初始化，关于blk_init_queue有如下描述

 * Description:
 *    If a block device wishes to use the standard request handling procedures,
 *    which sorts requests and coalesces adjacent requests, then it must
 *    call blk_init_queue().  The function @rfn will be called when there
 *    are requests on the queue that need to be processed.

调用 elevator_init 初始化 request_queue rq

调用 add_disk(gendisk *)，把struct gendisk 在内核注册

调用 blktap_device_configure，对tapdevXXX设备进行配置，其中blktap_params 参数由copy_from_user从user space得到：

    set_capacity： 设置gendisk 磁盘大小 = 传入的 capacity

    blk_queue_logical_block_size： set logical block size = 传入的sector_size

    blk_queue_max_sectors：max_sectors 最小为8， 最大为1024个sector。注意这里的sector大小是块驱动认为的固定大小 512 bytes

    blk_queue_segment_boundary / blk_queue_max_segment_size ： per segment 的 size是 4K

    blk_queue_max_phys_segments / blk_queue_max_hw_segments ： request_queue 每个 request 最多有11个segment，每个segment 4k，相当于8个sectors大小

    

blktap_device_destroy

blktapXXX设备执行ioctl, command为BLKTAP2_IOCTL_REMOVE_DEVICE时，执行blktap_device_destroy。

blktap_device_destroy会调用 blk_cleanup_queue，这是内核的通用函数

void blk_cleanup_queue(struct request_queue *q)
{
/*
* We know we have process context here, so we can be a little
* cautious and ensure that pending block actions on this device
* are done before moving on. Going into this function, we should
* not have processes doing IO to this device.
*/
blk_sync_queue(q);

mutex_lock(&q->sysfs_lock);
queue_flag_set_unlocked(QUEUE_FLAG_DEAD, q);
mutex_unlock(&q->sysfs_lock);

if (q->elevator)
elevator_exit(q->elevator);

blk_put_queue(q);
}

我们知道request_queue里的IO请求都是异步的，在关闭tapdevXXX 设备的时候，这些请求是需要进行清理的。这通过blk_sync_queue来实现。

/**
 * blk_sync_queue - cancel any pending callbacks on a queue
 * @q: the queue
 *
 * Description:
 *     The block layer may perform asynchronous callback activity
 *     on a queue, such as calling the unplug function after a timeout.
 *     A block device may call blk_sync_queue to ensure that any
 *     such activity is cancelled, thus allowing it to release resources
 *     that the callbacks might use. The caller must already have made sure
 *     that its ->make_request_fn will not re-add plugging prior to calling
 *     this function.
 *
 */
void blk_sync_queue(struct request_queue *q)
{
del_timer_sync(&q->unplug_timer);
del_timer_sync(&q->timeout);
cancel_work_sync(&q->unplug_work);
}

blk_sync_queue应该对于没有返回的IO请求，取消之前的注册行为，相当于discard这些请求了。

blktap_device_fail_queue

该函数调用 __blktap_next_queued_rq 遍历 request_queue，对每个请求调用 __blktap_end_queued_rq(rq, -EIO)

我们回顾下blktapXXX设备提供了如下操作

static struct file_operations blktap_ring_file_operations = {
.owner    = THIS_MODULE,
.open     = blktap_ring_open,
.release  = blktap_ring_release,
.ioctl    = blktap_ring_ioctl,
.mmap     = blktap_ring_mmap,
.poll     = blktap_ring_poll,
};

blktap_ring_poll

blktap_ring_poll 会调用 blktap_device_run_queue，里面又是一个循环，对request_queue里的所有request, 调用 blktap_device_make_request 。 

blktap_device_make_request 首先调用blktap_ring_make_request，生成 blktap_request 结构，然后调用 blktap_request_get_pages 为blktap_request 分配页框，最后调用 blktap_ring_submit_request 

blktap_device_do_request 是 tapdevXXX 块设备初始化函数 blk_init_queue 传入的函数指针。这个指针具体做什么的请参考内核块设备。blktap_device_do_request 调用了blktap_ring_kick_user，用来 wake_up 一个 blktap_ring->poll_wait 结构。还记得之前的blktap_ring_poll函数么，该函数调用 poll_wait(filp, &ring->poll_wait, wait) 一直阻塞在 poll_wait 这个wait_queue list 上。所以可以认为 blktap_ring_kick_user 用来唤醒 blktap_ring_poll 函数，把request_queue里的request submit上去。

blktap_ring_submit_request 把请求放到IO环里，下一步应该是tapdisk2 来处理这些IO请求了