libvirt和SanLock中的租约配置

最新推荐文章于 2024-01-11 16:58:46 发布

weixin_33973609

最新推荐文章于 2024-01-11 16:58:46 发布

阅读量181

点赞数

文章标签：运维 5g python

原文链接：https://my.oschina.net/LastRitter/blog/1540124

版权

2019独角兽企业重金招聘Python工程师标准>>>

在之前的测试中，sanlock的租约保存在NFS中（disk_lease_dir = "/var/lib/libvirt/sanlock"），并使用libvirt自动租约管理功能（auto_disk_leases=1），下面我们将尝试一些新的配置方法。

把租约直接保存在共享存储中（自动租约管理）

在libvirt与sanlock相关的配置和文档中并未找到可以把租约直接保存在共享存储中的方法，只有在使用自动租约管理时，可以配置租约位置的“disk_lease_dir”变量，不知是否可以通过修改这个配置为共享存储来自动把租约保存在共享存储上？如果可以，源码中必定会有相关的判断语句才对，因此进去探查一番。

分析源码

分析一下libvirt对“disk_lease_dir”变量的解析。

查看源码中对“disk_lease_dir”变量进行参考的位置：

$ tar xvf libvirt-1.3.4.tar.gz
$ cd libvirt-1.3.4/

$ grep -rHn disk_lease_dir
src/locking/test_libvirt_sanlock.aug.in:6:{ "disk_lease_dir" = "/var/lib/libvirt/sanlock" }
src/locking/lock_driver_sanlock.c:134:    p = virConfGetValue(conf, "disk_lease_dir");
src/locking/lock_driver_sanlock.c:135:    CHECK_TYPE("disk_lease_dir", VIR_CONF_STRING);
src/locking/libvirt_sanlock.aug:20:   let entry = str_entry "disk_lease_dir"
src/locking/sanlock.conf:34:#disk_lease_dir = "/var/lib/libvirt/sanlock"
src/locking/sanlock.conf:40:# at 'disk_lease_dir' *MUST* be given a different host ID.
src/locking/sanlock.conf:66:# content of disk_lease_dir) to make sure sanlock daemon can
ChangeLog:80123:	$ augtool get /files/etc/libvirt/qemu-sanlock.conf/disk_lease_dir
ChangeLog:80124:	/files/etc/libvirt/qemu-sanlock.conf/disk_lease_dir = /var/lib/libvirt/sanlock
tools/virt-sanlock-cleanup.in:29:LOCKDIR=`augtool get '/files@sysconfdir@/libvirt/qemu-sanlock.conf/disk_lease_dir'`

凭经验可以推测出真正解析“disk_lease_dir”变量的位置可能就在lock_driver_sanlock.c中，进去查看一下：

$ vi src/locking/lock_driver_sanlock.c

可以通过“S_ISDIR(st.st_mode)”这判断语句初步推断“disk_lease_dir”变量必须是目录，否则会直接报错。

...

static int virLockManagerSanlockLoadConfig(const char *configFile)
{
    virConfPtr conf;
    virConfValuePtr p;
    char *tmp;

...

    p = virConfGetValue(conf, "auto_disk_leases");
    CHECK_TYPE("auto_disk_leases", VIR_CONF_ULONG);
    if (p) driver->autoDiskLease = p->l;

    p = virConfGetValue(conf, "disk_lease_dir");
    CHECK_TYPE("disk_lease_dir", VIR_CONF_STRING);
    if (p && p->str) {
        VIR_FREE(driver->autoDiskLeasePath);
        if (VIR_STRDUP(driver->autoDiskLeasePath, p->str) < 0) {
            virConfFree(conf);
            return -1;
        }
    }

...

static int virLockManagerSanlockSetupLockspace(void)
{
    int fd = -1;
    struct stat st;
    int rv;
    struct sanlk_lockspace ls;
    char *path = NULL;
    char *dir = NULL;
    int retries = LOCKSPACE_RETRIES;

    if (virAsprintf(&path, "%s/%s",
                    driver->autoDiskLeasePath,
                    VIR_LOCK_MANAGER_SANLOCK_AUTO_DISK_LOCKSPACE) < 0)
        goto error;

...

        if (stat(dir, &st) < 0 || !S_ISDIR(st.st_mode)) {
            virReportError(VIR_ERR_INTERNAL_ERROR,
                           _("Unable to create lockspace %s: parent directory"
                             " does not exist or is not a directory"),
                           path);
            goto error;
        }

...

总结：

从源码简单分析来看，如果“disk_lease_dir”变量不是目录，便无法使用，没有自动区分是“共享存储”还是“NFS”的语句。

实际测试

在共享存储上创建一个卷来保存租约。

任选一个节点创建卷：

$ lvcreate -n sanlock -L 5G storage

在另外一个节点激活卷：

$ pvscan --cache
$ lvchange -ay storage/sanlock

设置在这个卷上自动管理租约：

$ augtool -s set /files/etc/libvirt/qemu-sanlock.conf/auto_disk_leases 1
$ augtool -s set /files/etc/libvirt/qemu-sanlock.conf/disk_lease_dir "/dev/storage/sanlock"

重启libvirt服务：

$ systemctl restart libvirtd
$ systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since 五 2017-04-21 18:45:56 CST; 1s ago
     Docs: man:libvirtd(8)
           http://libvirt.org
  Process: 59501 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 59501 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/libvirtd.service
           ├─12993 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           └─12994 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

4月 21 18:45:56 Test2 systemd[1]: Starting Virtualization daemon...
4月 21 18:45:56 Test2 systemd[1]: Started Virtualization daemon.
4月 21 18:45:56 Test2 dnsmasq[12993]: read /etc/hosts - 6 addresses
4月 21 18:45:56 Test2 dnsmasq[12993]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
4月 21 18:45:56 Test2 dnsmasq-dhcp[12993]: read /var/lib/libvirt/dnsmasq/default.hostsfile
4月 21 18:45:56 Test2 libvirtd[59501]: libvirt version: 2.0.0, package: 10.el7_3.5 (CentOS BuildSystem <http://bugs.centos.org>, 2017-03-0...tos.org)
4月 21 18:45:56 Test2 libvirtd[59501]: hostname: test2
4月 21 18:45:56 Test2 libvirtd[59501]: internal error: 无法生成锁空间 /dev/storage/sanlock/__LIBVIRT__DISKS__：上级目录不存在或者不是一个目录
4月 21 18:45:56 Test2 libvirtd[59501]: QEMU 的初始化状态驱动器失败：internal error: 无法生成锁空间 /dev/storage/sanlock/__LIBVIRT__DISKS__…是一个目录
4月 21 18:45:56 Test2 libvirtd[59501]: 驱动器状态初始化失败

这说明无法通过设置“disk_lease_dir”变量为一个卷来直接在共享存储上自动管理租约，与之前的源码分析结果一致。

手动租约管理

配置libvirt

禁用自动租约管理：

$ augtool -s set /files/etc/libvirt/qemu-sanlock.conf/auto_disk_leases 0

$ augtool -s rm /files/etc/libvirt/qemu-sanlock.conf/disk_lease_dir

允许磁盘没有配置租约：

$ augtool -s set /files/etc/libvirt/qemu-sanlock.conf/require_lease_for_disks 0

重启libvirt服务：

$ systemctl restart libvirtd
$ systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since 五 2017-04-21 19:00:03 CST; 5s ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 60745 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           ├─12993 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           ├─12994 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           └─60745 /usr/sbin/libvirtd

4月 21 19:00:03 Test2 systemd[1]: Starting Virtualization daemon...
4月 21 19:00:03 Test2 systemd[1]: Started Virtualization daemon.
4月 21 19:00:03 Test2 dnsmasq[12993]: read /etc/hosts - 6 addresses
4月 21 19:00:03 Test2 dnsmasq[12993]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
4月 21 19:00:03 Test2 dnsmasq-dhcp[12993]: read /var/lib/libvirt/dnsmasq/default.hostsfile

创建租约文件（在Test1或者Test2节点）

准备在共享存储/dev/mapper/raw上创建租约，修改其权限：

$ chown sanlock:sanlock /dev/mapper/raw

创建Lockspace，名字为“libvirt”：

$ sanlock direct init -s libvirt:0:/dev/mapper/raw:0
init done 0

创建名字为“test1”的资源，属于名字为libvirt的Lockspace：

$ sanlock direct init -r libvirt:test1:/dev/mapper/raw:1048576

加入Lockspace（Test1和Test2节点）

两个节点分别使用不同的Host ID加入到Lockspace中（每次开机后都需要重新添加）。

Test1 节点：

$ sanlock client add_lockspace -s libvirt:1:/dev/mapper/raw:0
add_lockspace
add_lockspace done 0

$ sanlock client status
daemon 581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1
p -1 helper
p -1 listener
p -1 status
s libvirt:1:/dev/mapper/raw:0
s __LIBVIRT__DISKS__:1:/var/lib/libvirt/sanlock/__LIBVIRT__DISKS__:0

Test2 节点：

$ sanlock client add_lockspace -s libvirt:2:/dev/mapper/raw:0
add_lockspace
add_lockspace done 0

$ sanlock client status
daemon b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2
p -1 helper
p -1 listener
p -1 status
s libvirt:2:/dev/mapper/raw:0
s __LIBVIRT__DISKS__:2:/var/lib/libvirt/sanlock/__LIBVIRT__DISKS__:0

查看创建好的租约：

$ sanlock direct dump /dev/mapper/raw
  offset                            lockspace                                         resource  timestamp  own  gen lver
00000000                              libvirt       581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1 0000038190 0001 0001
00000512                              libvirt       b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2 0000037427 0002 0001
01048576                              libvirt                                            test1 0000000000 0000 0000 0

创建虚拟机磁盘（Test1和Test2节点）

在共享存储上创建一个卷来作为磁盘。

任选一个节点创建并初始化卷：

$ lvcreate -n test4_sanlock -L 5G storage
$ dd if=cirros-0.3.4-x86_64-disk.img of=/dev/storage/test4_sanlock

在另外一个节点激活卷：

$ pvscan --cache
$ lvchange -ay storage/test4_sanlock

创建新的虚拟机（Test1和Test2节点）

配置虚拟机：

$ vi test4_sanlock.xml

<domain type='kvm'>
    <name>test4_sanlock</name>
    <memory>262144</memory>
    <vcpu>1</vcpu>

    <os>
        <type arch='x86_64' machine='pc'>hvm</type>
        <boot dev='hd'/>
    </os>

    <devices>
        <disk type='file' device='disk'>
            <driver name='qemu' type='qcow2'/>
            <source file='/dev/storage/test4_sanlock'/>
            <target dev='hda' bus='ide'/>
        </disk>
        
        <lease>
            <lockspace>libvirt</lockspace>
            <key>test1</key>
            <target path='/dev/mapper/raw' offset='1048576'/>
        </lease>
        
        <input type='tablet' bus='usb'/>
        <input type='mouse' bus='ps2'/>

        <graphics type='vnc' port='-1' listen = '0.0.0.0' autoport='yes' keymap='en-us'/>
    </devices>
</domain>

定义虚拟机：

$ virsh define test4_sanlock.xml
定义域 test4_sanlock（从 test4_sanlock.xml）

$ virsh list --all
 Id    名称                         状态
----------------------------------------------------
 -     test1_sanlock                  关闭
 -     test2_sanlock                  关闭
 -     test3_sanlock                  关闭
 -     test4_sanlock                  关闭
 -     test_sanlock                   关闭

启动虚拟机（Test1和Test2节点）

在Test1节点上启动虚拟机test4_sanlock：

$ virsh start test4_sanlock
域 test4_sanlock 已开始

$ virsh list --all
 Id    名称                         状态
----------------------------------------------------
 2     test4_sanlock                  running
 -     test1_sanlock                  关闭
 -     test2_sanlock                  关闭
 -     test3_sanlock                  关闭
 -     test_sanlock                   关闭

$ sanlock direct dump /dev/mapper/raw
  offset                            lockspace                                         resource  timestamp  own  gen lver
00000000                              libvirt       581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1 0000040118 0001 0001
00000512                              libvirt       b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2 0000039355 0002 0001
01048576                              libvirt                                            test1 0000040075 0001 0001 1

启动正常，资源test1所有者属于Host ID 1，即Test1节点。

在Test2节点上启动虚拟机test4_sanlock：

$ virsh start test4_sanlock
错误：开始域 test4_sanlock 失败
错误：resource busy: 请求锁失败：错误 -243

$ virsh list --all
 Id    名称                         状态
----------------------------------------------------
 -     test1_sanlock                  关闭
 -     test2_sanlock                  关闭
 -     test3_sanlock                  关闭
 -     test4_sanlock                  关闭
 -     test_sanlock                   关闭

$ sanlock direct dump /dev/mapper/raw
  offset                            lockspace                                         resource  timestamp  own  gen lver
00000000                              libvirt       581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1 0000040200 0001 0001
00000512                              libvirt       b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2 0000039437 0002 0001
01048576                              libvirt                                            test1 0000040075 0001 0001 1

$ sanlock client log_dump
...
2017-04-21 20:02:54+0800 39406 [24937]: s2:r3 resource libvirt:test1:/dev/mapper/raw:1048576 for 2,10,65662
2017-04-21 20:02:54+0800 39406 [24937]: r3 paxos_acquire begin 2 0 0
2017-04-21 20:02:54+0800 39406 [24937]: r3 paxos_acquire leader 1 owner 1 1 40075 max mbal[0] 1 our_dblock 0 0 0 0 0 0
2017-04-21 20:02:54+0800 39406 [24937]: r3 paxos_acquire owner 1 1 40075 host_status 1 1 40139 wait_start 39396
2017-04-21 20:02:54+0800 39406 [24937]: r3 paxos_acquire owner 1 delta 1 1 40159 alive
2017-04-21 20:02:54+0800 39406 [24937]: r3 acquire_disk rv -243 lver 1 at 40075
2017-04-21 20:02:54+0800 39406 [24937]: r3 acquire_token held error -243
2017-04-21 20:02:54+0800 39406 [24937]: r3 release_token r_flags 0 lver 0
2017-04-21 20:02:54+0800 39406 [24937]: r3 release_token done r_flags 0
2017-04-21 20:02:54+0800 39406 [24937]: r3 cmd_acquire 2,10,65662 acquire_token -243 lease owned by other host
2017-04-21 20:02:54+0800 39406 [24937]: cmd_acquire 2,10,65662 result -243 pid_dead 0
2017-04-21 20:02:54+0800 39406 [24933]: client_pid_dead 2,10,65662 cmd_active 0 suspend 0
2017-04-21 20:05:43+0800 39573 [24933]: cmd_register ci 2 fd 10 pid 65914
2017-04-21 20:05:43+0800 39573 [24933]: cmd_restrict ci 2 fd 10 pid 65914 flags 1
2017-04-21 20:05:43+0800 39574 [24938]: cmd_inquire 2,10,65914 ci_in 3 fd 15
2017-04-21 20:05:43+0800 39574 [24938]: cmd_inquire 2,10,65914 result 0 pid_dead 0 res_count 0 cat_count 0 strlen 0
2017-04-21 20:05:43+0800 39574 [24937]: cmd_acquire 2,10,65914 ci_in 4 fd 16 count 1 flags 0
...

启动失败，无法获取test1的所有者。

在Test1节点上关闭虚拟机test4_sanlock：

$ killall -9 qemu-kvm

$ virsh list --all
 Id    名称                         状态
----------------------------------------------------
 -     test1_sanlock                  关闭
 -     test2_sanlock                  关闭
 -     test3_sanlock                  关闭
 -     test4_sanlock                  关闭
 -     test_sanlock                   关闭

$ sanlock direct dump /dev/mapper/raw
  offset                            lockspace                                         resource  timestamp  own  gen lver
00000000                              libvirt       581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1 0000040262 0001 0001
00000512                              libvirt       b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2 0000039498 0002 0001
01048576                              libvirt                                            test1 0000000000 0001 0001 1

关闭虚拟机后，资源test1的时间戳变为0。

在Test2节点上启动虚拟机test4_sanlock：

$ virsh start test4_sanlock
域 test4_sanlock 已开始

$ virsh list --all
 Id    名称                         状态
----------------------------------------------------
 2     test4_sanlock                  running
 -     test1_sanlock                  关闭
 -     test2_sanlock                  关闭
 -     test3_sanlock                  关闭
 -     test_sanlock                   关闭

$ sanlock direct dump /dev/mapper/raw
  offset                            lockspace                                         resource  timestamp  own  gen lver
00000000                              libvirt       581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1 0000040344 0001 0001
00000512                              libvirt       b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2 0000039580 0002 0001
01048576                              libvirt                                            test1 0000039574 0002 0001 2

启动成功，资源test1的所有者变为Host ID 2，即Test2节点。

总结

使用自动租约管理，只能使用NFS保存租约文件，锁的粒度是“磁盘”，必须所有虚拟机都使用或者都不使用sanlock，租约文件只能集中保存在一个地方；
使用手动租约管理，可以使用共享存储或NFS保存租约文件，粒度是“虚拟机”，每个虚拟机可以根据需要选择是否使用sanlock，租约文件可以根据需要自己选择保存位置，可分散保存也可集中保存。

转载于:https://my.oschina.net/LastRitter/blog/1540124