Docker存储配置切换loop-lvm到direct-lvm

生产环境下的存储配置

CentOS下全部使用device mapper的driver存储容器数据以及镜像，推荐使用direct-lvm模式。主要是效率，数据安全性等方面的考虑，效率方面可以参考Comprehensive Overview of Storage Scalability in Docker，数据安全性等内容可以阅读官方文档。

切换loop-lvm到direct-lvm

需要额外预留一个200g的分区（不挂载），并用lvm命令创建thinpool（OPS已经定义了装机模板）。

安装device mapper和lvm

$ yum install -yq lvm2 device-mapper

创建pv/vg

$ pvcreate /dev/sda2  # 初始化一个physical volume（假设sda2是未挂载的200g分区） $ vgcreate docker /dev/sda2  # 创建volume cgroup

创建两个pool

$ lvcreate --wipesignatures y -n thinpool docker -l  80 %VG $ lvcreate --wipesignatures y -n thinpoolmeta docker -l  5 %VG

创建thinpool

$ lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta

创建thinpool后检查下状态：

$ lvs -o+seg_monitor $ lsblk NAME                                                                                           MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda                                                                                               8 : 0     0    1 .1T   0  disk ├─sda1                                                                                            8 : 1     0    500M   0  part /boot ├─sda2                                                                                            8 : 2     0    200G   0  part │ ├─docker_vg-thinpool_tmeta                                                                    253 : 0     0     10G   0  lvm  │ │ └─docker_vg-thinpool                                                                        253 : 2     0    160G   0  lvm   │ └─docker_vg-thinpool_tdata                                                                    253 : 1     0    160G   0  lvm  │   └─docker_vg-thinpool                                                                        253 : 2     0    160G   0  lvm    ├─sda3                                                                                            8 : 3     0     20G   0  part /var ├─sda4                                                                                            8 : 4     0      1K   0  part ├─sda5                                                                                            8 : 5     0     10G   0  part / ├─sda6                                                                                            8 : 6     0      8G   0  part [SWAP] ├─sda7                                                                                            8 : 7     0      2G   0  part /home └─sda8                                                                                            8 : 8     0  877 .3G   0  part /home/q

docker使用thinpool

$ dockerd --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_removal= true

运维相关

dockerd运行成功后，创建容器，并用lsblk以及lvs命令观察当前的分配情况：

$ lvs -a    LV               VG        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert    [lvol0_pmspare]  docker_vg ewi-------   10 .00g                                                       thinpool         docker_vg twi-aot---  160 .00g              4.74    0.04                               [thinpool_tdata] docker_vg Twi-ao----  160 .00g                                                       [thinpool_tmeta] docker_vg ewi-ao----   10 .00g   $ lsblk NAME                                                                                           MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT sda                                                                                               8 : 0     0    1 .1T   0  disk ├─sda1                                                                                            8 : 1     0    500M   0  part /boot ├─sda2                                                                                            8 : 2     0    200G   0  part │ ├─docker_vg-thinpool_tmeta                                                                    253 : 0     0     10G   0  lvm  │ │ └─docker_vg-thinpool                                                                        253 : 2     0    160G   0  lvm  │ │   ├─docker- 8 : 8 - 2415920137 -fbaa790aaa4464ec26893f2ef8c0df0aa4ff4c3656e96bdaf1fe26433c043560  253 : 3     0     10G   0  dm   │ │   ├─docker- 8 : 8 - 2415920137 -23638258c17119a569a1ccf6b1a91f73e15167cf1f7917b6298ca1423a8f5db8  253 : 4     0     10G   0  dm   │ │   ├─docker- 8 : 8 - 2415920137 -10e51b2d98421c93d721b32add56699c5ac55baa58b19e80ed6c96a6e977d728  253 : 5     0     10G   0  dm   │ │   ├─docker- 8 : 8 - 2415920137 -1e6a5e0122b39967605523d12dbc276f8d34ca0e2b6f50a3feebad2c4f09b891  253 : 6     0     10G   0  dm   │ │   ├─docker- 8 : 8 - 2415920137 -ad3129f820e200588b5189f83f2b875155688c5d8fb6e02a8e8034628006663e  253 : 7     0     10G   0  dm   │ │   ├─docker- 8 : 8 - 2415920137 -1012bad184a9cc1a4de495a465ba690c32ba43ad3f3393d1af5c33bf17db6be5  253 : 8     0     10G   0  dm   │ │   ├─docker- 8 : 8 - 2415920137 -35584c11675f78a7a469a5637ab8251113f6d05695661a2f76985f8cc3e66207  253 : 9     0     10G   0  dm   │ │   ├─docker- 8 : 8 - 2415920137 -964da701e42b4734cec85ee043436451437b97bc03129e7a3c3e38d9196a7acc  253 : 10    0     10G   0  dm   │ │   ├─docker- 8 : 8 - 2415920137 -37e327e0e23cc07f42400e0d7e83748b9c16cea72c4c1e9d39063b4e38983ac0  253 : 11    0     10G   0  dm   │ │   ├─docker- 8 : 8 - 2415920137 -94aac673d3b9098ef00d594329a85338872dee6321dd4e3fbca38c1ca975b026  253 : 12    0     10G   0  dm   │ │   ├─docker- 8 : 8 - 2415920137 -ed3587d218ad3241457fb19f1196e64dee3e3bf4f6737472748e53219030d6cb  253 : 13    0     10G   0  dm   │ │   ├─docker- 8 : 8 - 2415920137 -02eba21c96277023cc330d333c907aa6d69f8fb860ea784b8f25d2c1015d59ba  253 : 14    0     10G   0  dm   │ │   └─docker- 8 : 8 - 2415920137 -e7f1449e05caab6ba37543bf3192ff9933325587928d29f835322223d78631df  253 : 15    0     10G   0  dm   │ └─docker_vg-thinpool_tdata                                                                    253 : 1     0    160G   0  lvm  │   └─docker_vg-thinpool                                                                        253 : 2     0    160G   0  lvm  │     ├─docker- 8 : 8 - 2415920137 -fbaa790aaa4464ec26893f2ef8c0df0aa4ff4c3656e96bdaf1fe26433c043560  253 : 3     0     10G   0  dm   │     ├─docker- 8 : 8 - 2415920137 -23638258c17119a569a1ccf6b1a91f73e15167cf1f7917b6298ca1423a8f5db8  253 : 4     0     10G   0  dm   │     ├─docker- 8 : 8 - 2415920137 -10e51b2d98421c93d721b32add56699c5ac55baa58b19e80ed6c96a6e977d728  253 : 5     0     10G   0  dm   │     ├─docker- 8 : 8 - 2415920137 -1e6a5e0122b39967605523d12dbc276f8d34ca0e2b6f50a3feebad2c4f09b891  253 : 6     0     10G   0  dm   │     ├─docker- 8 : 8 - 2415920137 -ad3129f820e200588b5189f83f2b875155688c5d8fb6e02a8e8034628006663e  253 : 7     0     10G   0  dm   │     ├─docker- 8 : 8 - 2415920137 -1012bad184a9cc1a4de495a465ba690c32ba43ad3f3393d1af5c33bf17db6be5  253 : 8     0     10G   0  dm   │     ├─docker- 8 : 8 - 2415920137 -35584c11675f78a7a469a5637ab8251113f6d05695661a2f76985f8cc3e66207  253 : 9     0     10G   0  dm   │     ├─docker- 8 : 8 - 2415920137 -964da701e42b4734cec85ee043436451437b97bc03129e7a3c3e38d9196a7acc  253 : 10    0     10G   0  dm   │     ├─docker- 8 : 8 - 2415920137 -37e327e0e23cc07f42400e0d7e83748b9c16cea72c4c1e9d39063b4e38983ac0  253 : 11    0     10G   0  dm   │     ├─docker- 8 : 8 - 2415920137 -94aac673d3b9098ef00d594329a85338872dee6321dd4e3fbca38c1ca975b026  253 : 12    0     10G   0  dm   │     ├─docker- 8 : 8 - 2415920137 -ed3587d218ad3241457fb19f1196e64dee3e3bf4f6737472748e53219030d6cb  253 : 13    0     10G   0  dm   │     ├─docker- 8 : 8 - 2415920137 -02eba21c96277023cc330d333c907aa6d69f8fb860ea784b8f25d2c1015d59ba  253 : 14    0     10G   0  dm   │     └─docker- 8 : 8 - 2415920137 -e7f1449e05caab6ba37543bf3192ff9933325587928d29f835322223d78631df  253 : 15    0     10G   0  dm   ├─sda3                                                                                            8 : 3     0     20G   0  part /var ├─sda4                                                                                            8 : 4     0      1K   0  part ├─sda5                                                                                            8 : 5     0     10G   0  part / ├─sda6                                                                                            8 : 6     0      8G   0  part [SWAP] ├─sda7                                                                                            8 : 7     0      2G   0  part /home └─sda8                                                                                            8 : 8     0  877 .3G   0  part /home/q

同时收集device-mapper的日志，用于持续跟踪direct-lvm工作情况，本机上可以使用:

journalctl -fu dm-event.service

如果发现error的日志，配合device mapper events的文档看一下，160g的空间一般不会发生dm.min_free_space相关的问题，除非当前机器上的垃圾镜像/容器内数据膨胀的很大了。

其他问题

性能

direct-lvm的block大小与性能有比较大的关系，写入是Allocate-on-demand，更新已存在的数据是Copy-on-write，前者与块存储很像，一次写入的数据量小于block size的话，会造成浪费。后者则是创建snapshot的时候每次都copy block size大小的数据，所以不适合在direct-lvm上频繁更新/写入小文件（小于block size），性能很低。

扩容

lvm扩容的需求暂时看不到，等CM的发布上来以后，估计lvm扩容是个常态了。

扩容的话需要使用vgextend/lvextend：

$ vgextend vg-docker /dev/sdb1 $ lvextend -l+ 100 %FREE -n vg-docker/data

然后根据blockdev --getsize64拿到vg的大小，除以512,算出新的table sectors，重新写回dmsetup table里。

docker-storage-setup脚本

部分CentOS7的docker-engine的RPM包带了一个docker-storage-setup的脚本，可以自动初始化direct-lvm，咱们这边没看到有，可以忽略这个脚本。

systemd MountFlags

docker1.12以后修改了systemd的service unit中的MountFlags参数，从slave改成了shared，这样就导致了容器内的mountinfo会被自动继承到宿主机的/etc/mtab里。df -h就会变成如下的样子：

Filesystem      Size  Used Avail Use% Mounted on /dev/sda5        10G   1 .7G   8 .4G   17 % / devtmpfs         63G      0    63G    0 % /dev tmpfs            63G   12K   63G    1 % /dev/shm tmpfs            63G  323M   63G    1 % /run tmpfs            63G      0    63G    0 % /sys/fs/cgroup /dev/sda7        2 .0G   33M   2 .0G    2 % /home /dev/sda8        2 .0T  366G   1 .6T   19 % /home/q /dev/sda4        20G  876M   20G    5 % /var /dev/sda2       497M  127M  370M   26 % /boot /dev/dm- 3         10G  837M   9 .2G    9 % /home/q/docker/devicemapper/mnt/c59f5124cf4f2c055c0fb8a545b9a36430a62484bd17e40c65fa759677a1c494 shm              64M      0    64M    0 % /home/q/docker/containers/c34787539d49fc2b8c9b1692ffadac2c887f11d0eb0f583ea15de7969d8dcd05/shm /dev/dm- 5         10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/45b205be0b5868f2abf518529f1cf5a57cfaeef930b4a93246d2ff577cc2729b shm              64M      0    64M    0 % /home/q/docker/containers/cd691d6274a9ed0755baa0aa9164a137cbed9f66d2c86a7710a8322a30ed4932/shm /dev/dm- 7         10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/1411a2dd3212fff1ccbd5dad4631c3039974ecadbf721b21367215b52d41a05f shm              64M      0    64M    0 % /home/q/docker/containers/20ef7dd79679475c43b5d96b2582c50a0fb8e278aacc38789c4c7b079baffa67/shm /dev/dm- 13        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/635e8495ae7a78b6528fd9815990d8ffe3c313c2e9e39556838b0d1e09505188 shm              64M      0    64M    0 % /home/q/docker/containers/b27bb93c915ca41a20ebfbf3946a84158d81d0e0e547fc7b29993b93d71ba70c/shm /dev/dm- 8         10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/433cd2bef34f59a936b01dd6e533f3a5f6c46c5c777a72e03f8273efc0f84782 shm              64M      0    64M    0 % /home/q/docker/containers/d843b21f0f872b257fb33ec4ec7a45013810b65e17e2e03dda4a718b43123e5b/shm /dev/dm- 9         10G  234M   9 .8G    3 % /home/q/docker/devicemapper/mnt/5fa277cc9ccab68ea3476197c0ad8a8e65556ea489579386ee9721dd7cc9e742 shm              64M      0    64M    0 % /home/q/docker/containers/ed9fa870d9e5d718fa9ef2ef37399521e6baa98cb823dd265f6bd7a3c7d31143/shm /dev/dm- 12        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/fbac533312232de3d3081c536f66f3d39d809ad02310fd9899fe3c5ec4403785 shm              64M      0    64M    0 % /home/q/docker/containers/87585b4ce2b1e9757b7e7626927bca66d0352ccdc0979c79cec7763124a72c70/shm /dev/dm- 11        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/ff5943eea966f25aba19a4f01549681386739e01ca59d9856209c0f8a02c49bd shm              64M      0    64M    0 % /home/q/docker/containers/88318b626dcb1f758104b59a617f72321fe89fea1014e8769ec1646333819b6d/shm /dev/dm- 6         10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/a6f3263a91f34fccffb933144b2d7adf119ee2a11f2309d9dba50ca56c3c95ef shm              64M      0    64M    0 % /home/q/docker/containers/83d5aa6bd318e9279b07ca93882b3a6e61c40b5a63b2a57ad23ca05eda90ec13/shm /dev/dm- 19        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/8e6dd15f468c0daf1a84742ddb6518bc43ade296db9b11b295113a081c097f20 shm              64M      0    64M    0 % /home/q/docker/containers/3ef4f4bee627e25f38bbfeb6585d5f0b1521c999ff70594403361db40c60d521/shm /dev/dm- 20        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/1101767d5c8a2976fba2ad801b070a0aee462747a24b2604ee3c0e58f268d7c7 shm              64M      0    64M    0 % /home/q/docker/containers/8a3d3533b73e4f2e73b2ebf8948a153aa0238f841f06a04258a1cd93e4a3185f/shm /dev/dm- 21        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/b8cf8ba6575d7a22f9112a456a7ccd3489b873af3fe8297fd48a8c632f4bc755 shm              64M      0    64M    0 % /home/q/docker/containers/065b671662d9485f85177e8d1d8efa318b775d16cce8f1764e3ef322e5577d58/shm /dev/dm- 17        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/746ea6dffda2d73dc50f38a0e888ef94c6fe32a6d50530ed340d4b7e45c4d01b shm              64M      0    64M    0 % /home/q/docker/containers/d0c1615128973e527862801cbbaec912c1c2c49612792f113c60473d6f8e6e1f/shm /dev/dm- 14        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/f05fc9e9d0c46cf3487361cc23315d8f052a984fcd83ad07db6ced2b9d056fab shm              64M      0    64M    0 % /home/q/docker/containers/0fd60f4b75328441c98989fc5c475a0535a5be69708c724122c56d75b272ee4f/shm /dev/dm- 15        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/8263d710eb71391c7565f42895547ab54b6be7134ff6a53504b27fb0827ad8cc shm              64M      0    64M    0 % /home/q/docker/containers/3d930b65aff13f1b4c4ba5881094264379ba24b4f9513862383f6b330a99ca3c/shm /dev/dm- 16        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/a18bbc47ee0aef6cf64a694a8ef22ebdd336ae0fbb85646a12971039d2b705d8 shm              64M      0    64M    0 % /home/q/docker/containers/8809b728ecdfba50ca7b0dad62678f73e79bb394cc908d405c7e519386ba2969/shm /dev/dm- 18        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/2ec7de4bfb4e4470acc2084eb7088d461e2b6f9a881c694f4fca17a13d5346aa shm              64M      0    64M    0 % /home/q/docker/containers/3d2821e9b0c7a955fc8a413eaf45d2335f445582a2ff12674ecd1bb52cc94472/shm /dev/dm- 22        10G  548M   9 .5G    6 % /home/q/docker/devicemapper/mnt/c69dfe3fab712c4c4406cd2b15361517a0f08213fdcd3c768d90fff1aef38d98 shm              64M      0    64M    0 % /home/q/docker/containers/d13a45b5a66aa072df0b3aeeef679753ff1a381c1ff71f228ee17745f88ba633/shm /dev/dm- 4         10G   62M   10G    1 % /home/q/docker/devicemapper/mnt/f614499952f86ef6d6f0a12b08e98791bce567e249d0057c40a380e71f2a3e53 shm              64M      0    64M    0 % /home/q/docker/containers/0151dbaaa28a6d30a27932f1cafe5f54bf83e78d8870bd78c18ea5352c68e01a/shm /dev/dm- 10        10G   1 .4G   8 .7G   14 % /home/q/docker/devicemapper/mnt/9f226b2388cb91dced03aa32e623fb47d068060ff264d1a081976371c76cf353 shm              64M      0    64M    0 % /home/q/docker/containers/fc59743e16ac9134e729e531f4ae28460b408501f4bb243847bea5976bbb8a06/shm tmpfs            13G      0    13G    0 % /run/user/ 30303

这里会引申出两个问题，第一个是会出现proc的错误挂载（原因还未找到），导致我们的check_disk报错：

/etc/mtab

proc net:[ 4026531968 ] proc rw,nosuid,nodev,noexec,relatime  0  0

DISK CRITICAL - net:[ 4026531968 ] is not accessible: No such file or directory

需要修改nrpe的配置：

/home/q/nrpe/libexec/check_disk -w  15 % -c  10 % -A -l -I proc