问题：
制作镜像的时候报错
devmapper: Thin Pool has 162394 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
解决方案：
运行下面三个命令：
// 注意，以下三个命令执行时可能出错是正常的。
清理exited进程：
docker rm $(docker ps -q -f status=exited)
清理dangling volumes：
docker volume rm $(docker volume ls -qf dangling=true)
清理dangling image：
docker rmi $(docker images --filter "dangling=true" -q --no-trunc)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
修改device mapper为direct-lvm
1.停止Docker服务
首先运行docker info 查看当前系统的docker配置：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
|
[root@localhost ~]
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-8:3-135618250-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data
file
:
/dev/loop0
Metadata
file
:
/dev/loop1
Data Space Used: 11.8 MB
Data Space Total: 107.4 GB
Data Space Available: 51.43 GB
Metadata Space Used: 581.6 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported:
true
Deferred Removal Enabled:
true
Deferred Deletion Enabled:
true
Deferred Deleted Device Count: 0
Data loop
file
:
/var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged
for
production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop
file
:
/var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume:
local
Network: host bridge null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 1.954 GiB
Name: localhost.localdomain
ID: 5PPL:6TWG:X2JA:VSVV:YYFD:X7BG:EBP5:GYWB:S3FJ:QET6:T3ED:5HTM
Docker Root Dir:
/var/lib/docker
Debug Mode (client):
false
Debug Mode (server):
false
Registry: https:
//index
.docker.io
/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0
/8
Registries: docker.io (secure)
[root@localhost ~]
#
|
发现Storage Driver是Devicemapper，Data File和Metadata File都是loop设备，下面我们将docker停掉：
|
1
|
# systemctl stop docker
|
2. 添加磁盘并创建thin-pool lv
查看新添加的磁盘
|
1
2
3
4
5
6
7
8
|
[root@localhost ~]
# fdisk -l /dev/sdb
Disk
/dev/sdb
: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
[root@localhost ~]
#
|
创建pv
|
1
2
3
|
[root@localhost ~]
# pvcreate /dev/sdb
Physical volume
"/dev/sdb"
successfully created.
[root@localhost ~]
#
|
创建vg
|
1
2
3
|
[root@localhost ~]
# vgcreate docker /dev/sdb
Volume group
"docker"
successfully created
[root@localhost ~]
#
|
创建data lv
|
1
2
3
|
[root@localhost ~]
# lvcreate --wipesignatures y -n thinpool docker -l 95%VG
Logical volume
"thinpool"
created.
[root@localhost ~]
#
|
创建metadata lv
|
1
2
3
|
[root@localhost ~]
# lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
Logical volume
"thinpoolmeta"
created.
[root@localhost ~]
#
|
[root@docker ~]#
数据LV大小为VG的95%,元数据LV大小为VG的1%,剩余的空间用来自动扩展。
注意:作为meta的pool大小不能超过16GB！！！
|
1
2
3
4
5
|
[root@localhost ~]
# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
thinpool docker -wi-a----- <19.00g
thinpoolmeta docker -wi-a----- 204.00m
[root@localhost ~]
#
|
将pool 转换为thin-pool
将 thinpool lv 的 chunksize 改为 512KB,并且将前 4KB 字节清零。
|
1
2
3
4
5
6
|
[root@localhost ~]
# lvconvert -y --zero n -c 512k --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta
Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
WARNING: Converting logical volume docker
/thinpool
and docker
/thinpoolmeta
to thin pool's data and metadata volumes with metadata wiping.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Converted docker
/thinpool_tdata
to thin pool.
[root@localhost ~]
#
|
创建一个thinpool的profile
|
1
2
3
4
5
6
7
8
9
10
11
12
|
[root@localhost ~]
# vim /etc/lvm/profile/docker-thinpool.profile
[root@localhost ~]
# cat /etc/lvm/profile/docker-thinpool.profile
activation {
thin_pool_autoextend_threshold=80
thin_pool_autoextend_percent=20
}
[root@localhost ~]
#
定义一个百分比的阈值，表明触发 lvm 自动扩容前，已用空间占比。
thin_pool_autoextend_threshold = 80
每次扩容 thin pool 空间的比例
thin_pool_autoextend_percent = 20
|
应用配置
|
1
2
3
|
[root@localhost ~]
# lvchange --metadataprofile docker-thinpool docker/thinpool
Logical volume docker
/thinpool
changed.
[root@localhost ~]
#
|
注意： docker-thinpool 即刚才创建的 profile 文件名的前缀，不需要加.profile,而且要在etc/lvm/profile 目录下运行此命令。 执行完毕后不要mount，不要格式化 lv。
3. 查看lv状态
|
1
2
3
4
5
|
[root@localhost ~]
# lvs -o+seg_monitor
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Monitor
thinpool docker twi-a-t--- <19.00g 0.00 0.03 monitored
[root@localhost ~]
#
看到有Meta 和Data下面都有数字，代表刚才创建的thinpool创建成功。
|
4. 查看磁盘状态
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
[root@localhost ~]
# fdisk -l
Disk
/dev/sda
: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk label
type
: dos
Disk identifier: 0x00075587
Device Boot Start End Blocks Id System
/dev/sda1
* 2048 411647 204800 83 Linux
/dev/sda2
411648 1460223 524288 82 Linux swap / Solaris
/dev/sda3
1460224 104857599 51698688 83 Linux
Disk
/dev/sdb
: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk
/dev/mapper/docker-thinpool_tmeta
: 213 MB, 213909504 bytes, 417792 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk
/dev/mapper/docker-thinpool_tdata
: 20.4 GB, 20396900352 bytes, 39837696 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk
/dev/mapper/docker-thinpool
: 20.4 GB, 20396900352 bytes, 39837696 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 524288 bytes / 524288 bytes
[root@localhost ~]
#
|
5. 配置docker
方式一： 调整 docker.service 的配置参数
添加如下参数到docker服务的启动项里:
|
1
2
3
4
5
|
--storage-driver=devicemapper --storage-opt dm.thinpooldev=
/dev/mapper/docker-thinpool
--storage-opt dm.use_deferred_removal=
true
--storage-opt dm.use_deferred_del
etion=
true
或者：
--storage-driver=devicemapper --storage-opt=dm.thinpooldev=
/dev/mapper/docker-thinpool
--storage-opt=dm.use_deferred_removal=
true
--storage-opt=dm.use_deferred_del
etion=
true
|
完整的配置文件内容如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
|
[root@localhost ~]
# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http:
//docs
.docker.com
After=network.target
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer
[Service]
Type=notify
NotifyAccess=all
EnvironmentFile=-
/run/containers/registries
.conf
EnvironmentFile=-
/etc/sysconfig/docker
EnvironmentFile=-
/etc/sysconfig/docker-storage
EnvironmentFile=-
/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=
/usr/libexec/docker
:
/usr/bin
:
/usr/sbin
ExecStart=
/usr/bin/dockerd-current
\
--add-runtime docker-runc=
/usr/libexec/docker/docker-runc-current
\
--default-runtime=docker-runc \
--
exec
-opt native.cgroupdriver=systemd \
--userland-proxy-path=
/usr/libexec/docker/docker-proxy-current
\
--storage-driver=devicemapper --storage-opt=dm.thinpooldev=
/dev/mapper/docker-thinpool
--storage-opt=dm.use_deferred_removal=
true
--storage-opt=dm.use_deferred_deletion=
true
\
$OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$ADD_REGISTRY \
$BLOCK_REGISTRY \
$INSECURE_REGISTRY\
$REGISTRIES
ExecReload=
/bin/kill
-s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
MountFlags=slave
KillMode=process
[Install]
WantedBy=multi-user.target
[root@localhost ~]
#
|
或者：
|
1
|
[root@docker ~]
# sed -i '/^ExecStart=/c\ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true \\' /lib/systemd/system/docker.service
|
完整的配置文件内容如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
|
[root@docker ~]
# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http:
//docs
.docker.com
After=network.target
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer
[Service]
Type=notify
NotifyAccess=all
EnvironmentFile=-
/run/containers/registries
.conf
EnvironmentFile=-
/etc/sysconfig/docker
EnvironmentFile=-
/etc/sysconfig/docker-storage
EnvironmentFile=-
/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=
/usr/libexec/docker
:
/usr/bin
:
/usr/sbin
ExecStart=
/usr/bin/dockerd
--storage-driver=devicemapper --storage-opt=dm.thinpooldev=
/dev/mapper/docker-thinpool
--storage-opt=dm.use_deferred_removal=
true
--storage-opt=dm.use_deferred_deletion=
true
\
--add-runtime docker-runc=
/usr/libexec/docker/docker-runc-current
\
--default-runtime=docker-runc \
--
exec
-opt native.cgroupdriver=systemd \
--userland-proxy-path=
/usr/libexec/docker/docker-proxy-current
\
$OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$ADD_REGISTRY \
$BLOCK_REGISTRY \
$INSECURE_REGISTRY\
$REGISTRIES
ExecReload=
/bin/kill
-s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
MountFlags=slave
KillMode=process
[Install]
WantedBy=multi-user.target
[root@docker ~]
#
|
方式二： 在 daemon.json 中配置参数
|
1
2
3
4
5
6
7
8
9
10
11
|
[root@localhost ~]
# cat /etc/docker/daemon.json
{
"storage-driver"
:
"devicemapper"
,
"storage-opts"
: [
"dm.thinpooldev=/dev/mapper/docker-thinpool"
,
"dm.use_deferred_removal=true"
,
"dm.use_deferred_deletion=true"
]
}
[root@localhost ~]
# systemctl daemon-reload
[root@localhost ~]
# systemctl restart docker
|
6. 清除Graphdriver
在启动docker之前，需要将之前残留的docker文件删除掉，要不然会有以下报错：
|
1
2
3
4
|
Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem
verification failed: devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed
rm
-rf
/var/lib/docker/
*
|
7. 启动docker，并查看docker详情
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
|
[root@localhost ~]
# systemctl daemon-reload
[root@localhost ~]
# systemctl start docker
[root@localhost ~]
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data
file
:
Metadata
file
:
Data Space Used: 20.45 MB
Data Space Total: 20.4 GB
Data Space Available: 20.38 GB
Metadata Space Used: 61.44 kB
Metadata Space Total: 213.9 MB
Metadata Space Available: 213.8 MB
Thin Pool Minimum Free Space: 2.039 GB
Udev Sync Supported:
true
Deferred Removal Enabled:
true
Deferred Deletion Enabled:
true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume:
local
Network: host bridge overlay null
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 1.954 GiB
Name: localhost.localdomain
ID: 5PPL:6TWG:X2JA:VSVV:YYFD:X7BG:EBP5:GYWB:S3FJ:QET6:T3ED:5HTM
Docker Root Dir:
/var/lib/docker
Debug Mode (client):
false
Debug Mode (server):
false
Registry: https:
//index
.docker.io
/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0
/8
Registries: docker.io (secure)
[root@localhost ~]
#
|
查看 devicemapper 的资源,发现 docker-thinpool 与 docker info 显示的 Pool Name 一致,代表启用direct-lvm 成功。
|
1
2
3
4
5
|
[root@localhost ~]
# dmsetup ls
docker-thinpool_tdata (253:1)
docker-thinpool_tmeta (253:0)
docker-thinpool (253:2)
[root@localhost ~]
#
|
8.查看对应的对应的设备
|
1
2
3
4
5
6
7
8
|
[root@localhost ~]
# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 20G 0 disk
├─docker-thinpool_tmeta 253:0 0 204M 0 lvm
│ └─docker-thinpool 253:2 0 19G 0 lvm
└─docker-thinpool_tdata 253:1 0 19G 0 lvm
└─docker-thinpool 253:2 0 19G 0 lvm
[root@localhost ~]
#
|
9.docker info 中其它的警告取消
默认禁用网桥上的netfilter，所以出现以下警告
|
1
2
|
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
|
修改配置文件 /usr/lib/sysctl.d/00-system.conf，将net.bridge.bridge参数值设定为1，然后重启系统；
|
1
2
3
|
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
|
10.再次查看docker信息
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
|
[root@localhost ~]
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data
file
:
Metadata
file
:
Data Space Used: 20.45 MB
Data Space Total: 20.4 GB
Data Space Available: 20.38 GB
Metadata Space Used: 61.44 kB
Metadata Space Total: 213.9 MB
Metadata Space Available: 213.8 MB
Thin Pool Minimum Free Space: 2.039 GB
Udev Sync Supported:
true
Deferred Removal Enabled:
true
Deferred Deletion Enabled:
true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume:
local
Network: overlay null bridge host
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 1.954 GiB
Name: localhost.localdomain
ID: 5PPL:6TWG:X2JA:VSVV:YYFD:X7BG:EBP5:GYWB:S3FJ:QET6:T3ED:5HTM
Docker Root Dir:
/var/lib/docker
Debug Mode (client):
false
Debug Mode (server):
false
Registry: https:
//index
.docker.io
/v1/
Insecure Registries:
127.0.0.0
/8
Registries: docker.io (secure)
[root@localhost ~]
#
|
3083
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
