Docker-使用 overlayfs 作为后端存储遇到的问题

最新推荐文章于 2024-07-04 11:09:42 发布

Aaron.com

最新推荐文章于 2024-07-04 11:09:42 发布

阅读量614

点赞数

分类专栏： Docker

本文链接：https://blog.csdn.net/iov_aaron/article/details/97128010

版权

Docker 专栏收录该内容

12 篇文章 0 订阅

订阅专栏

文章目录

问题描述

在一个测试环境中，发现容器中无法删除镜像中只读层的文件，现象：
问题描述

问题原因

参考 https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html

相关知识点

1、XFS 文件系统作为 overlayfs 底层时，该文件系统创建时必须使用 -n ftype=1 标志，使用 xfs_info 命令查看 xfs 文件系统的标志：

$ xfs_info /var/lib/docker
meta-data=/dev/mapper/centos_k8s--master-root isize=256    agcount=4, agsize=4653056 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=18612224, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=9088, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
$ xfs_info /var/lib/docker | grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0

在一台正常的机器上：

$ xfs_info /var/lib/docker
meta-data=/dev/mapper/cl-root    isize=512    agcount=4, agsize=2956800 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=11827200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=5775, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
$ xfs_info /var/lib/docker |grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1

2、通过 man mkfs.xfs 命令，查看 -n ftype 的含义：是否允许将 inode 类型存储在目录结构中，以便 readdir，getdents 不需要查找 inode 就可知道 inode 类型，默认为0，不存在目录结构中。
在这里插入图片描述
3、在 docker-1.13+ 版本中，使用 docker info 命令可以看到对 xfs 的 ftype 特性是否支持的信息 https://github.com/moby/moby/pull/27433

如果supports d_type为false，会有如下告警信息

解决办法

使用独立的 LV 并使用 mkfs.xfs -n ftype=1 命令格式化新的分区，然后挂载到 /var/lib/docker
如果 /var/lib/docker 已经使用了单独的 LV，需要先删除原来的 LV，再重新创建一个：

$ systemctl stop docker && systemctl disable docker
$ rm -rf /var/lib/docker/*
$ umount /dev/mapper/centos_k8s--node1-var_lib_docker
$ lvremove /dev/mapper/centos_k8s--node1-var_lib_docker
$ lvcreate -L 180G -n docker-lv centos_k8s-node1
$ mkfs.xfs -n ftype=1 /dev/centos_k8s-node1/docker-lv
$ mount /dev/mapper/centos_k8s--node1-docker--lv /var/lib/docker
$ sed '/centos_k8s--node1-var_lib_docker/d' /etc/fstab
$ echo "/dev/mapper/centos_k8s--node1-docker--lv  /var/lib/docker  xfs     defaults        0 0" >> /etc/fstab
$ systemctl start docker && systemctl enable docker

如果 /var/lib/docker 没有使用单独的 LV，推荐使用新的 LV 挂载到 /var/lib/docker 目录：

$ systemctl stop docker && systemctl disable docker
$ rm -rf /var/lib/docker/*
# 为虚拟机添加一块新磁盘 /dev/sdb 大小 50GB
$ fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x50e1bb21.
Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-104857599, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-104857599, default 104857599):
Using default value 104857599
Partition 1 of type Linux and of size 50 GiB is set
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'
Command (m for help): p
Disk /dev/sdb: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x50e1bb21
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   104857599    52427776   8e  Linux LVM
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
# 强制内核写入新的分区表
$ partprobe
# 创建PV
$ pvcreate /dev/sdb1
  Physical volume "/dev/sdb1" successfully created.
$ pvs
  PV         VG                Fmt  Attr PSize  PFree
  /dev/sda2  centos_k8s-master lvm2 a--  98.88g  4.00m
  /dev/sdb1                    lvm2 ---  50.00g 50.00g
# 创建 VG
$ vgcreate docker-vg /dev/sdb1
  Volume group "docker-vg" successfully created
$ vgs
  VG                #PV #LV #SN Attr   VSize  VFree
  centos_k8s-master   1   3   0 wz--n- 98.88g  4.00m
  docker-vg           1   0   0 wz--n- 50.00g 50.00g
# 创建 LV
$ lvcreate -L 45G -n docker-lv docker-vg
  Logical volume "docker-lv" created.
$ lvs
  LV        VG                Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home      centos_k8s-master -wi-ao---- 20.00g                                                   
  root      centos_k8s-master -wi-ao---- 71.00g                                                   
  swap      centos_k8s-master -wi-ao----  7.88g                                                   
  docker-lv docker-vg         -wi-a----- 45.00g
  
###### 使用 mkfs.xfs -n ftype=1 格式化新的 LV
$ mkfs.xfs -n ftype=1 /dev/docker-vg/docker-lv
meta-data=/dev/docker-vg/docker-lv isize=512    agcount=4, agsize=2949120 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=11796480, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=5760, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
$ [[ ! -d /var/lib/docker ]] && mkdir /var/lib/docker
# 将 /dev/docker-vg/docker-lv 挂载到 /var/lib/docker
$ mount /dev/docker-vg/docker-lv /var/lib/docker
# 查看 /var/lib/docker 的 ftype 参数值
$ xfs_info /var/lib/docker |grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
# 设置开机自动挂载
$ echo "/dev/mapper/docker--vg-docker--lv   /var/lib/docker         xfs     defaults        0 0" >> /etc/fstab
$ systemctl start docker && systemctl enable docker

现在重新测试一下：

$ docker run --rm -it centos:7.3.1611
Unable to find image 'centos:7.3.1611' locally
7.3.1611: Pulling from library/centos
45a2e645736c: Pull complete
Digest: sha256:c577af3197aacedf79c5a204cd7f493c8e07ffbce7f88f7600bf19c688c38799
Status: Downloaded newer image for centos:7.3.1611
[root@325a7b4bc2d8 /]# ls
anaconda-post.log  bin  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@325a7b4bc2d8 /]# rm -f anaconda-post.log
[root@325a7b4bc2d8 /]# ll
total 0
lrwxrwxrwx.   1 root root   7 Dec 14 14:57 bin -> usr/bin
drwxr-xr-x.   5 root root 360 May 18 04:42 dev
drwxr-xr-x.   1 root root  66 May 18 04:42 etc
drwxr-xr-x.   2 root root   6 Nov  5  2016 home
lrwxrwxrwx.   1 root root   7 Dec 14 14:57 lib -> usr/lib
lrwxrwxrwx.   1 root root   9 Dec 14 14:57 lib64 -> usr/lib64
drwx------.   2 root root   6 Dec 14 14:57 lost+found
drwxr-xr-x.   2 root root   6 Nov  5  2016 media
drwxr-xr-x.   2 root root   6 Nov  5  2016 mnt
drwxr-xr-x.   2 root root   6 Nov  5  2016 opt
dr-xr-xr-x. 243 root root   0 May 18 04:42 proc
dr-xr-x---.   2 root root 114 Dec 14 14:59 root
drwxr-xr-x.  10 root root 130 Dec 14 14:59 run
lrwxrwxrwx.   1 root root   8 Dec 14 14:57 sbin -> usr/sbin
drwxr-xr-x.   2 root root   6 Nov  5  2016 srv
dr-xr-xr-x.  13 root root   0 May 18 04:37 sys
drwxrwxrwt.   7 root root 132 Dec 14 14:59 tmp
drwxr-xr-x.  13 root root 155 Dec 14 14:57 usr
drwxr-xr-x.  18 root root 238 Dec 14 14:59 var
[root@325a7b4bc2d8 /]#

现在一些都正常了。

参考资料：

1、https://github.com/moby/moby/issues/27358

2、https://github.com/moby/moby/issues/31283

3、https://github.com/moby/moby/issues/31445

4、https://github.com/moby/moby/pull/27433

5、https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html

Aaron.com

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Docker-使用 overlayfs 作为后端存储遇到的问题

文章目录问题描述问题原因相关知识点解决办法问题描述在一个测试环境中，发现容器中无法删除镜像中只读层的文件，现象：问题原因参考 https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html...
复制链接

扫一扫