Apiserver采用高可用方式安装部署。这篇文章接上一篇《Kubernetes1.9生产环境高可用实践–002-apiserver高可用安装部署》。
在这一篇,我们着重写在node节点中如何安装Docker,以及Docker的配置。
kubernetes结点的配置我们会放到下一节中进行讨论。
这篇在很多的文章中都写的比较简单。我们在这里写的相对麻烦一些,不过,如果要是在生产环境中配置,还是见意使用这样的配置。
我们在这里安装三个节点:
192.168.3.56 yds-dev-svc02-node01
192.168.3.57 yds-dev-svc02-node02
192.168.3.58 yds-dev-svc02-node03
三个节点的安装过程是一样的,因此,我们在这里只配置yds-dev-svc02-node01,其他节点采用相同的配置。
01 服务器配置
01.01 配置服务器名和IP
[root@localhost ~]# hostnamectl set-hostname yds-dev-svc02-node01
[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens32
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens32
UUID=7d6fb2ed-364c-415f-9b02-0e54436ff1ec
DEVICE=ens32
ONBOOT=yes
IPADDR=192.168.3.56
NETMASK=255.255.255.0
GATEWAY=192.168.3.1
DNS1=192.168.3.10
DNS2=61.139.2.69
配置完成后,退出重新登录。
02 Docker安装
02.01 安装epel-release
yum update -y
yum install -y epel-release wget lsof
02.02 安装Docker
yum install -y docker
[root@yds-dev-svc02-node01 ~]# docker --version
Docker version 1.13.1, build 774336d/1.13.1
02.03 启动docker
systemctl enable docker
systemctl start docker
systemctl status docker
02.04 检查docker
[root@yds-dev-svc02-node01 ~]# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 1.78 GiB
Name: yds-dev-svc02-node01
ID: YKWT:7Y6M:O3FB:C7BC:KU3Q:ZI5I:KM7E:QGTW:7TZV:2WF4:S5LD:ROKB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)
现在Docker默认的Storage Driver为overlay2,只适用于测试 环境。但我们在生产环境,需要把这一项改为devicemapper的direct-lvm模式,不要在生产中使用loop-lvm模式。针对Docker的存储驱动,网上有很多的关于这方面的讲解,如果时间充足,会翻译几篇关于这方面的对比文档。
接下来,我们把docker的存储改为direct-lvm。
03 配置Docker direct-lvm模式
03.01 硬盘检查
我们先查看一下硬盘信息。
[root@yds-dev-svc02-node01 ~]# fdisk -l
Disk /dev/sda: 128.8 GB, 128849018880 bytes, 251658240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a9fcf
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 251658239 124779520 8e Linux LVM
Disk /dev/sdb: 257.7 GB, 257698037760 bytes, 503316480 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-home: 71.9 GB, 71932313600 bytes, 140492800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
可以看到/dev/sdb为未使用的硬盘。
现在,我们来处理/dev/sdb来存放docker数据。
03.02 停止Docker
systemctl stop docker
03.03 安装软件
yum install -y lvm2 device-mapper-persistent-data
03.04 创建物理卷
[root@yds-dev-svc02-node01 ~]# pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created.
03.05 创建Docker卷组
[root@yds-dev-svc02-node01 ~]# vgcreate docker /dev/sdb
Volume group "docker" successfully created
03.06 创建逻辑卷
在这里需要创建二个逻辑卷,名称为thinpool和thinpoolmeta。
[root@yds-dev-svc02-node01 ~]# lvcreate --wipesignatures y -n thinpool docker -l 95%VG
Logical volume "thinpool" created.
[root@yds-dev-svc02-node01 ~]# lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
Logical volume "thinpoolmeta" created.
03.07 卷转换
Convert the volumes to a thin pool and a storage location for metadata for the thin pool, using the lvconvert command.
将刚创建的卷转换为thin pool,并使用刚创建的thinpoolmeta卷。
[root@yds-dev-svc02-node01 ~]# lvconvert -y \
--zero n \
-c 512K \
--thinpool docker/thinpool \
--poolmetadata docker/thinpoolmeta
Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
WARNING: Converting logical volume docker/thinpool and docker/thinpoolmeta to thin pool's data and metadata volumes with metadata wiping.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Converted docker/thinpool_tdata to thin pool.
03.08 自动扩展配置
需要配置的选项有thin_pool_autoextend_threshold和thin_pool_autoextend_percent。
thin_pool_autoextend_threshold: 当使用量达到百分之多少是会尝试进行自动扩容,使用已经存在的空间。配置为100表示为不扩容(Disable)。
thin_pool_autoextend_percent: 在扩容是增加百分之多少空间。
The example below adds 20% more capacity when the disk usage reaches 80%.
下面配置是当卷的使用量达到80%的时候增加20%的容量。
activation {
thin_pool_autoextend_threshold=80
thin_pool_autoextend_percent=20
}
现在我们把这个配置写到配置文件/etc/lvm/profile/docker-thinpool.profile中。
[root@yds-dev-svc02-node01 ~]# cat /etc/lvm/profile/docker-thinpool.profile
activation {
thin_pool_autoextend_threshold=80
thin_pool_autoextend_percent=20
}
03.09 应用LVM配置文件
[root@yds-dev-svc02-node01 ~]# lvchange --metadataprofile docker-thinpool docker/thinpool
Logical volume docker/thinpool changed.
03.10 启用LVM监控
如果不启用lvm监控,刚才我们配置的自动扩容是不生效的。
[root@yds-dev-svc02-node01 ~]# lvs -o+seg_monitor
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Monitor
home centos -wi-ao---- 66.99g
root centos -wi-ao---- 50.00g
swap centos -wi-ao---- 2.00g
thinpool docker twi-a-t--- <228.00g 0.00 0.01 monitored
03.11 消除Docker数据
如果存在/var/lib/docker,将里面的文件备份或清空。
mkdir /var/lib/docker.bk
mv /var/lib/docker/* /var/lib/docker.bk
当配置完成后,如果出不错,就可以删除掉目录/var/lib/docker.bk
03.12 配置Docker存储驱动
如果在配置前/etc/docker/daemon.json为空。现在我们把这个文件修改成以下内容.
[root@yds-dev-svc02-node01 docker]# cat /etc/docker/daemon.json
{
"storage-driver": "devicemapper",
"storage-opts": [
"dm.thinpooldev=/dev/mapper/docker-thinpool",
"dm.use_deferred_removal=true",
"dm.use_deferred_deletion=true"
]
}
如果在/etc/sysconfig/docker-storage在有下面配置,删除掉.
DOCKER_STORAGE_OPTIONS=”–storage-driver devicemapper”
[root@yds-dev-svc02-node01 docker]# cat /etc/sysconfig/docker-storage
[root@yds-dev-svc02-node01 docker]#
查看/etc/sysconfig/docker-storage-setup中信息.
[root@yds-dev-svc02-node01 docker]# cat /etc/sysconfig/docker-storage-setup
STORAGE_DRIVER=devicemapper
03.13 启动Docker
systemctl start docker
systemctl status docker
03.13 验证配置
[root@yds-dev-svc02-node01 docker]# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 20.45 MB
Data Space Total: 244.8 GB
Data Space Available: 244.8 GB
Metadata Space Used: 311.3 kB
Metadata Space Total: 2.575 GB
Metadata Space Available: 2.575 GB
Thin Pool Minimum Free Space: 24.48 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 1.78 GiB
Name: yds-dev-svc02-node01
ID: YKWT:7Y6M:O3FB:C7BC:KU3Q:ZI5I:KM7E:QGTW:7TZV:2WF4:S5LD:ROKB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)
如果配置正确,Data file和Metadata file两是是空的,pool name 是docker-thinpool。
03.13 清理
当验证完我们配置后,可以将前面我们备份的目录删除掉.
rm -rf /var/lib/docker.bk
04 Docker的一些配置
由于默认的Base Device Size为10G,而经常Docker的大小会超过10G,需要修改这个值的大小。
我们这里把Base Device Size修改为30G。
只需要在/etc/docker/daemon.json中增加参数:dm.basesize=20G
[root@yds-dev-svc02-node01 docker]# cat /etc/docker/daemon.json
{
"storage-driver": "devicemapper",
"storage-opts": [
"dm.thinpooldev=/dev/mapper/docker-thinpool",
"dm.use_deferred_removal=true",
"dm.use_deferred_deletion=true",
"dm.basesize=20G"
]
}
重启Docker然后验证如下:
[root@yds-dev-svc02-node01 docker]# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 21.47 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 29.36 MB
Data Space Total: 244.8 GB
Data Space Available: 244.8 GB
Metadata Space Used: 311.3 kB
Metadata Space Total: 2.575 GB
Metadata Space Available: 2.575 GB
Thin Pool Minimum Free Space: 24.48 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 1.78 GiB
Name: yds-dev-svc02-node01
ID: YKWT:7Y6M:O3FB:C7BC:KU3Q:ZI5I:KM7E:QGTW:7TZV:2WF4:S5LD:ROKB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)
05 一些docker的自定义配置
05.01 /etc/sysconfig/docker配置文件
[root@yds-dev-svc02-node01 ~]# cat /etc/sysconfig/docker
# /etc/sysconfig/docker
# Modify these options if you want to change the way the docker daemon runs
OPTIONS=''
if [ -z "${DOCKER_CERT_PATH}" ]; then
DOCKER_CERT_PATH=/etc/docker
fi
# Do not add registries in this file anymore. Use /etc/containers/registries.conf
# from the atomic-registries package.
#
# On an SELinux system, if you remove the --selinux-enabled option, you
# also need to turn on the docker_transition_unconfined boolean.
# setsebool -P docker_transition_unconfined 1
# Location used for temporary files, such as those created by
# docker load and build operations. Default is /var/lib/docker/tmp
# Can be overriden by setting the following environment variable.
# DOCKER_TMPDIR=/var/tmp
# Controls the /etc/cron.daily/docker-logrotate cron job status.
# To disable, uncomment the line below.
# LOGROTATE=false
# docker-latest daemon can be used by starting the docker-latest unitfile.
# To use docker-latest client, uncomment below lines
#DOCKERBINARY=/usr/bin/docker-latest
#DOCKERDBINARY=/usr/bin/dockerd-latest
#DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest
#DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest
05.02 docker-storage-setup 配置文件
[root@yds-dev-svc02-node01 ~]# cat /etc/sysconfig/docker-storage-setup
STORAGE_DRIVER=devicemapper
05.03 /etc/docker/daemon.json 配置文件
[root@yds-dev-svc02-node01 ~]# cat /etc/docker/daemon.json
{
"storage-driver": "devicemapper",
"storage-opts": [
"dm.thinpooldev=/dev/mapper/docker-thinpool",
"dm.use_deferred_removal=true",
"dm.use_deferred_deletion=true",
"dm.basesize=20G"
],
"log-driver": "json-file",
"log-opts": {
"max-size": "200m",
"max-file": "5",
"labels": "prod"
},
"insecure-registries": [
"192.168.0.0/16"
],
"dns": [
"10.254.0.2",
"61.139.2.69"
],
"selinux-enabled": false,
"dns-search": [
"default.svc.cluster.local",
"svc.cluster.local"
],
"dns-opt": [
"ndots:2",
"timeout:2",
"attempts:2"
]
}
你的支持,是笔者最大的动力: