Kubernetes1.9生产环境高可用实践--003-node中docker安装及配置

最新推荐文章于 2023-04-21 12:28:31 发布

wenwst

最新推荐文章于 2023-04-21 12:28:31 发布

阅读量1.4k

点赞数

分类专栏： kubernetes k8s docker 文章标签：集群 kubernetes docker 高可用生产环境

本文链接：https://blog.csdn.net/wenwst/article/details/79885556

版权

kubernetes 同时被 3 个专栏收录

34 篇文章 0 订阅

订阅专栏

k8s

26 篇文章 0 订阅

订阅专栏

docker

17 篇文章 0 订阅

订阅专栏

Apiserver采用高可用方式安装部署。这篇文章接上一篇《Kubernetes1.9生产环境高可用实践–002-apiserver高可用安装部署》。

在这一篇，我们着重写在node节点中如何安装Docker，以及Docker的配置。
kubernetes结点的配置我们会放到下一节中进行讨论。

这篇在很多的文章中都写的比较简单。我们在这里写的相对麻烦一些，不过，如果要是在生产环境中配置，还是见意使用这样的配置。

我们在这里安装三个节点：
192.168.3.56 yds-dev-svc02-node01
192.168.3.57 yds-dev-svc02-node02
192.168.3.58 yds-dev-svc02-node03

三个节点的安装过程是一样的，因此，我们在这里只配置yds-dev-svc02-node01，其他节点采用相同的配置。

01 服务器配置

01.01 配置服务器名和IP

[root@localhost ~]# hostnamectl set-hostname yds-dev-svc02-node01
[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens32 
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens32
UUID=7d6fb2ed-364c-415f-9b02-0e54436ff1ec
DEVICE=ens32
ONBOOT=yes
IPADDR=192.168.3.56
NETMASK=255.255.255.0
GATEWAY=192.168.3.1
DNS1=192.168.3.10
DNS2=61.139.2.69

配置完成后，退出重新登录。

02 Docker安装

02.01 安装epel-release

yum update -y 
yum install -y epel-release wget lsof

02.02 安装Docker

yum install -y docker

[root@yds-dev-svc02-node01 ~]# docker --version
Docker version 1.13.1, build 774336d/1.13.1

02.03 启动docker

systemctl enable docker
systemctl start docker
systemctl status docker

02.04 检查docker

[root@yds-dev-svc02-node01 ~]# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
 selinux
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 1.78 GiB
Name: yds-dev-svc02-node01
ID: YKWT:7Y6M:O3FB:C7BC:KU3Q:ZI5I:KM7E:QGTW:7TZV:2WF4:S5LD:ROKB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)

现在Docker默认的Storage Driver为overlay2，只适用于测试环境。但我们在生产环境，需要把这一项改为devicemapper的direct-lvm模式，不要在生产中使用loop-lvm模式。针对Docker的存储驱动，网上有很多的关于这方面的讲解，如果时间充足，会翻译几篇关于这方面的对比文档。

接下来，我们把docker的存储改为direct-lvm。

03 配置Docker direct-lvm模式

03.01 硬盘检查

我们先查看一下硬盘信息。

[root@yds-dev-svc02-node01 ~]# fdisk -l

Disk /dev/sda: 128.8 GB, 128849018880 bytes, 251658240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a9fcf

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   251658239   124779520   8e  Linux LVM

Disk /dev/sdb: 257.7 GB, 257698037760 bytes, 503316480 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-home: 71.9 GB, 71932313600 bytes, 140492800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

可以看到/dev/sdb为未使用的硬盘。
现在，我们来处理/dev/sdb来存放docker数据。

03.02 停止Docker

systemctl stop docker

03.03 安装软件

yum install -y lvm2 device-mapper-persistent-data

03.04 创建物理卷

[root@yds-dev-svc02-node01 ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.

03.05 创建Docker卷组

[root@yds-dev-svc02-node01 ~]# vgcreate docker /dev/sdb
  Volume group "docker" successfully created

03.06 创建逻辑卷

在这里需要创建二个逻辑卷，名称为thinpool和thinpoolmeta。

[root@yds-dev-svc02-node01 ~]# lvcreate --wipesignatures y -n thinpool docker -l 95%VG
  Logical volume "thinpool" created.

[root@yds-dev-svc02-node01 ~]# lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
  Logical volume "thinpoolmeta" created.

03.07 卷转换

Convert the volumes to a thin pool and a storage location for metadata for the thin pool, using the lvconvert command.
将刚创建的卷转换为thin pool，并使用刚创建的thinpoolmeta卷。

[root@yds-dev-svc02-node01 ~]# lvconvert -y \
--zero n \
-c 512K \
--thinpool docker/thinpool \
--poolmetadata docker/thinpoolmeta
  Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
  WARNING: Converting logical volume docker/thinpool and docker/thinpoolmeta to thin pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  Converted docker/thinpool_tdata to thin pool.

03.08 自动扩展配置

需要配置的选项有thin_pool_autoextend_threshold和thin_pool_autoextend_percent。

thin_pool_autoextend_threshold: 当使用量达到百分之多少是会尝试进行自动扩容，使用已经存在的空间。配置为100表示为不扩容(Disable)。
thin_pool_autoextend_percent: 在扩容是增加百分之多少空间。
The example below adds 20% more capacity when the disk usage reaches 80%.
下面配置是当卷的使用量达到80%的时候增加20%的容量。

activation {
  thin_pool_autoextend_threshold=80
  thin_pool_autoextend_percent=20
}

现在我们把这个配置写到配置文件/etc/lvm/profile/docker-thinpool.profile中。

[root@yds-dev-svc02-node01 ~]# cat /etc/lvm/profile/docker-thinpool.profile
activation {
  thin_pool_autoextend_threshold=80
  thin_pool_autoextend_percent=20
}

03.09 应用LVM配置文件

[root@yds-dev-svc02-node01 ~]# lvchange --metadataprofile docker-thinpool docker/thinpool
  Logical volume docker/thinpool changed.

03.10 启用LVM监控

如果不启用lvm监控，刚才我们配置的自动扩容是不生效的。

[root@yds-dev-svc02-node01 ~]# lvs -o+seg_monitor
  LV       VG     Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Monitor  
  home     centos -wi-ao----   66.99g                                                              
  root     centos -wi-ao----   50.00g                                                              
  swap     centos -wi-ao----    2.00g                                                              
  thinpool docker twi-a-t--- <228.00g             0.00   0.01                             monitored

03.11 消除Docker数据

如果存在/var/lib/docker，将里面的文件备份或清空。

mkdir /var/lib/docker.bk
mv /var/lib/docker/* /var/lib/docker.bk

当配置完成后，如果出不错，就可以删除掉目录/var/lib/docker.bk

03.12 配置Docker存储驱动

如果在配置前/etc/docker/daemon.json为空。现在我们把这个文件修改成以下内容.

[root@yds-dev-svc02-node01 docker]# cat /etc/docker/daemon.json
{
    "storage-driver": "devicemapper",
    "storage-opts": [
    "dm.thinpooldev=/dev/mapper/docker-thinpool",
    "dm.use_deferred_removal=true",
    "dm.use_deferred_deletion=true"
    ]
}

如果在/etc/sysconfig/docker-storage在有下面配置，删除掉.
DOCKER_STORAGE_OPTIONS=”–storage-driver devicemapper”

[root@yds-dev-svc02-node01 docker]# cat /etc/sysconfig/docker-storage
[root@yds-dev-svc02-node01 docker]#

查看/etc/sysconfig/docker-storage-setup中信息.

[root@yds-dev-svc02-node01 docker]# cat /etc/sysconfig/docker-storage-setup 
STORAGE_DRIVER=devicemapper

03.13 启动Docker

systemctl start docker
systemctl status docker

03.13 验证配置

[root@yds-dev-svc02-node01 docker]# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 20.45 MB
 Data Space Total: 244.8 GB
 Data Space Available: 244.8 GB
 Metadata Space Used: 311.3 kB
 Metadata Space Total: 2.575 GB
 Metadata Space Available: 2.575 GB
 Thin Pool Minimum Free Space: 24.48 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: journald
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
 selinux
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 1.78 GiB
Name: yds-dev-svc02-node01
ID: YKWT:7Y6M:O3FB:C7BC:KU3Q:ZI5I:KM7E:QGTW:7TZV:2WF4:S5LD:ROKB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)

如果配置正确，Data file和Metadata file两是是空的，pool name 是docker-thinpool。

03.13 清理

当验证完我们配置后，可以将前面我们备份的目录删除掉.

rm -rf /var/lib/docker.bk

04 Docker的一些配置

由于默认的Base Device Size为10G，而经常Docker的大小会超过10G，需要修改这个值的大小。
我们这里把Base Device Size修改为30G。
只需要在/etc/docker/daemon.json中增加参数：dm.basesize=20G

[root@yds-dev-svc02-node01 docker]# cat /etc/docker/daemon.json 
{
    "storage-driver": "devicemapper",
    "storage-opts": [
    "dm.thinpooldev=/dev/mapper/docker-thinpool",
    "dm.use_deferred_removal=true",
    "dm.use_deferred_deletion=true",
    "dm.basesize=20G"
    ]
}

重启Docker然后验证如下:

[root@yds-dev-svc02-node01 docker]# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3 kB
 Base Device Size: 21.47 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 29.36 MB
 Data Space Total: 244.8 GB
 Data Space Available: 244.8 GB
 Metadata Space Used: 311.3 kB
 Metadata Space Total: 2.575 GB
 Metadata Space Available: 2.575 GB
 Thin Pool Minimum Free Space: 24.48 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: journald
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
 selinux
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 1.78 GiB
Name: yds-dev-svc02-node01
ID: YKWT:7Y6M:O3FB:C7BC:KU3Q:ZI5I:KM7E:QGTW:7TZV:2WF4:S5LD:ROKB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)

05 一些docker的自定义配置

05.01 /etc/sysconfig/docker配置文件

[root@yds-dev-svc02-node01 ~]# cat /etc/sysconfig/docker
# /etc/sysconfig/docker

# Modify these options if you want to change the way the docker daemon runs
OPTIONS=''
if [ -z "${DOCKER_CERT_PATH}" ]; then
    DOCKER_CERT_PATH=/etc/docker
fi

# Do not add registries in this file anymore. Use /etc/containers/registries.conf
# from the atomic-registries package.
#

# On an SELinux system, if you remove the --selinux-enabled option, you
# also need to turn on the docker_transition_unconfined boolean.
# setsebool -P docker_transition_unconfined 1

# Location used for temporary files, such as those created by
# docker load and build operations. Default is /var/lib/docker/tmp
# Can be overriden by setting the following environment variable.
# DOCKER_TMPDIR=/var/tmp

# Controls the /etc/cron.daily/docker-logrotate cron job status.
# To disable, uncomment the line below.
# LOGROTATE=false

# docker-latest daemon can be used by starting the docker-latest unitfile.
# To use docker-latest client, uncomment below lines
#DOCKERBINARY=/usr/bin/docker-latest
#DOCKERDBINARY=/usr/bin/dockerd-latest
#DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest
#DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest

05.02 docker-storage-setup 配置文件

[root@yds-dev-svc02-node01 ~]# cat /etc/sysconfig/docker-storage-setup 
STORAGE_DRIVER=devicemapper

05.03 /etc/docker/daemon.json 配置文件

[root@yds-dev-svc02-node01 ~]# cat /etc/docker/daemon.json 
{
    "storage-driver": "devicemapper",
    "storage-opts": [
        "dm.thinpooldev=/dev/mapper/docker-thinpool",
        "dm.use_deferred_removal=true",
        "dm.use_deferred_deletion=true",
        "dm.basesize=20G"
    ],
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "200m",
        "max-file": "5",
        "labels": "prod"
    },
    "insecure-registries": [
        "192.168.0.0/16"
    ],
    "dns": [
        "10.254.0.2",
        "61.139.2.69"
    ],
        "selinux-enabled": false,
        "dns-search": [
                "default.svc.cluster.local",
                "svc.cluster.local"
         ],
        "dns-opt": [
                "ndots:2",
                "timeout:2",
                "attempts:2"
         ]
}

你的支持，是笔者最大的动力：
这里写图片描述

wenwst

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Kubernetes1.9生产环境高可用实践--003-node中docker安装及配置

Apiserver采用高可用方式安装部署。这篇文章接上一篇《Kubernetes1.9生产环境高可用实践–002-apiserver高可用安装部署》。在这一篇，我们着重写在node节点中如何安装Docker，以及Docker的配置。 kubernetes结点的配置我们会放到下一节中进行讨论。这篇在很多的文章中都写的比较简单。我们在这里写的相对麻烦一些，不过，如果要是在...
复制链接

扫一扫

专栏目录