说明:本部署文章参照了 https://github.com/opsnull/follow-me-install-kubernetes-cluster ,欢迎给作者star
docker 是容器的运行环境,管理它的生命周期。kubelet 通过 Container Runtime Interface (CRI) 与 docker 进行交互
下载和分发 docker 二进制文件
到 https://download.docker.com/linux/static/stable/x86_64/ 页面下载最新发布包:
wget https://download.docker.com/linux/static/stable/x86_64/docker-18.06.3-ce.tgz tar -xvf docker-18.06.3-ce.tgz
分发二进制文件到所有 worker 节点:
cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.170 192.168.161.171 192.168.161.172 do echo ">>> ${node_ip}" scp docker/* root@${node_ip}:/opt/k8s/bin/ ssh root@${node_ip} "chmod +x /opt/k8s/bin/*" done
创建和分发 systemd unit 文件
cd /opt/k8s/work cat > docker.service <<"EOF" [Unit] Description=Docker Application Container Engine Documentation=http://docs.docker.io [Service] WorkingDirectory=##DOCKER_DIR## Environment="PATH=/opt/k8s/bin:/bin:/sbin:/usr/bin:/usr/sbin" EnvironmentFile=-/run/flannel/docker ExecStart=/opt/k8s/bin/dockerd $DOCKER_NETWORK_OPTIONS ExecReload=/bin/kill -s HUP $MAINPID Restart=on-failure RestartSec=5 LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity Delegate=yes KillMode=process [Install] WantedBy=multi-user.target EOF
-
EOF 前后有双引号,这样 bash 不会替换文档中的变量,如 $DOCKER_NETWORK_OPTIONS;
-
dockerd 运行时会调用其它 docker 命令,如 docker-proxy,所以需要将 docker 命令所在的目录加到 PATH 环境变量中;
-
flanneld 启动时将网络配置写入
/run/flannel/docker
文件中,dockerd 启动前读取该文件中的环境变量DOCKER_NETWORK_OPTIONS
,然后设置 docker0 网桥网段; -
如果指定了多个
EnvironmentFile
选项,则必须将/run/flannel/docker
放在最后(确保 docker0 使用 flanneld 生成的 bip 参数); -
docker 需要以 root 用于运行;
-
docker 从 1.13 版本开始,可能将 iptables FORWARD chain的默认策略设置为DROP,从而导致 ping 其它 Node 上的 Pod IP 失败,遇到这种情况时,需要手动设置策略为
ACCEPT
:
sudo iptables -P FORWARD ACCEPT
并且把以下命令写入 /etc/rc.local
文件中,防止节点重启iptables FORWARD chain的默认策略又还原为DROP
/sbin/iptables -P FORWARD ACCEPT
完整 unit 见 docker.service
分发 systemd unit 文件到所有 worker 机器:
cd /opt/k8s/work source /opt/k8s/bin/environment.sh sed -i -e "s/##DOCKER_DIR##/${DOCKER_DIR}/" docker.service for node_ip in 192.168.161.170 192.168.161.171 192.168.161.172 do echo ">>> ${node_ip}" scp docker.service root@${node_ip}:/etc/systemd/system/ done
配置和分发 docker 配置文件
cd /opt/k8s/work source /opt/k8s/bin/environment.sh cat > docker-daemon.json <<EOF { "registry-mirrors": ["https://hub-mirror.c.163.com", "https://docker.mirrors.ustc.edu.cn"], "insecure-registries": ["docker02:35000"], "max-concurrent-downloads": 20, "live-restore": true, "max-concurrent-uploads": 10, "debug": true, "data-root": "${DOCKER_DIR}/data", "exec-root": "${DOCKER_DIR}/exec", "log-opts": { "max-size": "100m", "max-file": "5" } } EOF
分发 docker 配置文件到所有 work 节点:
cd /opt/k8s/work source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.170 192.168.161.171 192.168.161.172 do echo ">>> ${node_ip}" ssh root@${node_ip} "mkdir -p /etc/docker/ ${DOCKER_DIR}/{data,exec}" scp docker-daemon.json root@${node_ip}:/etc/docker/daemon.json done
启动 docker 服务
source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.170 192.168.161.171 192.168.161.172 do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl stop firewalld && systemctl disable firewalld" ssh root@${node_ip} "/usr/sbin/iptables -F && /usr/sbin/iptables -X && /usr/sbin/iptables -F -t nat && /usr/sbin/iptables -X -t nat" ssh root@${node_ip} "/usr/sbin/iptables -P FORWARD ACCEPT" ssh root@${node_ip} "systemctl daemon-reload && systemctl enable docker && systemctl restart docker" #ssh root@${node_ip} 'for intf in /sys/devices/virtual/net/docker0/brif/*; do echo 1 > $intf/hairpin_mode; done' ssh root@${node_ip} "sudo sysctl -p /etc/sysctl.d/kubernetes.conf" done
- 关闭 firewalld(centos7)/ufw(ubuntu16.04),否则可能会重复创建 iptables 规则;
- 清理旧的 iptables rules 和 chains 规则;
- 开启 docker0 网桥下虚拟网卡的 hairpin 模式;
检查服务运行状态
source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.170 192.168.161.171 192.168.161.172 do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl status docker|grep Active" done
确保状态为 active (running)
,否则查看日志,确认原因:
journalctl -u docker
检查 docker0 网桥
source /opt/k8s/bin/environment.sh for node_ip in 192.168.161.170 192.168.161.171 192.168.161.172 do echo ">>> ${node_ip}" ssh root@${node_ip} "/usr/sbin/ip addr show flannel.1 && /usr/sbin/ip addr show docker0" done
确认各 work 节点的 docker0 网桥和 flannel.1 接口的 IP 处于同一个网段中(如下):