K8S 集群环境准备及Docker 安装

二进制搭建环境

一 环境准备

192.168.56.107 k8s-master01
192.168.56.108 k8s-master02
192.168.56.109 k8s-master02
192.168.56.236 k8s-master-lb
192.168.56.110 k8s-node01
192.168.56.111 k8s-node02

k8s service 网段： 10.96.0.0/12

k8s pod 网段： 172.168.0.0/12

二 修改文件配置源

五台设备一起更改

vi /etc/hosts

192.168.56.107 k8s-master01
192.168.56.108 k8s-master02
192.168.56.109 k8s-master03
192.168.56.236 k8s-master-lb
192.168.56.110 k8s-node01
192.168.56.111 k8s-node02

五台设备centos yum 源

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

五台设备安装必备的工具

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git -y

五台设备关闭selinux 防火墙

systemctl disable --now firewalld 
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

五台设备确认selinux 处于disabled 状态，然而并不生效，需要关机重启才生效

getenforce

五台设备关闭swap 分区， fstab 注释swap

swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

五台设备下载同步时间组件

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install ntpdate -y

五台设备同步时间

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com

五台设备加到cornjob里

crontab -e

 */5 * * * * /usr/sbin/ntpdate time2.aliyun.com

五台设备配置limit

ulimit -SHn 65535

vim /etc/security/limits.conf

* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited　

master01 配置ssh key 免密登录到其它的节点，需要传送一些生成的证书文件,安装过程中生成的证书全在master01 上面生成，aws 阿里云需要单独的一台kubectl 进行操作，完成如下操作

master01 节点上执行命令，xshell关闭全局执行命令传输

ssh-keygen -t rsa

免密登录到其它节点

for i in k8s-node01 k8s-node02 k8s-master02 k8s-master03;do ssh-copy-id -i .ssh/id_rsa.pub $i; done

clone 基础组件配置文件，下载到master01节点上

 cd /root/ ; git clone https://github.com/dotbalo/k8s-ha-install.git

三 升级内核章节

五台设备需要升级内核 ,跟新yum 源 重启设备

yum update -y --exclude=kernel* && reboot  

下载内核，安装内核,master01 下载内核，然后再传输到其它节点上

cd /root
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm

拷贝到其它的节点上

cd /root/

for i in  k8s-master02 k8s-master03  k8s-node01 k8s-node02;do scp kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/ ; done

所有节点安装内核

cd /root && yum localinstall -y kernel-ml*

所有节点更改内核启动顺序

grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

查看默认内核是否为4.19

grubby --default-kernel

做完操作后，所有节点要重启 reboot

reboot
uname -a

所有节点安装ipvsadm ，ipvs 比iptable 效率高，因此都安装ipvs

yum install ipvsadm ipset sysstat conntrack libseccomp -y

修改ipvs 内核配置，所有节点;内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4

vim /etc/modules-load.d/ipvs.conf 

ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip

所有节点服务生效,待重新启动

systemctl enable --now systemd-modules-load.service 

开启k8s 中所有节点需要的内核参数，k8s必要的参数,然后让配置生效

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
net.ipv4.conf.all.route_localnet = 1

vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF

sysctl --system

所有节点配置完成后，重新启动然后查看是否加载成功

reboot
lsmod | grep --color=auto -e ip_vs -e nf_conntrack

四 安装docker runtime 环境

所有节点安装docke-ce

yum install docker-ce-19.03.*  -y

新版kubectl 建议使用 systemd ,将docker CGroupdriver改用为systemd

mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

所有节点设置为开机自启动

systemctl daemon-reload && systemctl enable --now docker

打开安装的changelog

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

master01节点下载上面k8s 二进制安装文件