Centos8下Kubernetes 1.19 环境搭建实战（iptables篇）

使用 kubeadm 搭建Kubernetes 1.19.0单节点集群实战（基于CentOS Linux release 8.2.2004 (Core)）

录屏视频请访问：https://www.bilibili.com/video/bv1Pf4y1X7js

详细步骤：

1，把IP地址改为你想使用的IP，比如，192.168.31.12，然后修改hosts，

sudo su -

vi /etc/hosts

追加enp0s3的ip和hostname，例如

192.168.31.12 k8s119-master

hostnamectl set-hostname k8s119-master

2，安装docker

2.1，Uninstall old versions

sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

2.2，Set up the repository

sudo yum install -y yum-utils

sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

2.3，Install docker engine

sudo yum install -y docker-ce docker-ce-cli https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm

2.4，Set docker to start on boot and start docker

sudo systemctl enable --now docker

2.5，Add docker group to normal user

sudo usermod -a -G docker oracle
sudo su - oracle

2.6，Verify that Docker Engine

docker run hello-world

2.7，Set cgroupdriver=systemd

sudo su -

cat > /etc/docker/daemon.json <<EOF
{
   "exec-opts": ["native.cgroupdriver=systemd"],
   "log-driver": "json-file",
   "log-opts": {
     "max-size": "100m"
   },
   "storage-driver": "overlay2",
   "storage-opts": [
     "overlay2.override_kernel_check=true"
   ]
}
EOF

exit
sudo systemctl daemon-reload
sudo systemctl restart docker

3，安装kubernetes

3.1，Turn off swap

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

3.2，Set selinux

# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

3.3，Letting iptables see bridged traffic

lsmod | grep br_netfilter
sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

3.3，Check required ports

sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=2379-2380/tcp
sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --permanent --add-port=10251/tcp
sudo firewall-cmd --permanent --add-port=10252/tcp
sudo firewall-cmd --permanent --add-port=10255/tcp
sudo firewall-cmd --permanent --add-masquerade 
# only if you want NodePorts exposed on control plane IP as well
sudo firewall-cmd --permanent --add-port=30000-32767/tcp
sudo firewall-cmd --reload

3.4，Installing kubeadm, kubelet and kubectl

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

3.5，Initializing control-plane node

sudo su -

export IP_ADDR=$(ip addr show enp0s3 | grep -Po 'inet \K[\d.]+')
echo $IP_ADDR
kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=$IP_ADDR --kubernetes-version=stable-1.19

 3.6，To start using your cluster, you need to run the following as a regular user

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config


echo "source <(kubectl completion bash)" >> ~/.bashrc
echo "alias k=kubectl" >> ~/.bashrc
echo "complete -F __start_kubectl k" >> ~/.bashrc
source ~/.bashrc

3.7，Installing Calico for policy and flannel (aka Canal) for networking

curl https://docs.projectcalico.org/manifests/canal.yaml -O
kubectl apply -f canal.yaml

3.8，Control plane node isolation

kubectl taint nodes --all node-role.kubernetes.io/master-

3.9，Verify kubernetes cluster

[oracle@k8s119-master ~]$ kubectl get nodes
NAME            STATUS   ROLES    AGE    VERSION
k8s119-master   Ready    master   7m3s   v1.19.0

 

[oracle@k8s119-master ~]$ kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-5bc4fc6f5f-jbw87   1/1     Running   0          3m50s
kube-system   canal-7rwqf                                2/2     Running   0          3m50s
kube-system   coredns-f9fd979d6-kffgc                    1/1     Running   0          7m
kube-system   coredns-f9fd979d6-pqdsc                    1/1     Running   0          7m
kube-system   etcd-k8s119-master                         1/1     Running   0          7m11s
kube-system   kube-apiserver-k8s119-master               1/1     Running   0          7m11s
kube-system   kube-controller-manager-k8s119-master      1/1     Running   0          7m11s
kube-system   kube-proxy-jq48r                           1/1     Running   0          7m
kube-system   kube-scheduler-k8s119-master               1/1     Running   0          7m11s

3.10，Deploy nginx

kubectl create deployment nginx --image=nginx

[oracle@k8s119-master ~]$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE            NOMINATED NODE   READINESS GATES
nginx-6799fc88d8-8qdrg   1/1     Running   0          51s   10.244.0.2   k8s119-master   <none>           <none>

[oracle@k8s119-master ~]$ curl 10.244.0.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

恭喜，kubernetes 1.19.0正常安装完成。

接下来是清除步骤，请根据需要执行。 

4，Clean up worker node

4.1，Remove the node

kubectl drain <node name> --delete-local-data --force --ignore-daemonsets

4.2，Before removing the node, reset the state installed by kubeadm:

sudo su -
kubeadm reset

 4.3，The reset process does not reset or clean up iptables rules or IPVS tables. If you wish to reset iptables, you must do so manually

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

4.4，If you want to reset the IPVS tables, you must run the following command:

ipvsadm -C

 4.5，Now remove the node:

kubectl delete node <node name>

5，Clean up the control plane  

sudo su -
kubeadm reset

6，Joining your nodes

kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>

6.1，If you do not have the token, you can get it by running the following command on the control-plane node:

kubeadm token list

6.2，By default, tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, you can create a new token by running the following command on the control-plane node:

kubeadm token create

6.3，If you don't have the value of --discovery-token-ca-cert-hash, you can get it by running the following command chain on the control-plane node:

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'