Kubernetes快速部署

Kubernetes快速部署

kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具。

安装要求

在开始之前，部署Kubernetes集群机器需要满足以下几个条件：

-至少3台机器，操作系统 CentOS7+

• 硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘20GB或更多
• 集群中所有机器之间网络互通
• 可以访问外网，需要拉取镜像
• 禁止swap分区

环境：

主机	IP地址
master	192.168.244.146
node1	192.168.244.147
node2	192.168.244.148

准备工作：

关闭三主机上的防火墙和selinux

[root@master ~]# systemctl disable --now firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@master ~]# vim /etc/selinux/config 
[root@master ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

删除或注释swap（所有主机）

[root@master ~]# cat /etc/fstab
.......
/dev/mapper/cl-root     /                       xfs     defaults        0 0
UUID=d9b2bc8d-92af-4609-b95e-2bafce46ab1b /boot                   xfs     defaults        0 0
/dev/mapper/cl-home     /home                   xfs     defaults        0 0
#/dev/mapper/cl-swap     none                    swap    defaults        0 0
........

在master添加hosts主机映射

[root@master ~]# vim /etc/hosts 
[root@master ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.244.146 master master.example.com
192.168.244.147 node1 node1.example.com
192.168.244.148 nose2 node2.example.com

[root@master ~]# ping master
PING master (192.168.244.146) 56(84) bytes of data.
64 bytes from master (192.168.244.146): icmp_seq=1 ttl=64 time=0.031 ms
64 bytes from master (192.168.244.146): icmp_seq=2 ttl=64 time=0.044 ms
^C
--- master ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1037ms
rtt min/avg/max/mdev = 0.031/0.037/0.044/0.008 ms


[root@master ~]# ping node1
PING node1 (192.168.244.147) 56(84) bytes of data.
64 bytes from node1 (192.168.244.147): icmp_seq=1 ttl=64 time=0.700 ms
64 bytes from node1 (192.168.244.147): icmp_seq=2 ttl=64 time=0.397 ms
^C
--- node1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1050ms
rtt min/avg/max/mdev = 0.397/0.548/0.700/0.153 ms

[root@master ~]# ping node2
PING node2 (192.168.244.148) 56(84) bytes of data.
64 bytes from node2 (192.168.244.148): icmp_seq=1 ttl=64 time=1.29 ms
64 bytes from node2 (192.168.244.148): icmp_seq=2 ttl=64 time=0.970 ms
64 bytes from node2 (192.168.244.148): icmp_seq=3 ttl=64 time=0.390 ms
^C
--- node2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.390/0.882/1.288/0.373 ms
[root@master ~]# 

在master上将桥接的IPv4流量传递到iptables的链：

[root@master ~]# cat /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
[root@master ~]# sysctl --system 
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-coredump.conf ...
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
kernel.kptr_restrict = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.promote_secondaries = 1
net.core.default_qdisc = fq_codel
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ...
net.core.optmem_max = 81920
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/k8s.conf ...
* Applying /etc/sysctl.conf ...

时间同步

（所有主机）

[root@master ~]# cat /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
pool time1.aliyun.com iburst
#重启服务
[root@master ~]# systemctl enable --now chronyd

免密认证

在master上生成密钥传给node

[root@master ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:3pttk58gSDH7bRkrCXQTEw0/+9dw0Ts0otUNNB8yn1U root@master.example.com
The key's randomart image is:
+---[RSA 3072]----+
|          =+ ++.E|
|           +. =+*|
|        + o oo B+|
|       . = .ooo +|
|        S  .o .o.|
|       o = o = oo|
|        o * *.. o|
|           B+. o |
|          o..oo  |
+----[SHA256]-----+

[root@master ~]# ssh-copy-id node1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'node1 (192.168.244.147)' can't be established.
ECDSA key fingerprint is SHA256:L+Q5+H8loZWMsfvfYfpoiRN3S4H+mlHu/RICihCRTv4.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@node1's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'node1'"
and check to make sure that only the key(s) you wanted were added.

[root@master ~]# ssh-copy-id node2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'node2 (192.168.244.148)' can't be established.
ECDSA key fingerprint is SHA256:0vl3YceXpjhPzNdTBo0pMWoWTRMwYn06jbjVNdnzK4M.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@node2's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'node2'"
and check to make sure that only the key(s) you wanted were added.

查看时间同步

[root@master ~]# for i in master node1 node2;do ssh $i 'date';done
Sat Dec 18 21:29:34 CST 2021
Sat Dec 18 21:29:34 CST 2021
Sat Dec 18 21:29:34 CST 2021

重启所有主机让服务生效

所有节点安装Docker/kubeadm/kubelet

安装docker

#安装docker源
[root@master ~]#  wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
#安装docker 
[root@master ~]# yum -y install docker-ce


添加镜像加速器
[root@master ~]# cat /etc/docker/daemon.json
{
  "registry-mirrors": ["https://k654a2v7.mirror.aliyuncs.com"],  #加速器
  "exec-opts": ["native.cgroupdriver=systemd"],  #驱动
  "log-driver": "json-file",    #日志类型
  "log-opts": {
    "max-size": "100m"  #日志大小 超过100M就滚动，生成一个新的日志文件
  },
  "storage-driver": "overlay2"
}

#启动容器
[root@master ~]# systemctl enable --now docker 
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.

添加kubernetes阿里云YUM软件源

[root@master ~]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

安装kubeadm，kubelet和kubectl

由于版本更新频繁，这里指定版本号部署：

[root@master ~]# yum  -y install kubelet-1.20.0 kubeadm-1.20.0 kubectl-1.20.0
[root@master ~]# systemctl enable kubelet
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /usr/lib/systemd/system/kubelet.service.

部署Kubernetes Master

在192.168.244.146（Master）执行。

[root@master ~]# kubeadm init --apiserver-advertise-address=192.168.244.146 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.20.0 --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.20.0
[preflight] Running pre-flight checks
        [WARNING FileExisting-tc]: tc not found in system path
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.12. Latest validated version: 19.03
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master.example.com] and IPs [10.96.0.1 192.168.244.146]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master.example.com] and IPs [192.168.244.146 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master.example.com] and IPs [192.168.244.146 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 8.002225 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master.example.com as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node master.example.com as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: brts9h.197chej3t5t3c946
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.244.146:6443 --token brts9h.197chej3t5t3c946 \
    --discovery-token-ca-cert-hash sha256:e0769d77cbc853ff92ad79420ee556c72d8a21ed44a3812e94edc3b7929e3416 



[root@master ~]# echo 'export KUBECONFIG=/etc/kubernetes/admin.conf' > /etc/profile.d/k8s.sh
[root@master ~]# source /etc/profile.d/k8s.sh

由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址。

安装Pod网络插件（CNI）

[root@master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

#或者下载
[root@master ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
[root@master ~]#  kubectl apply -f  kube-flannel.yml

加入Kubernetes Node

在node1和2上执行

[root@node1 ~]# kubeadm join 192.168.244.146:6443 --token brts9h.197chej3t5t3c946 \
>     --discovery-token-ca-cert-hash sha256:e0769d77cbc853ff92ad79420ee556c72d8a21ed44a3812e94edc3b7929e3416 

在master查看

[root@master ~]# kubectl get nodes
NAME                 STATUS   ROLES                  AGE     VERSION
master.example.com   Ready    control-plane,master   39m     v1.20.0
node1.example.com    Ready    <none>                 2m23s   v1.20.0
node2.example.com    Ready    <none>                 2m19s   v1.20.0

创建容器验证

[root@master ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@master ~]# kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed
[root@master ~]# kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-6799fc88d8-cfgpr   1/1     Running   0          2m38s
[root@master ~]# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP           NODE                NOMINATED NODE   READINESS GATES
nginx-6799fc88d8-cfgpr   1/1     Running   0          2m52s   10.244.2.2   node2.example.com   <none>           <none>
[root@master ~]# curl 10.244.2.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>