第一篇：kubernetes部署实操

说明： 总的目标是在k8s集群部署gitlab、jenkins，并且在本地提交代码到gitlab后jenkin流水线可以自动编译打包成为docker镜像然后部署到k8s中并实现客户端外部域名访问，在文档分为多个部分，其中涉及的技术有docker安装、k8s搭建、部署gitlab、部署jenkins、部署sonarqube、gitlab和jenkin联动、jenkins和sonarqube联动、pipline脚本编写、istio部署、istio服务网关等…

这篇文档讲解的是k8s的部署

1. 环境说明

我在本地电脑起了三个虚拟机，其中一台为k8s主节点，两台是k8s工作节点
系统： ubuntu-20.04.4
k8s版本：1.22.20
docker版本：20.10.12
ip地址规划如下：

主机名	k8s角色	ip地址
k8s-master	主节点	192.168.100.230
k8s-work01	工作节点	192.168.100.231
k8s-work02	工作节点	192.168.100.232

2. 更改主机名

三台主机名都需要更改，且需要更改为不一样（切记）

root@ubuntu:~# hostnamectl set-hostname k8s-master

root@ubuntu:~# hostnamectl set-hostname k8s-work01

root@ubuntu:~# hostnamectl set-hostname k8s-work02

注意： 更改完成后重新连接console终端后主机名就可以看到了

3. 更改hosts文件

k8s-master节点如下

// 文件位置/etc/hosts
root@k8s-master:~# cat /etc/hosts
127.0.0.1       localhost
127.0.1.1       k8s-master

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

k8s-work01节点如下

// 文件位置/etc/hosts
root@k8s-master:~# cat /etc/hosts
127.0.0.1       localhost
127.0.1.1       k8s-work01

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

k8s-work02节点如下

// 文件位置/etc/hosts
root@k8s-master:~# cat /etc/hosts
127.0.0.1       localhost
127.0.1.1       k8s-work02

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouter

4. 安装依赖软件

// 三台机器都需要执行
root@k8s-master:~# apt-get install -y \
>     apt-transport-https \
>     ca-certificates \
>     curl \
>     gnupg \
>     lsb-release

5. 安装docker

// 三台机器都需要执行
root@k8s-master:~# apt-get install -y docker.io

6. 安装ntpdate和ntp并同步时间

k8s所有节点时间必须一致， 否则后期会报错
#首先安装ntpdate和ntp

// 三台机器都需要执行
root@k8s-work01:~# apt-get install -y ntpdate ntp

#关闭三台机器的ntp服务

// 三台机器都需要执行
root@k8s-work01:~# systemctl stop ntp

#调整时间

// 三台机器都需要执行
root@k8s-master:~# ntpdate ntp.ubuntu.com
 9 Jan 00:52:55 ntpdate[20513]: adjust time server 91.189.91.157 offset -0.010919 sec

#开启三台机器的ntp服务

// 三台机器都需要执行
root@k8s-work01:~# systemctl start ntp

7. 安装kubernetes

以下命令三台机器都需要执行安装

#首先添加安装源

root@k8s-master:~# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
> deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
> EOF
root@k8s-master:~#

#gpg添加密钥

root@k8s-master:~# gpg --keyserver keyserver.ubuntu.com --recv-keys FEEA9169307EA071
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: key FEEA9169307EA071: 1 duplicate signature removed
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key FEEA9169307EA071: public key "Rapture Automatic Signing Key (cloud-rapture-signing-key-2021-03-01-08_01_09.pub)" imported
gpg: Total number processed: 1
gpg:               imported: 1
root@k8s-master:~# gpg --export --armor FEEA9169307EA071 | sudo apt-key add -
OK
root@k8s-master:~# gpg --keyserver keyserver.ubuntu.com --recv-keys 8B57C5C2836F4BEB
gpg: key 8B57C5C2836F4BEB: public key "gLinux Rapture Automatic Signing Key (//depot/google3/production/borg/cloud-rapture/keys/cloud-rapture-pubkeys/cloud-rapture-signing-key-2020-12-03-16_08_05.pub) <glinux-team@google.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
root@k8s-master:~# gpg --export --armor 8B57C5C2836F4BEB | sudo apt-key add -
OK
root@k8s-master:~#

上面总共四条命令如下：
gpg --keyserver keyserver.ubuntu.com --recv-keys FEEA9169307EA071
gpg --export --armor FEEA9169307EA071 | sudo apt-key add -
gpg --keyserver keyserver.ubuntu.com --recv-keys 8B57C5C2836F4BEB
gpg --export --armor 8B57C5C2836F4BEB | sudo apt-key add -

#安装kubernetes组件

root@k8s-master:~# apt-get update
root@k8s-master:~# apt-get install -y kubelet=1.22.2-00 kubeadm=1.22.2-00 kubectl=1.22.2-00
root@k8s-master:~# apt-mark hold kubelet kubeadm kubectl

8. 拉取kubernetes镜像

在主节点k8s-master上执行

root@k8s-master:~# kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.22.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.22.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.22.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.22.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.22.2
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.5
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.0-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.4
root@k8s-master:~#

9. 关闭linux系统swap

注意k8s默认不支持linux系统的swap， 需要关闭， 否则会出错

// 三台机器都需要执行
root@k8s-master:~# swapoff -a

注意：这里的swapoff -a只能时临时关闭swap， 重启机器后就会失效，永久生效的办法时注释掉/etc/fstab文件关于swap的哪一行， 我是用的untube系统注释如下：

// 注释的是倒数第二行内容
root@k8s-master:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda5 during installation
UUID=0d0de8b6-d6bb-42af-a24e-25733ade5b8b /               ext4    errors=remount-ro 0       1
# /boot/efi was on /dev/sda1 during installation
UUID=1B64-6D0D  /boot/efi       vfat    umask=0077      0       1
#/swapfile                                 none            swap    sw              0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto,exec,utf8 0       0
root@k8s-master:~#

10. 更改docker的cgroup驱动类型

docker默认的cgroup驱动类型为cgroupfs 需要更改为systemd

// 三个节点都需要执行
vim /etc/docker/daemon.json
#增加下面配置
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

#重启docker
systemctl restart docker

可以使用docker info命令查看是否更改正确 如下就是正确的

 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 1

11. 初始化kubernetes集群

root@k8s-master:~# kubeadm init --kubernetes-version=v1.22.2 --apiserver-advertise-address=192.168.100.230   --image-repository registry.aliyuncs.com/google_containers
[init] Using Kubernetes version: v1.22.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.100.230]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.100.230 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.100.230 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 8.504169 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: dt4ufl.pv8ivosct9z5gyzk
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.100.230:6443 --token dt4ufl.pv8ivosct9z5gyzk \
        --discovery-token-ca-cert-hash sha256:fbf9f6ef77e5f52ae15ab36af1d20a8b947d06fd67ee39b3d568c8fc3abb4144
root@k8s-master:~#

上面集群初始化命令是：kubeadm init --kubernetes-version=v1.22.2 --apiserver-advertise-address=192.168.100.230 --image-repository registry.aliyuncs.com/google_containers

–kubernetes-version： 指安装版本号
-apiserver-advertise-address： 安装集群主节点的ip地址（我们这里是单主模式）
-image-repository： 指仓库地址

12. 配置kubernetes集群管理信息

第11步初始化完成后会在控制台生成一些信息，在主节点继续执行上面生成的命令

// 主节点执行， 下面三条命令是11步周初始化集群中生成的
root@k8s-master:mkdir -p $HOME/.kube
root@k8s-master:sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
root@k8s-master:sudo chown $(id -u):$(id -g) $HOME/.kube/config

配置完成后我们就可以在k8s-master主机上使用kubelet命令了

我们还需要将主节点下的/root/.kube/config 远程copy到其他两个工作节点（这么做是为了后面jenkin构建容器可以挂载本地的config配置文件，是其可以部署kubernetes资源）
root@k8s-master：scp config root@192.168.100.231:/root/.kube
root@k8s-master：scp config root@192.168.100.232:/root/.kube

13. 工作节点加入kubernetes集群

第11步初始化完成后会在控制台最后面生成加入集群的信息，在两个工作节点（k8s-work01和k8s-work02节点）执行

root@k8s-work01:~# kubeadm join 192.168.100.230:6443 --token dt4ufl.pv8ivosct9z5gyzk \
>         --discovery-token-ca-cert-hash sha256:fbf9f6ef77e5f52ae15ab36af1d20a8b947d06fd67ee39b3d568c8fc3abb4144
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
root@k8s-work01:~#

注意：如我们在初始化完集群后没有即使保存或者过一段时间需要在新的工作节点执行加入集群， 这时我们可能忘了保存初始化时加入集群的命令或者命令中token过期了，我们可以实时生成加入集群命令 如下：
kubeadm token create --print-join-command

// 主节点执行，这是直接生成工作节点加入集群命令的方式
root@k8s-master:~# kubeadm token create --print-join-command
kubeadm join 192.168.100.230:6443 --token z801fl.w6pkk3dydpz7dts8 --discovery-token-ca-cert-hash sha256:fbf9f6ef77e5f52ae15ab36af1d20a8b947d06fd67ee39b3d568c8fc3abb4144
root@k8s-master:~#
//生成完成后 就可以直接在新的工作节点执行加入命令：kubeadm join 192.168.100.230:6443 --token z801fl.w6pkk3dydpz7dts8 --discovery-token-ca-cert-hash sha256:fbf9f6ef77e5f52ae15ab36af1d20a8b947d06fd67ee39b3d568c8fc3abb4144

14. 设置kubectl自动补全

#首先需要安装bash-completion包

root@k8s-master:~# apt-get install -y bash-completion
Reading package lists... Done
Building dependency tree
Reading state information... Done
bash-completion is already the newest version (1:2.10-1ubuntu1).
bash-completion set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 86 not upgraded.
root@k8s-master:~#

#设置自动补全

root@k8s-master:~# source <(kubectl completion bash)
root@k8s-master:~# echo "source <(kubectl completion bash)" >> ~/.bashrc

#设置完成后我们在输入命令是就可以使用Tab键自动补全命令了

15. 验证kubernetes安装结果

做完以上步骤之后我们就可以使用命令检查最终结果了
命令： kubectl get node

// 在k8s-master主节点上执行
root@k8s-master:~# kubectl get pod -A
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-7f6cbbb7b8-svhdg             0/1     Pending   0          20h
kube-system   coredns-7f6cbbb7b8-vmm2h             0/1     Pending   0          20h
kube-system   etcd-k8s-master                      1/1     Running   0          20h
kube-system   kube-apiserver-k8s-master            1/1     Running   0          20h
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          20h
kube-system   kube-proxy-4dvgq                     1/1     Running   0          20h
kube-system   kube-proxy-5jhkj                     1/1     Running   0          20h
kube-system   kube-proxy-5jtwf                     1/1     Running   0          20h
kube-system   kube-scheduler-k8s-master            1/1     Running   0          20h

最终结果如上上面安装成功
注意：我们发现上面coredns状态是Pending， 这里是正常的 当我们后面安装了网络插件就好了

PS. 安装过程中可能遇到的问题

ps1 如果执行init命令的主节点k8s-master上有etcd服务在运行的话， 这里会报错的哦；

ps2 创建k8s集群时主机节点都需要有默认路由（也就是需要配置网关）， 要不coredns容器起不来；

ps3 coredns在init后会创建， 但是状态一直会不正常， 直到部署了网络插件（第二篇中我们部署的是calico插件）后dns状态将会正常

ps4 安装完成后coredns一直处于CrashLoopBackOff状态，并且查看pod日志报错信息为“ plugin/loop: Loop (127.0.0.1:43201 -> :53) detected for zone “.” ”

原因： 因为宿主机/etc/resolv.conf文件中有nameserver：127.0.0.1造成的

解决办法： 将所有k8s主机节点的/etc/resolv.conf中的127.0.0.1更改成其他ip地址
kubectl edit deployment coredns -n kube-system
将replicates改为0保存退出，从而停止已经启动的coredns pod，等停止后再将replicates改为2（默认是2）， 这里其实相当于重启了coredns的pod容器

注意：这里更改完成后再删除dns的pod是不起效的， 所以按照上面的来就行

ps5 工作节点node状态一直是NoteReadly， 此时可以使用kubectl describe node k8s-work02（这里的k8s-work02是集群节点主机名）查看原因
如系统缺少/run/systemd/resolve/resolv.conf 和/etc/resolv.conf 时工作节点会报错，
解决办法： 可以手动创建这两个文件（里面内容可以为空）

ps6 如果工作和主节点时间不一致会导致工作节点加入集群失败，
解决办法：首先可以使用tzselect命令将时区设置正确， 然后cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime， 然后根据文档第6步 安装ntpdate和ntp解决问题

ps7 工作节点加入都成功， 但是再主节点执行kubectl get node 发现有的节点不能显示
原因： 可能是工作节点主机名一样造成的
解决办法：根据文档第3步 更改各个主机的主机名

接下来一章将讲解部署网络插件calico，第二篇：kubernetes部署部署calico网络插件