K8S部署

 KubernetesKubernetes 是用于自动部署，扩展和管理容器化应用程序的开源系统。 它将组成应用程序的容器组合成逻辑单元，以便于管理和服务发现。Kubernetes 源自Google 15 年生产环境的运维经验，同时凝聚了社区的最佳创意和实践。 星际尺度 Google 每周运行数十亿个容器，Kubernetes 基于与之相同的原则来设计，能够在不扩张运维团队的情况下进行规模扩展。 处处适用 无论是本地测试，还是跨国公司，Kubernetes 的灵活性都能让你在应对复杂系统时得心应手。 永不过时 Kubernetes 是开源系统，可以自由地部署在企业内部，私有云、混合云或公有云，让您轻松地做出合适的选择。 The Challenges of Migrating 150+ Microservices to Kubernetes -- 将 150+ 微服务迁移到 Kubernetes 上的挑战 By Sarah Wells, Technical Director for Operations and Reliability, Financial Times-- Sarah Wells, 运营和可靠性技术总监, 金融时报Watch Video Attend KubeCon in Shanghai on Nov. 13-15, 2018 -- 参加11月13日到15日的上海 KubeCon Attend KubeCon in Seattle on Dec.https://kubernetes.io/zh/

一、环境

节点CPU核数必须是：>= 2核，否则k8s无法启动

DNS网络：最好设置为本地网络连通的DNS,否则网络不通，无法下载一些镜像

linux内核： linux内核必须是 4 版本以上，因此必须把linux核心进行升级

1、服务器说明

主机名	操作系统版本	ip	docker version	k8s	备注
master01	CentOS Linux release 7.6.1810 (Core)	192.168.23.160	20.10.12	V1.22.2	master主机
node01	CentOS Linux release 7.6.1810 (Core)	192.168.23.161	20.10.12	V1.22.2	node节点
node02	CentOS Linux release 7.6.1810 (Core)	192.168.23.162	20.10.12	V1.22.2	node节点

2、其他说明

查看系统版本  cat /etc/redhat-release

查看系统内核  uname -r

VMware Workstation版本   Pro 16.2.1 build-18811642

3、给节点设置主机名称

[root@k8s-master01 ~]# hostnamectl set-hostname k8s-master01
[root@k8s-node01 ~]# hostnamectl set-hostname k8s-node01
[root@k8s-node02 ~]# hostnamectl set-hostname k8s-node02

二、服务器基础配置

所有节点

1、配置IP HOST映射关系

cat >> /etc/hosts <<EOF
192.168.23.160 k8s-master01
192.168.23.161 k8s-node01
192.168.23.162 k8s-node02
EOF

2、安装依赖

yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git iproute lrzsz bash-completion tree bridge-utils unzip bind-utils gcc

4、关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

5、禁用swap

swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

6、关闭selinux

setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

7、升级Linux内核（内核大于4，忽略）

查看内核命令

uname -r

下载内核

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm

安装内核

yum --enablerepo=elrepo-kernel install -y kernel-lt

设置开机从新内核启动

查看已安装内核命令　　rpm -qa | grep -i kernel  或 rpm -qa kernel 需要重启才可以看到最新安装版本

设置默认启动内核

grub2-set-default 'kernel-lt-5.4.170-1.el7.elrepo.x86_64'

三、Docker安装

所有节点都需要安装docker

1、安装依赖

yum install -y yum-utils device-mapper-persistent-data lvm2

 2、设置Docker源

配置一个稳定（stable）的仓库，仓库配置会保存到/etc/yum.repos.d/docker-ce.repo文件中

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

3、安装Docker CE

docker安装版本查看

yum list docker-ce --showduplicates | sort -r

更新与安装Docker 

yum update -y && yum install docker-ce

4、命令补全

安装bash-completion

yum -y install bash-completion

加载bash-completion

source /etc/profile.d/bash_completion.sh

5、配置daemon.json文件

由于Docker Hub的服务器在国外，下载镜像会比较慢，可以配置镜像加速器。主要的加速器有：Docker官方提供的中国registry mirror、阿里云加速器、DaoCloud 加速器，本文以阿里加速器配置为例

 创建目录

mkdir /etc/docker

配置daemon.json文件
 

cat > /etc/docker/daemon.json <<EOF
{"exec-opts": ["native.cgroupdriver=systemd"],"log-driver": "json-file","log-opts": {"max-size": "100m"},"registry-mirrors": ["https://98zyayk8.mirror.aliyuncs.com"]}
EOF

说明"registry-mirrors": ["https://98zyayk8.mirror.aliyuncs.com"]是配置镜像加速器

6、启动Docker

systemctl daemon-reload && systemctl restart docker && systemctl enable docker

7、查看Docker

docker --version

四、K8S安装

1、调整内核参数

cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

将优化内核文件拷贝到/etc/sysctl.d/文件夹下，这样优化文件开机的时候能够被调用

cp kubernetes.conf /etc/sysctl.d/kubernetes.conf

2、创建日志配置

创建日志目录

mkdir /var/log/journal

创建配置文件存放目录

mkdir /etc/systemd/journald.conf.d

cat > /etc/systemd/journald.conf.d/99-prophet.conf <<EOF
[Journal]
Storage=persistent
Compress=yes
SyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000
SystemMaxUse=10G
SystemMaxFileSize=200M
MaxRetentionSec=2week
ForwardToSyslog=no
EOF

说明

#持久化保存到磁盘

Storage=persistent   

#压缩历史日志

Compress=yes

SyncIntervalSec=5m

RateLimitInterval=30s

RateLimitBurst=1000

#最大占用空间

SystemMaxUse=10G

#单个日志文件大小

SystemMaxFileSize=200M

#日志保存时间2周

MaxRetentionSec=2week

#不将日志转发Syslog

ForwardToSyslog=no

重启systemd journald的配置

systemctl restart systemd-journald

3、打开文件数调整 (可忽略，不执行)

echo "* soft nofile 65536" >> /etc/security/limits.confecho "* hard nofile 65536" >> /etc/security/limits.conf

4、kube-proxy 开启 ipvs 前置条件

modprobe br_netfilter

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

使用lsmod命令查看这些文件是否被引导

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

5、设置kubernetes源

安装kubernetes的时候，需要安装kubelet, kubeadm等包，但k8s官网给的yum源是packages.cloud.google.com，国内访问不了，此时我们可以使用阿里云的yum仓库镜像

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

说明

[] 中括号中的是repository id，唯一，用来标识不同仓库
name 仓库名称，自定义
baseurl 仓库地址
enable 是否启用该仓库，默认为1表示启用
gpgcheck 是否验证从该仓库获得程序包的合法性，1为验证
repo_gpgcheck 是否验证元数据的合法性 元数据就是程序包列表，1为验证
gpgkey=URL 数字签名的公钥文件所在位置，如果gpgcheck值为1，此处就需要指定gpgkey文件的位置，如果gpgcheck值为0就不需要此项了

6、安装kubelet、kubeadm和kubectl

版本查看

yum list kubelet --showduplicates | sort -r 

安装

yum install -y  kubeadm-1.22.2 kubelet-1.22.2 kubectl-1.22.2

若不指定版本直接运行‘yum install -y kubelet kubeadm kubectl’则默认安装最新版

说明

• kubelet 运行在集群所有节点上，用于启动Pod和容器等对象的工具
• kubeadm 用于初始化集群，启动集群的命令工具
• kubectl 用于和集群通信的命令行，通过kubectl可以部署和管理应用，查看各种资源，创建、删除和更新各种组件

启动 kubelet并设置开机启动

systemctl enable kubelet && systemctl start kubelet

kubelet命令补全

echo "source <(kubectl completion bash)" >> ~/.bash_profile
source .bash_profile 

7、下载镜像

镜像下载的脚本

Kubernetes几乎所有的安装组件和Docker镜像都放在goolge自己的网站上,直接访问可能会有网络问题，这里的解决办法是从阿里云镜像仓库下载镜像，拉取到本地以后改回默认的镜像tag

新建脚本image.sh

#!/bin/bash
url=registry.cn-hangzhou.aliyuncs.com/google_containers
version=v1.22.2
images=(`kubeadm config images list --kubernetes-version=$version|awk -F '/' '{print $2}'`)
for imagename in ${images[@]} ; do
  docker pull $url/$imagename
  docker tag $url/$imagename k8s.gcr.io/$imagename
  docker rmi -f $url/$imagename
done

url为阿里云镜像仓库地址，version为安装的kubernetes版本

下载镜像

行脚本image.sh，下载指定版本的镜像，运行脚本前先赋权

[root@k8s-master01 ~]# chmod u+x image.sh
[root@k8s-master01 ~]# ./image.sh
[root@k8s-master01 ~]# docker images

拷贝到node01节点

[root@k8s-master01 ~]# scp -r image.sh root@k8s-node01:/root/

执行脚本

[root@k8s-node01 ~]# ./image.sh

拷贝到node02节点

[root@k8s-master01 ~]# scp -r image.sh root@k8s-node02:/root/

执行脚本

[root@k8s-node02 ~]# ./image.sh

8、初始化Master

[root@k8s-master01 ~]#  kubeadm init --kubernetes-version=1.22.2 \--apiserver-advertise-address=192.168.23.160 \--image-repository registry.aliyuncs.com/google_containers \--service-cidr=10.1.0.0/16 \--pod-network-cidr=10.244.0.0/16 | tee kubeadm-init.log

命令解析：

• --pod-network-cidr: 定义pod网段为：10.244.0.0/16
• --kubernetes-version=1.22.2  版本设置
• --apiserver-advertise-address指定master的interface
• --apiserver-advertise-address：master主机内网IP地址
• --image-repository：指定阿里云镜像仓库地址。由于kubeadm 默认从官网http://k8s.grc.io下载所需镜像，国内无法访问，因此需要通过–image-repository指定阿里云镜像仓库地址。（国外主机可以不用设定）

加载环境变量

[root@k8s-master01 ~]# mkdir -p $HOME/.kube
[root@k8s-master01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

9、安装pod网络

[root@k8s-master01 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

10、node节点加入集群

在所有node节点执行上文master节点部署成功后生成的信息

查看信息

[root@k8s-master01 ~]# cat kubeadm-init.log

node01

[root@k8s-node01 ~]# kubeadm join 192.168.23.160:6443 --token p8x62i.rc2czn80qfr4en8s \
> --discovery-token-ca-cert-hash sha256:b0c8d888ed199961940490f510021e6b0dc30a750b42ffab26b45742c37e9716

node02

[root@k8s-node02 ~]# kubeadm join 192.168.23.160:6443 --token p8x62i.rc2czn80qfr4en8s \
> --discovery-token-ca-cert-hash sha256:b0c8d888ed199961940490f510021e6b0dc30a750b42ffab26b45742c37e9716

master01查看集群

[root@k8s-master01 ~]# kubectl get node
NAME           STATUS   ROLES                  AGE     VERSION
k8s-master01   Ready    control-plane,master   15m     v1.22.2
k8s-node01     Ready    <none>                 5m46s   v1.22.2
k8s-node02     Ready    <none>                 5m42s   v1.22.2
[root@k8s-master01 ~]# 

查询工作空间中pod容器的详细信息

[root@k8s-master01 ~]# kubectl get pod -n kube-system -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP               NODE           NOMINATED NODE   READINESS GATES
coredns-7f6cbbb7b8-47hqw               1/1     Running   0          16m     10.244.0.2       k8s-master01   <none>           <none>
coredns-7f6cbbb7b8-cns6k               1/1     Running   0          16m     10.244.0.3       k8s-master01   <none>           <none>
etcd-k8s-master01                      1/1     Running   0          16m     192.168.23.160   k8s-master01   <none>           <none>
kube-apiserver-k8s-master01            1/1     Running   0          16m     192.168.23.160   k8s-master01   <none>           <none>
kube-controller-manager-k8s-master01   1/1     Running   0          16m     192.168.23.160   k8s-master01   <none>           <none>
kube-flannel-ds-684pr                  1/1     Running   0          6m52s   192.168.23.161   k8s-node01     <none>           <none>
kube-flannel-ds-l2mk7                  1/1     Running   0          6m48s   192.168.23.162   k8s-node02     <none>           <none>
kube-flannel-ds-z5mvl                  1/1     Running   0          8m4s    192.168.23.160   k8s-master01   <none>           <none>
kube-proxy-2th5p                       1/1     Running   0          6m52s   192.168.23.161   k8s-node01     <none>           <none>
kube-proxy-56jxv                       1/1     Running   0          16m     192.168.23.160   k8s-master01   <none>           <none>
kube-proxy-6c9jb                       1/1     Running   0          6m48s   192.168.23.162   k8s-node02     <none>           <none>
kube-scheduler-k8s-master01            1/1     Running   0          16m     192.168.23.160   k8s-master01   <none>           <none>
[root@k8s-master01 ~]# 

五、Dashboard安装

Web 界面 (Dashboard) | Kubernetes

浏览器访问https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml

1、新建recommended.yaml

# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: v1
kind: Namespace
metadata:
  name: kubernetes-dashboard

---

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  selector:
    k8s-app: kubernetes-dashboard

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-certs
  namespace: kubernetes-dashboard
type: Opaque

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-csrf
  namespace: kubernetes-dashboard
type: Opaque
data:
  csrf: ""

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-key-holder
  namespace: kubernetes-dashboard
type: Opaque

---

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-settings
  namespace: kubernetes-dashboard

---

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
rules:
  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
    verbs: ["get", "update", "delete"]
    # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["kubernetes-dashboard-settings"]
    verbs: ["get", "update"]
    # Allow Dashboard to get metrics.
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["heapster", "dashboard-metrics-scraper"]
    verbs: ["proxy"]
  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
    verbs: ["get"]

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
rules:
  # Allow Metrics Scraper to get metrics from the Metrics server
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list", "watch"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
        - name: kubernetes-dashboard
          image: kubernetesui/dashboard:v2.4.0
          imagePullPolicy: Always
          ports:
            - containerPort: 8443
              protocol: TCP
          args:
            - --auto-generate-certificates
            - --namespace=kubernetes-dashboard
            # Uncomment the following line to manually specify Kubernetes API server Host
            # If not specified, Dashboard will attempt to auto discover the API server and connect
            # to it. Uncomment only if the default does not work.
            # - --apiserver-host=http://my-address:port
          volumeMounts:
            - name: kubernetes-dashboard-certs
              mountPath: /certs
              # Create on-disk volume to store exec logs
            - mountPath: /tmp
              name: tmp-volume
          livenessProbe:
            httpGet:
              scheme: HTTPS
              path: /
              port: 8443
            initialDelaySeconds: 30
            timeoutSeconds: 30
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      volumes:
        - name: kubernetes-dashboard-certs
          secret:
            secretName: kubernetes-dashboard-certs
        - name: tmp-volume
          emptyDir: {}
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 8000
      targetPort: 8000
  selector:
    k8s-app: dashboard-metrics-scraper

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: dashboard-metrics-scraper
  template:
    metadata:
      labels:
        k8s-app: dashboard-metrics-scraper
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: dashboard-metrics-scraper
          image: kubernetesui/metrics-scraper:v1.0.7
          ports:
            - containerPort: 8000
              protocol: TCP
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /
              port: 8000
            initialDelaySeconds: 30
            timeoutSeconds: 30
          volumeMounts:
          - mountPath: /tmp
            name: tmp-volume
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      volumes:
        - name: tmp-volume
          emptyDir: {}

2、修改配置recommended.yaml

 3、部署Dashboard

[root@k8s-master01 ~]# kubectl apply -f recommended.yaml

4、查看状态查看

[root@k8s-master01 ~]# kubectl get all -n kubernetes-dashboard

5、使用token认证进行登录

创建serviceAccount，假如用户名为dashboard-admin

kubectl create serviceaccount dashboard-admin -n kube-system

通过clusterrolebinding绑定到内置的cluster-admin角色上

kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin

令牌查看

kubectl describe secrets -n kube-system dashboard-admin

访问

https://192.168.23.160:30001/#/login

参考

k8s实践(一)：Centos7.6部署k8s(v1.14.2)集群【附源码】_loong576_51CTO博客

kubeadm 安装kubernetes - 知乎

部署CNI网络插件 The connection to the server raw.githubusercontent.com was refused - did you specify the r_小小草_8741的博客-CSDN博客

kubeadm 报错 error execution phase preflight: couldn’t validate the identity of the API Server: abort connecting to API servers after timeout of 5m0s - 司家勇 - 博客园