kubernetes系列之《集群部署（下）》

最新推荐文章于 2024-07-16 11:44:26 发布

被遗忘的区域

最新推荐文章于 2024-07-16 11:44:26 发布

阅读量514

点赞数

分类专栏： docker，k8s，虚拟化

西西工作室

本文链接：https://blog.csdn.net/songxi_bo/article/details/105250322

版权

docker，k8s，虚拟化专栏收录该内容

55 篇文章 3 订阅

订阅专栏

本文继承上一篇文章《集群部署（上）》, 继续Kubernetes集群实战
本文将实现：

完整部署一套Kubernetes
部署Kubernetes UI（Dashboard）
部署多Master节点的Kubernetes集群
部署Kubernetes集群内部DNS（CoreDNS ）
七、部署Master组件
根据环境规划，Master节点应有以下组件：

kube-apiserver
kube-controller-manager
kube-sheduler
etcd
etcd在上面已经部署完成，本文里Master节点是k8s-master-128，那该节点就剩下其他三个组件需部署。

7.1、获取二进制包
官网：https://kubernetes.io/，
Github: https://github.com/kubernetes/kubernetes/releases
二进制包：https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz

注意：二进制包比较大，400M大小，且有墙，我这里手动下载然后上传到服务器；

7.2、部署二进制文件

部署Master组件

[root@k8s-master-128 ~]# cd soft/
[root@k8s-master-128 soft]# tar zxf kubernetes-server-linux-amd64.tar.gz
[root@k8s-master-128 soft]# cp kubernetes/server/bin/{kubectl,kube-scheduler,kube-controller-manager,kube-apiserver} /opt/kubernetes/bin/
[root@k8s-master-128 soft]# echo “export PATH=$PATH:/opt/kubernetes/bin” >>/etc/profile
[root@k8s-master-128 soft]# source /etc/profile
[root@k8s-master-128 soft]# which kubectl
/opt/kubernetes/bin/kubectl

部署Node组件（方便后面使用）

[root@k8s-master-128 soft]# scp kubernetes/server/bin/{kubelet,kube-proxy} k8s-node-129:/opt/kubernetes/bin/
7.3、生成Kubeconfig文件
7.3.1、api-server认证方式
这个是知识了解，api-server有如下几种认证方式：

CA证书认证
Token认证：token-auth
基本认证：basic-auth
kubernetes 认证主要分为上面三种，可以同时配置多种认证方式，只要其中任意一个方式认证通过即可。

参考：

Kubernetes 的安全机制 APIServer 认证、授权、准入控制
Kubernetes apiserver认证
本次部署使用第二种：Token认证。

7.3.2、创建配置文件
需要创建以下三个配置文件。

1、创建TLS Bootstrapping Token
2、创建kubelet kubeconfig
3、创建kube-proxy kubeconfig
我们使用一个脚本将所有的配置文件一起创建出来

[root@k8s-master-128 ~]# mkdir ssl/k8s-cret
[root@k8s-master-128 ~]# cd ssl/k8s-cret/
[root@k8s-master-128 k8s-cret]# vim kubeconfig.sh

创建 TLS Bootstrapping Token

export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ’ ') # 生成随机token（随机字符串）

cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,“system:kubelet-bootstrap”
EOF

#----------------------

创建kubelet bootstrapping kubeconfig

export KUBE_APISERVER=“https://172.16.194.128:6443” # 这个脚本唯一需要更改的就是这个地方，填写成你的Master IP地址

设置集群参数

kubectl config set-cluster kubernetes
–certificate-authority=./ca.pem
–embed-certs=true
–server=${KUBE_APISERVER}
–kubeconfig=bootstrap.kubeconfig

设置客户端认证参数

kubectl config set-credentials kubelet-bootstrap
–token=${BOOTSTRAP_TOKEN}
–kubeconfig=bootstrap.kubeconfig

设置上下文参数

kubectl config set-context default
–cluster=kubernetes
–user=kubelet-bootstrap
–kubeconfig=bootstrap.kubeconfig

设置默认上下文

kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#----------------------

创建kube-proxy kubeconfig文件

kubectl config set-cluster kubernetes
–certificate-authority=./ca.pem
–embed-certs=true
–server=${KUBE_APISERVER}
–kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy
–client-certificate=./kube-proxy.pem
–client-key=./kube-proxy-key.pem
–embed-certs=true
–kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default
–cluster=kubernetes
–user=kube-proxy
–kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
执行脚本，创建配置文件：

[root@k8s-master-128 k8s-cret]# chmod +x kubeconfig.sh
[root@k8s-master-128 k8s-cret]# ./kubeconfig.sh
Cluster “kubernetes” set.
User “kubelet-bootstrap” set.
Context “default” created.
Switched to context “default”.
Cluster “kubernetes” set.
User “kube-proxy” set.
Context “default” created.
Switched to context “default”.
配置文件的用途：总共创建了三个文件，用途分别为：

bootstrap.kubeconfig # Node节点的kubelet组件使用
kube-proxy.kubeconfig # Node节点的kube-proxy组件使用
token.csv # Master节点的apiserver组件使用
根据上面的用途说明，将生成的文件推送到各个节点的目录中（以备后面部署组件使用）：

Master节点只需要token.csv

[root@k8s-master-128 k8s-cret]# cp token.csv /opt/kubernetes/cfg/

Node节点推送文件

[root@k8s-master-128 k8s-cret]# scp bootstrap.kubeconfig kube-proxy.kubeconfig k8s-node-129:/opt/kubernetes/cfg/
[root@k8s-master-128 k8s-cret]# scp bootstrap.kubeconfig kube-proxy.kubeconfig k8s-node-130:/opt/kubernetes/cfg/
7.3.3、查看配置文件
token.csv

[root@k8s-master-128 k8s-cret]# cat token.csv
38435b41e3861251dce8c2cbf968ca67,kubelet-bootstrap,10001,“system:kubelet-bootstrap”
注解：

38435b41e3861251dce8c2cbf968ca67：随机生成的token，也可以固定一个Token来使用
kubelet-bootstrap ：用户名
10001：用户ID
system:kubelet-bootstrap：用户组
bootstrap.kubeconfig

[root@k8s-master-128 k8s-cret]# cat bootstrap.kubeconfig
apiVersion: v1 # api的版本
clusters: # 集群的内容

cluster:
certificate-authority-data: # 集群CA数字证书，有很大一堆，略······
server: https://172.16.194.128:6443 # server的地址
name: kubernetes # 集群的名字
contexts: # 上下文内容
context:
cluster: kubernetes
user: kubelet-bootstrap # k8s用户（可改变，是从token.csv里定义的）
name: default # 上下文名称
current-context: default # 当前默认使用的上下文
kind: Config
preferences: {}
users: # 用户的信息
name: kubelet-bootstrap # 用户名
user:
token: 38435b41e3861251dce8c2cbf968ca67 # token，关键！必须要与token.csv里的token对应！不对应则没有相应的权限，会认证失败。
kube-proxy.kubeconfig

[root@k8s-master-128 k8s-cret]# cat kube-proxy.kubeconfig
apiVersion: v1
clusters:

cluster:
certificate-authority-data: # 集群CA数字证书，有很大一堆，略······
server: https://192.16.194.128:6443
name: kubernetes
contexts:
context:
cluster: kubernetes
user: kube-proxy
name: default
current-context: default
kind: Config
preferences: {}
users:
name: kube-proxy
user:
client-certificate-data: # 客户端CA数字证书，有很大一堆，略······
client-key-data: # 客户端key证书，有很大一堆，略······
7.4、启动Master组件
7.4.1、启动kube-apiserver
1、启动的脚本

[root@k8s-master-128 ~]# cat apiserver.sh
#!/bin/bash
MASTER_ADDRESS= ${1:-"192.168.1.195"} ETCD_SERVERS=$ {2:-“http://127.0.0.1:2379”}

cat </opt/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="–logtostderr=false
–log-dir=/opt/kubernetes/logs
–v=4
–etcd-servers= $KaTeX parse error: Undefined control sequence: \ at position 16: {ETCD_SERVERS} \̲ ̲--insecure-bind…$ {MASTER_ADDRESS}
–insecure-port=8080
–secure-port=6443
–advertise-address=${MASTER_ADDRESS}
–allow-privileged=true
–service-cluster-ip-range=10.0.0.0/24
–service-node-port-range=30000-50000
–admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction
–authorization-mode=RBAC,Node
–kubelet-https=true
–enable-bootstrap-token-auth
–token-auth-file=/opt/kubernetes/cfg/token.csv
–tls-cert-file=/opt/kubernetes/ssl/server.pem
–tls-private-key-file=/opt/kubernetes/ssl/server-key.pem
–client-ca-file=/opt/kubernetes/ssl/ca.pem
–service-account-key-file=/opt/kubernetes/ssl/ca-key.pem
–etcd-cafile=/opt/etcd/ssl/ca.pem
–etcd-certfile=/opt/etcd/ssl/server.pem
–etcd-keyfile=/opt/etcd/ssl/server-key.pem"
EOF

cat </usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
2、启动apiserver

apiserver.sh 接收两个参数，第一个是apiserver的地址，第二个是ETCD集群的地址，我们集群有三个节点，都填写上。

[root@k8s-master-128 ~]# chmod +x apiserver.sh
[root@k8s-master-128 ~]# ./apiserver.sh 172.16.194.128 https://172.16.194.128:2379,https://172.16.194.129:2379,https://172.16.194.130:2379
[root@k8s-master-128 ~]# systemctl daemon-reload
[root@k8s-master-128 ~]# systemctl enable kube-apiserver
[root@k8s-master-128 ~]# systemctl start kube-apiserver
[root@k8s-master-128 ~]# systemctl status kube-apiserver
3、查看启动状态

[root@k8s-master-128 logs]# ls -lh /opt/kubernetes/logs/ # 详细日志
[root@k8s-master-128 logs]# netstat -lntup|grep kube
tcp 0 0 172.16.194.128:6443 0.0.0.0:* LISTEN 28201/kube-apiserve
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 28201/kube-apiserve
[root@k8s-master-128 logs]# ps -ef|grep kube-apiserver

信息太多，不贴了

注意：

如果你照着上面的脚本执行后，如没有进程存活，你应该查看下日志和检查上面的证书都是否存在。
一定要先启动apiserver！因为后面两个组件依赖apiserver，controller-manager和scheduler可以没有先后顺序。
7.4.2、启动kube-controller-manager
1、启动的脚本

[root@k8s-master-128 ~]# cat controller-manager.sh
#!/bin/bash

MASTER_ADDRESS=${1:-“127.0.0.1”}

cat </opt/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="–logtostderr=false
–log-dir=/opt/kubernetes/logs
–v=4
–master=${MASTER_ADDRESS}:8080
–leader-elect=true
–address=127.0.0.1
–service-cluster-ip-range=10.0.0.0/24
–cluster-name=kubernetes
–cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem
–cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem
–root-ca-file=/opt/kubernetes/ssl/ca.pem
–service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem
–experimental-cluster-signing-duration=87600h0m0s"
EOF

cat </usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
2、启动controller-manager

[root@k8s-master-128 ~]# chmod +x controller-manager.sh
[root@k8s-master-128 ~]# ./controller-manager.sh
[root@k8s-master-128 ~]# systemctl daemon-reload
[root@k8s-master-128 ~]# systemctl enable kube-controller-manager
[root@k8s-master-128 ~]# systemctl start kube-controller-manager
启动失败请检查这两个文件是否配置正确，以及启动日志:
/opt/kubernetes/cfg/kube-controller-manager
/usr/lib/systemd/system/kube-controller-manager.service
/opt/kubernetes/logs/

3、查看启动状态

controller-manager启动的端口是10252

[root@k8s-master-128 ~]# netstat -lntup|grep kube-c
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 7699/kube-controlle
tcp6 0 0 :::10257 ::😗 LISTEN 7699/kube-controlle
[root@k8s-master-128 ~]# ps -ef|grep controller-manager
7.4.3、启动kube-sheduler
1、启动的脚本

[root@k8s-master-128 ~]# cat scheduler.sh
#!/bin/bash

MASTER_ADDRESS=${1:-“127.0.0.1”}

cat </opt/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="–logtostderr=false
–log-dir=/opt/kubernetes/logs
–v=4
–master=${MASTER_ADDRESS}:8080
–leader-elect"
EOF

cat </usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
2、启动scheduler

[root@k8s-master-128 ~]# chmod +x scheduler.sh
[root@k8s-master-128 ~]# ./scheduler.sh
[root@k8s-master-128 ~]# systemctl daemon-reload
[root@k8s-master-128 ~]# systemctl enable kube-scheduler
[root@k8s-master-128 ~]# systemctl start kube-scheduler
3、查看启动状态

scheduler启动的端口是10251

[root@k8s-master-128 ~]# netstat -lntup|grep kube-sc
tcp6 0 0 :::10251 ::😗 LISTEN 7897/kube-scheduler
tcp6 0 0 :::10259 ::😗 LISTEN 7897/kube-scheduler
[root@k8s-master-128 ~]# ps -ef|grep scheduler
7.5、查看组件运行状态
kubectl get cs # 查看k8s集群资源的健康信息
相关阅读：Kubernetes kubectl get 命令详解

[root@k8s-master-128 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-1 Healthy {“health”:“true”}
etcd-0 Healthy {“health”:“true”}
etcd-2 Healthy {“health”:“true”}
看到以上信息都为健康状态，到此，Kubernetes集群的Master节点即部署完成。

八、部署Node组件
根据前面环境规划，Node节点将启动kubelet和kube-proxy组件。

8.1、创建Token租户并绑定角色
这是我们创建的tuken信息，需要把租户信息创建出来，Node节点在部署kubelet组件时需要通过token租户进行权限验证

[root@k8s-master-128 ~]# kubectl create --help
[root@k8s-master-128 ~]# kubectl create clusterrolebinding --help
Usage:
kubectl create clusterrolebinding NAME --clusterrole=NAME [–user=username] [–group=groupname]
[–serviceaccount=namespace:serviceaccountname] [–dry-run] [options]

[root@k8s-master-128 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
[root@k8s-master-128 ~]# kubectl get clusterrole # 查看集群角色（system:node-bootstrapper由此得来）
8.2、部署Kubelet
8.2.1、启动Kubelet组件
1、启动的脚本

[root@k8s-node-129 ~]# cat kubelet.sh
#!/bin/bash

NODE_ADDRESS= $KaTeX parse error: Expected 'EOF', got '#' at position 18: \dots:-"127.0.0.1"} #̲ 节点的IP地址，通过参数传入\dots$ {2:-“10.0.0.2”} # DNS地址，在apiserver.sh里指定的地址端，可以写10.10.10.0/24段的任意IP

cat </opt/kubernetes/cfg/kubelet
KUBELET_OPTS="–logtostderr=false
–log-dir=/opt/kubernetes/logs
–v=4
–address= $KaTeX parse error: Undefined control sequence: \ at position 16: {NODE_ADDRESS} \̲ ̲--hostname-over…$ {NODE_ADDRESS}
–kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig
–experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig
–cert-dir=/opt/kubernetes/ssl
–allow-privileged=true
–cluster-dns=${DNS_SERVER_IP}
–cluster-domain=cluster.local
–fail-swap-on=false
–pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
EOF

cat </usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target
EOF
2、启动kubelet

两个Node节点同时启动kubelet

[root@k8s-node-129 ~]# chmod +x kubelet.sh
[root@k8s-node-129 ~]# ./kubelet.sh 172.16.194.129
[root@k8s-node-130 ~]# ./kubelet.sh 172.16.194.130

启动kubelet

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
3、查看启动状态

[root@k8s-node-129 ~]# ps -ef|grep kubelet
[root@k8s-node-129 ~]# ls -lh /opt/kubernetes/cfg/kubelet.kubeconfig
ls: 无法访问/opt/kubernetes/cfg/kubelet.kubeconfig: 没有那个文件或目录
注意：你会发现没有kubelet.kubeconfig文件，这是为什么呢？是因为k8s-Master需要给Node节点的kubelet组件颁发证书，Node节点才会生成这个证书文件。

承上，我们继续到Master节点看看证书请求状况:

[root@k8s-master-128 ~]# kubectl get csr # 查看Node节点的kubelet证书申请请求
NAME AGE REQUESTOR CONDITION
node-csr-K50rilZHx2Gn_MHesRDr_wIJ5hOrg3buFjWuq3RNRhE 3m45s kubelet-bootstrap Pending
node-csr-xW7Q15gU-88wPvObRXmia6y-eMyx5dbcMHbGIwINzD0 15m kubelet-bootstrap Pending # 等待颁发状态
8.2.2、k8s集群为kubelet颁发证书
[root@k8s-master-128 ~]# kubectl --help|grep certificate
certificate 修改 certificate 资源.
[root@k8s-master-128 ~]# kubectl certificate --help
approve 同意一个自签证书请求
deny 拒绝一个自签证书请求

[root@k8s-master-128 ~]# kubectl certificate approve --help
Usage:
kubectl certificate approve (-f FILENAME | NAME) [options]

上面几步help我们找到了用法

[root@k8s-master-128 ~]# kubectl certificate approve node-csr-K50rilZHx2Gn_MHesRDr_wIJ5hOrg3buFjWuq3RNRhE
certificatesigningrequest.certificates.k8s.io/node-csr-K50rilZHx2Gn_MHesRDr_wIJ5hOrg3buFjWuq3RNRhE approved
[root@k8s-master-128 ~]# kubectl certificate approve node-csr-xW7Q15gU-88wPvObRXmia6y-eMyx5dbcMHbGIwINzD0
certificatesigningrequest.certificates.k8s.io/node-csr-xW7Q15gU-88wPvObRXmia6y-eMyx5dbcMHbGIwINzD0 approved

[root@k8s-master-128 ~]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-K50rilZHx2Gn_MHesRDr_wIJ5hOrg3buFjWuq3RNRhE 7m8s kubelet-bootstrap Approved,Issued
node-csr-xW7Q15gU-88wPvObRXmia6y-eMyx5dbcMHbGIwINzD0 18m kubelet-bootstrap Approved,IssuedCRGvCskSEnG13F4BPrqQbDiR2epHRg4 7m kubelet-bootstrap Approved,Issued # 同意办法证书后，变为：批准状态。
我这里有两个Node节点，都为其颁发证书允许加入集群。

8.2.3、查看Node节点加入k8s集群
[root@k8s-master-128 ~]# kubectl get nodes # 查看集群中的Node节点信息，Ready表示该节点健康，已准备就绪
NAME STATUS ROLES AGE VERSION
172.16.194.129 Ready 54s v1.14.0
172.16.194.130 Ready 63s v1.14.0
8.2.4、查看Node节点签发的证书
[root@k8s-node-129 ~]# ls -lh /opt/kubernetes/cfg/kubelet.kubeconfig # 已经有这个文件了，你可以cat查看内容细节
-rw------- 1 root root 2.3K 5月 7 17:04 /opt/kubernetes/cfg/kubelet.kubeconfig

ssl目录生成如下证书

[root@k8s-node-129 ~]# ls -lh /opt/kubernetes/ssl/kubelet*
-rw------- 1 root root 1.3K 5月 7 17:04 /opt/kubernetes/ssl/kubelet-client-2019-05-07-17-04-27.pem
lrwxrwxrwx 1 root root 58 5月 7 17:04 /opt/kubernetes/ssl/kubelet-client-current.pem -> /opt/kubernetes/ssl/kubelet-client-2019-05-07-17-04-27.pem
-rw-r–r-- 1 root root 2.2K 5月 7 16:36 /opt/kubernetes/ssl/kubelet.crt
-rw------- 1 root root 1.7K 5月 7 16:36 /opt/kubernetes/ssl/kubelet.key
整个自签发证书颁发流程完结。

8.3、部署kube-proxy
1、启动的脚本

[root@k8s-node-129 ~]# cat kube-proxy.sh
#!/bin/bash
NODE_ADDRESS=${1:-“127.0.0.1”} # kube-proxy的节点IP

cat </opt/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="–logtostderr=false
–log-dir=/opt/kubernetes/logs
–v=4
–hostname-override=${NODE_ADDRESS}
–kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
EOF

cat </usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
2、启动kube-proxy

两个Node节点都启动kube-proxy

[root@k8s-node-129 ~]# chmod +x kube-proxy.sh

[root@k8s-node-129 ~]# ./kube-proxy.sh 172.16.194.129
[root@k8s-node-130 ~]# ./kube-proxy.sh 172.16.194.130

启动kube-proxy

systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy
3、查看启动状态

[root@k8s-node-129 ~]# ps -ef|grep kube-proxy
[root@k8s-node-130 ~]# systemctl status kube-proxy
九、部署一个测试示例
9.1、创建一个Nginx pods
[root@k8s-master-128 ~]# kubectl run --help # 能看到有很多用法
[root@k8s-master-128 ~]# kubectl run nginx --image=nginx
[root@k8s-master-128 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-7db9fccd9b-kckml 0/1 ContainerCreating 0 45s # 第一次创建K8s会在Node节点上拉取镜像，启动时间稍长；
[root@k8s-master-128 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-7db9fccd9b-kckml 1/1 Running 0 3m57s
查看pod被分配到了哪个节点上：

[root@k8s-master-128 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-7db9fccd9b-bhldc 1/1 Running 0 7m7s 172.17.64.2 172.16.194.129
9.2、创建一个services
用于将pod封装成一个service，提供外界访问；

[root@k8s-master-128 ~]# kubectl expose --help
[root@k8s-master-128 ~]# kubectl expose deployment nginx --port=80 --target-port=80 --type=NodePort
service/nginx exposed
[root@k8s-master-128 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 443/TCP 22h
nginx NodePort 10.0.0.86 80:43364/TCP 13s
9.3、访问services
[root@k8s-node-129 ~]# curl 10.0.0.86 # 对内访问
[root@k8s-master-128 ~]# curl 172.16.194.129:43364 # 对外访问

查看访问日志

[root@k8s-master-128 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-7db9fccd9b-bhldc 1/1 Running 0 13m
[root@k8s-master-128 ~]# kubectl logs nginx-7db9fccd9b-bhldc
十、部署Web UI（Dashboard）
部署UI这部分网上有很多方法，如果下文有你看不懂的地方，可以不用参考我这里的方案

在kubernetes源码里有关于Dashboard的部署文件：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dashboard
https://github.com/kubernetes/dashboard

文件的细节这里不展开解释，需要具备一定阅读能力。

10.1、获取YAML文件
从源码包获取文件

[root@k8s-master-128 ~]# cd /root/soft/kubernetes
[root@k8s-master-128 kubernetes]# tar zxf kubernetes-src.tar.gz
[root@k8s-master-128 kubernetes]# ls -lh cluster/addons/dashboard/ # 在源码包里获取这些文件
总用量 32K
-rw-rw-r-- 1 root root 264 3月 21 13:51 dashboard-configmap.yaml
-rw-rw-r-- 1 root root 1.8K 3月 21 13:51 dashboard-controller.yaml
-rw-rw-r-- 1 root root 1.4K 3月 21 13:51 dashboard-rbac.yaml
-rw-rw-r-- 1 root root 551 3月 21 13:51 dashboard-secret.yaml
-rw-rw-r-- 1 root root 322 3月 21 13:51 dashboard-service.yaml
10.2、更改YAML文件
dashboard-controller配置文件里引用的国外镜像源，因此需要更改成国内的源才可启动

[root@k8s-master-128 kubernetes]# cd cluster/addons/dashboard
[root@k8s-master-128 dashboard]# vim dashboard-controller.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
addonmanager.kubernetes.io/mode: Reconcile
name: kubernetes-dashboard
namespace: kube-system

apiVersion: apps/v1
kind: Deployment
metadata:
name: kubernetes-dashboard
namespace: kube-system
labels:
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: “true”
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ‘’
seccomp.security.alpha.kubernetes.io/pod: ‘docker/default’
spec:
priorityClassName: system-cluster-critical
containers:
- name: kubernetes-dashboard
image: registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1 # 将这里的官网镜像更改成国内镜像
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 50m
memory: 100Mi
ports:
- containerPort: 8443
protocol: TCP
args:
# PLATFORM-SPECIFIC ARGS HERE
- --auto-generate-certificates
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
- name: tmp-volume
mountPath: /tmp
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
tolerations:
- key: “CriticalAddonsOnly”
operator: “Exists”
dashboard-service.yaml 文件里需要新增：type: NodePort来对外提供服务；

[root@k8s-master-128 dashboard]# cat dashboard-service.yaml
apiVersion: v1
kind: Service
metadata:
name: kubernetes-dashboard
namespace: kube-system
labels:
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: “true”
addonmanager.kubernetes.io/mode: Reconcile
spec:
type: NodePort
selector:
k8s-app: kubernetes-dashboard
ports:

port: 443
targetPort: 8443
10.3、执行部署
[root@k8s-master-128 dashboard]# kubectl create -f dashboard-configmap.yaml
configmap/kubernetes-dashboard-settings created
[root@k8s-master-128 dashboard]# kubectl create -f dashboard-rbac.yaml
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
[root@k8s-master-128 dashboard]# kubectl create -f dashboard-secret.yaml
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-key-holder created
[root@k8s-master-128 dashboard]# kubectl create -f dashboard-controller.yaml
serviceaccount/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
[root@k8s-master-128 dashboard]# kubectl create -f dashboard-service.yaml
service/kubernetes-dashboard created
查看部署情况

[root@k8s-master-128 dashboard]# kubectl get pods -n kube-system # 指定命名空间查看Pod
NAME READY STATUS RESTARTS AGE
kubernetes-dashboard-7d5f7c58f5-mn44f 1/1 Running 0 2m44s

[root@k8s-master-128 dashboard]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.0.0.47 443:34756/TCP 11s # 对外暴露了38276端口，这是能访问UI界面的端口

[root@k8s-master-128 dashboard]# kubectl describe pods/kubernetes-dashboard-7d5f7c58f5-mn44f -n kube-system # 这个命令能查看pods创建过程信息
10.4、UI界面
https://Node节点IP:34756（节点IP之间，能负载均衡，所有随便访问哪个节点IP+端口都能访问到UI）
https://172.16.194.130:34756

创建Service account并绑定默认cluster-admin管理员集群角色：

$ kubectl create serviceaccount dashboard-admin -n kube-system
$ kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
$ kubectl describe secrets -n kube-system $(kubectl get secrets -n kube-system |awk ‘/dashboard-admin/{print $1}’)

输出：
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tOXA4ODciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMmRkODQyNzUtN2M2OS0xMWU5LTkzYzQtMDAwYzI5ZjRkYWE5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.e96EQXa1YIbhRXeckNthFpqNaVw7jrQFqlQTgvVIropqCGZyw_PiM9CF3F5fWwGKZmrojo1_OV4xPNEgxkA1pA-FizwBsUo6flPeVnvkxgaSw2ME6D_z0nj8rIPwyNoDe6x2mLGZNtZi4JNDu5ehoKhZqSF60-rDsoYmlwhY8WVq6uuNVSu086i25q3UU8Wz963TyRZgiywP5fTIbCfikBA5Aj7Mjar9IcCsrPGKeWOm0CxaF_IFPidMWR0scNOZfwdTHC2gU6MUxwMjAQ3KRcC1j7sNQnjXd_mPuJg96SDJsWT8T9IKaMXfXa0etb_b9F5FEZ3qAdFFsKjh-pbJ7g
将这个token放入Dashboard认证，即可登录到UI页面。

十一、部署多Master节点集群
环境规划中，选定172.16.194.127来作为另一个Master节点，主要是实现kube-apiserver的负载均衡，首先对Master节点进行部署，然后使用nginx来进行对kube-apiserver进行负载代理，实现apiserver的高可用；

11.1、部署Master组件
在172.16.194.128服务器上已经部署好了一台Master所需要的组件，这里可以直接将配置文件和启动服务文件直接拉过来即可使用

11.1.1、拷贝文件

配置免密要环境

[root@k8s-master-128 ~]# ssh-copy-id k8s-master-127

kube-apiserver需要用到etcd的证书，因此我们将128上etcd所有文件拷贝到127

[root@k8s-master-128 ~]# ssh k8s-master-127 mkdir -p /opt/etcd/{bin,cfg,ssl}
[root@k8s-master-128 ~]# scp -r /opt/etcd/{bin,cfg,ssl} k8s-master-127:/opt/etcd/

拷贝k8s配置文件

[root@k8s-master-128 ~]# ssh k8s-master-127 mkdir -p /opt/kubernetes/{bin,cfg,logs,ssl}
[root@k8s-master-128 ~]# scp -r /opt/kubernetes/{bin,cfg,ssl} k8s-master-127:/opt/kubernetes/
[root@k8s-master-128 ~]# scp -r /usr/lib/systemd/system/kube-{apiserver,controller-manager,scheduler}.service k8s-master-127:/usr/lib/systemd/system/

更改配置信息

[root@k8s-master-127 ~]# cd /opt/kubernetes/cfg/
[root@k8s-master-127 cfg]# vim kube-apiserver

将以下参数的IP值更改为对应主机IP，其他的不变

–bind-address=172.16.194.127
–advertise-address=172.16.194.127

kube-controller-manager与kube-schedule组件的配置不用更改，并且这两个组件都有负载均衡功能：

–leader-elect
11.1.2、启动组件
[root@k8s-master-127 ~]# systemctl start kube-apiserver
[root@k8s-master-127 ~]# systemctl start kube-controller-manager
[root@k8s-master-127 ~]# systemctl start kube-scheduler
[root@k8s-master-127 ~]# systemctl enable kube-apiserver
[root@k8s-master-127 ~]# systemctl enable kube-controller-manager
[root@k8s-master-127 ~]# systemctl enable kube-scheduler

[root@k8s-master-127 ~]# netstat -lntup|grep kube
tcp 0 0 172.16.194.127:6443 0.0.0.0:* LISTEN 7344/kube-apiserver
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 7362/kube-controlle
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 7344/kube-apiserver
tcp6 0 0 :::10251 ::😗 LISTEN 7375/kube-scheduler
tcp6 0 0 :::10257 ::😗 LISTEN 7362/kube-controlle
tcp6 0 0 :::10259 ::😗 LISTEN 7375/kube-scheduler
启动完成。
配置环境变量和查看集群状态：

[root@k8s-master-127 ~]# echo “export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/opt/kubernetes/bin” >>/etc/profile
[root@k8s-master-127 ~]# source /etc/profile
[root@k8s-master-127 ~]# which kubectl
/opt/kubernetes/bin/kubectl

[root@k8s-master-127 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-7db9fccd9b-bhldc 1/1 Running 0 4d7h
[root@k8s-master-127 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
kubernetes-dashboard-7d5f7c58f5-dvz5m 1/1 Running 5 3d13h
[root@k8s-master-127 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 443/TCP 5d6h
nginx NodePort 10.0.0.86 80:43364/TCP 4d7h
[root@k8s-master-127 ~]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.0.0.47 443:34756/TCP 4d6h
[root@k8s-master-127 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {“health”:“true”}
etcd-0 Healthy {“health”:“true”}
etcd-2 Healthy {“health”:“true”}
能看到127节点可以正常接入集群并使用。

11.2、Nginx负载Master节点
官网安装文档：http://nginx.org/en/linux_packages.html

选择自己的OS环境并进行nginx安装，因为实验环境机器有限，这里使用127和128两台k8s的Master节点来安装Nginx来进行实验。与前期环境规划一致。

安装：

[root@k8s-master-127 ~]# vim /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/ $r e l e a s e v e r /$ basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key

[root@k8s-master-127 ~]# yum install nginx -y
启动nginx：

[root@k8s-master-127 ~]# systemctl start nginx
[root@k8s-master-127 ~]# systemctl enable nginx
配置：
nginx配置目录在/etc/nginx/下，主配置文件nginx.conf，生产环境需要对这个配置文件进行调优，而这里做实验的话默认配置就行。

[root@k8s-master-127 nginx]# cat nginx.conf
···略···
events {
worker_connections 1024;
}

stream模块用于代理4层请求，且不能存在http模块内

stream {
log_format main ‘$remote_addr $upstream_addr $time_local $status’;
access_log /var/log/nginx/k8s-apiserver.com.access_log main;
error_log /var/log/nginx/k8s-apiserver.com.error_log warn;
upstream k8s-apiserver {
server 172.16.194.127:6443;
server 172.16.194.128:6443;
}
server {
listen 172.16.194.127:6444;
proxy_pass k8s-apiserver; # 代理4层请求，不用加http://
}
}

http {
···略···
}
···略···
查看启动：

[root@k8s-master-127 nginx]# netstat -lntup|grep 6444
tcp 0 0 172.16.194.127:6444 0.0.0.0:* LISTEN 7061/nginx: master
因为本机有6443，想要与代理理共存的话，nginx更改下端口即可。

11.3、Node节点配置使用
更改Node节点的配置：

[root@k8s-node-129 ~]# cd /opt/kubernetes/cfg/
[root@k8s-node-129 cfg]# grep “6443” ./* # 将这三个文件的地址更改为api-server的代理地址
./bootstrap.kubeconfig: server: https://172.16.194.128:6443
./kubelet.kubeconfig: server: https://172.16.194.128:6443
./kube-proxy.kubeconfig: server: https://172.16.194.128:6443

更改后：(所有节点都需要更改)

[root@k8s-node-129 cfg]# grep “6444” ./*
./bootstrap.kubeconfig: server: https://172.16.194.127:6444
./kubelet.kubeconfig: server: https://172.16.194.127:6444
./kube-proxy.kubeconfig: server: https://172.16.194.127:6444
重启Node节点组件：

[root@k8s-node-129 cfg]# systemctl restart kubelet
[root@k8s-node-129 cfg]# systemctl restart kube-proxy
查看nginx代理日志：

[root@k8s-master-127 nginx]# tail -f k8s-apiserver.com.access_log
172.16.194.130 172.16.194.127:6443 13/May/2019:03:12:07 +0800 200
172.16.194.130 172.16.194.127:6443 13/May/2019:03:12:07 +0800 200
172.16.194.130 172.16.194.128:6443 13/May/2019:03:12:13 +0800 200
172.16.194.130 172.16.194.128:6443 13/May/2019:03:12:13 +0800 200
172.16.194.130 172.16.194.128:6443 13/May/2019:03:12:13 +0800 200
代理成功。

查看集群Node节点状态：

[root@k8s-master-127 nginx]# kubectl get node
NAME STATUS ROLES AGE VERSION
172.16.194.129 Ready 5d10h v1.14.0
172.16.194.130 Ready 5d10h v1.14.0
只有代理连接是正常的，集群节点才会是Ready状态，否则就是上面配置环境有问题。

11.4、Keepalived实现Nginx高可用
目前只部署了一个Nginx节点，只有一个节点就存在单点故障的风险，一旦这个节点挂掉，那么整个集群将会不可用，
市场有些可选的高可用开源解决方案，如：Keepalived、heartbeat等，这里我们选择使用前者。

大致的思路就是：
1、将上面的Nginx代理节点配置多个（两个以上）；
2、通过Keepalived控制VIP来进行漂移，如果Nginx代理节点挂掉后，Keepalived会将VIP漂移到正常的Nginx代理节点上，从而实现集群高可用，提高k8s集群的健壮性。

127节点已经安装和配置好Nginx，128节点按照上面的配置即可。

11.4.1、Keepalived安装与配置
127Master节点部署：

[root@k8s-master-127 ~]# yum install keepalived -y
[root@k8s-master-127 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}

vrrp_script check_nginx {
script “/etc/keepalived/check_nginx.sh”
interval 2
weight 2
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 50
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.194.111
}
track_script {
check_nginx
}
}
检查nginx的脚本：

[root@k8s-master-127 ~]# cat /etc/keepalived/check_nginx.sh
count=$(ps -C nginx --no-header |wc -l)

if [ $count -eq 0 ];then
systemctl stop keepalived
fi
[root@k8s-master-127 ~]# chmod +x /etc/keepalived/check_nginx.sh
启动主节点的Keepalived：

[root@k8s-master-127 ~]# systemctl enable keepalived
[root@k8s-master-127 ~]# systemctl start keepalived
注意：Keepalived不可设置成开机自启动，一旦发生VIP漂移，则需要运维工程师介入排查问题，如果设置成开启启动，有可能会给业务带来二次伤害。

查看Keepalived绑定的VIP：

[root@k8s-master-127 ~]# ip a
···略···
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:41:09:16 brd ff:ff:ff:ff:ff:ff
inet 172.16.194.127/24 brd 172.16.194.255 scope global noprefixroute eth0 # 这是本机IP
valid_lft forever preferred_lft forever
inet 172.16.194.111/32 scope global eth0 # 这是VIP
valid_lft forever preferred_lft forever
···略···
128Backup节点部署：

[root@k8s-master-128 nginx]# yum install keepalived -y
[root@k8s-master-128 nginx]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

vrrp_script check_nginx {
script “/etc/keepalived/check_nginx.sh”
interval 2
weight 2
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 50 # 这个参数要与Master一样，不然不能识别及通讯
priority 50
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.194.111/24
}
track_script {
check_nginx
}
}
检查nginx的脚本：

[root@k8s-master-128 nginx]# vim /etc/keepalived/check_nginx.sh
count=$(ps -C nginx --no-header |wc -l)

if [ $count -eq 0 ];then
systemctl stop keepalived
fi
[root@k8s-master-128 nginx]# chmod +x /etc/keepalived/check_nginx.sh
启动备节点的Keepalived：

[root@k8s-master-128 nginx]# systemctl enable keepalived
[root@k8s-master-128 nginx]# systemctl start keepalived
11.4.2、故障模拟测试
现在状态：127Master节点和128Backup节点都已经启动，VIP绑定在127的eth0网卡上；
理论上，将nginx停掉，VIP会从当前节点消失，漂移到备节点上，那来看实际情况：

模拟故障停掉nginx，看看VIP是否会漂移到备用节点上

[root@k8s-master-127 ~]# systemctl stop nginx
[root@k8s-master-127 ~]# ip a
···略···
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:41:09:16 brd ff:ff:ff:ff:ff:ff
inet 172.16.194.127/24 brd 172.16.194.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
···略···

查看备节点是否有VIP

[root@k8s-master-128 ~]# ip a
···略···
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:f4:da:a9 brd ff:ff:ff:ff:ff:ff
inet 172.16.194.128/24 brd 172.16.194.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 172.16.194.111/24 scope global secondary eth0
valid_lft forever preferred_lft forever
···略···
跟理论情况一样，因故障后，VIP漂移正常，那也就意味着负载均衡加高可用集群搭建完成。

11.5、k8s集群接入负载均衡使用
kube-apiserver的负载均衡主备节点都已经搭建完成并测试成功，接下来就需要配置下k8s的Node组件使用即可

11.5.1、更改Nginx配置
127和128节点的nginx更改如下

[root@k8s-master-127 ~]# vim /etc/nginx/nginx.conf
stream {
log_format main ‘$remote_addr $upstream_addr $time_local $status’;
access_log /var/log/nginx/k8s-apiserver.com.access_log main;
error_log /var/log/nginx/k8s-apiserver.com.error_log warn;
upstream k8s-apiserver {
server 172.16.194.127:6443;
server 172.16.194.128:6443;
}
server {
listen 0.0.0.0:6444; # 将这里更改为监听所有网段
proxy_pass k8s-apiserver;
}
}
重新加载nginx，使配置生效：

[root@k8s-master-127 ~]# systemctl reload nginx
[root@k8s-master-127 ~]# netstat -lntup|grep nginx
tcp 0 0 0.0.0.0:6444 0.0.0.0:* LISTEN 20523/nginx: master

[root@k8s-master-128 ~]# systemctl reload nginx
[root@k8s-master-128 ~]# netstat -lntup|grep nginx
tcp 0 0 0.0.0.0:6444 0.0.0.0:* LISTEN 21604/nginx: master
11.5.2、Node节点配置使用
更改api-server地址的配置

[root@k8s-node-129 cfg]# grep ‘111’ ./*
./bootstrap.kubeconfig: server: https://172.16.194.111:6444
./kubelet.kubeconfig: server: https://172.16.194.111:6444
./kube-proxy.kubeconfig: server: https://172.16.194.111:6444

[root@k8s-node-130 cfg]# grep ‘111’ ./*
./bootstrap.kubeconfig: server: https://172.16.194.111:6444
./kubelet.kubeconfig: server: https://172.16.194.111:6444
./kube-proxy.kubeconfig: server: https://172.16.194.111:6444
重启kubelet和kube-proxy组件：

[root@k8s-node-129 cfg]# systemctl restart kubelet
[root@k8s-node-129 cfg]# systemctl restart kube-proxy
[root@k8s-node-130 cfg]# systemctl restart kubelet
[root@k8s-node-130 cfg]# systemctl restart kube-proxy
查看127上的代理日志：

[root@k8s-master-127 ~]# tail -f /var/log/nginx/k8s-apiserver.com.access_log
172.16.194.130 172.16.194.127:6443 13/May/2019:05:32:55 +0800 200
172.16.194.129 172.16.194.128:6443 13/May/2019:05:32:55 +0800 200
172.16.194.129 172.16.194.127:6443 13/May/2019:05:32:55 +0800 200
172.16.194.130 172.16.194.128:6443 13/May/2019:05:32:55 +0800 200
nginx默认是轮训代理。

11.5.3、K8集群状态
能看到Node节点状态是Ready就没事

[root@k8s-master-128 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
172.16.194.129 Ready 7d v1.14.0
172.16.194.130 Ready 7d v1.14.0
遇到的坑：
1、在生成server证书的时候，需要将VIP填写进去并生成证书，不然就会报错如下：

启动kubelet查看/var/log/messages

certificate is valid for 127.0.0.1, 10.0.0.1, 172.16.194.127, 172.16.194.128, 172.16.194.129, 172.16.194.130, not 172.16.194.111
解决办法：将server证书重新生成（填写VIP进去），并重启api-server进程即可解决。

十二、部署集群内部DNS解析服务（CoreDNS）
在kubernetes1.12之后的版本中，使用了CoreDNS作为默认的DNS；
官网：https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/coredns/

12.1、下载CoreDNS部署文件
https://github.com/coredns/deployment/tree/master/kubernetes

[root@k8s-master-128 ~]# mkdir coredns && cd coredns
[root@k8s-master-128 coredns]# wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed
[root@k8s-master-128 coredns]# wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh
deploy.sh是一个便捷的脚本，用于生成用于在当前运行标准kube-dns的集群上运行CoreDNS的清单。使用coredns.yaml.sed文件作为模板，它创建一个ConfigMap和一个CoreDNS deployment，然后更新 Kube-DNS service selector以使用CoreDNS deployment。通过重新使用现有服务，服务请求不会中断。

12.2、部署CoreDNS
[root@k8s-master-128 coredns]# chmod +x deploy.sh
[root@k8s-master-128 coredns]# ./deploy.sh -i 10.0.0.2 >coredns.yaml
tips:
少了个jq命令：yum install -y jq

[root@k8s-master-128 coredns]# kubectl create -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
查看生成的文件：

apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
rules:

apiGroups:
- “”
  resources:
- endpoints
- services
- pods
- namespaces
  verbs:
- list
- watch
apiGroups:
- “”
  resources:
- nodes
  verbs:
- get

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: “true”
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:

kind: ServiceAccount
name: coredns
namespace: kube-system

apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}

apiVersion: apps/v1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/name: “CoreDNS”
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
spec:
priorityClassName: system-cluster-critical
serviceAccountName: coredns
tolerations:
- key: “CriticalAddonsOnly”
operator: “Exists”
nodeSelector:
beta.kubernetes.io/os: linux
containers:
- name: coredns
image: coredns/coredns:1.5.0
imagePullPolicy: IfNotPresent
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
args: [ “-conf”, “/etc/coredns/Corefile” ]
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
readOnly: true
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /ready
port: 8181
scheme: HTTP
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile

apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
annotations:
prometheus.io/port: “9153”
prometheus.io/scrape: “true”
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: “true”
kubernetes.io/name: “CoreDNS”
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.0.0.2
ports:

name: dns
port: 53
protocol: UDP
name: dns-tcp
port: 53
protocol: TCP
name: metrics
port: 9153
protocol: TCP
早在部署kubelet的时候就通过–cluster-dns=10.0.0.2 --cluster-domain=cluster.local参数指定了dns地址和解析的域，因此直接部署即可使用，如果kubelet启动参数里没有配置这两个dns参数的话，上面部署coredns之后还需要将所有kubelet重新添加配置并重启进程。

12.3、查看CoreDNS状态
[root@k8s-master-128 coredns]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-55f46dd959-q47j5 1/1 Running 0 3m59s
coredns-55f46dd959-vcj4w 1/1 Running 0 3m59s
kubernetes-dashboard-7d5f7c58f5-xqvmh 1/1 Running 1 94m

[root@k8s-master-128 coredns]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.0.0.2 53/UDP,53/TCP,9153/TCP 4m12s
kubernetes-dashboard NodePort 10.0.0.47 443:34756/TCP 14d

[root@k8s-master-128 coredns]# kubectl get deploy -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
coredns 2/2 2 2 5m32s
kubernetes-dashboard 1/1 1 1 14d

[root@k8s-master-128 coredns]# kubectl get ep -n kube-system kube-dns
NAME ENDPOINTS AGE
kube-dns 172.17.17.4:53,172.17.50.2:53,172.17.17.4:53 + 3 more… 10m

[root@k8s-master-128 coredns]# kubectl -n kube-system get configmap coredns
NAME DATA AGE
coredns 1 12m
CoreDNS所创建的服务均已正常启动。

12.4、测试CoreDNS
在安装完Kubernetes cluster环境后，如何验证coreDNS是否在正常工作?这是一项很重要的工作，将会影响将来在容器中部署的服务能否被正常调用。

我们可以通过创建一个busybox 的pod，再在busybox里去解析服务名的方式来验证coreDNS是否正常工作。

具体可参考kubernetes官方文档《Debugging DNS Resolution》

busybox的yaml文件：

[root@k8s-master-128 coredns]# cat busybox.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:

name: busybox
image: busybox:1.28
command:
- sleep
- “3600”
  imagePullPolicy: IfNotPresent
  restartPolicy: Always
  创建Buxybox pod：

[root@k8s-master-128 coredns]# kubectl create -f busybox.yaml
pod/busybox created
[root@k8s-master-128 coredns]# kubectl get pods busybox
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 26s
busybox的resolv.conf内容：

[root@k8s-master-128 coredns]# kubectl exec busybox cat /etc/resolv.conf
nameserver 10.0.0.2
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
在busybox 的pod里解析不同名字空间的服务：
解析规则是：my-svc.my-namespace.svc.cluster.local
因此每个空间的服务都需要指明自己所在的名字空间才可进行访问

[root@k8s-master-128 coredns]# kubectl exec -it busybox nslookup kubernetes.default
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name: kubernetes
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
[root@k8s-master-128 coredns]# kubectl exec -it busybox nslookup kube-dns.kube-system
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name: kube-dns.kube-system
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
实验证明，用 my-svc.my-namespace.svc.cluster.local方法即可访问服务。

在busybox 的pod里解析公网：

[root@k8s-master-128 coredns]# kubectl exec -it busybox nslookup qq.com
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name: qq.com
Address 1: 111.161.64.48 dns48.online.tj.cn
Address 2: 111.161.64.40 dns40.online.tj.cn
[root@k8s-master-128 coredns]# kubectl exec -it busybox nslookup www.baidu.com
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name: www.baidu.com
Address 1: 220.181.38.150
Address 2: 220.181.38.149
在busybox 的pod里解析外部IP地址，按照前文CoreDNS的配置，是通过pod所在node上的/etc/resolv.conf 来代理解析的。
通过以上例子可见，coredns工作正常。coredns既可以管理新生成的service的域名，又可以解析出外部域名。