部署k8s ssl集群实践9:部署高可用 kube-scheduler 集群

参考文档:
https://github.com/opsnull/follow-me-install-kubernetes-cluster
感谢作者的无私分享。
集群环境已搭建成功跑起来。
文章是部署过程中遇到的错误和详细操作步骤记录。如有需要对比参考，请按照顺序阅读和测试。

注:二进制文件前面已经分发到各个节点

9.1
创建 kube-scheduler 证书和私钥
创建证书签名请求

[root@k8s-master scheduler]# cat kube-scheduler-csr.json
{
"CN": "system:kube-scheduler",
"hosts": [
"127.0.0.1",
"192.168.1.92",
"192.168.1.93",
"192.168.1.95"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SZ",
"L": "SZ",
"O": "system:kube-scheduler",
"OU": "4Paradigm"
}
]
}
[root@k8s-master scheduler]#

生成证书和私钥

[root@k8s-master scheduler]# cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
[root@k8s-master scheduler]# ls
kube-scheduler.csr  kube-scheduler-csr.json  kube-scheduler-key.pem  kube-scheduler.pem
[root@k8s-master scheduler]#

创建和分发 kubeconfig 文件
kubeconfig 文件包含访问 apiserver 的所有信息，如 apiserver 地址、CA 证书和自身使
用的证书

[root@k8s-master scheduler]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://192.168.1.94:8443 --kubeconfig=kube-scheduler.kubeconfig
Cluster "kubernetes" set.

[root@k8s-master scheduler]# kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig
User "system:kube-scheduler" set.
[root@k8s-master scheduler]#

[root@k8s-master scheduler]# kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
Context "system:kube-scheduler" created.

[root@k8s-master scheduler]# kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
Switched to context "system:kube-scheduler".
[root@k8s-master scheduler]#

9.2
分发kubeconfig到所有节点

[root@k8s-master scheduler]# cp kube-scheduler.kubeconfig /etc/kubernetes/
[root@k8s-master scheduler]# scp kube-scheduler.kubeconfig root@k8s-node1:/etc/kubernetes/
kube-scheduler.kubeconfig                                                                      100% 6370    4.3MB/s  00:00   
[root@k8s-master scheduler]# scp kube-scheduler.kubeconfig root@k8s-node2:/etc/kubernetes/
kube-scheduler.kubeconfig                                                                      100% 6370    3.5MB/s  00:00   
[root@k8s-master scheduler]#

改好权限

[root@k8s-master scheduler]# ll /etc/kubernetes/
总用量 24
drwxr-xr-x 2 k8s  root 4096 8月  23 14:48 cert
-rwxr-xr-x 1 k8s  root  240 8月  23 11:59 encryption-config.yaml
-rwx--x--x 1 k8s  root 6446 8月  23 15:52 kube-controller-manager.kubeconfig
-rw------- 1 root root 6370 8月  29 11:25 kube-scheduler.kubeconfig
[root@k8s-master scheduler]#

参考下面：

[root@k8s-master scheduler]# ll /etc/kubernetes/
总用量 24
drwxr-xr-x 2 k8s root 4096 8月  23 14:48 cert
-rwxr-xr-x 1 k8s root  240 8月  23 11:59 encryption-config.yaml
-rwx--x--x 1 k8s root 6446 8月  23 15:52 kube-controller-manager.kubeconfig
-rwx--x--x 1 k8s root 6370 8月  29 11:25 kube-scheduler.kubeconfig
[root@k8s-master scheduler]#

9.3
创建和分发 kube-scheduler systemd unit 文件

[root@k8s-master scheduler]# cat kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/opt/k8s/bin/kube-scheduler \
--address=127.0.0.1 \
--kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
--leader-elect=true \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2
Restart=on-failure
RestartSec=5
User=k8s
[Install]
WantedBy=multi-user.target
[root@k8s-master scheduler]#

--address ：在 127.0.0.1:10251 端口接收 http /metrics 请求；kube-scheduler
目前还不支持接收 https 请求；
--kubeconfig ：指定 kubeconfig 文件路径，kube-scheduler 使用它连接和验证
kube-apiserver；
--leader-elect=true ：集群运行模式，启用选举功能；被选为 leader 的节点负
责处理工作，其它节点为阻塞状态；
User=k8s ：使用 k8s 账户运行；

分发 systemd unit 文件到所有 master 节点

[root@k8s-master scheduler]# cp kube-scheduler.service /etc/systemd/system
[root@k8s-master scheduler]# scp kube-scheduler.service root@k8s-node1:/etc/systemd/system
kube-scheduler.service                                                                          100%  418  596.8KB/s  00:00   
[root@k8s-master scheduler]# scp kube-scheduler.service root@k8s-node2:/etc/systemd/system
kube-scheduler.service                                                                          100%  418  524.5KB/s  00:00   
[root@k8s-master scheduler]#

9.4
启动服务

[root@k8s-master scheduler]# systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /etc/systemd/system/kube-scheduler.service.
[root@k8s-master scheduler]#

[root@k8s-master scheduler]# systemctl status kube-scheduler.service
● kube-scheduler.service - Kubernetes Scheduler
  Loaded: loaded (/etc/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
  Active: active (running) since 三 2018-08-29 11:33:07 CST; 37s ago
    Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 22590 (kube-scheduler)
    Tasks: 8
  Memory: 10.5M
  CGroup: /system.slice/kube-scheduler.service
          └─22590 /opt/k8s/bin/kube-scheduler --address=127.0.0.1 --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig --lea...
8月 29 11:33:07 k8s-master kube-scheduler[22590]: I0829 11:33:07.136661  22590 server.go:126] Version: v1.11.0
8月 29 11:33:07 k8s-master kube-scheduler[22590]: I0829 11:33:07.136967  22590 factory.go:949] Creating scheduler from a...ider'
8月 29 11:33:07 k8s-master kube-scheduler[22590]: I0829 11:33:07.136978  22590 factory.go:1049] Creating scheduler with ...eCoun
8月 29 11:33:07 k8s-master kube-scheduler[22590]: W0829 11:33:07.139592  22590 authorization.go:47] Authorization is disabled
8月 29 11:33:07 k8s-master kube-scheduler[22590]: W0829 11:33:07.140027  22590 authentication.go:55] Authentication is disabled
8月 29 11:33:07 k8s-master kube-scheduler[22590]: I0829 11:33:07.140038  22590 insecure_serving.go:47] Serving healthz i...10251
8月 29 11:33:08 k8s-master kube-scheduler[22590]: I0829 11:33:08.043409  22590 controller_utils.go:1025] Waiting for cac...oller
8月 29 11:33:08 k8s-master kube-scheduler[22590]: I0829 11:33:08.143594  22590 controller_utils.go:1032] Caches are sync...oller
8月 29 11:33:08 k8s-master kube-scheduler[22590]: I0829 11:33:08.143681  22590 leaderelection.go:185] attempting to acqu...er...
8月 29 11:33:12 k8s-master kube-scheduler[22590]: E0829 11:33:12.165221  22590 leaderelection.go:238] error initially cr...xists
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s-master scheduler]#

[root@k8s-master scheduler]# curl -s http://127.0.0.1:10251/metrics |head
# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0
apiserver_client_certificate_expir

9.5
测试 kube-scheduler 集群的高可用
随便找一个或两个 master 节点，停掉 kube-scheduler 服务，看其它节点是否获取了
leader 权限（systemd 日志）。

查看当前的 leader

[root@k8s-master scheduler]# kubectl get endpoints kube-scheduler --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-node1_405a1599-ab3c-11e8-ba05-000c2933952f","leaseDurationSeconds":15,"acquireTime":"2018-08-29T03:33:11Z","renewTime":"2018-08-29T03:35:54Z","leaderTransitions":0}'
  creationTimestamp: 2018-08-29T03:33:11Z
  name: kube-scheduler
  namespace: kube-system
  resourceVersion: "384663"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler
  uid: 4104efc4-ab3c-11e8-88d6-000c2933952f
[root@k8s-master scheduler]#

可以看到当前是k8s-node1
停止k8s-node1的kube-scheduler.service
[root@k8s-node1 ~]# systemctl stop kube-scheduler.service
[root@k8s-node1 ~]#

已经切换到master了

[root@k8s-node1 ~]# kubectl get endpoints kube-scheduler --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master_3e987454-ab3c-11e8-860c-000c29b6aeef","leaseDurationSeconds":15,"acquireTime":"2018-08-29T03:37:09Z","renewTime":"2018-08-29T03:37:11Z","leaderTransitions":1}'
  creationTimestamp: 2018-08-29T03:33:11Z
  name: kube-scheduler
  namespace: kube-system
  resourceVersion: "384755"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler
  uid: 4104efc4-ab3c-11e8-88d6-000c2933952f
[root@k8s-node1 ~]#

到这里三master节点高可用已经部署完成了。

[root@k8s-master1 kube-scheduler-key]# cd
[root@k8s-master1 ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-2               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
[root@k8s-master1 ~]# kubectl cluster-info
Kubernetes master is running at https://192.168.211.127:8443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@k8s-master1 ~]# 

接下来开始部署work节点。

转载于:https://blog.51cto.com/goome/2165880