unable to load server certificate: open /etc/kubernetes/pki/apiserver.crt: no such file or directory

本人使用kubeadm部署k8s集群，并扩容kube-apiserver到3节点,报错

[root@k8s-master pki]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                   READY     STATUS             RESTARTS   AGE
      5h
kube-system   kube-apiserver                         0/1       CrashLoopBackOff   70         5h
kube-system   kube-apiserver-k8s-master              1/1       Running            0          5h
kube-system   kube-apiserver-k8s-node-1              0/1       CrashLoopBackOff   7          15m
kube-system   kube-apiserver-k8s-node-2              0/1       CrashLoopBackOff   7          15m

有三个pod报错CrashLoopBackOff，第一个kube-apiserver暂时没找到原因，直接删除

kubectl delete pod kube-apiserver -n kube-system

后面两个报错参考1

配置 kube-apiserver.yaml

### 拷贝原 master 上的配置：
[root@k8s-master ~]# mkdir -p ~/k8s_install/master/manifests
[root@k8s-master ~]# cd !$
[root@k8s-master manifests]# cp -a /etc/kubernetes/manifests/kube-apiserver.yaml k8s-node-1.kube-apiserver.yaml
### 替换 ip 信息：
[root@k8s-master manifests]# sed -i 's#192.168.203.201#192.168.203.202#' k8s-node-1.kube-apiserver.yaml

[root@k8s-master manifests]# cp -a /etc/kubernetes/manifests/kube-apiserver.yaml tvm-02.kube-apiserver.yaml
### 替换 ip 信息：
[root@k8s-master manifests]# sed -i 's#192.168.203.201#192.168.203.203#' k8s-node-2.kube-apiserver.yaml

### 启动：
[root@k8s-master manifests ~]# scp k8s-node-1.kube-apiserver.yaml 192.168.203.202:/etc/kubernetes/manifests/kube-apiserver.yaml
[root@k8s-master manifests ~]# scp k8s-node-2.kube-apiserver.yaml 192.168.203.203:/etc/kubernetes/manifests/kube-apiserver.yaml

### 但，查看 pods 的状态发现，需要证书
[root@k8s-master ~]# kubectl get pods --all-namespaces |grep 'kube-apiserver-k8s'
kube-system   kube-apiserver-k8s-master            1/1       Running            0          2h
kube-system   kube-apiserver-k8s-node-1            0/1       CrashLoopBackOff   5          5m
kube-system   kube-apiserver-k8s-node-2            0/1       CrashLoopBackOff   5          3m
[root@k8s-master ~]# kubectl logs kube-apiserver-k8s-node-1 -n kube-system 
I1226 10:35:46.521561       1 server.go:121] Version: v1.9.0
unable to load server certificate: open /etc/kubernetes/pki/apiserver.crt: no such file or directory

 分析 kube-apiserver 依赖的证书

### 查看下 kube-apiserver 依赖了哪些证书：
[root@k8s-master pki]# cat /etc/kubernetes/manifests/kube-apiserver.yaml |grep '=/etc/kubernetes/pki' |sort
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

### 其中，可以直接拿来复用的有以下几个：
ca.crt
apiserver-kubelet-client.crt
apiserver-kubelet-client.key
front-proxy-ca.crt
front-proxy-client.key
sa.pub

### 需要为每个 master 节点重新生成证书的有以下几个：
apiserver.crt
apiserver.key

### 基本思路确定了，开始干活
### 先查看当前 master 上的 apiserver.crt 证书里边有什么内容

[root@k8s-master ~]# cd /etc/kubernetes/pki/
[root@k8s-master pki]# openssl x509 -noout -text -in apiserver.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1949677591623098936 (0x1b0ea570933be238)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Dec 22 08:24:18 2017 GMT
            Not After : Dec 22 08:24:19 2018 GMT
        Subject: CN=kube-apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                ###（输出略）
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                DNS:k8s-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:192.168.203.201
    Signature Algorithm: sha256WithRSAEncryption
    ###（输出略）

 为新节点生成专属证书

### 准备工作，先拷贝当前 master 上的 key 并移除 apiserver 证书，后续复用此处的证书：
[root@k8s-master ~]# mkdir -p ~/k8s_install/master/pki/original
[root@k8s-master ~]# cd !$
[root@k8s-master original]# cp -a /etc/kubernetes/pki/* . && rm -f apiserver.key apiserver.crt

### 准备工作，复用证书，生成本节点专属的 apiserver 证书：
[root@k8s-master ~]# mkdir -p ~/k8s_install/master/pki/k8s-node-1
[root@k8s-master ~]# cd !$
[root@k8s-master k8s-node-1]# cp -a ../original/* .
### 生成2048位的密钥对：
[root@k8s-master k8s-node-1]# openssl genrsa -out apiserver.key 2048
Generating RSA private key, 2048 bit long modulus
.....+++
..........................................................+++
e is 65537 (0x10001)

### 生成证书签署请求文件：
[root@k8s-master k8s-node-1]# openssl req -new -key apiserver.key -subj "/CN=kube-apiserver," -out apiserver.csr

### 生成 ext 文件：
[root@k8s-master k8s-node-1]# echo 'subjectAltName = DNS:k8s-node-1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP:10.96.0.1, IP:192.168.203.202' >apiserver.ext

### 使用 ca.key 和 ca.crt 签署上述请求：
[root@k8s-master k8s-node-1]# openssl x509 -req -in apiserver.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out apiserver.crt -days 365 -extfile apiserver.ext
Signature ok
subject=/CN=kube-apiserver,
Getting CA Private Key

### 查看新生成的证书：
[root@k8s-master k8s-node-1]# openssl x509 -noout -text -in apiserver.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 12945856570840330031 (0xb3a8ebf60a182f2f)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Dec 26 10:54:23 2017 GMT
            Not After : Dec 26 10:54:23 2018 GMT
        Subject: CN=kube-apiserver,
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                （输出略）
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:k8s-node-1, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:192.168.203.202
    Signature Algorithm: sha1WithRSAEncryption
    （输出略）

### 另一个节点类似：
### 准备工作，复用证书，生成本节点专属的 apiserver 证书：
[root@k8s-master ~]# mkdir -p ~/k8s_install/master/pki/k8s-node-2
[root@k8s-master ~]# cd !$

[root@k8s-master k8s-node-2]# cp -a ../original/* .
### 生成2048位的密钥对：
[root@k8s-master k8s-node-2]# openssl genrsa -out apiserver.key 2048
### 生成证书签署请求文件：
[root@k8s-master k8s-node-2]# openssl req -new -key apiserver.key -subj "/CN=kube-apiserver," -out apiserver.csr
### 生成 ext 文件：
[root@k8s-master k8s-node-2]# echo 'subjectAltName = DNS:k8s-node-2,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP:10.96.0.1, IP:192.168.203.203' >apiserver.ext

### 使用 ca.key 和 ca.crt 签署上述请求：
[root@k8s-master k8s-node-2]# openssl x509 -req -in apiserver.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out apiserver.crt -days 365 -extfile apiserver.ext

### 查看新生成的证书：
[root@k8s-master k8s-node-2]# openssl x509 -noout -text -in apiserver.crt

 下发证书到对应的节点

[root@k8s-master ~]# cd ~/k8s_install/master/pki
[root@k8s-master pki]# scp k8s-node-1/* 192.168.203.202:/etc/kubernetes/pki/
[root@k8s-master pki]# scp k8s-node-2/* 192.168.203.203:/etc/kubernetes/pki/

 确认每个节点的 apiserver 是否处于 Running 状态，然而还是不行

### 查看 pods 的状态
[root@k8s-master ~]# kubectl get pods --all-namespaces |grep 'kube-apiserver-tvm'
kube-system   kube-apiserver-k8s-master            1/1       Running           0          17h
kube-system   kube-apiserver-k8s-node-1            0/1       CrashLoopBackOff  184        15h
kube-system   kube-apiserver-k8s-node-2            0/1       CrashLoopBackOff  183        15h

 查看原因

[root@k8s-k8s-master pki]# kubectl describe kube-apiserver-k8s-node-1 -n kube-system
the server doesn't have a resource type "kube-apiserver-k8s-node-1"

参考2

[root@k8s-master pki]# sudo cp /etc/kubernetes/admin.conf ~/.kube/config

 即可

### 查看 pods 的状态
[root@k8s-master ~]# kubectl get pods --all-namespaces |grep 'kube-apiserver-tvm'
kube-system   kube-apiserver-k8s-master            1/1       Running  0          17h
kube-system   kube-apiserver-k8s-node-1            1/1       Running  184        15h
kube-system   kube-apiserver-k8s-node-2            1/1       Running  183        15h