k8s踩坑(三)、kubeadm证书/etcd证书过期处理

故障现象

使用kubeadm部署的集群,在运行了一年之后今天,出现k8s api无法调取的现象,使用kubectl命令获取资源均返回如下报错:

Unable to connect to the server: x509: certificate has expired or is not yet valid

故障排查

查看apiserver.crt证书的签署日期和过期日期:

root@9027:/etc/etcd/ssl# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
            Not Before: Mar  8 03:49:29 2018 GMT
            Not After : Mar  7 05:44:27 2020 GMT

发现恰好刚到过期日期,去github查询才得知kubeadm默认的证书签署过期时间为1年,大坑!google找到方法说可以通过修改kubeadm源代码调整证书签署的过期时间然后重新编译,即可预防这个问题。但是现在证书过期的问题已经发生了,只能通过更换证书的办法解决,在github上有详细的说明,issue链接如下:
https://github.com/kubernetes/kubeadm/issues/581

开始替换apiserver证书

进入master节点

cd /etc/kubernetes
# 备份证书和配置
mkdir ./pki_bak
mkdir ./conf_bak
mv pki/apiserver* ./pki_bak/
mv pki/front-proxy-client.* ./pki_bak/
 mv ./admin.conf ./conf_bak/
 mv ./kubelet.conf ./conf_bak/
 mv ./controller-manager.conf ./conf_bak/
 mv ./scheduler.conf ./conf_bak/

# 创建证书
kubeadm alpha phase certs apiserver --apiserver-advertise-address ${MASTER_API_SERVER_IP}
kubeadm alpha phase certs apiserver-kubelet-client
kubeadm alpha phase certs front-proxy-client

# 生成新配置文件
kubeadm alpha phase kubeconfig all --apiserver-advertise-address ${MASTER_API_SERVER_IP}

# 将新生成的admin配置文件覆盖掉原本的admin文件
mv $HOME/.kube/config $HOME/.kube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
sudo chmod 777 $HOME/.kube/config

完成上方操作后,docker restart重启kube-apiserver,kube-controller,kube-scheduler这3个容器

查看证书的过期时间:

root@009027:~# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
            Not Before: Mar  8 03:49:29 2018 GMT
            Not After : Mar  7 05:44:27 2020 GMT

可以看到证书有效期延长了一年

如果有多台master节点,先仿照上方将证书文件和配置文件进行备份,然后将这一台配置完成的master上的证书和配置scp过去:

scp pki/* root@${other_master}:/etc/kubernetes/pki/
scp admin.conf kubelet.conf controller-manager.conf scheduler.conf root@${other_master}:/etc/kubernetes/

scp完成后记得重启相应master上的3个kubernetes组件容器。

验证

kubectl命令发现还是无法查看资源,检查apiserver的日志: kubectl logs ${api-server_container_id}

I0308 06:26:52.956024       1 serve.go:89] Serving securely on [::]:6443
I0308 06:26:52.956168       1 available_controller.go:262] Starting AvailableConditionController
I0308 06:26:52.956180       1 apiservice_controller.go:112] Starting APIServiceRegistrationController
I0308 06:26:52.956244       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
E0308 06:26:52.956279       1 cache.go:35] Unable to sync caches for APIServiceRegistrationController controller
I0308 06:26:52.956291       1 apiservice_controller.go:116] Shutting down APIServiceRegistrationController
I0308 06:26:52.956325       1 crd_finalizer.go:242] Starting CRDFinalizer
I0308 06:26:52.956195       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0308 06:26:52.956342       1 crd_finalizer.go:246] Shutting down CRDFinalizer
I0308 06:26:52.956363       1 customresource_discovery_controller.go:152] Starting DiscoveryController
E0308 06:26:52.956376       1 customresource_discovery_controller.go:155] timed out waiting for caches to sync
I0308 06:26:52.956392       1 naming_controller.go:274] Starting NamingConditionController
E0308 06:26:52.956399       1 cache.go:35] Unable to sync caches for AvailableConditionController controller
I0308 06:26:52.956403       1 naming_controller.go:278] Shutting down NamingConditionController
I0308 06:26:52.956435       1 controller.go:84] Starting OpenAPI AggregationController
I0308 06:26:52.956457       1 controller.go:90] Shutting down OpenAPI AggregationController
I0308 06:26:52.956476       1 crdregistration_controller.go:110] Starting crd-autoregister controller
I0308 06:26:52.956487       1 controller_utils.go:1019] Waiting for caches to sync for crd-autoregister controller
E0308 06:26:52.956504       1 controller_utils.go:1022] Unable to sync caches for crd-autoregister controller
I0308 06:26:52.957364       1 customresource_discovery_controller.go:156] Shutting down DiscoveryController
I0308 06:26:52.958437       1 available_controller.go:266] Shutting down AvailableConditionController
I0308 06:26:52.959508       1 crdregistration_controller.go:115] Shutting down crd-autoregister controller

发现kube-apiserver容器无法正常启动,怀疑是证书配置过程中出错,重复了如上步骤多次,发现问题依旧。etcd和kubeadm是同一时间部署的,这个时候考虑etcd是否有问题

etcd证书过期处理

检查etcd的证书,发现etcd同样也过期了:

root@yksp009027:/etc/etcd/ssl# openssl x509 -in etcd.pem -noout -text |grep ' Not '
            Not Before: Mar  7 00:59:00 2018 GMT
            Not After : Mar  7 00:59:00 2019 GMT

查看此前部署的时候配置etcd的ca配置文件,发现指定的证书签署过期时间居然是1年(8760h)。。太坑了!(不清楚证书生成以及etcd安装流程的,可以参考此前的文章:部署篇)

root@009027:~/ssl# cat config.json
{
"signing": {
    "default": {
      "expiry": "8760h"
      },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "8760h"
      }
    }
}
}

居然和kubeadm生产的api-server证书在同一时间过期了,两个坑连在一起,简直防不胜防!

知道问题所在了,那么现在开始替换etcd的证书:

首先备份etcd数据:

cd /var/lib
tar -zvcf etcd.tar.gz etcd/

修改ca配置文件,将默认证书签署过期时间修改为10年:


root@009027:~/ssl# cat config.json
{
"signing": {
    "default": {
      "expiry": "87600h"
      },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
}
}

生成新的证书:

#删除过期证书
rm -f /etc/etcd/ssl/*

# 创建新证书
cfssl gencert -ca=ca.pem   -ca-key=ca-key.pem   -config=config.json   -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
cp etcd.pem etcd-key.pem  ca.pem /etc/etcd/ssl/

# 查看新证书
root@009027:/etc/etcd/ssl# openssl x509 -in etcd.pem -noout -text |grep ' Not '
            Not Before: Mar  8 06:59:00 2019 GMT
            Not After : Mar  5 06:59:00 2029 GMT

#拷贝到其他etcd节点
scp -r /etc/etcd/ssl root@${other_node}:/etc/etcd/

# 重启etcd服务(记住,要3个节点一起重启,不然会hang住)
systemctl restart etcd

处理结果

etcd证书更换完毕后,再次尝试重启apiserver的容器,启动成功,集群api-server恢复,故障解决。连续2个证书过期,一坑接一坑,卡在apiserver一直无法恢复的时候,一度有些恐慌,担心整个集群由此崩溃数据丢失,还在后面顺利解决。在部署之初没有注意到这些细节问题,例如很明显的是etcd证书的签署ca配置的超时时间居然只有1年,才导致后面才踩了这些坑。平时也要及时备份重要数据,要多多考虑极端情况下的服务恢复方案。

展开阅读全文
©️2020 CSDN 皮肤主题: 技术黑板 设计师: CSDN官方博客 返回首页
实付0元
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值