kubeadm安装的k8s集群证书过期处理
kubeadm安装的k8s证书过期处理
一、背景说明
- kubeadm默认证书为一年,一年过期后,会导致api service不可用,使用过程中会出现:
x509: certificate has expired or is not yet valid
- kubelet证书分为
server
和client
两种, k8s1.9开始默认启用了client证书的自动轮换,但server证书自动轮换需要用户配置开启
二、开启server证书自动轮换
此方案适用于证书还未过期
1. 增加kubelet参数
#在/etc/sysconfig/kubelet增加,若多master,都需要配置:
KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true --rotate-server-certificates=true
2. 配置kube-controller-manager
cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-controller-manager
tier: control-plane
name: kube-controller-manager
namespace: kube-system
spec:
containers:
- command:
- kube-controller-manager
- --experimental-cluster-signing-duration=87600h0m0s #增加证书颁发时间参数
- --feature-gates=RotateKubeletServerCertificate=true #开启server证书签发
- --allocate-node-cidrs=true
3. 创建rbac对象,允许节点轮换kubelet server证书
cat > ca-update.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/selfnodeserver
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:node-autoapprove-certificate-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
EOF
kubectl apply -f ca-update.yaml
4. 重启kubelet
systemctl restart kubelet
#查看csr,状态会由Pending to Approved
kubectl get csr
5. 多master其他节点一直处于Pending
#出于安全原因,处于pending状态的master节点需要手动审批
kubectl certificate approve <name>
三、替换server证书
此方案适用于证书已过期,处理完成后,再执行
开启server证书自动轮换
1. 报错信息
kubectl get po
Unable to connect to the server: x509: certificate has expired or is not yet valid
2. 证书备份
cp -Ra /etc/kubernetes /opt/kubernetes-backup-time
3. 删除过期证书
#apiserver证书
rm -f /etc/kubernetes/pki/apiserver*
#front-proxy-client证书
rm -f /etc/kubernetes/pki/front-proxy-client.*
#etcd证书,若etcd是部署在集群外自签证书,不执行以下命令
rm -rf /etc/kubernetes/pki/etcd/healthcheck-client.*
rm -rf /etc/kubernetes/pki/etcd/server.*
rm -rf /etc/kubernetes/pki/etcd/peer.*
4. 重新生成证书
#下载对应版本的kubeadm
wget https://dl.k8s.io/release/v1.10.1/bin/linux/amd64/kubeadm
chmod a+x kubeadm
#生成证书,若使用HA需要配置成vip地址
./kubeadm alpha phase certs all --apiserver-advertise-address <IP address of your master server>
5. 重新生成配置文件
#备份配置文件
mv /etc/kubernetes/*.conf /tmp
#生成配置文件
./kubeadm alpha phase kubeconfig all --apiserver-advertise-address <IP address of your master server>
6. 重启kubelet
systemctl restart kubelet
7. 验证集群
#查看证书过期时间
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
#集群节点状态
kubectl get no