阿里云ACK的etcd证书过期手工升级操作

最新推荐文章于 2024-07-28 00:02:30 发布
最新推荐文章于 2024-07-28 00:02:30 发布
阅读量660 收藏
点赞数 1
分类专栏： k8s 文章标签： 阿里云 etcd 云计算 kubernetes
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/rendongxingzhe/article/details/131844743
版权

一.问题现象

阿里云ACK的etcd证书过期，通过图形化界面升级提示升级失败，考虑通过脚本的方式升级ETCD相关的证书。由于在前期做类似的升级ETCD证书失败导致整个集群业务出现访问异常，所有在升级之前做好对应的备份操作是很有必要的

二.前期准备

1.由于升级操作是在master节点操作，涉及的证书文件变更都在master节点，所以针对阿里云ACK的3台master节点做快照。

2.针对阿里云ACK的3台master节点涉及的证书目录做备份，必要情况下备份到其他服务器上。

3.针对阿里云ACK的etcd数据库做备份操作

date;
CACERT="/var/lib/etcd/cert/ca.pem"
CERT="/var/lib/etcd/cert/etcd-server.pem"
EKY="/var/lib/etcd/cert/etcd-server-key.pem"
ENDPOINTS="172.16.0.87:2379"

#etcdctl snapshot save /data/etcd_backup_dir/etcd-snapshot-`date +%Y%m%d`.db

ETCDCTL_API=3 etcdctl \
--cacert="${CACERT}" --cert="${CERT}" --key="${EKY}" \
--endpoints=${ENDPOINTS} \
snapshot save /data/etcd_backup_dir/etcd-snapshot-`date +%Y%m%d`.db

备份ETCD的数据目录 

cp -r /var/lib/etcd /var/lib/etcd-`date +%Y%m%d`.bak

4.针对阿里云ACK的所有部署资源的yaml文件做备份，涉及service deploy ingress configmap secret job cronjob daemonset statefulset pvc等，备份脚本如下：

#!/bin/bash
#define variable
BACKUP_PATH=/data/k8s-backup
BACKUP_PATH_DATA=$BACKUP_PATH/yaml/`date +%Y%m%d%H%M%S`
BACKUP_PATH_LOG=$BACKUP_PATH/log
BACKUP_LOG_FILE=$BACKUP_PATH_LOG/k8s-backup-`date +%Y%m%d%H%M%S`.log
# base function
function printlog(){
 echo "`date +'%Y-%m-%d %H:%M:%S'` $1"
 echo "`date +'%Y-%m-%d %H:%M:%S'` $1" >> $BACKUP_LOG_FILE 2>&1 
}
function printlogonly(){
 echo "`date +'%Y-%m-%d %H:%M:%S'` $1" >> $BACKUP_LOG_FILE 2>&1 
}
# set K8s type（此处可根据集群资源自行修改）
CONFIG_TYPE="service deploy ingress configmap secret job cronjob daemonset statefulset pvc"
# make dir
mkdir -p $BACKUP_PATH_DATA
mkdir -p $BACKUP_PATH_LOG
cd $BACKUP_PATH_DATA
# set namespace list
ns_list=`kubectl get ns | awk '{print $1}' | grep -v NAME`
if [ $# -ge 1 ]; then
ns_list="$@"
fi
# define counters
COUNT_NS=0
COUNT_ITEM_IN_NS=0
COUNT_ITEM_IN_TYPE=0
COUNT_ITEM_ALL=0
# print hint
printlog "Backup kubernetes config in namespaces: ${ns_list}"
printlog "Backup kubernetes config for [type: ${CONFIG_TYPE}]."
printlog "If you want to read the record of backup, please input command ' tail -100f ${BACKUP_LOG_FILE} '"
# ask and answer
message="This will backup resources of kubernetes cluster to yaml files."
printlog ${message}ls

# loop for namespaces
for ns in $ns_list;
do
COUNT_NS=`expr $COUNT_NS + 1`
printlog "Backup No.${COUNT_NS} namespace [namespace: ${ns}]."
COUNT_ITEM_IN_NS=0

## loop for types
for type in $CONFIG_TYPE; 
do
printlogonly "Backup type [namespace: ${ns}, type: ${type}]."
item_list=`kubectl -n $ns get $type | awk '{print $1}' | grep -v NAME | grep -v "No "`
COUNT_ITEM_IN_TYPE=0

## loop for items
for item in $item_list; 
do 
file_name=$BACKUP_PATH_DATA/${ns}_${type}_${item}.yaml
printlogonly "Backup kubernetes config yaml [namespace: ${ns}, type: ${type}, item: ${item}] to file: ${file_name}"
kubectl -n $ns get $type $item -o yaml > $file_name
COUNT_ITEM_IN_NS=`expr $COUNT_ITEM_IN_NS + 1`
COUNT_ITEM_IN_TYPE=`expr $COUNT_ITEM_IN_TYPE + 1`
COUNT_ITEM_ALL=`expr $COUNT_ITEM_ALL + 1`
printlogonly "Backup No.$COUNT_ITEM_ALL file done."
done;

done;
printlogonly "Backup $COUNT_ITEM_IN_TYPE files in [namespace: ${ns}, type: ${type}]."

printlog "Backup ${COUNT_ITEM_IN_NS} files done in [namespace: ${ns}]."
done;

# show stats
printlog "Backup ${COUNT_ITEM_ALL} yaml files in all."
printlog "kubernetes Backup completed, all done."
exit 0

三.升级证书操作步骤

1.确认集群Master节点之间配置了root用户的免密登录。

在Master上通过SSH方式登录其他任意Master节点，如果提示输入密码，请您按如下方式配置Master节点之间的免密登录。

1. ssh-keygen -t rsa  # 生成密钥。
2. ssh-copy-id -i ~/.ssh/id_rsa.pub $(internal-ip)     # 使用ssh-copy-id工具传输公钥到其他所有Master节点，$（internal-ip）为其他Master节点的内网IP。

2.分别复制以下脚本内容，保存并命名为restart-apiserver.sh和rotate-etcd.sh，然后将两者保存到同一个文件夹下

restart-apiserver.sh

 #! /bin/bash

function restart_apiserver() {
  apiserverID=$(/usr/bin/docker ps | grep kube-apiserver | grep -v NAME | awk '{print $1}')
  /usr/bin/docker stop $apiserverID
  rm -rf /root/kube-apiserver.yaml
  mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml
  while true; do
    NUM=$(docker ps | grep kube-apiserver | wc -l)
    if [[ $NUM == 0 ]]; then
      break
    fi
    sleep 1
  done
  /usr/bin/docker ps -a | grep kube-apiserver | awk '{print $1}' | xargs docker stop
  /usr/bin/docker ps -a | grep kube-apiserver | awk '{print $1}' | xargs docker rm

  mv /root/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
  while true; do
    NUM=$(docker ps | grep kube-apiserver | grep -v pause | wc -l)
    if [[ $NUM == 1 ]]; then
      break
    fi
    sleep 1
  done

  k8s::wait_apiserver_ready
}

k8s::wait_apiserver_ready() {
	set -e
	for i in $(seq 180); do
		if kubectl get po &>/dev/null; then
			return 0
		else
			echo "wait apiserver to be ready, retry ${i}th after 1s"
			sleep 1
		fi
	done

	echo "failed to wait apiserver to be ready"
	return 1
}

function restart_container() {
  crictl pods | grep kube-apiserver | awk '{print $1}' | xargs -I '{}' crictl stopp {} || true
}

if [[ -f /usr/bin/docker ]]; then
  restart_apiserver
else
  restart_container
  k8s::wait_apiserver_ready
fi
echo "API Server restarted"

 rotate-etcd.sh

#!/bin/bash

set -eo pipefail

dir=/tmp/etcdcert
KUBE_CERT_PATH=/etc/kubernetes/pki
ETCD_CERT_DIR=/var/lib/etcd/cert
ETCD_HOSTS=""
function get_etcdhosts() {
  name1=$(ls $ETCD_CERT_DIR | grep name | grep name-1.pem | sed 's/-name-1.pem//g')
  name2=$(ls $ETCD_CERT_DIR | grep name | grep name-2.pem | sed 's/-name-2.pem//g')
  name3=$(ls $ETCD_CERT_DIR | grep name | grep name-3.pem | sed 's/-name-3.pem//g')

  echo "hosts: " $name1 $name2 $name3
  #  ETCD_HOSTS=($name1 $name2 $name3)
  ETCD_HOSTS=$name1" "$name2" "$name3
}

function gencerts() {
  echo "generate ssl cert ..."
  rm -rf $dir
  mkdir -p "$dir"
  hosts=$(echo $ETCD_HOSTS | tr -s " " ",")

  echo "-----generate ca"
  echo '{"CN":"CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' |
    cfssl gencert -initca - | cfssljson -bare $dir/ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/ca-config.json

  echo "-----generate etcdserver"
  export ADDRESS=$hosts,ext1.example.com,coreos1.local,coreos1
  export NAME=etcd-server
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME
  export ADDRESS=
  export NAME=etcd-client
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME

  # gen peer-ca
  echo "-----generate peer certificates"
  echo '{"CN":"Peer-CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' | cfssl gencert -initca - | cfssljson -bare $dir/peer-ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/peer-ca-config.json
  i=0
  for host in $ETCD_HOSTS; do
    ((i = i + 1))
    export MEMBER=${host}-name-$i
    echo '{"CN":"'${MEMBER}'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
      cfssl gencert -ca=$dir/peer-ca.pem -ca-key=$dir/peer-ca-key.pem -config=$dir/peer-ca-config.json -profile=peer \
        -hostname="$hosts,${MEMBER}.local,${MEMBER}" - | cfssljson -bare $dir/${MEMBER}
    #-hostname="$host,${MEMBER}.local,${MEMBER}" - | cfssljson -bare $dir/${MEMBER}
  done

  ## backup
  TIMESTAMP=$(date "+%Y%m%d%H%M%S")
  \cp -r $ETCD_CERT_DIR $ETCD_CERT_DIR_$TIMESTAMP
  \cp -r $KUBE_CERT_PATH/etcd $KUBE_CERT_PATH/etcd_$TIMESTAMP

  # 制作bundle ca
  cat $KUBE_CERT_PATH/etcd/ca.pem >>$dir/bundle_ca.pem
  cat $ETCD_CERT_DIR/ca.pem >>$dir/bundle_ca.pem
  cat $dir/ca.pem >>$dir/bundle_ca.pem

  # 制作bundle peer-ca
  cat $ETCD_CERT_DIR/peer-ca.pem >$dir/bundle_peer-ca.pem
  cat $dir/peer-ca.pem >>$dir/bundle_peer-ca.pem

  # chown
  chown -R etcd:etcd $dir
  chmod 0644 $dir/*
}

function rotate_etcd_ca() {
  # Update certs on etcd nodes.
  for ADDR in $ETCD_HOSTS; do
    TIMESTAMP=$(date "+%Y%m%d%H%M%S")
    ssh -o StrictHostKeyChecking=no root@$ADDR cp -r $ETCD_CERT_DIR $ETCD_CERT_DIR_$TIMESTAMP
    echo "update etcd CA on node $ADDR"
    scp -o StrictHostKeyChecking=no $dir/bundle_ca.pem root@$ADDR:$ETCD_CERT_DIR/ca.pem
    scp -o StrictHostKeyChecking=no $dir/bundle_ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
    scp -o StrictHostKeyChecking=no $dir/etcd-client.pem root@$ADDR:$KUBE_CERT_PATH/etcd/etcd-client.pem
    scp -o StrictHostKeyChecking=no $dir/etcd-client-key.pem root@$ADDR:$KUBE_CERT_PATH/etcd/etcd-client-key.pem
    scp -o StrictHostKeyChecking=no $dir/bundle_peer-ca.pem root@$ADDR:$ETCD_CERT_DIR/peer-ca.pem
    ssh -o StrictHostKeyChecking=no root@$ADDR chown -R etcd:etcd $ETCD_CERT_DIR
    ssh -o StrictHostKeyChecking=no root@$ADDR chmod 0644 $ETCD_CERT_DIR/*
    echo "restart etcd on node $ADDR"
    ssh -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
    echo "etcd on node $ADDR restarted"
    ssh -o StrictHostKeyChecking=no root@$ADDR /usr/bin/bash /tmp/restart-apiserver.sh
    echo "apiserver on node $ADDR restarted"
    sleep 10
  done
}

function rotate_etcd_certs() {
  for ADDR in $ETCD_HOSTS; do
    echo "update etcd peer certs on node $ADDR"
    scp -o StrictHostKeyChecking=no \
      $dir/{peer-ca-key.pem,etcd-server.pem,etcd-server-key.pem,etcd-client.pem,etcd-client-key.pem,ca-key.pem,*-name*.pem} root@$ADDR:$ETCD_CERT_DIR/

    ssh -o StrictHostKeyChecking=no root@$ADDR chown -R etcd:etcd $ETCD_CERT_DIR

    ssh -o StrictHostKeyChecking=no root@$ADDR \
       chmod 0400 $ETCD_CERT_DIR/{peer-ca-key.pem,etcd-server.pem,etcd-server-key.pem,etcd-client.pem,etcd-client-key.pem,ca-key.pem,*-name*.pem}

    echo "restart etcd on node $ADDR"
    ssh -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
    echo "etcd on node $ADDR restarted"
    sleep 10
  done
}

function recover_etcd_ca() {
  # Update certs on etcd nodes.
  for ADDR in $ETCD_HOSTS; do
    echo "replace etcd CA on node $ADDR"
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$ETCD_CERT_DIR/ca.pem
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
    scp -o StrictHostKeyChecking=no $dir/peer-ca.pem root@$ADDR:$ETCD_CERT_DIR/peer-ca.pem
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
    ssh -o StrictHostKeyChecking=no root@$ADDR chown -R etcd:etcd $ETCD_CERT_DIR
    echo "restart apiserver on node $ADDR"
    ssh -o StrictHostKeyChecking=no root@$ADDR bash /tmp/restart-apiserver.sh
    echo "apiserver on node $ADDR restarted"
    echo "restart etcd on node $ADDR"
    sleep 5
    ssh -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
    echo "etcd on node $ADDR restarted"
    sleep 5
  done
}

function renew_k8s_certs() {
  # 更新K8s证书，根据集群Region替换下面cn-hangzhou的默认镜像地域。
  for ADDR in $ETCD_HOSTS; do
    echo "renew k8s components cert on node $ADDR"
    #compatible containerd
    set +e
    ssh -o StrictHostKeyChecking=no root@$ADDR docker run --privileged=true  -v /:/alicoud-k8s-host --pid host --net host \
             registry.cn-hangzhou.aliyuncs.com/acs/etcd-rotate:v2.0.0 /renew/upgrade-k8s.sh --role master

    ssh -o StrictHostKeyChecking=no root@$ADDR ctr image pull registry.cn-hangzhou.aliyuncs.com/acs/etcd-rotate:v2.0.0
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr run --privileged=true --mount type=bind,src=/,dst=/alicoud-k8s-host,options=rbind:rw \
            --net-host registry.cn-hangzhou.aliyuncs.com/acs/etcd-rotate:v2.0.0 cert-rotate /renew/upgrade-k8s.sh --role master
    set -e
    echo "finished renew k8s components cert on $ADDR"
    sleep 5
  done
}

function generate_cm() {
    echo "generate status configmap"

    cat <<-"EOF" > /tmp/ack-rotate-etcd-ca-cm.yaml.tpl
apiVersion: v1
kind: ConfigMap
metadata:
  name: ack-rotate-etcd-status
  namespace: kube-system
data:
  status: "success"
  hosts: "$hosts"
EOF

    sed -e "s#\$hosts#$ETCD_HOSTS#" /tmp/ack-rotate-etcd-ca-cm.yaml.tpl | kubectl apply -f -
}

get_etcdhosts
echo "${ETCD_HOSTS[@]}"

echo "---renew k8s components certs---"
renew_k8s_certs
echo "---end to renew k8s components certs---"

# Update certs on etcd nodes.
for ADDR in $ETCD_HOSTS; do
  scp -o StrictHostKeyChecking=no restart-apiserver.sh root@$ADDR:/tmp/restart-apiserver.sh
  ssh -o StrictHostKeyChecking=no root@$ADDR chmod +x /tmp/restart-apiserver.sh
done

gencerts
echo "---rotate etcd ca and etcd client ca---"
rotate_etcd_ca
echo "---end to rotate etcd ca and etcd client ca---"
echo
echo "---rotate etcd peer and certs---"
rotate_etcd_certs
echo "---end to rotate etcd peer and certs---"

echo
echo "---replace etcd ca---"
recover_etcd_ca
echo "---end to replace etcd ca---"

generate_cm
echo "etcd CA and certs have succesfully rotated!"

提前下载镜像文件 registry.cn-hangzhou.aliyuncs.com/acs/etcd-rotate:v2.0.0

镜像里面的/renew/upgrade-k8s.sh

#!/bin/sh
set -xe

if [ -d "/alicoud-k8s-host" ]; then
	rm -rf /alicoud-k8s-host/usr/local/k8s-upgrade
	mkdir -p /alicoud-k8s-host/usr/local/k8s-upgrade
	cp -r /renew/* /alicoud-k8s-host/usr/local/k8s-upgrade
	ls -l /alicoud-k8s-host/usr/local/k8s-upgrade
	chmod  -R +x /alicoud-k8s-host/usr/local/k8s-upgrade/
	chroot /alicoud-k8s-host /usr/local/k8s-upgrade/rotate.sh "$@"
fi

rotate.sh

#!/bin/sh
set -xe

if [ -d "/alicoud-k8s-host" ]; then
	rm -rf /alicoud-k8s-host/usr/local/k8s-upgrade
	mkdir -p /alicoud-k8s-host/usr/local/k8s-upgrade
	cp -r /renew/* /alicoud-k8s-host/usr/local/k8s-upgrade
	ls -l /alicoud-k8s-host/usr/local/k8s-upgrade
	chmod  -R +x /alicoud-k8s-host/usr/local/k8s-upgrade/
	chroot /alicoud-k8s-host /usr/local/k8s-upgrade/rotate.sh "$@"
fi





[root@master01 ~]# docker run -it registry.cn-hangzhou.aliyuncs.com/acs/etcd-rotate:v2.0.0 cat /renew/rotate.sh
#!/usr/bin/env bash

set -e -x

public::common::log() {
  echo $(date +"[%Y%m%d %H:%M:%S]: ") $1
}

function retry() {
  local n=0
  local try=$1
  local cmd="${@:2}"
  [[ $# -le 1 ]] && {
    echo "Usage $0 <retry_number> <Command>"
  }
  set +e
  until
    [[ $n -ge $try ]]
  do
    $cmd && break || {
      echo "Command Fail.."
      ((n++))
      echo "retry $n :: [$cmd]"
      sleep 2
    }
  done
  set -e
}

public::upgrade::backupmaster() {
  local backup_dir=/etc/kubeadm/backup-rotate-$(date +%F)
  if [ ! -f $backup_dir/kubelet.conf ]; then
    mkdir -p $backup_dir $backup_dir/kubelet $backup_dir/etcd
    cp -rf /etc/kubernetes/ $backup_dir/
    cp /etc/kubeadm/kubeadm.cfg $backup_dir/
    cp /etc/systemd/system/kubelet.service.d/10-kubeadm.conf $backup_dir
    cp /etc/kubernetes/kubelet.conf $backup_dir
    cp -rf /var/lib/kubelet/pki/* $backup_dir/kubelet
    cp -rf /var/lib/etcd/cert/* $backup_dir/etcd
  else
    public::common::log "master configuration is already backup, skip."
  fi
}

public::upgrade::backupnode() {
  public::common::log "Begin the node backup working."

  local backup_dir=/etc/kubeadm/backup-rotate-$(date +%F)
  if [ ! -f $backup_dir/10-kubeadm.conf ]; then
    mkdir -p $backup_dir $backup_dir/kubelet
    cp -rf /etc/kubernetes/ $backup_dir/
    cp /etc/kubernetes/kubelet.conf $backup_dir
    cp /etc/systemd/system/kubelet.service.d/10-kubeadm.conf $backup_dir
    cp -rf /var/lib/kubelet/pki/* $backup_dir/kubelet
  else
    public::common::log "node configuration is already backup, skip."
  fi
}

public::main::master-rotate() {
  ls -l /usr/local
  pwd
  local backup_dir=/etc/kubeadm/backup-rotate-$(date +%F)
  echo "mode is $MODE"
  if ! grep "rotate-certificates" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf; then
    ./usr/local/k8s-upgrade/cert-rotate -mode=$MODE -role=master -nodeip=$NODE_IP -auto-rotate=false >$backup_dir/renew.log
  else
    ./usr/local/k8s-upgrade/cert-rotate -mode=$MODE -role=master -nodeip=$NODE_IP >$backup_dir/renew.log
  fi
  if [[ "$MODE" = "etcd" ]]; then
    public::common::log "Successful update cert on $(hostname)"
    exit 0
  fi

  sleep 1
  #renew the dashboard certs
  if [ -d /etc/kubernetes/pki/dashboard ]; then
    cp -rf /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/dashboard/dashboard.crt
    cp -rf /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/dashboard/dashboard.key
    cp -rf /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/dashboard/dashboard-ca.crt
    cat /etc/kubernetes/pki/client-ca.crt >>/etc/kubernetes/pki/dashboard/dashboard-ca.crt
    if [ -f /etc/kubernetes/pki/user-ca.crt ]; then
      cat /etc/kubernetes/pki/user-ca.crt >>/etc/kubernetes/pki/dashboard/dashboard-ca.crt
    fi
    cp -rf /etc/kubernetes/pki/dashboard/dashboard-ca.crt /etc/kubernetes/pki/apiserver-ca.crt
  else
    cp -rf /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/apiserver-ca.crt
    cat /etc/kubernetes/pki/client-ca.crt >>/etc/kubernetes/pki/apiserver-ca.crt
    if [ -f /etc/kubernetes/pki/user-ca.crt ]; then
      cat /etc/kubernetes/pki/user-ca.crt >>/etc/kubernetes/pki/apiserver-ca.crt
    fi
  fi

  # /etc/kubernetes/manifests pod can not be pull up automatically. use this to workaround.
  set +e
  docker ps | grep kube-controller-manager | awk '{print $1}' | xargs -I '{}' docker restart {} || true
  crictl pods | grep kube-controller-manager | awk '{print $1}' | xargs -I '{}' crictl stopp {} || true
  set -e

  sleep 1
  #restart kubelet
  service kubelet restart
  sleep 1

  # /etc/kubernetes/manifests pod can not be pull up automatically. use this to workaround.
  set +e
  docker ps | grep kube-apiserver | awk '{print $1}' | xargs -I '{}' docker restart {} || true
  crictl pods | grep kube-apiserver | awk '{print $1}' | xargs -I '{}' crictl stopp {} || true
  sleep 1
  docker ps | grep kube-scheduler | awk '{print $1}' | xargs -I '{}' docker restart {} || true
  crictl pods | grep kube-scheduler | awk '{print $1}' | xargs -I '{}' crictl stopp {} || true
  set -e

  public::common::log "Successful update cert on $(hostname)"
}

public::main::node-rotate() {
  if [ -f /etc/kubernetes/pki/apiserver.crt ]; then
    public::common::log "Skip node rotate on master node"
    exit 0
  fi
  local backup_dir=/etc/kubeadm/backup-rotate-$(date +%F)

  if ! grep "rotate-certificates" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf; then
    ./usr/local/k8s-upgrade/cert-rotate -mode=$MODE -role=worker -auto-rotate=false -key=$KEY >$backup_dir/renew.log
  else
    ./usr/local/k8s-upgrade/cert-rotate -mode=$MODE -role=worker -key=$KEY >$backup_dir/renew.log
  fi
  sleep 1
  #restart kubelet
  service kubelet restart
  sleep 1

  public::common::log "Successful update cert on $(hostname)"
}

public::main::master() {
  public::upgrade::backupmaster
  public::main::master-rotate
}

public::main::node() {
  public::upgrade::backupnode
  public::main::node-rotate
}

main() {
  #use renew mode in default
  export MODE=renew

  while
    [[ $# -gt 0 ]]
  do
    key="$1"

    case $key in
    --role)
      export ROLE=$2
      shift
      ;;
    --mode)
      export MODE=$2
      shift
      ;;
    --nodeip)
      export NODE_IP=$2
      shift
      ;;
    --rootkey)
      export KEY=$2
      shift
      ;;
    *)
      public::common::log "unkonw option [$key]"
      exit 1
      ;;
    esac
    shift
  done

  mkdir -p /etc/kubeadm/backup-rotate-$(date +%F)
  #public::upgrade::backup

  ######################################################
  case $ROLE in

  "source")
    public::common::log "source scripts"
    ;;
  "master")
    public::main::master
    ;;
  "node")
    public::main::node
    ;;
  *)
    echo "
        Usage:
            $0 --role master|node --mode renew|rotate
            ./rotate.sh
        "
    ;;
  esac
}

main "$@"

 

3.在任意Master节点上运行bash rotate-etcd.sh。

当看到命令行输出etcd CA and certs have succesfully rotated!时，表示所有Master节点上的证书和K8s证书已经轮转完成。

脚本执行过程中，会将etcd相关的服务端和客户端证书备份在如下目录中：

  • /var/lib/etcd/cert_$时间戳后缀

  • /etc/kubernetes/pki/etcd_$时间戳后缀

4.查看证书有效期

openssl x509 -in cert.crt -noout -dates

 

忍冬行者
关注
  • 1
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
kubeadm证书/etcd证书过期处理
小啊宇的博客
10-14 2285
今天突然测试环境的Kubernetes 持续集成/持续发布出了问题了，然后上测试环境服务器排查，发现kubectl指令执行出现问题， Unable to connect to the server: x509: certificate has expired or is not yet valid 然后翻译了一下提示证书已过期，网上查了下资料，说是：kubernetes的apiServer 与kubelet的访问授权证书是一年，官方的解释是：通过这种方式，让用户不断的升级版本。给出的解决方案有以下几种：
阿里云容器服务ACK产品介绍.pdf
05-11
### 阿里云容器服务ACK产品介绍 #### 一、概述 阿里云容器服务ACK（Alibaba Cloud Container Service for Kubernetes）是阿里云提供的一种基于Kubernetes的企业级容器管理服务。它不仅支持标准的Kubernetes API，...
ETCD和api-server证书过期时间证书生成
风水道人
09-19 247
ETCD和api-server证书过期时间证书生成
k8s1.19.1证书续期（etcd外设版）
weixin_50763319的博客
06-21 1107
• 工作节点：工作节点主要是指kubelet连接apiserver所需的客户端证书，这个证书由controller-manager组件自动颁发，默认是一年，如果到期，kubelet将无法使用过期的证书连接apiserver，从而导致无法正常工作，日志会_kubelet-client-2023-01-18-22-31-02.pem。Kubernetes Kubeadm Kubelet 证书自动续签_kubelet-client-2023-01-18-22-31-02.pem-CSDN博客。
kops etcd证书过期问题
qq522044637的博客
08-26 731
前言 etcd-manager 在云或裸机上管理 etcd 集群。它借鉴了 etcd-operator 的思想，但避免了依赖 kubernetes（本身依赖 etcd）的循环依赖。 当 etcd-manager 首次启动时，它会为每个主节点颁发证书。这些证书的硬编码期限为一年。如果证书过期，apiserver 将无法再访问本地 etcd 成员。 etcd-manager 首次在 Kops 1.12 中引入，该版本于 2019 年 5 月 15 日发布。这意味着，如果你升级了1.12以上的版本，并且不知
k8s踩坑(三)、kubeadm证书/etcd证书过期处理
热门推荐
ywq935的博客
03-08 2万+
故障现象 使用kubeadm部署的集群，在运行了一年之后今天，出现k8s api无法调取的现象，使用kubectl命令获取资源均返回如下报错: Unable to connect to the server: x509: certificate has expired or is not yet valid 故障排查 查看apiserver.crt证书的签署日期和过期日期： root@9027:...
kubernetes,ETCD灾备恢复，重新生成证书，重新生成配置文件
qq_22648091的博客
08-08 1821
备份原来的证书帮助信息生成新的证书。
阿里云服务器 使用Certbot申请免费 HTTPS 证书及自动续期
江湖行骗老中医
04-17 3336
Certbot是一款免费且开源的自动化安全证书管理工具，由电子前沿基金会（EFF）开发和维护，是在Linux、Apache和Nginx服务器上配置和管理SSL/TLS证书的一种机制。Certbot可以自动完成域名的认证并安装证书。
阿里云ACK技术简介及实践.pptx
10-11
阿里云ACK技术简介及实践.pptx
阿里云ACK在线教育行业解决方案.pdf
09-30
阿里云ACK在线教育行业解决方案旨在帮助在线教育平台应对新冠疫情带来的挑战，提供稳定、弹性且经济高效的云服务。该方案的核心是阿里云容器服务ACK（Alibaba Cloud Container Service for Kubernetes），它基于...
阿里云ACK在线教育行业解决方案.pptx
09-30
阿里云ACK在线教育行业解决方案.pptx
ack-etcd备份资源
01-15
在“ACK-Etcd备份资源”中提到的`ack-etcd-backup-operator`可能是一个专门为阿里云ACK设计的工具或服务，它简化了Etcd备份流程，提供了与阿里云基础设施集成的自动化备份解决方案。这个操作员可能会监控Etcd集群，...
gitlab突然无法访问，报证书过期/404
sanqianqiqi的博客
09-01 1121
首先查看hosts文件有没有修改，前几天因为项目调用第三方的时候连接不通所以修改了hosts文件，导致文件后缀变成了txt，所以尽管配置了gitlab的host也无法读取。突然之间无法访问远程gitlab网址，一开始是报错：证书过期，后来在界面上输入。运行完之后重新ping一下gitlab，看通没通即可。就变成了404界面。邀请大佬帮忙之后，分分钟解决。解决方法：打开终端，输入以下命令行。，在命令行前面加上sudo就可以。
解决kubernetes集群证书过期的问题
jiaohuizhuang6019的博客
11-08 390
解决kubernetes集群证书过期的问题
[k8s]二进制部署k8s 更换etcd证书(100年有效期)
yyq2272的博客
04-16 360
[k8s]二进制部署k8s 更换etcd证书(100年有效期)
k8s 证书过期解决
jiangbenchu的博客
11-16 9495
K8S CA证书是10年，但是组件证书的日期只有1年，为了证书一直可用状态需要更新，目前主流的一共有3种： 1、版本升级，只要升级就会让各个证书延期1年，官方设置1年有效期的目的就是希望用户在一年内能升级1次，详见：k8s升级 2、通过命令续期 （这种只能延长一年） 3、编译源码Kubeadm，设置10年 一、查看证书过期时间 vim test.sh for item in `find /etc/kubernetes/pki -maxdepth 2 -name "*.crt"`; do openssl x5
k8s证书续期及版本升级
最新发布
geriletuma
07-28 365
k8s证书续期及版本升级
K8S二进制部署之定义CA证书与ETCD
l17605229954的博客
11-01 611
1、 服务器单向认证：只需要服务器端提供证书，客户端通过服务器端证书验证服务的身份，但服务器并不验证客户端的身份。④ 客户端私钥：客户端证书中包含的公钥所对应的私钥，同理，客户端使用该私钥来向服务器端证明自己是客户端证书的拥有者。⑤ 服务器端 CA 根证书：签发服务器端证书的 CA 根证书，客户端使用该 CA 根证书来验证服务器端证书的合法性。⑥ 客户端端 CA 根证书：签发客户端证书的 CA 根证书，服务器端使用该 CA 根证书来验证客户端证书的合法性。
阿里云ack 部署oceanbase
01-31
根据提供的引用内容，以下是在阿里云ACK上部署OceanBase的步骤： 1. 在ACK集群内创建一个Java应用。 2. 在pod的spec.template.metadata字段中添加两个Pod标签，armsPilotCreateAppName和armsPilotAutoEnable。 3. 将armsPilotCreateAppName设置为要接入ARMS的应用名称，可以与Deployment名称保持一致。 4. 将armsPilotAutoEnable设置为on，以启用ARMS的自动注入和配置。 5. 可以直接编辑Deployment的Yaml文件来添加Pod标签。 请注意，这些步骤是基于提供的引用内容进行推断的，具体的部署步骤可能会因为具体的环境和需求而有所不同。建议您参考阿里云的官方文档或者咨询阿里云的技术支持来获取更详细和准确的部署指南。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

忍冬行者

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值