etcd集群扩容与故障替换

集群信息：

isLeader	主机名	主机IP
false	k8s-master01	10.211.55.3
false	k8s-master02	10.211.55.4
true	k8s-master03	10.211.55.5

一、集群扩容

1.1 申请两台和etcd集群相同配置的机器或容器k8s-master04(10.211.55.6)和k8s-master05(10.211.55.7)用于部署新节点

1.2 etcdctl 3.x以后的版本和etcd集群交互有V2和V3两种API,在V2版本中，etcd集群使用ssl的时候etcdctl后面需要显示的加上证书访问，而V3不需要，笔者自己的etcdctl版本是3.3.12的，etcd集群也是3.0以上的，所以使用etcdctl的V3版本

[root@k8s-master01 ~]# etcdctl -v
etcdctl version: 3.3.12
API version: 2

设置etcdctl使用的api版本

[root@k8s-master01 ~]# export ETCDCTL_API=3

查看集群的节点信息

[root@k8s-master01 ~]# etcdctl --write-out=table  member list
+------------------+---------+--------+--------------------------+--------------------------+
|        ID        | STATUS  |  NAME  |        PEER ADDRS        |       CLIENT ADDRS       |
+------------------+---------+--------+--------------------------+--------------------------+
| 78a30be28647bb39 | started | infra2 | https://10.211.55.4:2380 | https://10.211.55.4:2379 |
| caf3e8da9311ea76 | started | infra3 | https://10.211.55.5:2380 | https://10.211.55.5:2379 |
| fcf8f1c47e6d256c | started | infra1 | https://10.211.55.3:2380 | https://10.211.55.3:2379 |
+------------------+---------+--------+--------------------------+--------------------------+

1.3 登陆集群其中一个节点进行数据备份

[root@k8s-master01 ~]# etcdctl backup --data-dir /var/lib/etcd -backup-dir /tmp/etcd_backup

1.4 在k8s-master01上执行添加第一个新节点,添加节点用到etcdctl的member add参数，我们通过执行 etcdctl member add --help可以看到下面的输出，是我们给新节点起的名字，一般按照etcdctl --write-out=table member list的输出中NAME规律往上叠加即可，[options]就是–peer-urls=""，后面的值就是要加的节点的IP和端口

[root@k8s-master01 ~]# etcdctl member add --help
NAME:
	member add - Adds a member into the cluster

USAGE:
	etcdctl member add <memberName> [options]

OPTIONS:
      --peer-urls=""	comma separated peer URLs for the new member.

GLOBAL OPTIONS:
      --cacert=""				verify certificates of TLS-enabled secure servers using this CA bundle
      --cert=""					identify secure client using this TLS certificate file
      --command-timeout=5s			timeout for short running command (excluding dial timeout)
      --debug[=false]				enable client-side debug logging
      --dial-timeout=2s				dial timeout for client connections
  -d, --discovery-srv=""			domain name to query for SRV records describing cluster endpoints
      --endpoints=[127.0.0.1:2379]		gRPC endpoints
      --hex[=false]				print byte strings as hex encoded strings
      --insecure-discovery[=true]		accept insecure SRV records describing cluster endpoints
      --insecure-skip-tls-verify[=false]	skip server certificate verification
      --insecure-transport[=true]		disable transport security for client connections
      --keepalive-time=2s			keepalive time for client connections
      --keepalive-timeout=6s			keepalive timeout for client connections
      --key=""					identify secure client using this TLS key file
      --user=""					username[:password] for authentication (prompt if password is not supplied)
  -w, --write-out="simple"			set the output format (fields, json, protobuf, simple, table)

具体操作和结果输出如下

[root@k8s-master01 ~]# etcdctl member add infra4 --peer-urls="https://10.211.55.6:2380"
Member  29c2088642a8819 added to cluster 20d679faaae7becc

ETCD_NAME="infra4"
ETCD_INITIAL_CLUSTER="infra4=https://10.211.55.6:2380,infra2=https://10.211.55.4:2380,infra3=https://10.211.55.5:2380,infra1=https://10.211.55.3:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.211.55.6:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

添加后查看集群状态，因为新节点还未启动，状态是unstarted，注意这一步是关键，先在集群里把要加的节点加进来，然后再启动节点，启动节点前需要修改etcd配置。

[root@k8s-master01 ~]# etcdctl --write-out=table  member list
+------------------+-----------+--------+--------------------------+--------------------------+
|        ID        |  STATUS   |  NAME  |        PEER ADDRS        |       CLIENT ADDRS       |
+------------------+-----------+--------+--------------------------+--------------------------+
|  29c2088642a8819 | unstarted |        | https://10.211.55.6:2380 |                          |
| 78a30be28647bb39 |   started | infra2 | https://10.211.55.4:2380 | https://10.211.55.4:2379 |
| caf3e8da9311ea76 |   started | infra3 | https://10.211.55.5:2380 | https://10.211.55.5:2379 |
| fcf8f1c47e6d256c |   started | infra1 | https://10.211.55.3:2380 | https://10.211.55.3:2379 |
+------------------+-----------+--------+--------------------------+--------------------------+

1.5 修改etcd集群启动参数，在原有变量基础上，再添加新节点的配置信息:
–initial-cluster infra1=https://10.211.55.3:2380,infra2=https://10.211.55.4:2380,infra3=https://10.211.55.5:2380,infra4=https://10.211.55.6:2380
将启动参数中–initial-cluster-state new 改为 --initial-cluster-state existing

1.6 启动新的节点etcd实例，确认集群状态是否正常，确认新etcd节点添加到集群是否成功。

二、节点故障替换

现在假设k8s-master02 这个节点的etcd实例挂掉，并且无法恢复，下面是替换这个节点的步骤
2.1 删除老的etcd节点故障节点k8s-master02
这一步是关键，因为如果不先删除，直接添加新的节点，会造成集群中有两个不可用的节点(老故障节点，新加节点添加时未启动)，四个节点有两台不可用，集群是不允许的，一般情况下集群的自我保护机制会添加失败
2.2 删除老节点后，按照上面的集群扩容步骤添加新的节点，添加成功后，一定要通过etcdctl --write-out=table member list查看集群中的节点是三个，而且status都是started
2.3 最后是更新旧的两台旧节点，将故障节点替换成新节点，ETCD_NAME改成新加的名字，挨个重启旧节点即完成故障节点替换
2.4 通过etcdctl工具可以测试一下读写操作