【运维】K8S集群部署系列之ETCD集群搭建（四）

ETCD集群扩容和缩容

本文将介绍生产环境下如何对ETCD集群进行扩容和缩容。

新节点环境准备（node3）

下载安装包并初始化环境

mkdir /home/k8s
cd /home/k8s
wget https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz
tar -zxvf etcd-v3.3.13-linux-amd64.tar.gz
mv etcd-v3.3.13-linux-amd64 etcd
chmod -R +x etcd/
cp -f ./{etcd,etcdctl} /usr/bin/
cp -f ./{etcd,etcdctl} /usr/local/bin/
mkdir -p /opt/etcd/{etc,data,pki}

网络准备

cat >>/etc/hosts<< EOF
192.168.159.3 master
192.168.159.4 node1
192.168.159.5 node2
192.168.159.6 node3
EOF

# 防火墙设置，开放2379和2380端口,如果启动防火墙但未放开端口则集群状态为“degraded”，
# 开启防火墙的节点状态为“are all unreachable”
# 2379端口提供给客户端访问集群，客户端如：etcdctl
# 2380端口提供给集群节点间通信
systemctl start firewalld
firewall-cmd --zone=public --add-port=2379/tcp --permanent
firewall-cmd --zone=public --add-port=2380/tcp --permanent
firewall-cmd --reload
firewall-cmd --list-all

生成node3对等证书

由于已有集群开启了服务端内部的TLS安全认证，因此需要在nodes准备好证书，
但旧的证书中hosts列表并不包含node3节点的主机IP，因此需要重新生成node3节点的peer证书。

cat > /home/k8s/cfssl/ssl/etcd4-peer-csr.json << EOF 
{
    "CN": "etcd4-peer",
    "hosts": [
        "192.168.159.6"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "ChengDu",
            "O": "JSQ",
            "OU": "k8s",
            "ST": "SiChuan"
        }
    ]
}
EOF

cfssl gencert --ca=ca.pem --ca-key=ca-key.pem --config=ca-config.json --profile=peer etcd4-peer-csr.json  | cfssljson -bare etcd4-peer
scp etcd4-* root@192.168.159.6:/opt/etcd/pki/

注意：此处hosts列表中只包含了node3节点的主机IP，实际上为了更加方便的进行动态的扩容和缩容操作，对等证书和服务器证书的hosts列表最好只包含本机的IP地址。

向集群中添加普通节点node3

添加新节点

命令调用方式：etcdctl member add <memberName> <peerURLS>；
注意此处添加的节点名和链接应该与新增节点的配置一致；
在已有集群的任意节点（最好为集群主节点，即isLeader=true的节点）执行如下命令。

etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member add etcd-4 http://192.168.159.6:2380

查看集群状态

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member list
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false
e1b7f9d6e4ff0f36[unstarted]: peerURLs=https://192.168.159.6:2380

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-health
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member 6bdd9302771bc9c5 is healthy: got healthy result from https://192.168.159.5:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
member e1b7f9d6e4ff0f36 is unreachable: no available published client urls
cluster is healthy

添加node3节点配置

此处ETCD_INITIAL_CLUSTER_STATE必须为existing，表示向已有集群新增节点；
由于已有集群已开启TLS安全验证，因此必须配置相关证书，如果是普通集群则无须配置；
ETCD_PEER_CLIENT_CERT_AUTH=false表示集群内部访问该节点服务端无须进行TLS验证。

mkdir -p /opt/etcd/{data,etc}
cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-4"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.6:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.159.6:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.6:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.159.6:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380,etcd-4=http://192.168.159.6:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="existing"

#[Security]
ETCD_PEER_CERT_FILE="/opt/etcd/pki/etcd4-peer.pem"    
ETCD_PEER_KEY_FILE="/opt/etcd/pki/etcd4-peer-key.pem"   
ETCD_PEER_CLIENT_CERT_AUTH="false"
ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem" 
EOF

添加node3节点服务文件

cat >  /usr/lib/systemd/system/etcd.service << EOF    
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--peer-cert-file=${ETCD_PEER_CERT_FILE} \
--peer-key-file=${ETCD_PEER_KEY_FILE} \
--peer-client-cert-auth=${ETCD_PEER_CLIENT_CERT_AUTH} \
--peer-trusted-ca-file=${ETCD_PEER_TRUSTED_CA_FILE}
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

启动node3节点

systemctl daemon-reload && systemctl start etcd

再次查看集群状态

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member list
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false
e1b7f9d6e4ff0f36: name=etcd-4 peerURLs=http://192.168.159.6:2380 clientURLs=http://192.168.159.6:2379 isLeader=false

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-health
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member 6bdd9302771bc9c5 is healthy: got healthy result from https://192.168.159.5:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
member e1b7f9d6e4ff0f36 is healthy: got healthy result from http://192.168.159.6:2379
cluster is healthy

至此完成向集群添加一个普通节点的操作，下一步继续进行节点的移除操作。

将node3节点从集群中移除

集群节点查看

通过查看集群节点确定需要移除节点的ID。

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member list
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false
e1b7f9d6e4ff0f36: name=etcd-4 peerURLs=http://192.168.159.6:2380 clientURLs=http://192.168.159.6:2379 isLeader=false

移除节点

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member remove e1b7f9d6e4ff0f36
Removed member e1b7f9d6e4ff0f36 from cluster

查看集群状态

可以看到节点etcd-4已经从集群移除。

[root@master pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member list
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false

[root@master pki]# etcdctl -ca-file=ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem cluster-health
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member 6bdd9302771bc9c5 is healthy: got healthy result from https://192.168.159.5:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
cluster is healthy

查看node3节点的服务状态

可以看到服务已被节点a3ec213779ea2c81通知停止通信。

[root@localhost ~]# systemctl status etcd
● etcd.service
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; bad; vendor preset: disabled)
   Active: inactive (dead)

8月 13 16:36:22 localhost.localdomain etcd[1344]: failed to dial a3ec213779ea2c81 on stream MsgApp v2 (context canceled)
8月 13 16:36:22 localhost.localdomain etcd[1344]: peer a3ec213779ea2c81 became inactive (message send to peer failed)
8月 13 16:36:22 localhost.localdomain etcd[1344]: stopped streaming with peer a3ec213779ea2c81 (stream MsgApp v2 reader)
8月 13 16:36:22 localhost.localdomain etcd[1344]: stopped streaming with peer a3ec213779ea2c81 (stream Message reader)
8月 13 16:36:22 localhost.localdomain etcd[1344]: stopped peer a3ec213779ea2c81
8月 13 16:39:07 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/etcd.service:1] Assignment outside of section. Ignoring.
8月 13 16:39:07 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/etcd.service:2] Assignment outside of section. Ignoring.
8月 13 16:39:07 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/etcd.service:3] Assignment outside of section. Ignoring.
8月 13 16:39:07 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/etcd.service:4] Assignment outside of section. Ignoring.
8月 13 16:39:07 localhost.localdomain systemd[1]: [/usr/lib/systemd/system/etcd.service:5] Assignment outside of section. Ignoring.

向集群中添加安全节点node3

生成node3节点的服务端证书

由于集群已开启服务端的TLS认证，因此node3节点需要有相应的服务器证书；
但旧的服务端证书中hosts列表并不包含node3节点的主机IP，因此需要重新生成node3节点的server证书

cat > etcd4-csr.json << EOF 
{
    "CN": "etcd4",
    "hosts": [
        "192.168.159.6"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "ChengDu",
            "O": "JSQ",
            "OU": "k8s",
            "ST": "SiChuan"
        }
    ]
}
EOF

cfssl gencert --ca=ca.pem --ca-key=ca-key.pem --config=ca-config.json --profile=server etcd4-csr.json  | cfssljson -bare etcd4
scp etcd4* root@192.168.159.6:/opt/etcd/pki/

[root@localhost pki]# ls etcd4*
etcd4.csr  etcd4-csr.json  etcd4-key.pem  etcd4-peer.csr  etcd4-peer-csr.json  etcd4-peer-key.pem  etcd4-peer.pem  etcd4.pem

集群的客户端证书由于没有hosts主机列表，因此可以通用。

从普通节点升级为TLS认证的安全节点

参照【运维】K8S集群部署系列之ETCD集群搭建（三）

直接将node3添加为TLS认证的安全节点

以https方式添加node3节点

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member add etcd-4 https://192.168.159.6:2380
Added member named etcd-4 with ID 1e7da56305348d0d to cluster

ETCD_NAME="etcd-4"
ETCD_INITIAL_CLUSTER="etcd-4=https://192.168.159.6:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380,etcd-1=https://192.168.159.3:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

查看集群状态

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member list
1e7da56305348d0d[unstarted]: peerURLs=https://192.168.159.6:2380
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-health
member 1e7da56305348d0d is unreachable: no available published client urls
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member 6bdd9302771bc9c5 is healthy: got healthy result from https://192.168.159.5:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
cluster is degraded

修改node3节点配置

此处修改字段ETCD_LISTEN_PEER_URLS、ETCD_LISTEN_CLIENT_URLS、ETCD_INITIAL_ADVERTISE_PEER_URLS、ETCD_ADVERTISE_CLIENT_URLS、ETCD_INITIAL_CLUSTER、ETCD_PEER_CLIENT_CERT_AUTH；
新增字段ETCD_CERT_FILE、ETCD_KEY_FILE、ETCD_CLIENT_CERT_AUTH、ETCD_TRUSTED_CA_FILE；
注意字段ETCD_INITIAL_CLUSTER_STATE的值必须为existing。

cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-4"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.159.6:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.6:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.159.6:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.6:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380,etcd-4=https://192.168.159.6:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="existing"

#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd4.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd4-key.pem"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"

ETCD_PEER_CERT_FILE="/opt/etcd/pki/etcd4-peer.pem"    
ETCD_PEER_KEY_FILE="/opt/etcd/pki/etcd4-peer-key.pem"   
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem" 
EOF

安全启动node3的服务文件

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE} \
--client-cert-auth=${ETCD_CLIENT_CERT_AUTH} \
--trusted-ca-file=${ETCD_TRUSTED_CA_FILE} \
--peer-cert-file=${ETCD_PEER_CERT_FILE} \
--peer-key-file=${ETCD_PEER_KEY_FILE} \
--peer-client-cert-auth=${ETCD_PEER_CLIENT_CERT_AUTH} \
--peer-trusted-ca-file=${ETCD_PEER_TRUSTED_CA_FILE}
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

删除node3节点的旧数据

这一步很重要，否则node3节点无法正确启动。

rm -rf /opt/etcd/data/*

启动node3节点

systemctl daemon-reload && systemctl start etcd

再次查看集群状态

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem member list
1e7da56305348d0d: name=etcd-4 peerURLs=https://192.168.159.6:2380 clientURLs=https://192.168.159.6:2379 isLeader=false
46899d42c87d524e: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=true
6bdd9302771bc9c5: name=etcd-3 peerURLs=https://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
a3ec213779ea2c81: name=etcd-1 peerURLs=https://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=false

[root@master pki]# etcdctl --ca-file=ca.pem  --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-health
member 1e7da56305348d0d is healthy: got healthy result from https://192.168.159.6:2379
member 46899d42c87d524e is healthy: got healthy result from https://192.168.159.4:2379
member 6bdd9302771bc9c5 is healthy: got healthy result from https://192.168.159.5:2379
member a3ec213779ea2c81 is healthy: got healthy result from https://192.168.159.3:2379
cluster is healthy

至此ETCD集群搭建及其动态扩容和缩容介绍完毕，下一篇我们将对etcdctl的基本操作进行简单介绍。