ETCD集群部署

K8S基础环境四联：
（一）ETCD集群部署
（二）安装配置 Flannel Docker
（三）手工部署kubernetes-1.17.0
（四）K8S之HelloWorld

---

ETCD集群部署

附件

/opt/soft/etcd/etcd-v3.4.4-linux-amd64.tar.gz
下载地址：https://github.com/etcd-io/etcd/releases

服务器

192.168.1.54、192.168.1.65、192.168.1.105

安装

1、解压包(每台机器)

ETCD_VER=v3.4.4
cd /opt/soft/etcd
tar xzvf etcd-${ETCD_VER}-linux-amd64.tar.gz --strip-components=1
rm -f etcd-${ETCD_VER}-linux-amd64.tar.gz
./etcd --version
./etcdctl version

2、创建etcd配置文件(每台机器)
/opt/soft/etcd/etcd.conf
下面配置文件中的IP地址分别修改为本机IP地址，ETCD_NAME分别为etcd01、etcd02、etcd03

#[Member]
#ETCD_CORS=""
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.65:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.65:2379,https://127.0.0.1:2379"
#ETCD_WAL_DIR=""
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#ETCD_MAX_REQUEST_BYTES="1572864"
#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"
#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"
#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"
#
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.65:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.65:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.1.54:2380,etcd02=https://192.168.1.65:2380,etcd03=https://192.168.1.105:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
#ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"

3、创建etcd.service配置文件(每台机器)
/usr/lib/systemd/system/etcd.service
这个是systemd中etcd的启动配置文件，配置完之后就可以用systemd启停etc服务了

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/opt/soft/etcd/
EnvironmentFile=/opt/soft/etcd/etcd.conf
ExecStart=/opt/soft/etcd/etcd \
--initial-cluster-state=new \
--cert-file=/opt/soft/etcd/ssl/server.pem \
--key-file=/opt/soft/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/soft/etcd/ssl/server.pem \
--peer-key-file=/opt/soft/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/soft/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/soft/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

4、创建TLS证书
找一个有网络的机器生成证书，否则你就手工下载相关文件

mkdir -p /opt/soft/etcd/ssl && cd $_
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssl-certinfo /usr/local/bin/cfssljson

tls.sh 文件内容如下全部内容(先修改其中的IP地址,其中hosts尽可能多加)

# etcd
# cat ca-config.json
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

# cat ca-csr.json
cat > ca-csr.json <<EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

# cat server-csr.json
cat > server-csr.json <<EOF
{
    "CN": "etcd",
    "hosts": [
    "127.0.0.1",
    "192.168.1.65",
    "192.168.1.54",
    "192.168.1.105"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

执行如下命令

sh tls.sh

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

ls *.pem

然后将生成的4个pem文件证书复制到各个机器的/opt/soft/etcd/ssl目录中

5、启动服务

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

6、查看集群状态
集群状态主要是etcdctl endpoint status和etcdctl endpoint health两条命令

cd /opt/soft/etcd/ && ./etcdctl \
--endpoints="https://192.168.1.54:2379,https://192.168.1.65:2379,https://192.168.1.105:2379" \
--cacert=ssl/ca.pem \
--key=ssl/server-key.pem  \
--cert=ssl/server.pem  \
endpoint health 

输出内容is healthy(健康)

https://192.168.1.105:2379 is healthy: successfully committed proposal: took = 29.874089ms
https://192.168.1.65:2379 is healthy: successfully committed proposal: took = 29.799246ms
https://192.168.1.54:2379 is healthy: successfully committed proposal: took = 39.710904ms

测试(正好配置一个flanneld的网络信息)
因为flanneld(目前最新版v0.11.0)只支持V2版本命令，所以请使用下面的第二段V2命令操作

# V2命令(set设值,ls查看)
ETCDCTL_API=2 etcdctl \
--endpoints="https://192.168.1.54:2379,https://192.168.1.65:2379,https://192.168.1.105:2379"  \
--ca-file=ssl/ca.pem \
--key-file=ssl/server-key.pem  \
--cert-file=ssl/server.pem \
set /flannel/network/config '{"Network":"10.244.0.0/16", "SubnetMin": "10.244.1.0", "SubnetMax": "10.244.254.0", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'

ETCDCTL_API=2 etcdctl \
--endpoints="https://192.168.1.54:2379,https://192.168.1.65:2379,https://192.168.1.105:2379"  \
--ca-file=ssl/ca.pem \
--key-file=ssl/server-key.pem  \
--cert-file=ssl/server.pem \
ls /flannel/network

# 输出如下内容表示正常
/flannel/network/config
/flannel/network/subnets

---

etcd 3.4注意事项

ETCD3.4版本ETCDCTL_API=3 etcdctl 和 etcd --enable-v2=false 成为了默认配置，如要使用v2版本，执行etcdctl时候需要设置ETCDCTL_API环境变量，例如：ETCDCTL_API=2 etcdctl
ETCD3.4版本会自动读取环境变量的参数，所以EnvironmentFile文件中有的参数，不需要再次在ExecStart启动参数中添加，二选一，如同时配置，会触发以下类似报错“etcd: conflicting environment variable “ETCD_NAME” is shadowed by corresponding command-line flag (either unset environment variable or disable flag)”
flannel操作etcd使用的是v2的API，而kubernetes操作etcd使用的v3的API
注意：flannel操作etcd使用的是v2的API，而kubernetes操作etcd使用的v3的API，为了兼容flannel，将默认开启v2版本，故需要配置文件/opt/soft/etcd/etcd.conf中设置 ETCD_ENABLE_V2=“true”
在配合K8S使用的时候，写入的Network网段必须是 /16 段地址，必须与K8S的kube-controller-manager、kube-proxy的cluster-cidr参数值一致（原则上应该反过来说）

另外，flanneld 配置文件中要将集群的地址和证书都配置上，如下示例：

FLANNEL_ETCD_ENDPOINTS="https://192.168.1.54:2379,https://192.168.1.65:2379,https://192.168.1.105:2379"
FLANNEL_ETCD_PREFIX="/flannel/network"
FLANNEL_OPTIONS="-etcd-cafile=/opt/soft/etcd/ssl/ca.pem -etcd-keyfile=/opt/soft/etcd/ssl/server-key.pem  -etcd-certfile=/opt/soft/etcd/ssl/server.pem"

还有flanneld.service的片段：

ExecStart=/opt/soft/flannel/flanneld -ip-masq -etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS} $FLANNEL_OPTIONS -etcd-prefix=${FLANNEL_ETCD_PREFIX}

---

（END）