etcd 3.1 高可用集群搭建

环境信息
CentOS 7.3

192.168.61.11 node1
192.168.61.12 node2
192.168.61.13 node3
TLS密钥和证书
这里部署的etcd集群使用TLS证书对证书通信进行加密，并开启基于CA根证书签名的双向数字证书认证。

下面介绍使用cfssl生成所需要的私钥和证书.

安装cfssl
cfssl是使用Go语言开发的工具，如果系统中安装了Go，可以使用直接go get安装cfssl:

1
go get -u github.com/cloudflare/cfssl/cmd/…
会在$GOPATH/bin下安装cfssl, cfssjosn, mkbundle等工具。

CA证书和私钥
创建ca-config.json：

{
“signing”: {
“default”: {
“expiry”: “87600h”
},
“profiles”: {
“frognew”: {
“usages”: [
“signing”,
“key encipherment”,
“server auth”,
“client auth”
],
“expiry”: “87600h”
}
}
}
}
ca-config.json中可以定义多个profile，分别设置不同的expiry和usages等参数。如上面的ca-config.json中定义了名称为frognew的profile，这个profile的expiry 87600h为10年，useages中：

signing表示此CA证书可以用于签名其他证书，ca.pem中的CA=TRUE
server auth表示TLS Server Authentication
client auth表示TLS Client Authentication
创建CA证书签名请求配置ca-csr.json：

{
“CN”: “frognew”,
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“ST”: “BeiJing”,
“L”: “BeiJing”,
“O”: “frognew”,
“OU”: “cloudnative”
}
]
}
下面使用cfss生成CA证书和私钥：

1
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
1
2
ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
ca-key.pem和ca.pem需要保存在一个安全的地方，后边会用到。

etcd证书和私钥
创建etcd证书签名请求配置etcd-csr.json：

{
“CN”: “frognew”,
“hosts”: [
“127.0.0.1”,
“192.168.61.11”,
“192.168.61.12”,
“192.168.61.13”,
“node1”,
“node2”,
“node3”
],
“key”: {
“algo”: “rsa”,
“size”: 2048
},
“names”: [
{
“C”: “CN”,
“ST”: “BeiJing”,
“L”: “BeiJing”,
“O”: “frognew”,
“OU”: “cloudnative”
}
]
}
注意上面配置hosts字段中制定授权使用该证书的IP和域名列表，因为现在要生成的证书需要被etcd集群各个节点使用，所以这里指定了各个节点的IP和hostname。

下面生成etcd的证书和私钥：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=frognew etcd-csr.json | cfssljson -bare etcd

ls etcd*
etcd.csr etcd-csr.json etcd-key.pem etcd.pem
对生成的证书可以使用cfssl或openssl查看：

cfssl-certinfo -cert etcd.pem

openssl x509 -noout -text -in etcd.pem
安装etcd
将CA证书ca.pem, etcd秘钥etcd-key.pem, etcd证书etcd.pem拷贝到各节点的/etc/etcd/ssl目录中。

下载etcd二进制文件包：

wget https://github.com/coreos/etcd/releases/download/v3.1.6/etcd-v3.1.6-linux-amd64.tar.gz
解压缩etcd-v3.1.6-linux-amd64.tar.gz，将其中的etcd和etcdctl两个可执行文件复制到各节点的/usr/bin目录。

在各节点创建etcd的数据目录：

mkdir -p /var/lib/etcd
在每个节点上创建etcd的systemd unit文件/usr/lib/systemd/system/etcd.service，注意替换ETCD_NAME和INTERNAL_IP变量的值：

export ETCD_NAME=node1
export INTERNAL_IP=192.168.61.11
cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd
–name KaTeX parse error: Expected 'EOF', got '\ ' at position 13: {ETCD_NAME} \̲ ̲ --cert-file=/…{INTERNAL_IP}:2380
–listen-peer-urls https://KaTeX parse error: Expected 'EOF', got '\ ' at position 20: …ERNAL_IP}:2380 \̲ ̲ --listen-clie…{INTERNAL_IP}:2379,https://127.0.0.1:2379
–advertise-client-urls https://${INTERNAL_IP}:2379
–initial-cluster-token etcd-cluster-1
–initial-cluster node1=https://192.168.61.11:2380,node2=https://192.168.61.12:2380,node3=https://192.168.61.13:2380
–initial-cluster-state new
–data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
上面在启动参数中指定了etcd的工作目录和数据目录是/var/lib/etcd
–cert-file和–key-file分别指定etcd的公钥证书和私钥
–peer-cert-file和–peer-key-file分别指定了etcd的Peers通信的公钥证书和私钥。
–trusted-ca-file指定了客户端的CA证书
–peer-trusted-ca-file指定了Peers的CA证书
–initial-cluster-state new表示这是新初始化集群，–name指定的参数值必须在–initial-cluster中
启动etcd
在各节点上启动etcd：

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd
检查集群是否健康，在任一节点执行：

etcdctl
–ca-file=/etc/etcd/ssl/ca.pem
–cert-file=/etc/etcd/ssl/etcd.pem
–key-file=/etc/etcd/ssl/etcd-key.pem
–endpoints=https://node1:2379,https://node2:2379,https://node3:2379
cluster-health

2017-04-24 19:53:40.545148 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-04-24 19:53:40.546127 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 4f2f99d70000fc19 is healthy: got healthy result from https://192.168.61.12:2379
member 99a756f799eb4163 is healthy: got healthy result from https://192.168.61.11:2379
member a9aff19397de2e4e is healthy: got healthy result from https://192.168.61.13:2379
cluster is healthy
确保输出cluster is healthy的信息