部署ETCD集群
1、概述
- 大意:
etcd 是兼具一致性和高可用性的键值数据库,可以作为保存 Kubernetes 所有集群数据的后台数据库。
- 官方网址:
https://etcd.io/docs/
2、服务器配置
2.1 服务器配置信息
主机名 | cpu | 内存 | 存储 | 操作系统 |
---|
k8s01 | 4 | 4 | 100g | Ubuntu 18.04.5 LTS |
k8s02 | 4 | 4 | 100g | Ubuntu 18.04.5 LTS |
k8s03 | 4 | 4 | 100g | Ubuntu 18.04.5 LTS |
2.2 服务器参数配置
$ cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
3、准备cfssl证书生成工具
cfssl是一个开源的证书管理工具,使用json文件生成证书.
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
4、创建etcd相关目录
$ mkdir -pv /opt/kubernetes/etcd/{bin,cfg,ssl,data}
5、创建证书
5.1、创建ca证书json文件
$ cd /opt/kubernetes/etcd/ssl
$ vim ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
$ vim ca-csr.json
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
5.2、生成ca证书
$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
5.3、使用自签ca签发etcd证书
$ cd /opt/kubernetes/etcd/ssl
$ vim server-csr.json
{
"CN": "etcd",
"hosts": [
"192.168.1.241",
"192.168.1.242",
"192.168.1.243"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
$ ll |grep server
-rw-r--r-- 1 root root 1013 Sep 14 15:06 server.csr
-rw-r--r-- 1 root root 290 Sep 14 15:05 server-csr.json
-rw------- 1 root root 1679 Sep 14 15:06 server-key.pem
-rw-r--r-- 1 root root 1338 Sep 14 15:06 server.pem
6、下载etcd二进制文件
- 下载地址
https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz
7、部署ETCD集群
7.1、复制二进制文件到指定文件
$ tar xf etcd-v3.5.0-linux-amd64.tar.gz
$ cp etcd-v3.5.0-linux-amd64/{etcd,etcdctl,etcdutl} /opt/kubernetes/etcd/bin
7.2、创建etcd配置文件
$ vim /opt/kubernetes/etcd/cfg/etcd.conf
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/opt/kubernetes/etcd/data/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.241:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.241:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.241:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.241:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.1.241:2380,etcd-2=https://192.168.1.242:2380,etcd-3=https://192.168.1.243:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_NAME:节点名称,集群中唯一
ETCD_DATA_DIR:数据目录
ETCD_LISTEN_PEER_URLS:集群通信监听地址
ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEERURLS:集群通告地址
ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
ETCD_INITIAL_CLUSTER:集群节点地址
ETCD_INITIALCLUSTER_TOKEN:集群Token
ETCD_INITIALCLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集群
7.3、创建systemd文件
$ vim /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/etcd/cfg/etcd.conf
ExecStart=/opt/kubernetes/etcd/bin/etcd \
--cert-file=/opt/kubernetes/etcd/ssl/server.pem \
--key-file=/opt/kubernetes/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/kubernetes/etcd/ssl/server.pem \
--peer-key-file=/opt/kubernetes/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/kubernetes/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/kubernetes/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
8、分发etcd文件
$ scp -r /opt/kubernetes k8s02:/opt/
$ scp /etc/systemd/system/etcd.service k8s02:/etc/systemd/system/
$ scp -r /opt/kubernetes k8s03:/opt/
$ scp /etc/systemd/system/etcd.service k8s03:/etc/systemd/system/
9、分别启动etcd服务
$ systemctl start etcd.service
$ systemctl start etcd.service
$ systemctl start etcd.service
10、查看集群状态
$ cd /opt/kubernetes/etcd
$ ETCDCTL_API=3 ./bin/etcdctl --cacert=/opt/kubernetes/etcd/ssl/ca.pem --cert=/opt/kubernetes/etcd/ssl/server.pem --key=/opt/kubernetes/etcd/ssl/server-key.pem --endpoints="https://192.168.1.241:2379,https://192.168.1.242:2379,https://192.168.1.243:2379" endpoint health --write-out=table
+----------------------------+--------+-------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+----------------------------+--------+-------------+-------+
| https://192.168.1.241:2379 | true | 8.913068ms | |
| https://192.168.1.242:2379 | true | 9.757387ms | |
| https://192.168.1.243:2379 | true | 12.405075ms | |
+----------------------------+--------+-------------+-------+
11、测试
[root@etcd01 bin]
OK
[root@etcd01 bin]
foo
Hello World
12、数据备份与恢复
etcdctl --cacert=/opt/software/etcd/ssl/ca.pem --cert=/opt/software/etcd/ssl/server.pem --key=/opt/software/etcd/ssl/server-key.pem --endpoints="https://192.168.1.241:2379" snapshot save snapshot.db
etcdctl --cacert=/opt/software/etcd/ssl/ca.pem --cert=/opt/software/etcd/ssl/server.pem --key=/opt/software/etcd/ssl/server-key.pem --endpoints="https://192.168.1.241:2379" snapshot restore snapshot.db --data-dir=/root/data