【k8s】集群2--etcd

1.部署Master节点服务

主机名	角色	ip
h133.host.com	etcd lead	192.168.146.133
h134.host.com	etcd follow	192.168.146.134
h135.host.com	etcd follow	192.168.146.135

 etcd版本：3.4.16

1.1 先制作etcd用的证书

对象：h136

1.1.1 创建基于根证书的config配置文件

vim /opt/certs/ca-config.json

###解释3个段server，client，peer
###server段意味着，启动服务端需要证书
###client段意味，客户端主动找服务端通信需要证书，而服务端主动找客户端通信不需要证书
###peer段意味着客户端与服务端双向通信都需要证书
{
        "signing": {
                "default": {
                        "expiry": "175200h"
                },
                "profiles": {
                        "server": {
                                "expiry": "175200h",
                                "usages": [
                                        "signing",
                                        "key encipherment",
                                        "server auth"
                                ]
                        },
                        "client": {
                                "expiry": "175200h",
                                "usages": [
                                        "signing",
                                        "key encipherment",
                                        "client auth"
                                ]
                        },
                        "peer": {
                                "expiry": "175200h",
                                "usages": [
                                        "signing",
                                        "key encipherment",
                                        "server auth",
                                        "client auth"
                                ]
                        }
                }
        }
}

1.1.2 创建etcd证书请求文件

vim /opt/certs/etcd-peer-csr.json

{
        "CN": "k8s-etcd",
        "hosts": [
                "192.168.146.132",
                "192.168.146.133",
                "192.168.146.134",
                "192.168.146.135"
        ],
        "key": {
                "algo": "rsa",
                "size": 2048
        },
        "names": [
                {
                        "C": "CN",
                        "ST": "beijing",
                        "L": "beijing",
                        "O": "od",
                        "OU": "ops"
                }
        ]
}

1.1.3 签发证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json | cfssl-json -bare etcd-peer

最后生成下面两个文件

1.2 etcd部署

version: 3.4.16

1.2.1  创建目录、拷贝证书、私钥

对象h133,h134,h135（ h134 h135配置省略）

#创建不含家目录的etcd用户
useradd -s /sbin/nologin -M etcd

从github上下载etcd 3.4.16版本的pre-built binary文件，进行如下操作

cd /opt/src
wget https://github.com/etcd-io/etcd/releases/download/v3.4.16/etcd-v3.4.16-linux-amd64.tar.gz
tar xf etcd-v3.4.16-linux-amd64.tar.gz -C /opt/
ln -s etcd-v3.4.16-linux-amd64/ etcd

创建目录

mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server

启动etcd需要3个证书即，ca.pem（ca证书）、etcd-peer-key.pem（etcd-peer私钥）、etcd-peer.pem（etcd-peer证书）,这三个证书都在h136的/opt/certs下

cd /opt/etcd/certs
scp h136:/opt/certs/ca.pem .
scp h136:/opt/certs/etcd-peer.pem .
scp h136:/opt/certs/etcd-peer-key.pem .

创 建etcd启动脚本，

touch /opt/etcd/etcd-server-startup.sh
chmod a+x /opt/etcd/etcd-server-startup.sh
chown etcd:etcd /data/etcd/
chown etcd:etcd /data/logs/etcd/

 etcd-server-startup.sh脚本

###--name listen-peer-urls listen-client-urls initial-advertise-peer-urls
###--advertise-client-urls 参数的后接主机ip是各自部署的主机ip
###最后两个参数，一个是每6小时自动压缩元数据，一个是开启api的v2版本(这是因为后边要用的flannel只能用v2版本与etcd通信)
#! /bin/sh
./etcd --name etcd-server-h133 \
--data-dir /data/etcd/etcd-server-startup.sh \
--listen-peer-urls https://192.168.146.133:2380 \
--listen-client-urls https://192.168.146.133:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://192.168.146.133:2380 \
--advertise-client-urls https://192.168.146.133:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-h133=https://192.168.146.133:2380,etcd-server-h134=https://192.168.146.134:2380,etcd-server-h135=https://192.168.146.135:2380 \
--cert-file '/opt/etcd/certs/etcd-peer.pem' \
--key-file '/opt/etcd/certs/etcd-peer-key.pem' \
--client-cert-auth \
--trusted-ca-file /opt/etcd/certs/ca.pem \
--peer-cert-file /opt/etcd/certs/etcd-peer.pem \
--peer-key-file /opt/etcd/certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file /opt/etcd/certs/ca.pem \
--log-outputs stdout \
--auto-compaction-retention 6 \
--enable-v2=true

修改权限

chmod +x /opt/etcd/etcd-server-startup.sh
#由于/opt/etcd为软链接，因此直接修改原文件属主属组
chown -R etcd:etcd /opt/etcd-v3.4.16-linux-amd64/
chown -R etcd:etcd /data/etcd/
chown -R etcd:etcd /data/logs/etcd-server/

虽然有了启动脚本，但etcd需要运行在后端，因此需要安装supervisor

yum install -y supervisor
systemctl start supervisord
systemctl enable supervisord

创建在supervisor管理下需要启动的服务(etcd)的配置文件touch /etc/supervisord.d/etcd-server.ini

###下面这一行参数，注意对应不同主机
[program:etcd-server-h133]
command=/opt/etcd/etcd-server.sh
numprocs=1
directory=/opt/etcd
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=etcd
redirect_stderr=true
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=4
stdout_capture_maxbytes=1MB
stdout_events_enabled=false

执行下面

superviorctl update

查看etcd集群的leader

#查看etcd集群成员
./etcdctl member list
#下面命令需要在每个节点
./etcdctl endpoint status --write-out=table