kubernetes高可用集群(1)——etcd集群搭建

环境介绍：

服务器IP地址	主机名	角色
192.168.1.101	etcd-node01	etcd01/k8s_master
192.168.1.102	etcd-node02	etcd02/k8s_master
192.168.1.103	etcd-node03	etcd03/

 

一、下载二进制版本

在https://github.com/etcd-io/etcd/releases下载etcd二进制安装包:

wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz

二、安装cfssl工具

本文档使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 证书和秘钥文件，CA 是自签名的证书，用来签名后续创建的其它 TLS 证书

 

curl -s -L -o /bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 
curl -s -L -o /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 
curl -s -L -o /bin/cfssl-certinfo https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 
chmod +x /bin/cfssl*

三、安装准备

1、在/opt目录下，创建所需文件目录：

   mkdir -p /opt/etcd/{bin,cfg,ssl,data}

2、移动etcd可执行文件，将etcd相关命令加入环境变量：

   mv /opt/etcd-v3.3.10/etcd* /opt/etcd/bin/

   echo “export PATH=$PATH:/opt/etcd/bin” >> /etc/profile

   source /etc/profile

四、创建相关证书（在一个节点上操作即可）

1、进入/opt/etcd/ssl文件夹：cd /opt/etcd/ssl

2、创建CA配置文件、CA签名请求文件、etcd证书请求文件：

  ca配置文件：

cfssl print-defaults config > ca-config.json

  ca签名请求文件：

cfssl print-defaults csr > ca-csr.json

  etcd证书请求文件:

cfssl print-defaults csr > server-csr.json

3、修改这三个文件：

[root@etcd-node01 json]# cat ca-config.json

{

    "signing": {

        "default": {

            "expiry": "87600h"

        },

        "profiles": {

            "www": {

                "expiry": "87600h",

                "usages": [

                    "signing",

                    "key encipherment",

                    "server auth",

                    "client auth"

                ]

            }

        }

    }

}

 

 

[root@etcd-node01 json]# cat ca-csr.json

{

    "CN": "etcd-cluster CA",

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "L": "Shenzhen",

            "ST": "Shenzhen"

        }

    ]

}

 

   

[root@etcd-node01 json]# cat server-csr.json

{

    "CN": "etcd-cluster Server",

    "hosts": [

        "192.168.1.101",

        "192.168.1.102",

        "192.168.1.103"

    ],

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "L": "Shenzhen",

            "ST": "Shenzhen"

        }

    ]

}

 

 

 

4、生成CA证书和私钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

生成ca.pem、ca.csr、ca-key.pem(CA私钥,需妥善保管)

5、生成服务端证书和私钥

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

  生成server.csr、server-key.pem、server.pem

6、将生成的这4个*.pem证书文件拷贝到/opt/etcd/ssl目录下

mv *.pem /opt/etcd/ssl/

 

五、修改启动etcd配置文件

1、在/opt/etcd/cfg/目录下创建etcd配置文件etcd.conf：

[root@etcd-node01 etcd]# cat cfg/etcd.conf |grep -v "^#"

ETCD_DATA_DIR="/opt/etcd/data"

ETCD_LISTEN_PEER_URLS="https://192.168.1.101:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.1.101:2379,http://127.0.0.1:2379"

ETCD_NAME="etcd01"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.101:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.101:2379"

ETCD_INITIAL_CLUSTER="etcd01=https://192.168.1.101:2380,etcd02=https://192.168.1.102:2380,etcd03=https://192.168.1.103:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

 

2、systemd管理etcd启动脚本：

[root@etcd-node01 etcd]# cat /usr/lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

Documentation=https://github.com/coreos

 

[Service]

Type=notify

EnvironmentFile=/opt/etcd/cfg/etcd.conf

ExecStart=/opt/etcd/bin/etcd \

  --name ${ETCD_NAME} \

  --cert-file=/opt/etcd/ssl/server.pem \

  --key-file=/opt/etcd/ssl/server-key.pem \

  --peer-cert-file=/opt/etcd/ssl/server.pem \

  --peer-key-file=/opt/etcd/ssl/server-key.pem \

  --trusted-ca-file=/opt/etcd/ssl/ca.pem \

  --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \

  --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \

  --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \

  --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \

  --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \

  --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \

  --initial-cluster=${ETCD_INITIAL_CLUSTER} \

  --initial-cluster-state=new \

  --data-dir=${ETCD_DATA_DIR}

Restart=on-failure

RestartSec=3

LimitNOFILE=65536

 

[Install]

WantedBy=multi-user.target

 

3、将/opt/etcd/ssl下的证书、/opt/etcd/cfg下的配置文件、以及system管理文件etcd.service拷贝到另外两个节点对应目录下

 

4、在各节点启动etcd服务并设置开机启动

systemctl start etcd

systemctl enable etcd

 

5、查看etcd集群状态

 [root@etcd-node01 ssl]# etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem cluster-health

member 540fa0a1b3b843f3 is healthy: got healthy result from https://192.168.1.101:2379
member 95450cd9b0f8b300 is healthy: got healthy result from https://192.168.1.103:2379
member e16c23384aeb3678 is healthy: got healthy result from https://192.168.1.102:2379
cluster is healthy