k8s集群之etcd

最新推荐文章于 2024-05-03 09:41:03 发布

钟明家

最新推荐文章于 2024-05-03 09:41:03 发布

阅读量588

点赞数

本文链接：https://blog.csdn.net/weixin_36888575/article/details/98639171

版权

在正式环境我是三台主机做的集群，这里我就只能用一台虚拟机三个进程分别起etcd实际上原理是一样的。另外基础环境比如dns服务器（参考我前面dns文章）这里略过，做了正向、反向dns。

cfssl version
Version: 1.2.0
Revision: dev
Runtime: go1.6
etcd version：
etcdctl version: 3.3.13
API version: 2

1、安装 CFSSL

下载cfssl：
https://pkg.cfssl.org/R1.2/SHA256SUMS
https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

安装 CFSSL
chmod +x cfssl*
mv cfssl_linux-amd64 /usr/local/bin/cfssl -v
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson -v

2、准备etcd使用的证书

2、准备etcd使用的证书
mkdir /etc/k8s/ssl/ -pv
mkdir -pv /tmp/certs && cd /tmp/certs

ca配置文件：
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF

ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile； 
signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE； 
server auth：表示 client 可以用该 CA 对 server 提供的证书进行验证； 
client auth：表示 server 可以用该 CA 对 client 提供的证书进行验证；

创建CA证书签名请求
cat > /tmp/certs/ca-csr.json <<EOF
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "k8s",
      "OU": "k8s Security",
      "L": "ChengDU",
      "ST": "SiChuan",
      "C": "CN"
    }
  ],
  "CN": "kubernetes"
}
EOF
# 生成 CA 证书和私钥：
cfssl gencert --initca=true /tmp/certs/ca-csr.json | cfssljson --bare /tmp/certs/ca

CN：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)，浏览器使用该字段验证网站是否合法；
O：Organization，kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；
kube-apiserver 将提取的 User、Group 作为 RBAC 授权的用户标识；

# verify
openssl x509 -in /tmp/certs/ca.pem -text -noout


# 创建生成证书配置文件
cat > /tmp/certs/gencert.json <<EOF

{
  "signing": {
    "default": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "87600h"
    },
  "kubernetes": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "175200h"
    }
  }
}

EOF

cat > /tmp/certs/etcd-csr.json <<EOF
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "k8s",
      "OU": "k8s Security",
      "L": "ChengDU",
      "ST": "SiChuan",
      "C": "CN"
    }
  ],
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "localhost",
    "10.0.0.11",
    "etcd",
    "etcd1",
    "etcd2",
    "etcd3",
    "etcd.zmjcd.cc",
    "etcd1.zmjcd.cc",
    "etcd2.zmjcd.cc",
    "etcd3.zmjcd.cc"
  ]
}

EOF

hosts 字段指定授权使用该证书的 etcd 节点 IP 或域名列表，这里将 etcd 集群的三个节点 IP 都列在其中;

生成 CA 证书和私钥：
cfssl gencert \
  --ca /tmp/certs/ca.pem \
  --ca-key /tmp/certs/ca-key.pem \
  --config /tmp/certs/gencert.json \
  /tmp/certs/etcd-csr.json | cfssljson --bare /tmp/certs/etcd

# verify
openssl x509 -in /tmp/certs/etcd.pem -text -noout


将*.pem证书分发到3台etcd的/etc/kubernetes/cert目录下
scp ./* root@etcd3:/etc/k8s/ssl/
scp ./* root@etcd2:/etc/k8s/ssl/
scp ./* root@etcd1:/etc/k8s/ssl/

3、安装etcd集群

我用一台服务跑三个etcd进程来模拟三台etcd服务器。
下载etcd：
https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz

在三个节点都安装etcd，下面的操作需要再三个节点都执行一遍
tar -xf etcd-v3.3.13-linux-amd64.tar.gz -C /usr/local/ && cd /usr/local/etcd-v3.3.13-linux-amd64/
ln -sv /usr/local/etcd-v3.3.13-linux-amd64/etcd /usr/local/bin/
ln -sv /usr/local/etcd-v3.3.13-linux-amd64/etcdctl /usr/local/bin/


mkdir -pv /var/lib/etcd{1,2,3}

配置etcd：
cat > /tmp/etcd1.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd


[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
WorkingDirectory=/var/lib/etcd1/

ExecStart=/usr/local/bin/etcd \
--name etcd1   \
--data-dir /var/lib/etcd1   \
--listen-client-urls https://10.0.0.11:2379   \
--advertise-client-urls https://10.0.0.11:2379   \
--listen-peer-urls https://10.0.0.11:2380   \
--initial-advertise-peer-urls https://10.0.0.11:2380   \
--initial-cluster etcd1=https://10.0.0.11:2380,etcd2=https://10.0.0.11:22380,etcd3=https://10.0.0.11:32380   \
--initial-cluster-token tkn   \
--initial-cluster-state new   \
--client-cert-auth   \
--trusted-ca-file /etc/k8s/ssl/ca.pem   \
--cert-file /etc/k8s/ssl/etcd.pem   \
--key-file /etc/k8s/ssl/etcd-key.pem   \
--peer-client-cert-auth   \
--peer-trusted-ca-file /etc/k8s/ssl/ca.pem   \
--peer-cert-file /etc/k8s/ssl/etcd.pem   \
--peer-key-file /etc/k8s/ssl/etcd-key.pem

[Install]
WantedBy=multi-user.target

EOF
mv /tmp/etcd1.service /etc/systemd/system/etcd1.service


cat > /tmp/etcd2.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd


[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
WorkingDirectory=/var/lib/etcd2/

ExecStart=/usr/local/bin/etcd \
--name etcd2   \
--data-dir /var/lib/etcd2   \
--listen-client-urls https://10.0.0.11:22379   \
--advertise-client-urls https://10.0.0.11:22379   \
--listen-peer-urls https://10.0.0.11:22380   \
--initial-advertise-peer-urls https://10.0.0.11:22380   \
--initial-cluster etcd1=https://10.0.0.11:2380,etcd2=https://10.0.0.11:22380,etcd3=https://10.0.0.11:32380   \
--initial-cluster-token tkn   \
--initial-cluster-state new   \
--client-cert-auth   \
--trusted-ca-file /etc/k8s/ssl/ca.pem   \
--cert-file /etc/k8s/ssl/etcd.pem   \
--key-file /etc/k8s/ssl/etcd-key.pem   \
--peer-client-cert-auth   \
--peer-trusted-ca-file /etc/k8s/ssl/ca.pem   \
--peer-cert-file /etc/k8s/ssl/etcd.pem   \
--peer-key-file /etc/k8s/ssl/etcd-key.pem

[Install]
WantedBy=multi-user.target

EOF
mv /tmp/etcd2.service /etc/systemd/system/etcd2.service

cat > /tmp/etcd3.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd


[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
WorkingDirectory=/var/lib/etcd2/

ExecStart=/usr/local/bin/etcd \
--name etcd3   \
--data-dir /var/lib/etcd3   \
--listen-client-urls https://10.0.0.11:32379   \
--advertise-client-urls https://10.0.0.11:32379   \
--listen-peer-urls https://10.0.0.11:32380   \
--initial-advertise-peer-urls https://10.0.0.11:32380   \
--initial-cluster etcd1=https://10.0.0.11:2380,etcd2=https://10.0.0.11:22380,etcd3=https://10.0.0.11:32380   \
--initial-cluster-token tkn   \
--initial-cluster-state new   \
--client-cert-auth   \
--trusted-ca-file /etc/k8s/ssl/ca.pem   \
--cert-file /etc/k8s/ssl/etcd.pem   \
--key-file /etc/k8s/ssl/etcd-key.pem   \
--peer-client-cert-auth   \
--peer-trusted-ca-file /etc/k8s/ssl/ca.pem   \
--peer-cert-file /etc/k8s/ssl/etcd.pem   \
--peer-key-file /etc/k8s/ssl/etcd-key.pem

[Install]
WantedBy=multi-user.target

EOF
mv /tmp/etcd3.service /etc/systemd/system/etcd3.service


# to start service
sudo systemctl daemon-reload
sudo systemctl cat etcd1.service
sudo systemctl enable etcd1.service
sudo systemctl start etcd1.service

sudo systemctl cat etcd2.service
sudo systemctl enable etcd2.service
sudo systemctl start etcd2.service

sudo systemctl cat etcd3.service
sudo systemctl enable etcd3.service
sudo systemctl start etcd3.service

4、验证etcd集群

查看etcd2状态：
 ETCDCTL_API=3 /usr/local/bin/etcdctl  \
--endpoints 10.0.0.11:2379,10.0.0.11:22379,10.0.0.11:32379  \
--cacert /etc/k8s/ssl/ca.pem   \
--cert /etc/k8s/ssl/etcd.pem   \
--key /etc/k8s/ssl/etcd-key.pem   \
endpoint health

10.0.0.11:22379 is healthy: successfully committed proposal: took = 4.000979ms
10.0.0.11:32379 is healthy: successfully committed proposal: took = 4.907629ms
10.0.0.11:2379 is healthy: successfully committed proposal: took = 11.089059ms

查看那台为leader：
etcdctl  --cert-file=/etc/k8s/ssl/etcd.pem \
--key-file=/etc/k8s/ssl/etcd-key.pem \
--ca-file=/etc/k8s/ssl/ca.pem \
--endpoints https://10.0.0.11:2379,https://10.0.0.11:22379,https://10.0.0.11:32379 \
member list

6246c0e15cee547e: name=etcd2 peerURLs=https://10.0.0.11:22380 clientURLs=https://10.0.0.11:22379 isLeader=false
799f3a7a5ffa02a6: name=etcd1 peerURLs=https://10.0.0.11:2380 clientURLs=https://10.0.0.11:2379 isLeader=true
7d932a4207ff1146: name=etcd3 peerURLs=https://10.0.0.11:32380 clientURLs=https://10.0.0.11:32379 isLeader=false


关闭etcd2：
systemctl stop etcd1

etcdctl  --cert-file=/etc/k8s/ssl/etcd.pem \
--key-file=/etc/k8s/ssl/etcd-key.pem \
--ca-file=/etc/k8s/ssl/ca.pem \
--endpoints https://10.0.0.11:2379,https://10.0.0.11:22379,https://10.0.0.11:32379 \
cluster-health

member 6246c0e15cee547e is healthy: got healthy result from https://10.0.0.11:22379
failed to check the health of member 799f3a7a5ffa02a6 on https://10.0.0.11:2379: Get https://10.0.0.11:2379/health: dial tcp 10.0.0.11:2379: connect: connection refused
member 799f3a7a5ffa02a6 is unreachable: [https://10.0.0.11:2379] are all unreachable
member 7d932a4207ff1146 is healthy: got healthy result from https://10.0.0.11:32379
cluster is degraded


启动etcd1：
systemctl start etcd1

etcdctl  --cert-file=/etc/k8s/ssl/etcd.pem \
--key-file=/etc/k8s/ssl/etcd-key.pem \
--ca-file=/etc/k8s/ssl/ca.pem \
--endpoints https://10.0.0.11:2379,https://10.0.0.11:22379,https://10.0.0.11:32379 \
cluster-health

member 6246c0e15cee547e is healthy: got healthy result from https://10.0.0.11:22379
member 799f3a7a5ffa02a6 is healthy: got healthy result from https://10.0.0.11:2379
member 7d932a4207ff1146 is healthy: got healthy result from https://10.0.0.11:32379
cluster is healthy

钟明家

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
k8s集群之etcd

在正式环境我是三台主机做的集群，这里我就只能用一台虚拟机三个进程分别起etcd实际上原理是一样的。另外基础环境比如dns服务器（参考我前面dns文章）这里略过，做了正向、反向dns。cfssl versionVersion: 1.2.0Revision: devRuntime: go1.6etcd version：etcdctl version: 3.3.13API ve...
复制链接

扫一扫