k8s集群之etcd

在正式环境我是三台主机做的集群,这里我就只能用一台虚拟机三个进程分别起etcd实际上原理是一样的。另外基础环境比如dns服务器(参考我前面dns文章)这里略过,做了正向、反向dns。

cfssl version
Version: 1.2.0
Revision: dev
Runtime: go1.6
etcd version:
etcdctl version: 3.3.13
API version: 2

1、安装 CFSSL

下载cfssl:
https://pkg.cfssl.org/R1.2/SHA256SUMS
https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

安装 CFSSL
chmod +x cfssl*
mv cfssl_linux-amd64 /usr/local/bin/cfssl -v
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson -v

2、准备etcd使用的证书

2、准备etcd使用的证书
mkdir /etc/k8s/ssl/ -pv
mkdir -pv /tmp/certs && cd /tmp/certs

ca配置文件:
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF

ca-config.json:可以定义多个 profiles,分别指定不同的过期时间、使用场景等参数;后续在签名证书时使用某个 profile; 
signing:表示该证书可用于签名其它证书;生成的 ca.pem 证书中 CA=TRUE; 
server auth:表示 client 可以用该 CA 对 server 提供的证书进行验证; 
client auth:表示 server 可以用该 CA 对 client 提供的证书进行验证;

创建CA证书签名请求
cat > /tmp/certs/ca-csr.json <<EOF
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "k8s",
      "OU": "k8s Security",
      "L": "ChengDU",
      "ST": "SiChuan",
      "C": "CN"
    }
  ],
  "CN": "kubernetes"
}
EOF
# 生成 CA 证书和私钥:
cfssl gencert --initca=true /tmp/certs/ca-csr.json | cfssljson --bare /tmp/certs/ca

CN:Common Name,kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name),浏览器使用该字段验证网站是否合法;
O:Organization,kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group);
kube-apiserver 将提取的 User、Group 作为 RBAC 授权的用户标识;

# verify
openssl x509 -in /tmp/certs/ca.pem -text -noout


# 创建生成证书配置文件
cat > /tmp/certs/gencert.json <<EOF

{
  "signing": {
    "default": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "87600h"
    },
  "kubernetes": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "175200h"
    }
  }
}

EOF

cat > /tmp/certs/etcd-csr.json <<EOF
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "k8s",
      "OU": "k8s Security",
      "L": "ChengDU",
      "ST": "SiChuan",
      "C": "CN"
    }
  ],
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "localhost",
    "10.0.0.11",
    "etcd",
    "etcd1",
    "etcd2",
    "etcd3",
    "etcd.zmjcd.cc",
    "etcd1.zmjcd.cc",
    "etcd2.zmjcd.cc",
    "etcd3.zmjcd.cc"
  ]
}

EOF

hosts 字段指定授权使用该证书的 etcd 节点 IP 或域名列表,这里将 etcd 集群的三个节点 IP 都列在其中;

生成 CA 证书和私钥:
cfssl gencert \
  --ca /tmp/certs/ca.pem \
  --ca-key /tmp/certs/ca-key.pem \
  --config /tmp/certs/gencert.json \
  /tmp/certs/etcd-csr.json | cfssljson --bare /tmp/certs/etcd

# verify
openssl x509 -in /tmp/certs/etcd.pem -text -noout


将*.pem证书分发到3台etcd的/etc/kubernetes/cert目录下
scp ./* root@etcd3:/etc/k8s/ssl/
scp ./* root@etcd2:/etc/k8s/ssl/
scp ./* root@etcd1:/etc/k8s/ssl/

3、安装etcd集群

我用一台服务跑三个etcd进程来模拟三台etcd服务器。
下载etcd:
https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz

在三个节点都安装etcd,下面的操作需要再三个节点都执行一遍
tar -xf etcd-v3.3.13-linux-amd64.tar.gz -C /usr/local/ && cd /usr/local/etcd-v3.3.13-linux-amd64/
ln -sv /usr/local/etcd-v3.3.13-linux-amd64/etcd /usr/local/bin/
ln -sv /usr/local/etcd-v3.3.13-linux-amd64/etcdctl /usr/local/bin/


mkdir -pv /var/lib/etcd{1,2,3}

配置etcd:
cat > /tmp/etcd1.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd


[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
WorkingDirectory=/var/lib/etcd1/

ExecStart=/usr/local/bin/etcd \
--name etcd1   \
--data-dir /var/lib/etcd1   \
--listen-client-urls https://10.0.0.11:2379   \
--advertise-client-urls https://10.0.0.11:2379   \
--listen-peer-urls https://10.0.0.11:2380   \
--initial-advertise-peer-urls https://10.0.0.11:2380   \
--initial-cluster etcd1=https://10.0.0.11:2380,etcd2=https://10.0.0.11:22380,etcd3=https://10.0.0.11:32380   \
--initial-cluster-token tkn   \
--initial-cluster-state new   \
--client-cert-auth   \
--trusted-ca-file /etc/k8s/ssl/ca.pem   \
--cert-file /etc/k8s/ssl/etcd.pem   \
--key-file /etc/k8s/ssl/etcd-key.pem   \
--peer-client-cert-auth   \
--peer-trusted-ca-file /etc/k8s/ssl/ca.pem   \
--peer-cert-file /etc/k8s/ssl/etcd.pem   \
--peer-key-file /etc/k8s/ssl/etcd-key.pem

[Install]
WantedBy=multi-user.target

EOF
mv /tmp/etcd1.service /etc/systemd/system/etcd1.service


cat > /tmp/etcd2.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd


[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
WorkingDirectory=/var/lib/etcd2/

ExecStart=/usr/local/bin/etcd \
--name etcd2   \
--data-dir /var/lib/etcd2   \
--listen-client-urls https://10.0.0.11:22379   \
--advertise-client-urls https://10.0.0.11:22379   \
--listen-peer-urls https://10.0.0.11:22380   \
--initial-advertise-peer-urls https://10.0.0.11:22380   \
--initial-cluster etcd1=https://10.0.0.11:2380,etcd2=https://10.0.0.11:22380,etcd3=https://10.0.0.11:32380   \
--initial-cluster-token tkn   \
--initial-cluster-state new   \
--client-cert-auth   \
--trusted-ca-file /etc/k8s/ssl/ca.pem   \
--cert-file /etc/k8s/ssl/etcd.pem   \
--key-file /etc/k8s/ssl/etcd-key.pem   \
--peer-client-cert-auth   \
--peer-trusted-ca-file /etc/k8s/ssl/ca.pem   \
--peer-cert-file /etc/k8s/ssl/etcd.pem   \
--peer-key-file /etc/k8s/ssl/etcd-key.pem

[Install]
WantedBy=multi-user.target

EOF
mv /tmp/etcd2.service /etc/systemd/system/etcd2.service

cat > /tmp/etcd3.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd


[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
WorkingDirectory=/var/lib/etcd2/

ExecStart=/usr/local/bin/etcd \
--name etcd3   \
--data-dir /var/lib/etcd3   \
--listen-client-urls https://10.0.0.11:32379   \
--advertise-client-urls https://10.0.0.11:32379   \
--listen-peer-urls https://10.0.0.11:32380   \
--initial-advertise-peer-urls https://10.0.0.11:32380   \
--initial-cluster etcd1=https://10.0.0.11:2380,etcd2=https://10.0.0.11:22380,etcd3=https://10.0.0.11:32380   \
--initial-cluster-token tkn   \
--initial-cluster-state new   \
--client-cert-auth   \
--trusted-ca-file /etc/k8s/ssl/ca.pem   \
--cert-file /etc/k8s/ssl/etcd.pem   \
--key-file /etc/k8s/ssl/etcd-key.pem   \
--peer-client-cert-auth   \
--peer-trusted-ca-file /etc/k8s/ssl/ca.pem   \
--peer-cert-file /etc/k8s/ssl/etcd.pem   \
--peer-key-file /etc/k8s/ssl/etcd-key.pem

[Install]
WantedBy=multi-user.target

EOF
mv /tmp/etcd3.service /etc/systemd/system/etcd3.service


# to start service
sudo systemctl daemon-reload
sudo systemctl cat etcd1.service
sudo systemctl enable etcd1.service
sudo systemctl start etcd1.service

sudo systemctl cat etcd2.service
sudo systemctl enable etcd2.service
sudo systemctl start etcd2.service

sudo systemctl cat etcd3.service
sudo systemctl enable etcd3.service
sudo systemctl start etcd3.service
4、验证etcd集群
查看etcd2状态:
 ETCDCTL_API=3 /usr/local/bin/etcdctl  \
--endpoints 10.0.0.11:2379,10.0.0.11:22379,10.0.0.11:32379  \
--cacert /etc/k8s/ssl/ca.pem   \
--cert /etc/k8s/ssl/etcd.pem   \
--key /etc/k8s/ssl/etcd-key.pem   \
endpoint health

10.0.0.11:22379 is healthy: successfully committed proposal: took = 4.000979ms
10.0.0.11:32379 is healthy: successfully committed proposal: took = 4.907629ms
10.0.0.11:2379 is healthy: successfully committed proposal: took = 11.089059ms

查看那台为leader:
etcdctl  --cert-file=/etc/k8s/ssl/etcd.pem \
--key-file=/etc/k8s/ssl/etcd-key.pem \
--ca-file=/etc/k8s/ssl/ca.pem \
--endpoints https://10.0.0.11:2379,https://10.0.0.11:22379,https://10.0.0.11:32379 \
member list

6246c0e15cee547e: name=etcd2 peerURLs=https://10.0.0.11:22380 clientURLs=https://10.0.0.11:22379 isLeader=false
799f3a7a5ffa02a6: name=etcd1 peerURLs=https://10.0.0.11:2380 clientURLs=https://10.0.0.11:2379 isLeader=true
7d932a4207ff1146: name=etcd3 peerURLs=https://10.0.0.11:32380 clientURLs=https://10.0.0.11:32379 isLeader=false


关闭etcd2:
systemctl stop etcd1

etcdctl  --cert-file=/etc/k8s/ssl/etcd.pem \
--key-file=/etc/k8s/ssl/etcd-key.pem \
--ca-file=/etc/k8s/ssl/ca.pem \
--endpoints https://10.0.0.11:2379,https://10.0.0.11:22379,https://10.0.0.11:32379 \
cluster-health

member 6246c0e15cee547e is healthy: got healthy result from https://10.0.0.11:22379
failed to check the health of member 799f3a7a5ffa02a6 on https://10.0.0.11:2379: Get https://10.0.0.11:2379/health: dial tcp 10.0.0.11:2379: connect: connection refused
member 799f3a7a5ffa02a6 is unreachable: [https://10.0.0.11:2379] are all unreachable
member 7d932a4207ff1146 is healthy: got healthy result from https://10.0.0.11:32379
cluster is degraded


启动etcd1:
systemctl start etcd1

etcdctl  --cert-file=/etc/k8s/ssl/etcd.pem \
--key-file=/etc/k8s/ssl/etcd-key.pem \
--ca-file=/etc/k8s/ssl/ca.pem \
--endpoints https://10.0.0.11:2379,https://10.0.0.11:22379,https://10.0.0.11:32379 \
cluster-health

member 6246c0e15cee547e is healthy: got healthy result from https://10.0.0.11:22379
member 799f3a7a5ffa02a6 is healthy: got healthy result from https://10.0.0.11:2379
member 7d932a4207ff1146 is healthy: got healthy result from https://10.0.0.11:32379
cluster is healthy
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
在 Kubernetes (k8s) 集群中,Etcd 是作为核心组件之一的分布式键值存储系统,用于存储和管理集群配置数据和服务发现信息。以下是搭建 Etcd 集群的基本步骤: 1. **下载安装**: 首先需要从 Etcd 的官方 GitHub 仓库下载最新版本的二进制文件。你可以选择适合你的操作系统(如 Linux 或 macOS)的版本。 2. **设置环境变量**: 创建一个 `.bashrc` 或者 `.zshrc` 文件,并添加 Etcd 的路径到 `PATH` 变量中,便于后续命令行操作。 3. **初始化集群**: 对每个节点运行 Etcd 后台服务,使用 `-name` 参数指定本节点名称,例如: ``` etcd -name node1 --initial-cluster="node1=http://localhost:2380,node2=http://localhost:2381,node3=http://localhost:2382" ``` 这里创建了一个包含三个节点的集群,假设它们都在本地主机上运行。实际部署中,你需要使用对应的 IP 地址和端口。 4. **启动集群**: 分别在各个节点上执行上述命令,当所有节点都启动后,Etcd 集群会自动选举领导者并开始同步数据。 5. **验证集群状态**: 使用 `etcdctl` 工具连接到任意节点检查集群健康状况和数据一致性。 6. **配置 Kubernetes**: 在 k8s 中,你需要将 Etcd 集群的地址添加到 `kubelet` 和 `kubectl` 的配置中,通过修改 `/etc/kubernetes/manifests/kube-apiserver.yaml` 和 `/etc/kubernetes/manifests/kube-controller-manager.yaml` 等配置文件。 7. **重启服务**: 重启 k8s 控制平面(如 kube-apiserver、kube-controller-manager 等),让它们能够连接到新配置的 Etcd 集群
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值