（三）Kubernetes - 手动部署(二进制方式安装)

<接前文…>

环境准备

主机名	操作系统	IP 地址	所需组件
node-251	CentOS 7.9	192.168.71.251	所有组件都安装（合理利用资源）
node-252	CentOS 7.9	192.168.71.252	所有组件都安装
node-253	CentOS 7.9	192.168.71.253	docker kubelet kube-proxy

我们已经在node-251和node-252部署了etcd集群，并在所有机器安装了docker，接下来我们要来安装k8s其他组件

1 部署主节点

1.1 生成kube-apiserver证书

在前文中我们给etcd颁发了证书，接下来我们将为kube-apiserver颁发证书

1.1.1 自签证书颁发机构(CA)

cd ~/TLS/k8s

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF
cat > ca-csr.json << EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Shenyang",
            "ST": "Shenyang",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

生成证书：

[root@node-251 k8s]# /usr/local/bin/cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2023/05/06 17:26:02 [INFO] generating a new CA key and certificate from CSR
2023/05/06 17:26:02 [INFO] generate received request
2023/05/06 17:26:02 [INFO] received CSR
2023/05/06 17:26:02 [INFO] generating key: rsa-2048
2023/05/06 17:26:02 [INFO] encoded CSR
2023/05/06 17:26:02 [INFO] signed certificate with serial number 487885639304622094422215371699430439513865771259

目录下会生成 ca.pem 和 ca-key.pem

1.1.2 使用自签CA签发kube-apiserver https证书

创建证书申请文件：

cat > server-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.71.251",
      "192.168.71.252",
      "192.168.71.253",
      "192.168.71.254",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Shenyang",
            "ST": "Shenyang",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

说明:
上述文件中hosts字段中IP为所有Master/LB/VIP IP,一个都不能少,为了方便后期扩容可以多写几个预留的IP。

生成证书:

[root@node-251 k8s]# /usr/local/bin/cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
2023/05/06 17:28:46 [INFO] generate received request
2023/05/06 17:28:46 [INFO] received CSR
2023/05/06 17:28:46 [INFO] generating key: rsa-2048
2023/05/06 17:28:46 [INFO] encoded CSR
2023/05/06 17:28:46 [INFO] signed certificate with serial number 714464680386252024164975579217412192488569034346
2023/05/06 17:28:46 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

说明：
当前目录下会生成server.pem 和 server-key.pem文件。

1.2 下载k8s-server

下载地址:https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

如果能打开的话，选择如下图

1.3 解压二进制包

上传k8s软件包到服务器上

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} 
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
cp kubectl /usr/bin/

1.4 部署kube-apiserver

1.4.1 创建配置文件

cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=https://192.168.71.251:2379,https://192.168.71.252:2379 \\
--bind-address=192.168.71.251 \\
--secure-port=6443 \\
--advertise-address=192.168.71.251 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--service-account-issuer=https://kubernetes.default.svc.cluster.local \\
--service-account-signing-key-file=/opt/kubernetes/ssl/server-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\
--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\
--requestheader-allowed-names=kubernetes \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

说明:
上面两个\\，第一个是转义符,第二个是换行符,使用转义符是为了使用EOF保留换行符。

• –logtostderr ：启用日志
• –v ：日志等级
• –log-dir ：日志目录
• –etcd-servers ：etcd集群地址
• –bind-address ：监听地址
• –secure-port ：https安全端口
• –advertise-address ：集群通告地址
• –allow-privileged ：启动授权
• –service-cluster-ip-range ：Service虚拟IP地址段
• –enable-admission-plugins ： 准入控制模块
• –authorization-mode ：认证授权,启用RBAC授权和节点自管理
• –enable-bootstrap-token-auth ：启用TLS bootstrap机制
• –token-auth-file ：bootstrap token文件
• –service-node-port-range ：Service nodeport类型默认分配端口范围
• –kubelet-client-xxx ：apiserver访问kubelet客户端证书
• –tls-xxx-file ：apiserver https证书
• 1.20版本必须加的参数：–service-account-issuer,–service-account-signing-key-file
• –etcd-xxxfile ：连接etcd集群证书
• –audit-log-xxx ：审计日志
• 启动聚合层网关配置：–requestheader-client-ca-file,–proxy-client-cert-file,–proxy-client-key-file,–requestheader-allowed-names,–requestheader-extra-headers-prefix,–requestheader-group-headers,–requestheader-username-headers,–enable-aggregator-routing

1.4.2 拷贝刚才生成的证书

把刚才生成的证书拷贝到配置文件中的路径：

[root@node-251 bin]# cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/
[root@node-251 bin]# ll /opt/kubernetes/ssl/
total 16
-rw------- 1 root root 1675 May  6 18:56 ca-key.pem
-rw-r--r-- 1 root root 1363 May  6 18:56 ca.pem
-rw------- 1 root root 1679 May  6 18:56 server-key.pem
-rw-r--r-- 1 root root 1643 May  6 18:56 server.pem

1.4.3 启用TLS bootstrapping机制

TLS Bootstraping：Master apiserver启用TLS认证后，Node节点kubelet和kube-proxy要与kube-apiserver进行通信，必须使用CA签发的有效证书才可以，当Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式，目前主要用于kubelet，kube-proxy还是由我们统一颁发一个证书。

TLS bootstraping 工作流程：

创建上述配置文件中token文件：

cat > /opt/kubernetes/cfg/token.csv << EOF
4136692876ad4b01bb9dd0988480ebba,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

格式：token,用户名,UID,用户组

token也可自行生成替换：

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

1.4.4 systemd管理apiserver

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

1.4.5 启动并设置开机启动

systemctl daemon-reload
systemctl start kube-apiserver 
systemctl enable kube-apiserver

1.5 部署kube-controller-manager

1.5.1 创建配置文件

cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--cluster-signing-duration=87600h0m0s"
EOF

• –kubeconfig ：连接apiserver配置文件。
• –leader-elect ：当该组件启动多个时,自动选举(HA)
• –cluster-signing-cert-file ：自动为kubelet颁发证书的CA,apiserver保持一致
• –cluster-signing-key-file ：自动为kubelet颁发证书的CA,apiserver保持一致

1.5.2 生成kubeconfig文件

生成kube-controller-manager证书 ：

# 切换工作目录
cd ~/TLS/k8s

# 创建证书请求文件
cat > kube-controller-manager-csr.json << EOF
{
  "CN": "system:kube-controller-manager",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Shenyang", 
      "ST": "Shenyang",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

生成证书

[root@node-251 k8s]# /usr/local/bin/cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
2023/05/06 19:26:38 [INFO] generate received request
2023/05/06 19:26:38 [INFO] received CSR
2023/05/06 19:26:38 [INFO] generating key: rsa-2048
2023/05/06 19:26:38 [INFO] encoded CSR
2023/05/06 19:26:38 [INFO] signed certificate with serial number 596752905879086439660675748465218716933500301112
2023/05/06 19:26:38 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

生成kubeconfig文件(以下是shell命令,直接在shell终端执行)

KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig"
KUBE_APISERVER="https://192.168.71.251:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
  
kubectl config set-credentials kube-controller-manager \
  --client-certificate=./kube-controller-manager.pem \
  --client-key=./kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
  
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-controller-manager \
  --kubeconfig=${KUBE_CONFIG}
  
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

1.5.3 systemd管理controller-manager

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

1.5.4 启动并设置开机自启

[root@node-251 k8s]# systemctl daemon-reload
[root@node-251 k8s]# systemctl start kube-controller-manager.service
[root@node-251 k8s]# systemctl status kube-controller-manager.service
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2023-05-06 19:30:30 CST; 6s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 7963 (kube-controller)
    Tasks: 8
   Memory: 45.8M
   CGroup: /system.slice/kube-controller-manager.service
           └─7963 /opt/kubernetes/bin/kube-controller-manager --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect...

May 06 19:30:30 node-251 systemd[1]: Started Kubernetes Controller Manager.
[root@node-251 k8s]# systemctl enable kube-controller-manager.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.

有可能报错如下，如果是内部使用，可以忽略。如果上云使用，则需要配置cloud-provider参数了

May 09 03:04:51 node-251 kube-controller-manager[2543]: E0509 03:04:51.087617    2543 core.go:232] failed to start cloud node lifecycle controller: no cloud provider provided
May 09 03:05:01 node-251 kube-controller-manager[2543]: E0509 03:05:01.176846    2543 core.go:92] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail

1.6 部署 kube-scheduler

1.6.1 创建配置文件

cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect \\
--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\
--bind-address=127.0.0.1"
EOF

• –kubeconfig ：连接apiserver配置文件
• –leader-elect ：当该组件启动多个时,自动选举(HA)。

1.6.2 生成kubeconfig文件

生成kube-scheduler证书 ：

# 切换工作目录
cd ~/TLS/k8s

# 创建证书请求文件
cat > kube-scheduler-csr.json << EOF
{
  "CN": "system:kube-scheduler",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Shenyang",
      "ST": "Shenyang",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

生成证书

[root@node-251 k8s]# /usr/local/bin/cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
2023/05/06 19:34:46 [INFO] generate received request
2023/05/06 19:34:46 [INFO] received CSR
2023/05/06 19:34:46 [INFO] generating key: rsa-2048
2023/05/06 19:34:46 [INFO] encoded CSR
2023/05/06 19:34:46 [INFO] signed certificate with serial number 577369023958200029339462439762844831682746851754
2023/05/06 19:34:46 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

生成kubeconfig文件 ：

KUBE_CONFIG="/opt/kubernetes/cfg/kube-scheduler.kubeconfig"
KUBE_APISERVER="https://192.168.71.251:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
  
kubectl config set-credentials kube-scheduler \
  --client-certificate=./kube-scheduler.pem \
  --client-key=./kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
  
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-scheduler \
  --kubeconfig=${KUBE_CONFIG}
  
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

1.6.3 systemd管理scheduler

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

1.6.4 启动并设置开机启动

systemctl daemon-reload
systemctl start kube-scheduler
systemctl enable kube-scheduler

1.6.5 查看集群状态

生成kubectl连接集群的证书 ：

cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Shenyang",
      "ST": "Shenyang",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}
EOF

生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

生成kubeconfig文件 ：

mkdir /root/.kube

KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://192.168.71.251:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}
  
kubectl config set-credentials cluster-admin \
  --client-certificate=./admin.pem \
  --client-key=./admin-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}
  
kubectl config set-context default \
  --cluster=kubernetes \
  --user=cluster-admin \
  --kubeconfig=${KUBE_CONFIG}
  
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

通过kubectl工具查看当前集群组件状态 ：

[root@node-251 k8s]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true"}
etcd-1               Healthy   {"health":"true"}
controller-manager   Healthy   ok

如上说明主节点组件运行正常。

1.6.6 授权kubelet-bootstrap用户允许请求证书

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap