二进制搭建Kubernetes 1.28

已于 2023-11-06 17:59:56 修改
阅读量530 收藏 3
点赞数 1
分类专栏： K8s 文章标签： kubernetes 容器 云原生
于 2023-11-06 16:30:06 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_43666009/article/details/134248833
版权

二进制搭建Kubernetes 1.28

说明

节点信息

名称角色IP操作系统
masterControl Plane, Etcd192.168.10.60Centos7.9
master02Control Plane, Etcd192.168.10.63Centos7.9
master03Control Plane, Etcd192.168.10.64Centos7.9
worker01Worker192.168.10.61Centos7.9
worker02Worker192.168.10.62Centos7.9

证书信息

注意：所有生成证书的操作都只在master节点上

组件证书
Etcdetcd-ca.pem、etcd.pem、etcd-key.pem
Kubeletca.pem、kubeconfig
Kube-apiseverca.pem、kube-apiserver.pem、kube-apiserver-key.pem
Kube-controller-managerca.pem、kube-controller-manager.pem、kube-controller-manager-key.pem
Kube-scheduleca.pem、kube-schedule.pem、kube-schedule-key.pem
Kube-proxyca.pem、kubeconfig

主机初始化

主机名设置

hostnamectl set-hostname xxx
echo '''
192.168.10.60 master
192.168.10.63 master02
192.168.10.64 master03
192.168.10.61 worker01
192.168.10.62 worker02
''' >> /etc/hosts

安全配置

#关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
#关闭swap
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab
#设置seLinux
setenforce 0
vi /etc/selinux/config
SELINUX=enforcing ---> SELINUX=disabled

内核升级

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum install -y http://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm

yum --disablerepo="*" --enablerepo="elrepo-kernel" list available

yum --enablerepo=elrepo-kernel install kernel-lt -y
grub2-set-default 0 
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot
uname -r

时间同步

yum install chrony -y
systemctl start chronyd
systemctl enable chronyd
chronyc sources

额外配置

#模块配置
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

modprobe overlay
modprobe br_netfilter
systemctl restart systemd-modules-load.service

#配置limit
cat >> /etc/security/limits.conf <<EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* seft memlock unlimited
* hard memlock unlimitedd


#ipvs配置
yum install ipvsadm ipset sysstat conntrack libseccomp -y

cat >> /etc/modules-load.d/ipvs.conf <<EOF 
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

systemctl restart systemd-modules-load.service
lsmod | grep -e ip_vs -e nf_conntrack

安装containerd

下载二进制包

本次选择cri-containerd-cni-1.7.8-linux-amd64.tar.gz，否则需要额外安装cni与runc

mkdir -p /software/containerd/
cd /software/containerd/
wget https://github.com/containerd/containerd/releases/download/v1.7.8/cri-containerd-cni-1.7.8-linux-amd64.tar.gz

解压安装

tar -xvf cri-containerd-cni-1.7.8-linux-amd64.tar.gz
cp -r etc  opt  usr /

cat > /etc/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF

systemctl enable --now containerd

配置contained

添加配置

mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml

# 设置aliyun地址，不设置会连接不上, 如果无法下载镜像检查一下配置是否替换 cat /etc/containerd/config.toml |grep sandbox_image
sed -i "s#registry.k8s.io/pause#registry.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml
sed -i "s#k8s.gcr.io/pause#registry.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml

# 设置驱动为systemd
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# 设置dicker地址为aliyun镜像地址
sed -i '/\[plugins\."io\.containerd\.grpc\.v1\.cri"\.registry\.mirrors\]/a\      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]\n        endpoint = ["https://8aj710su.mirror.aliyuncs.com" ,"https://registry-1.docker.io"]' /etc/containerd/config.toml

查看结果

crictl info
crictl images

systemctl daemon-reload
systemctl restart containerd

安装cfssl

url=https://github.com/cloudflare/cfssl/releases/download
ver=1.6.1

curl -L -o /usr/local/bin/cfssl           ${url}/v${ver}/cfssl_${ver}_linux_amd64
curl -L -o /usr/local/bin/cfssljson       ${url}/v${ver}/cfssljson_${ver}_linux_amd64
curl -L -o /usr/local/bin/cfssl-certinfo  ${url}/v${ver}/cfssl-certinfo_${ver}_linux_amd64

chmod +x /usr/local/bin/cfssl*
export PATH=$PATH:$HOME/bin:/usr/local/bin
cfssl version

安装Etcd

下载二进制包

mkdir /software/etcd/
wget https://storage.googleapis.com/etcd/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz

生成ca证书

生成ca配置文件

mkdir -p /etc/etcd/ssl
cd /etc/etcd/ssl
cat > etcd-ca-config.json << EOF 
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

生成ca请求文件

cat > etcd-ca-csr.json  << EOF 
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

生成ca证书

cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca -
#输出结果
etcd-ca.csr、etcd-ca-key.pem、etcd-ca.pem

生成证书

配置证书请求文件

cat > etcd-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
    "192.168.10.60",
    "192.168.10.61",
    "192.168.10.62",
    "192.168.10.63",
    "192.168.10.64"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

使用ca和证书请求文件生成etcd的证书

cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=etcd-ca-config.json -profile=www etcd-csr.json | cfssljson -bare etcd
#输出结果
etcd.csr、etcd-key.pem、etcd.pem

将证书复制到其他两个节点

Master='master02 master03'
for NODE in $Master; do ssh $NODE "mkdir -p /etc/etcd/ssl"; for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}; done; done

安装

解压
cd /software/etcd/
tar -xf etcd*.tar.gz && mv etcd-*/etcd /usr/local/bin/ && mv etcd-*/etcdctl /usr/local/bin/
ls /usr/local/bin/
配置etcd

配置master节点

cat > /etc/etcd/etcd.config.yml << EOF 
name: 'master'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.10.60:2380'
listen-client-urls: 'https://192.168.10.60:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.10.60:2380'
advertise-client-urls: 'https://192.168.10.60:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'master=https://192.168.10.60:2380,master02=https://192.168.10.63:2380,master03=https://192.168.10.64:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

配置master02节点

cat > /etc/etcd/etcd.config.yml << EOF 
name: 'master02'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.10.63:2380'
listen-client-urls: 'https://192.168.10.63:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.10.63:2380'
advertise-client-urls: 'https://192.168.10.63:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'master=https://192.168.10.60:2380,master02=https://192.168.10.63:2380,master03=https://192.168.10.64:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

配置master03节点

cat > /etc/etcd/etcd.config.yml << EOF 
name: 'master03'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.10.64:2380'
listen-client-urls: 'https://192.168.10.64:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.10.64:2380'
advertise-client-urls: 'https://192.168.10.64:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'master=https://192.168.10.60:2380,master02=https://192.168.10.63:2380,master03=https://192.168.10.64:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF
创建service

创建etcd.service文件

cat > /usr/lib/systemd/system/etcd.service << EOF

[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service

EOF

创建证书目录

mkdir  -p /etc/kubernetes/pki/etcd
ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/

启动etcd

systemctl daemon-reload
systemctl enable --now etcd.service
systemctl restart etcd.service

查看etcd

systemctl status etcd.service
export ETCDCTL_API=3
etcdctl --endpoints="192.168.10.60:2379,192.168.10.63:2379,192.168.10.64:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table

在这里插入图片描述

安装K8S

下载二进制包

mkdir -p /software/k8s
cd /software/k8s
wget https://dl.k8s.io/v1.28.3/kubernetes-server-linux-amd64.tar.gz

创建工作目录

mkdir -p /etc/kubernetes/manifests/ /etc/kubernetes/cfg /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes

解压

master节点解压

tar -xf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
ls /usr/local/bin/

复制到其他的master节点和worker节点

Master='master02 master03'
Work='worker01 worker02'

# 拷贝master组件
for NODE in $Master; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done

# 拷贝work组件
for NODE in $Work; do echo $NODE; scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done

生成集群的ca证书

生成ca配置文件

mkdir -p /etc/kubernetes/pki
cd /etc/kubernetes/pki
cat > ca-config.json << EOF 
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

生成ca请求文件

cat > ca-csr.json  << EOF 
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

生成ca证书

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
#输出结果
ca.csr、ca-key.pem、ca.pem

安装kube-apiserver

生成kube-apiserver的证书

配置证书请求文件

cd /etc/kubernetes/pki
#注意10.96.0.1是定义的service段的集群地址
cat > kube-apiserver-csr.json << EOF
{
    "CN": "kube-apiserver",
    "hosts": [
    "10.96.0.1",
    "127.0.0.1",
    "192.168.10.60",
    "192.168.10.61",
    "192.168.10.62",
    "192.168.10.63",
    "192.168.10.64"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

使用ca和证书请求文件生成kube-apiserver的证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
#输出结果
kube-apiserver.csr、kube-apiserver-key.pem、kube-apiserver.pem

将证书复制到其他两个节点

Master='master02 master03'
for NODE in $Master; do ssh $NODE "mkdir -p /etc/kubernetes/pki"; for FILE in ca-key.pem  ca.pem  kube-apiserver-key.pem  kube-apiserver.pem; do scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done; done
安装
配置kube-apiserver

注意定义的service IP段
master节点

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
      --v=2  \\
      --allow-privileged=true  \\
      --bind-address=0.0.0.0  \\
      --secure-port=6443  \\
      --advertise-address=192.168.10.60 \\
      --service-cluster-ip-range=10.96.0.0/24  \\
      --service-node-port-range=30000-32767  \\
      --etcd-servers=https://192.168.10.60:2379,https://192.168.10.63:2379,https://192.168.10.64:2379 \\
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --tls-cert-file=/etc/kubernetes/pki/kube-apiserver.pem  \\
      --tls-private-key-file=/etc/kubernetes/pki/kube-apiserver-key.pem  \\
      --kubelet-client-certificate=/etc/kubernetes/pki/kube-apiserver.pem  \\
      --kubelet-client-key=/etc/kubernetes/pki/kube-apiserver-key.pem  \\
      --service-account-key-file=/etc/kubernetes/pki/ca-key.pem  \\
      --service-account-signing-key-file=/etc/kubernetes/pki/ca-key.pem  \\
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \\
      --enable-bootstrap-token-auth=true  \\
      --token-auth-file=/etc/kubernetes/cfg/token.csv \\
      --requestheader-client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --proxy-client-cert-file=/etc/kubernetes/pki/kube-apiserver.pem  \\
      --proxy-client-key-file=/etc/kubernetes/pki/kube-apiserver-key.pem  \\
      --requestheader-allowed-names=aggregator  \\
      --requestheader-group-headers=X-Remote-Group  \\
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
      --requestheader-username-headers=X-Remote-User \\
      --enable-aggregator-routing=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

EOF

master02节点

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
      --v=2  \\
      --allow-privileged=true  \\
      --bind-address=0.0.0.0  \\
      --secure-port=6443  \\
      --advertise-address=192.168.10.63 \\
      --service-cluster-ip-range=10.96.0.0/24  \\
      --service-node-port-range=30000-32767  \\
      --etcd-servers=https://192.168.10.60:2379,https://192.168.10.63:2379,https://192.168.10.64:2379 \\
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --tls-cert-file=/etc/kubernetes/pki/kube-apiserver.pem  \\
      --tls-private-key-file=/etc/kubernetes/pki/kube-apiserver-key.pem  \\
      --kubelet-client-certificate=/etc/kubernetes/pki/kube-apiserver.pem  \\
      --kubelet-client-key=/etc/kubernetes/pki/kube-apiserver-key.pem  \\
      --service-account-key-file=/etc/kubernetes/pki/ca-key.pem  \\
      --service-account-signing-key-file=/etc/kubernetes/pki/ca-key.pem  \\
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \\
      --enable-bootstrap-token-auth=true  \\
      --token-auth-file=/etc/kubernetes/cfg/token.csv \\
      --requestheader-client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --proxy-client-cert-file=/etc/kubernetes/pki/kube-apiserver.pem  \\
      --proxy-client-key-file=/etc/kubernetes/pki/kube-apiserver-key.pem  \\
      --requestheader-allowed-names=aggregator  \\
      --requestheader-group-headers=X-Remote-Group  \\
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
      --requestheader-username-headers=X-Remote-User \\
      --enable-aggregator-routing=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

EOF

master03节点

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
      --v=2  \\
      --allow-privileged=true  \\
      --bind-address=0.0.0.0  \\
      --secure-port=6443  \\
      --advertise-address=192.168.10.64 \\
      --service-cluster-ip-range=10.96.0.0/24  \\
      --service-node-port-range=30000-32767  \\
      --etcd-servers=https://192.168.10.60:2379,https://192.168.10.63:2379,https://192.168.10.64:2379 \\
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --tls-cert-file=/etc/kubernetes/pki/kube-apiserver.pem  \\
      --tls-private-key-file=/etc/kubernetes/pki/kube-apiserver-key.pem  \\
      --kubelet-client-certificate=/etc/kubernetes/pki/kube-apiserver.pem  \\
      --kubelet-client-key=/etc/kubernetes/pki/kube-apiserver-key.pem  \\
      --service-account-key-file=/etc/kubernetes/pki/ca-key.pem  \\
      --service-account-signing-key-file=/etc/kubernetes/pki/ca-key.pem  \\
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \\
      --enable-bootstrap-token-auth=true  \\
      --token-auth-file=/etc/kubernetes/cfg/token.csv \\
      --requestheader-client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --proxy-client-cert-file=/etc/kubernetes/pki/kube-apiserver.pem  \\
      --proxy-client-key-file=/etc/kubernetes/pki/kube-apiserver-key.pem  \\
      --requestheader-allowed-names=aggregator  \\
      --requestheader-group-headers=X-Remote-Group  \\
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
      --requestheader-username-headers=X-Remote-User \\
      --enable-aggregator-routing=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

EOF
生成TLS Bootstrapping的token
head -c 16 /dev/urandom | od -An -t x | tr -d ' '
# 创建配置文件中token文件：格式：token，用户名，UID，用户组
cat > /etc/kubernetes/cfg/token.csv << EOF
26efe2a33fcd598af2df97e21e4f442f,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

将token文件复制到其他两个节点

Master='master02 master03'
for NODE in $Master; do ssh $NODE "mkdir -p /etc/kubernetes/cfg"; for FILE in token.csv; do scp /etc/kubernetes/cfg/${FILE} $NODE:/etc/kubernetes/cfg/${FILE}; done; done
启动kube-apiserver
systemctl daemon-reload
systemctl enable --now kube-apiserver.service
systemctl restart kube-apiserver.service
systemctl status kube-apiserver.service

安装kube-controller-manager

生成kube-controller-manager的证书

配置证书请求文件

cd /etc/kubernetes/pki
cat > kube-controller-manager-csr.json << EOF
{
    "CN": "system:kube-controller-manager",
    "hosts": [
    "127.0.0.1",
    "192.168.10.60",
    "192.168.10.61",
    "192.168.10.62",
    "192.168.10.63",
    "192.168.10.64"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "system:masters",
            "OU": "System"
        }
    ]
}
EOF

使用ca和证书请求文件生成kube-controller-manager的证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
#输出结果
kube-controller-manager.csr、kube-controller-manager-key.pem、kube-controller-manager.pem

将证书复制到其他两个节点

Master='master02 master03'
for NODE in $Master; do ssh $NODE "mkdir -p /etc/kubernetes/pki"; for FILE in kube-controller-manager-key.pem  kube-controller-manager.pem; do scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done; done

配置kubeconfig集群项

kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.10.60:6443 \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
     
kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.10.63:6443 \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
     
kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.10.64:6443 \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

配置controller-manager 组件的客户端凭据

kubectl config set-credentials kube-controller-manager \
     --client-certificate=/etc/kubernetes/pki/kube-controller-manager.pem \
     --client-key=/etc/kubernetes/pki/kube-controller-manager-key.pem \
     --embed-certs=true \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

配置kubeconfig默认环境变量

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-controller-manager \
  --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config use-context default --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
安装
配置kube-controller-manager

注意定义的service IP段和cluster IP段

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\
      --v=2 \\
      --bind-address=127.0.0.1 \\
      --root-ca-file=/etc/kubernetes/pki/ca.pem \\
      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \\
      --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \\
      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \\
      --leader-elect=true \\
      --node-monitor-grace-period=40s \\
      --node-monitor-period=5s \\
      --controllers=*,bootstrapsigner,tokencleaner \\
      --allocate-node-cidrs=true \\
      --service-cluster-ip-range=10.96.0.0/24 \\
      --cluster-cidr=10.224.0.0/16

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

EOF
启动kube-controller-manager
systemctl daemon-reload
systemctl enable --now kube-controller-manager.service
systemctl restart kube-controller-manager.service
systemctl status kube-controller-manager.service

安装kube-schedule

生成kube-schedule的证书

配置证书请求文件

cd /etc/kubernetes/pki
cat > kube-schedule-csr.json << EOF
{
    "CN": "system:kube-schedule",
    "hosts": [
    "127.0.0.1",
    "192.168.10.60",
    "192.168.10.61",
    "192.168.10.62",
    "192.168.10.63",
    "192.168.10.64"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "system:masters",
            "OU": "System"
        }
    ]
}
EOF

使用ca和证书请求文件生成kube-schedule的证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-schedule-csr.json | cfssljson -bare kube-schedule
#输出结果
kube-schedule.csr、kube-schedule.pem、kube-schedule.pem

将证书复制到其他两个节点

Master='master02 master03'
for NODE in $Master; do ssh $NODE "mkdir -p /etc/kubernetes/pki"; for FILE in kube-schedule-key.pem  kube-schedule.pem; do scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done; done

配置kubeconfig集群项

kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.10.60:6443 \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
     
 kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.10.63:6443 \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
     
kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.10.64:6443 \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

配置kube-schedule 组件的客户端凭据

kubectl config set-credentials kube-schedule \
     --client-certificate=/etc/kubernetes/pki/kube-schedule.pem \
     --client-key=/etc/kubernetes/pki/kube-schedule-key.pem \
     --embed-certs=true \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

配置kubeconfig默认环境变量

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-schedule \
  --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config use-context default --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
安装
配置kube-scheduler
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler \\
      --v=2 \\
      --bind-address=127.0.0.1 \\
      --leader-elect=true \\
      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

EOF
启动kube-scheduler
systemctl daemon-reload
systemctl enable --now kube-scheduler.service
systemctl restart kube-scheduler.service
systemctl status kube-scheduler.service

生成连接集群的证书

配置证书请求文件

cd /etc/kubernetes/pki
cat > admin-csr.json << EOF
{
    "CN": "admin",
    "hosts": [
    "127.0.0.1",
    "192.168.10.60",
    "192.168.10.61",
    "192.168.10.62",
    "192.168.10.63",
    "192.168.10.64"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "system:masters",
            "OU": "System"
        }
    ]
}
EOF

使用ca和证书请求文件生成admin的证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
#输出结果
admin.csr、admin.pem、admin.pem

生成kubeconfig文件

mkdir /root/.kube

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/pki/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.60:6443 \
  --kubeconfig=/root/.kube/config
kubectl config set-credentials cluster-admin \
  --client-certificate=/etc/kubernetes/pki/admin.pem \
  --client-key=/etc/kubernetes/pki/admin-key.pem \
  --embed-certs=true \
  --kubeconfig=/root/.kube/config
kubectl config set-context default \
  --cluster=kubernetes \
  --user=cluster-admin \
  --kubeconfig=/root/.kube/config
kubectl config use-context default --kubeconfig=/root/.kube/config

查看集群组件

 kubectl get cs
 #输出结果
 NAME                 STATUS    MESSAGE                         ERROR
scheduler            Healthy   ok                              
controller-manager   Healthy   ok                              
etcd-1               Healthy   {"health":"true","reason":""}   
etcd-0               Healthy   {"health":"true","reason":""}   
etcd-2               Healthy   {"health":"true","reason":""}

授权kubelet-bootstrap用户允许请求证书

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

创建普通用户kubeconfig

普通用户的私钥

openssl genrsa -out myuser.key 2048
openssl req -new -key myuser.key -out myuser.csr -subj "/CN=myuser"

生成证书请求文件

key=$(cat myuser.csr | base64 | tr -d "\n")
cat > myuser-csr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: myuser
spec:
  groups:
  - system:authenticated
  signerName: kubernetes.io/kube-apiserver-client
  request: $key
  usages:
  - client auth
EOF
kubectl apply -f myuser-csr.yaml

批准csr

kubectl get csr
kubectl certificate approve myuser

获得证书

kubectl get csr myuser -o jsonpath='{.status.certificate}'| base64 -d > myuser.crt

创建角色和角色绑定

kubectl create role demo  --verb=get --verb=list --resource=pods
kubectl create rolebinding demo-binding-myuser --role=demo --user=myuser

创建kubeconfig文件

kubectl config set-cluster myuser --server=https://192.168.10.60:6443 --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --kubeconfig=myuser.config
kubectl config set-credentials myuser --client-key=myuser.key --client-certificate=myuser.crt --embed-certs=true --kubeconfig=myuser.config
kubectl config set-context myuser --cluster=myuser --user=myuser --kubeconfig=myuser.config
kubectl config use-context myuser --kubeconfig=myuser.config

验证

kubectl --kubeconfig=myuser.config get pods
#成功
kubectl --kubeconfig=myuser.config get svc
#失败，无权限

安装kubelet

配置kubelet

所有节点生成kubelet启动文件，以contained为runtime

cat > /usr/lib/systemd/system/kubelet.service << EOF

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
    --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig  \\
    --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
    --config=/etc/kubernetes/kubelet-conf.yml \\
    --container-runtime-endpoint=unix:///run/containerd/containerd.sock  \\
    --node-labels=node.kubernetes.io/node=

[Install]
WantedBy=multi-user.target
EOF

所有节点生成kubelet的配置文件
注意ClusterDNS的地址

cat > /etc/kubernetes/kubelet-conf.yml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
EOF
生成引导的kubeconfig文件
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/pki/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.60:6443 \
  --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
#注意token需要与生成TLS Bootstrapping的token一致
kubectl config set-credentials "kubelet-bootstrap" \
  --token=26efe2a33fcd598af2df97e21e4f442f \
  --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
  
kubectl config set-context default \
  --cluster=kubernetes \
  --user="kubelet-bootstrap" \
  --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
  
kubectl config use-context default --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
启动kubelet
systemctl daemon-reload
systemctl restart kubelet
systemctl enable kubelet
批准Node的kubelet证书申请并加入集群
kubectl get csr
kubectl certificate approve csr-vssd1

kubectl get nodes

复制相关证书文件到其他节点

Master='master02 master03'
for NODE in $Master; do ssh $NODE "mkdir -p /etc/kubernetes"; for FILE in bootstrap-kubelet.kubeconfig; do scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done

Worker='worker01 worker02'
for NODE in $Worker; do ssh $NODE "mkdir -p /etc/kubernetes"; for FILE in bootstrap-kubelet.kubeconfig; do scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done
for NODE in $Worker; do ssh $NODE "mkdir -p /etc/kubernetes/pki"; for FILE in ca.pem; do scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done; done

其他节点启动kubelet

systemctl daemon-reload
systemctl restart kubelet
systemctl enable kubelet

master节点批准其他节点的kubelet证书申请并加入集群

kubectl get csr
kubectl certificate approve csr-v29z9
kubectl get nodes

安装kube-proxy

生成kube-proxy的证书

配置证书请求文件

cd /etc/kubernetes/pki
cat > kube-proxy-csr.json << EOF
{
    "CN": "system:kube-proxy",
    "hosts": [
    "127.0.0.1",
    "192.168.10.60",
    "192.168.10.61",
    "192.168.10.62",
    "192.168.10.63",
    "192.168.10.64"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

使用ca和证书请求文件生成kube-proxy的证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
#输出结果
kube-proxy.csr、kube-proxy.pem、kube-proxy.pem

配置集群项

kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.10.60:6443 \
     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

配置Kubernetes 的 kube-proxy 组件的客户端凭据

kubectl config set-credentials kube-proxy \
     --client-certificate=/etc/kubernetes/pki/kube-proxy.pem \
     --client-key=/etc/kubernetes/pki/kube-proxy-key.pem \
     --embed-certs=true \
     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

配置默认环境变量

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

将文件复制到其他节点

Node='master02 master03 worker01 worker02'
for NODE in $Node; do ssh $NODE "mkdir -p /etc/kubernetes/pki"; for FILE in kube-proxy-key.pem  kube-proxy.pem; do scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done; done

for NODE in $Node; do ssh $NODE "mkdir -p /etc/kubernetes"; for FILE in kube-proxy.kubeconfig; do scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done
安装
配置kube-proxy

所有节点生成kube-proxy启动文件

cat >  /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
  --config=/etc/kubernetes/kube-proxy.yaml \\
  --v=2
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

EOF

所有节点生成kube-proxy的配置文件
注意定义的cluster IP段，本次使用的是ipvs模式

cat > /etc/kubernetes/kube-proxy.yaml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 10.224.0.0/16
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms
EOF
启动kube-proxy
systemctl daemon-reload
systemctl enable --now kube-proxy.service
systemctl restart kube-proxy.service
systemctl status kube-proxy.service

安装网络插件

Centos需要升级libseccomp版本高于2.4，否则安装网络插件会报错failed to create shim task: OCI runtime create failed

rpm -qa | grep libseccomp

yum -y install http://rpmfind.net/linux/centos/8-stream/BaseOS/x86_64/os/Packages/libseccomp-2.5.1-1.el8.x86_64.rpm
#或者
yum -y install https://mirrors.tuna.tsinghua.edu.cn/centos/8-stream/BaseOS/x86_64/os/Packages/libseccomp-2.5.1-1.el8.x86_64.rpm

rpm -qa | grep libseccomp
安装helm
wget https://mirrors.huaweicloud.com/helm/v3.12.3/helm-v3.12.3-linux-amd64.tar.gz
tar xvf helm-*-linux-amd64.tar.gz
cp linux-amd64/helm /usr/local/bin/
helm安装cilium
# 添加源
helm repo add cilium https://helm.cilium.io

# 修改为国内源
helm pull cilium/cilium
tar xvf cilium-*.tgz
cd cilium/
sed -i "s#quay.io/#m.daocloud.io/quay.io/#g" values.yaml

# 默认参数安装
helm install  cilium ./ -n kube-system

查看结果

kubectl get pods -n kube-system

安装CoreDNS

helm安装coredns
helm repo add coredns https://coredns.github.io/helm
helm pull coredns/coredns
tar xvf coredns-*.tgz
cd coredns/

# 修改IP地址
vim values.yaml
cat values.yaml | grep clusterIP:
clusterIP: "10.96.0.10"

查看结果

kubectl get pods -n kube-system

安装Metrics Server

安装高可用Metrics Server

下载相关的yaml

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability.yaml

修改配置

# 修改配置
vim high-availability.yaml
# 修改policy/v1beta1为policy/v1
# 如果是 Kubernetes 1.25+ 版本，需要将这里的 apiVersion 从 policy/v1beta1 更换为 policy/v1
# 否则会报错：no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
apiVersion: policy/v1
# 需要添加内容的3处地方
---
# 1
- args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls
        - --requestheader-client-ca-file=/etc/kubernetes/pki/ca.pem
        - --requestheader-username-headers=X-Remote-User
        - --requestheader-group-headers=X-Remote-Group
        - --requestheader-extra-headers-prefix=X-Remote-Extra-

# 2
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
        - name: ca-ssl
          mountPath: /etc/kubernetes/pki

# 3
      volumes:
      - emptyDir: {}
        name: tmp-dir
      - name: ca-ssl
        hostPath:
          path: /etc/kubernetes/pki

修改为国内镜像

# 修改为国内源
sed -i "s#registry.k8s.io/#m.daocloud.io/registry.k8s.io/#g" high-availability.yaml

安装

kubectl apply -f high-availability.yaml

查看结果

kubectl get pods -n kube-system
kubctl top pods -A
kubctl top nodes

问题总结

  • 创建pod时，可能会发生使用命令行查看logs失败，显示api-server无权限，此时添加一个clusterrolebinding即可；kubectl create clusterrolebinding kube-apiserver --clusterrole=cluster-admin --user=kube-apiserver
  • 创建网络插件时，如果是centos的操作系统，需要将libseccomp版本高于2.4，否则安装网络插件会报错failed to create shim task: OCI runtime create failed
  • 如果集群中的pod报错x509: certificate is valid for yyyy, not 10.96.0.1；可能就是apiserver的证书中，没有包含10.96.0.1这个IP，可以使用命令查看openssl x509 -noout -text -in kube-apiserver.pem

参考链接

[https://kubernetes.io/]：
[https://blog.csdn.net/lzb348110175/article/details/133807069]:
[https://blog.csdn.net/qq_36765991/article/details/128707005]:

qq_43666009
关注
  • 1
    点赞
  • 踩
  • 3
    收藏
    觉得还不错? 一键收藏
  • 3
    评论
专栏目录
k8s-docker二进制(1.28)的搭建
有一种信仰叫海阔天空
11-10 1215
k8s-docker二进制(1.28)的搭建
​二进制安装Kubernetes（k8s）v1.28.0
小云的博客
08-19 1106
二进制安装Kubernetes（k8s）v1.28.0https://github.com/cby-chen/Kubernetes 开源不易，帮忙点个star，谢谢了强烈建议在Github上查看文档 ！！！Github出问题会更新文档，并且后续尽可能第一时间更新新版本文档 ！！！地址：https://github.com/cby-chen/Kubernetes平台字数限制，内容不全，请在Githu...
【Kubernetes集群一主二从安装教程】
最新发布
Java程序员廖志伟
05-08 909
因此，在使用docker时，默认会使用fuse-overlayfs作为存储驱动，想要使用overlay2作为存储驱动，还需要手动配置docker，将存储驱动设置为overlay2， 在/etc/docker/daemon.json中添加以下内容：{ “storage-driver”: “overlay2” }，然后使用命令systemctl restart docker.service进行重启。因此，为了获得更好的Kubernetes使用体验，建议升级到官方推荐的Linux内核版本或者更高的版本。
k8s 1.28版本二进制安装
扛麻袋的少年的博客
10-13 6012
k8s 1.28版本二进制安装
Kubernetes二进制部署
Zlwriii的博客
12-16 2054
节点名称 ip地址 服务部署 master 192.168.32.14 kube-apiserver kube-controller-manager kube-scheduler etcd node1 192.168.32.15 kubelet kube-proxy docker flannel etcd no...
运维高级学习--Kubernetes(K8s 1.28.x)部署
c1429163893的博客
08-26 3950
至此，Kubernetes(K8s 1.28.x)部署完成。在仓库中新建项目：google_containers。接下来这一步只用在一个主机上操作。网络环境（在master主机上）定制软件源（所有主机操作）master节点初始化。
二进制搭建 Kubernetes v1.20 依赖包与脚本
08-03
k8s二进制安装依赖包与脚本
CentOS 使用二进制部署 Kubernetes 1.13集群.docx
04-24
本文档主要介绍了如何在 CentOS 系统上使用二进制方式部署 Kubernetes 1.13 集群。Kubernetes（简称 K8s）是一个开源的容器管理系统，能够自动化容器编排、部署和扩展。在选择部署方式时，文章建议避免使用像 ...
二进制部署kubernetes多master集群
08-30
每一步都很详细，欢迎来获取
二进制安装kubernetes中master环境内容
01-06
二进制安装kubernetes中master环境内容
kubernetes二进制搭建
03-21
kubernetes二进制搭建搭建文档分享 下载后按照文档搭建
libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包
12-06
libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccomp 2.5.1版本，修复Containerd不兼容问题的组件包。libseccv
kubernetes一步一步搭建与https访问
lxlmycsdnfree的博客
08-22 2978
原文链接：https://blog.csdn.net/xxb249/article/details/79437989 （1）kubernetes一步一步搭建 众所周知，kubernetes（简称k8s）是用于管理docker集群的，最近一段时间一直在折腾环境问题，在此写一篇博客，来帮助像我一样的小白，避免走弯路。 一、环境 集群环境 角色 IP地址 ...
【云原生 | Kubernetes 系列】— 部署K8S 1.28版本集群部署（基于Containerd容器运行）
云录
12-30 1385
部署K8S 1.28版本集群部署（基于Containerd容器运行）
【C站首发】全网最新Kubernetes（K8s）1.28版本探秘及部署 基于Containerd容器运行时（100年证书kubeadm）
weixin_47758895的博客
08-22 6670
如果运行runc命令时提示：runc: error while loading shared libraries: libseccomp.so.2: cannot open shared object file: No such file or directory，则表明runc没有找到libseccomp，需要检查libseccomp是否安装，本次安装默认就可以查询到。所有主机均需要操作。所有主机均需要操作。所有主机均需要操作。所有主机均需要操作。所有主机均需要操作。所有主机均需要操作。
openssl为k8s生成SSL证书
识途老码
03-25 5905
openssl为k8s生成SSL证书
云原生（第二篇）k8s-二进制搭建
Y_ht1209的博客
06-29 1332
二进制搭建k8s，烦但好用！
kubernets 使用OpenSSL生成CA证书
忘归的博客
07-09 805
之前dashboard升级时需要Https请求，需要自己生成CA证书；在此记录一下 创建一个2048bit的ca.key openssl genrsa -out ca.key 2048 根据上一步创建的ca.key文件生成ca.crt #-days设置有效时间 openssl req -x509 -new -nodes -key ca.key -subj "/CN=&lt;MASTER...
SSL基础：23:生成Kubernetes集群证书(OpenSSL方式)
知行合一 止于至善
12-15 1866
使用OpenSSL提供的命令则可非常容易地生成kubernetes集群创建所需要的证书，参照官方给的示例方法，再结合前面的文章对于OpenSSL的使用介绍，会发现openssl使用起来还是非常方便和简单的。
ic卡操作软件v1.28
10-16
IC卡操作软件v1.28是一款用于操作IC卡的软件。IC卡，即集成电路卡，是一种具有储存和处理信息能力的智能卡片。这款软件的版本号为1.28，说明它已经经历了一系列的更新和改进。 IC卡操作软件v1.28具有以下特点和功能。首先，它可以读取IC卡中的信息，包括储存的数据和应用程序。同时，它也可以向IC卡写入新的数据，比如更新卡片余额、更改卡片权限等。此外，该软件还能够对IC卡进行格式化、初始化操作，以确保卡片在使用之前处于良好的状态。 这款软件还具备一定的安全措施。它可以对IC卡进行加密和解密操作，保证卡片中的敏感信息不被非法获取。此外，软件还支持密码管理，用户可以设置访问和操作IC卡的密码，进一步保护卡片的安全性。 IC卡操作软件v1.28除了基本的IC卡操作功能外，还提供了一些额外的便利功能。比如，它可以对卡片进行备份和恢复，以防止数据丢失。此外，软件还具备查看操作记录、生成报告等功能，以方便用户对IC卡的使用情况进行监控和统计。 总之，IC卡操作软件v1.28是一款功能丰富、安全可靠的软件，适用于各种IC卡操作需求。无论是个人用户还是企业机构，都可以通过该软件方便地进行IC卡的读写和管理操作。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 3
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值