kubeadm安装k8s 1.22.4 高可用集群（ubuntu）

salarh

已于 2022-02-17 16:38:51 修改

阅读量1.3k

点赞数

分类专栏： k8s 文章标签： kubernetes 系统架构 ubuntu 运维容器

于 2022-02-17 16:35:39 首次发布

本文链接：https://blog.csdn.net/Datura_qing/article/details/121576788

版权

k8s 专栏收录该内容

13 篇文章 0 订阅

订阅专栏

五、安装kubeadm,kubectl和kubelet

1）centos 7 安装

2）Ubuntu 14.06安装

六、初始化master节点以及将work节点加入集群，增加网络插件

七、搭建集群出现的问题，以及解决方式

本次安装采用外部etcd集群

解耦了控制平面和Etcd，集群风险小，单独挂了一台master或etcd对集群影响很小。etcd在外部方便维护和恢复。

一、集群规划

主机	ip	角色
vip	192.168.50.200	虚拟VIP
master24	192.168.50.24	etcd1、master
master25	192.168.50.25	etcd2、master
node26	192.168.50.26	etcd3、node

二、初始化工作

1.安装docker（集群所有节点）

方式见之前的博客

2.环境准备

#先安装相关工具

apt-get update && apt-get install -y apt-transport-https curl

#禁用swap

swapoff -a

#关闭防火墙

sudo ufw disable

#禁用Selinux

apt install selinux-utils

setenforce 0

#设置网络:

tee /etc/sysctl.d/k8s.conf <<-'EOF'

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

net.ipv4.ip_forward = 1

EOF

modprobe br_netfilter

#查看 ipv4 与 v6 配置是否生效:

sysctl --system

#配置 iptables:

iptables -P FORWARD ACCEPT

vim /etc/rc.local

/usr/sbin/iptables -P FORWARD ACCEPT

三、keeplived集群安装测试

centos和ubuntu安装

1.安装keeplived（主备节点执行）

# centos:

yum install -y keepalived

#ubuntu

apt install keepalived

2.配置

ubuntu的这里需要做个配置(centos不需要)

cp /usr/share/doc/keepalived/samples/keepalived.conf.sample /etc/keepalived/keepalived.conf

master01

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

vrrp_instance VI_1 {

state MASTER #BACKUP上修改为BACKUP

interface eth0 #主网卡信息

virtual_router_id 44 #虚拟路由标识，主从相同

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass 1111 #主从认证密码必须一致

}

virtual_ipaddress { #虚拟IP（VIP）

192.168.50.240

}

virtual_server 192.168.50.240 6443 { #对外虚拟IP地址

delay_loop 6 #检查真实服务器时间，单位秒

lb_algo rr #设置负载调度算法，rr为轮训

lb_kind DR #设置LVS负载均衡NAT模式

protocol TCP #使用TCP协议检查realserver状态

real_server 192.168.50.24 6443 { #第一个节点

weight 3 #节点权重值

TCP_CHECK { #健康检查方式

connect_timeout 3 #连接超时

nb_get_retry 3 #重试次数

delay_before_retry 3 #重试间隔/S

}

real_server 192.168.50.25 6443 { #第二个节点

weight 3

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

master02

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

vrrp_instance VI_1 {

state MASTER #BACKUP上修改为BACKUP

interface eth0 #主网卡信息

virtual_router_id 44 #虚拟路由标识，主从相同

priority 80 #权重

advert_int 1

authentication {

auth_type PASS

auth_pass 1111 #主从认证密码必须一致

}

virtual_ipaddress { #虚拟IP（VIP）

192.168.50.240

}

virtual_server 192.168.50.240 6443 { #对外虚拟IP地址

delay_loop 6 #检查真实服务器时间，单位秒

lb_algo rr #设置负载调度算法，rr为轮训

lb_kind DR #设置LVS负载均衡NAT模式

protocol TCP #使用TCP协议检查realserver状态

real_server 192.168.50.24 6443 { #第一个节点

weight 3 #节点权重值

TCP_CHECK { #健康检查方式

connect_timeout 3 #连接超时

nb_get_retry 3 #重试次数

delay_before_retry 3 #重试间隔/S

}

real_server 192.168.50.25 6443 { #第二个节点

weight 3

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

}

3、启动服务

systemctl start keepalived

systemctl enable keepalived

systemctl status keepalived

4、查看

ip a

ip a

查看192.168.50.240是否在master01上

5.校验

systemctl stop keepalived

查看vip:192.168.50.240飘到master02上

四、etcd集群安装

1、在master1上安装cfssl（Kubernetes证书相关(CFSSL) - 简书）

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

chmod +x cfssl_linux-amd64 cfssljson_linux-amd64

mv cfssl_linux-amd64 /usr/local/bin/cfssl

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

2、安装etcd二进制文件

# 创建目录

mkdir -p /data/etcd/bin

# 下载

cd /tmp

wget https://storage.googleapis.com/etcd/v3.3.25/etcd-v3.3.25-linux-amd64.tar.gz

tar zxf etcd-v3.3.25-linux-amd64.tar.gz

cd etcd-v3.3.25-linux-amd64

mv etcd etcdctl /data/etcd/bin/

3、创建ca证书，客户端，服务端，节点之间的证书
Etcd属于server ,etcdctl 属于client，二者之间通过http协议进行通信。

ca证书自己给自己签名的权威证书，用来给其他证书签名
server证书 etcd的证书
client证书客户端，比如etcdctl的证书
peer证书节点与节点之间通信的证书
1）创建目录

mkdir -p /data/etcd/ssl

cd /data/etcd/ssl

2）创建ca证书
创建vim ca-config.json

{

"signing": {

"default": {

"expiry": "438000h"

},

"profiles": {

"server": {

"expiry": "438000h",

"usages": [

"signing",

"key encipherment",

"server auth",

"client auth"

]

},

"client": {

"expiry": "438000h",

"usages": [

"signing",

"key encipherment",

"client auth"

]

},

"peer": {

"expiry": "438000h",

"usages": [

"signing",

"key encipherment",

"server auth",

"client auth"

]

}

server auth表示client可以用该ca对server提供的证书进行验证
client auth表示server可以用该ca对client提供的证书进行验证
创建证书签名请求ca-csr.json
vim ca-csr.json

{

"CN": "etcd",

"key": {

"algo": "rsa",

"size": 2048

}

生成CA证书和私钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

# ls ca*

# ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem

3）生成客户端证书
vim client.json

{

"CN": "client",

"key": {

"algo": "ecdsa",

"size": 256

}

4）生成server，peer证书
创建配置 vim etcd.json

{

"CN": "etcd",

"hosts": [

"192.168.50.24",

"192.168.50.25",

"192.168.50.26"

],

"key": {

"algo": "ecdsa",

"size": 256

},

"names": [

{

"C": "CN",

"L": "BJ",

"ST": "BJ"

}

]

}

生成

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server etcd.json | cfssljson -bare server

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd.json | cfssljson -bare peer

5)配置/data/etcd/cfg/etcd.conf （其他的两个节点改到#的位置）

ETCD_DATA_DIR="/data/etcd/data"

ETCD_LISTEN_PEER_URLS="https://192.168.50.24:2380" #本机ip

ETCD_LISTEN_CLIENT_URLS="https://192.168.50.24:2379" #本机ip

ETCD_NAME=etc0 # etc名称

ETCD_SNAPSHOT_COUNT="500000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.50.24:2380" #本机ip

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.50.24:2379" #本机ip

ETCD_INITIAL_CLUSTER="etc0=https://192.168.50.24:2380,etc1=https://192.168.50.25:2380,etc2=https://192.168.50.26:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/data/etcd/ssl/server.pem"

ETCD_KEY_FILE="/data/etcd/ssl/server-key.pem"

ETCD_CLIENT_CERT_AUTH="True"

ETCD_TRUSTED_CA_FILE="/data/etcd/ssl/ca.pem"

ETCD_AUTO_TLS="True"

ETCD_PEER_CERT_FILE="/data/etcd/ssl/peer.pem"

ETCD_PEER_KEY_FILE="/data/etcd/ssl/peer-key.pem"

ETCD_PEER_CLIENT_CERT_AUTH="True"

ETCD_PEER_TRUSTED_CA_FILE="/data/etcd/ssl/ca.pem"

ETCD_PEER_AUTO_TLS="True"

ETCD_LOG_OUTPUT="default"

ETCD_AUTO_COMPACTION_RETENTION="1"

6）将master01的/data/etcd/ssl目录同步到master02和node01上 (master02 、node01 角色为etcd)

4、 systemd配置文件
vim /usr/lib/systemd/system/etcd.service(ubuntu 18.04 的是/lib/systemd/system)
三台主机配置不一样用的时候把注释最好删除

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

EnvironmentFile=/data/etcd/cfg/etcd.conf

ExecStart=/data/etcd/bin/etcd

Restart=always

RestartSec=15

LimitNOFILE=65536

OOMScoreAdjust=-999

[Install]

WantedBy=multi-user.target

5、启动 etcd

systemctl daemon-reload

systemctl enable etcd

systemctl start etcd

systemctl status etcd

6、验证是否成功

cd /data/etcd/ssl

# 查看状态

[root@localhost ssl]# ../bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.50.24:2379" cluster-health

member fa4ab5cabfae676 is healthy: got healthy result from http://192.168.50.25:2379

member 4dc32ddff8db8fb9 is healthy: got healthy result from http://192.168.50.24:2379

member 8f173179ae8613f7 is healthy: got healthy result from http://192.168.50.26:2379

# 查看集群主机

[root@localhost ssl]# ../bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.50.24:2379" member list

fa4ab5cabfae676: name=etc1 peerURLs=http://192.168.50.25:2380 clientURLs=http://192.168.50.25:2379 isLeader=false

4dc32ddff8db8fb9: name=etc0 peerURLs=http://192.168.50.24:2380 clientURLs=http://192.168.50.24:2379 isLeader=false

8f173179ae8613f7: name=etc2 peerURLs=http://192.168.50.26:2380 clientURLs=http://192.168.50.26:2379 isLeader=true

五、安装kubeadm,kubectl和kubelet

所有节点安装 kubeadm, kubelet 。kubectl是可选的，你可以安装在所有机器上，也可以只安装在一台master1上。

1）centos 7 安装

A.添加国内yum源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64

enabled=1

gpgcheck=0

repo_gpgcheck=0

gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg

http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

B.安装

yum install -y kubelet kubeadm kubectl

C、在所有安装kubelet的节点上，将kubelet设置为开机启动

systemctl enable kubelet

2）Ubuntu 14.06安装

A。添加证书

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -

B。添加apt源

cat <<EOF >/etc/apt/sources.list.d/kubernetes.list

deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main

EOF

apt-get update

C。查看可安装版本

apt-cache madison kubelet

D。安装

apt-get install -y kubelet=1.22.4-00 kubeadm=1.22.4-00 kubectl=1.22.4-00

E。设置开机自启动

sudo systemctl enable kubelet && sudo systemctl start kubelet

六、初始化master节点以及将work节点加入集群，增加网络插件

A。在master1上将搭建etcd时生成的的ca证书和客户端证书复制到指定地点并重命名，如下

[root@master01] ~$ mkdir -p /etc/kubernetes/pki/etcd/

#etcd集群的ca证书

[root@master01] ~$ cp /data/etcd/ssl/ca.pem /etc/kubernetes/pki/etcd/

#etcd集群的client证书，apiserver访问etcd使用

[root@master01] ~$ cp /data/etcd/ssl/client.pem /etc/kubernetes/pki/apiserver-etcd-client.pem

#etcd集群的client私钥

[root@master01] ~$ cp /data/etcd/ssl/client-key.pem /etc/kubernetes/pki/apiserver-etcd-client-key.pem

#确保

[root@master01] ~$ tree /etc/kubernetes/pki/

/etc/kubernetes/pki/

├── apiserver-etcd-client-key.pem

├── apiserver-etcd-client.pem

└── etcd

└── ca.pem

1 directory, 3 files

B。创建初始化配置文件

生成默认配置文件

cd /etc/kubernetes

kubeadm config print init-defaults > kubeadm-init.yaml

最后修改后 (修改部分是什么？？？)

apiVersion: kubeadm.k8s.io/v1beta3

bootstrapTokens:

- groups:

- system:bootstrappers:kubeadm:default-node-token

token: abcdef.0123456789abcdef

ttl: 24h0m0s

usages:

- signing

- authentication

kind: InitConfiguration

localAPIEndpoint:

advertiseAddress: 192.168.50.24 #本机ip

bindPort: 6443

nodeRegistration:

criSocket: /var/run/dockershim.sock

imagePullPolicy: IfNotPresent

#imagePullPolicy: Never

name: master01 #本机hostname

#taints: null

taints:

- effect: NoSchedule

key: node-role.kubernetes.io/master

---

apiServer:

timeoutForControlPlane: 4m0s

apiVersion: kubeadm.k8s.io/v1beta3

certificatesDir: /etc/kubernetes/pki

clusterName: kubernetes

controllerManager: {}

dns: {}

#dns:

# type: CoreDNS

etcd:

#local:

# dataDir: /var/lib/etcd

external:

endpoints:

- https://192.168.50.24:2379

- https://192.168.50.25:2379

- https://192.168.50.26:2379

caFile: /etc/kubernetes/pki/etcd/ca.pem #搭建etcd集群时生成的ca证书

certFile: /etc/kubernetes/pki/apiserver-etcd-client.pem #搭建etcd集群时生成的客户端证书

keyFile: /etc/kubernetes/pki/apiserver-etcd-client-key.pem #搭建etcd集群时生成的客户端密钥

#imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

imageRepository: k8s.gcr.io

kind: ClusterConfiguration

kubernetesVersion: 1.22.4

controlPlaneEndpoint: 192.168.50.200 # vip地址

networking:

dnsDomain: cluster.local

# add podSubnet

podSubnet: 10.244.0.0/16

serviceSubnet: 10.96.0.0/12

scheduler: {}

---

apiVersion: kubeproxy.config.k8s.io/v1alpha1

kind: KubeProxyConfiguration

mode: "ipvs"

C。安装所需要要的镜像（因为k8s.gcr.io下不下来）

# 查看所需的镜像

[root@master01 kubernetes]# kubeadm config images list

k8s.gcr.io/kube-apiserver:v1.22.4

k8s.gcr.io/kube-controller-manager:v1.22.4

k8s.gcr.io/kube-scheduler:v1.22.4

k8s.gcr.io/kube-proxy:v1.22.4

k8s.gcr.io/pause:3.5

k8s.gcr.io/etcd:3.5.0-0

k8s.gcr.io/coredns/coredns:v1.8.4

#去registry.cn-hangzhou.aliyuncs.com/google_containers下载，在改变

此处有个脚本

#!/bin/bash

images=(

kube-apiserver:v1.22.4

kube-controller-manager:v1.22.4

kube-scheduler:v1.22.4

kube-proxy:v1.22.4

pause:3.5

etcd:3.5.0-0

# coredns/coredns:v1.8.4

)

for imageName in ${images[@]};

do

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName

docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName

done

docker pull coredns/coredns:1.8.4

docker tag coredns/coredns:1.8.4 k8s.gcr.io/coredns/coredns:v1.8.4

docker rmi coredns/coredns:1.8.4

ubuntu运行脚本报错

root@node27:~# sh a.sh

a.sh: 2: a.sh: Syntax error: "(" unexpected

sudo dpkg-reconfigure dash 选择no

D。执行初始化

kubeadm init --config=kubeadm-init.yaml

E.配置kubectl

要使用 kubectl来管理集群操作集群，需要做如下配置：

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

测试下，kubectl是否正常，需要注意是此时master1的notready状态是正常的，因为我们还没有部署flannel网络插件

[root@master01 kubernetes]# kubectl get node

NAME STATUS ROLES AGE VERSION

master01 NotReady control-plane,master 12h v1.22.4

F。初始化master2

1）首先将 master1 中的生成的集群共用的ca 证书，scp 到其他 master 机器。

scp -r /etc/kubernetes/pki/* 192.168.50.25:/etc/kubernetes/pki/

2）将初始化配置文件复制到master2

scp kubeadm-init.yaml 192.168.50.25:/etc/kubernetes/

3）初始化master2

修改后初始化具体修改内容根据上面的标准文件注释修改

kubeadm init --config=kubeadm-init.yaml

H。将节点加入集群（node节点）

在master01上生成加入key

kubeadm token create --print-join-command

在节点主机直接执行添加（node节点）192.168.50.240:6443是keepalived vip 地址

kubeadm join 192.168.50.240:6443 --token 6rqvgm.ne3pygx84k7hq1zl --discovery-token-ca-cert-hash sha256:24894b47c94186bd597494c34665efdbbf6686566948c4c58c50a02cc2d74e53

I。测试集群

[root@master01 kubernetes]# kubectl get nodes

NAME STATUS ROLES AGE VERSION

master01 NotReady control-plane,master 13h v1.22.4

master02 NotReady control-plane,master 12h v1.22.4

node01 NotReady <none> 12h v1.22.4

组件状态

[root@master01 kubernetes]# kubectl get cs

Warning: v1 ComponentStatus is deprecated in v1.19+

NAME STATUS MESSAGE ERROR

scheduler Healthy ok

etcd-0 Healthy {"health":"true"}

controller-manager Healthy ok

etcd-1 Healthy {"health":"true"}

etcd-2 Healthy {"health":"true"}

服务账户

[root@master01 kubernetes]# kubectl get serviceaccount

NAME SECRETS AGE

default 1 13h

集群信息，注意这里的api地址正是我们搭建的vip地址。

[root@master01 kubernetes]# kubectl cluster-info

Kubernetes control plane is running at https://192.168.50.240:6443

CoreDNS is running at https://192.168.50.240:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

J。安装网络插件

[root@master01 kubernetes]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+

podsecuritypolicy.policy/psp.flannel.unprivileged created

clusterrole.rbac.authorization.k8s.io/flannel created

clusterrolebinding.rbac.authorization.k8s.io/flannel created

serviceaccount/flannel created

configmap/kube-flannel-cfg created

daemonset.apps/kube-flannel-ds created

H。查看

kubectl get pods -n kube-system

七、搭建集群出现的问题，以及解决方式

1.初始化报错：

error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR Swap]: running with swap on is not supported. Please disable swap

解决方式：

1. 关掉swapoff

swapoff -a

2. 注释掉配置

vi /etc/fstab

#

2.kubeadm init --config=kubeadm-init.yaml后报

[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.

解决方式：

vim /etc/docker/daemon.json

#增加，后保存

{

"registry-mirrors": ["https://dockerhub.azk8s.cn", "https://docker.mirrors.ustc.edu.cn"],

"max-concurrent-downloads": 10,

"log-driver": "json-file",

"log-level": "warn",

"log-opts": {

"max-size": "10m",

"max-file": "3"

},