安装kubernetes高可用集群
1.安装docker
1.1 查看支持的docker版本
yum list docker-ce --showduplicates|sort -r
1.2 安装docker
yum install -y docker-ce-20.10.7-3.el7
开启自启并且启动docker
systemctl enable docker && systemctl start docker
1.3 修改docker配置
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts":["native.cgroupdriver=systemd"],
"log-driver":"json-file",
"log-opts":{
"max-size":"100M"
},
"storage-opts":[
"overlay2.override_kernel_check=true"
]
}
EOF
附上docker配置文件解释 https://www.cnblogs.com/golinux/p/12759674.html
1.4重启docker
systemctl daemon-reload && systemctl restart docker
1.5 设置网桥包经IP Tables,core文件生成路径,配置永久生效
echo """
vm.swappiness = 0
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
""" > /etc/sysctl.conf
设置配置永久生效
sysctl -p
1.6 开启ipvs
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
启动并查看状态
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
有的开启的ipvs模块更多,暂时只开这几个,启动有问题再多增加几个
到这里安装k8s基础的环境配置就完成了。
1.7 安装kubernetes
通过yum search 最新的是1.21.2版本,这里选择和1.18.2,与视频版本保持一致
yum install kubeadm-1.18.2 kubelet-1.18.2 -y
设置开机自启动
systemctl enable kubelet
2. 安装组件镜像
2.1 安装master节点镜像
创建脚本文件
vim /etc/docker/pull.sh
#!/bin/bash
# download k8s 1.15.2 images
# get image-list by 'kubeadm config images list --kubernetes-version=v1.15.2'
# gcr.azk8s.cn/google-containers == k8s.gcr.io
images=(kube-apiserver:v1.18.2
kube-controller-manager:v1.18.2
kube-scheduler:v1.18.2
kube-proxy:v1.18.2
pause:3.2
etcd:3.4.3
coredns:1.6.7
)
for imageName in ${images[@]};do
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
done
改变文件权限
chmod 777 pull.sh
拉取镜像,通过阿里云镜像加速,貌似没有在/etc/docker/daemon.json 里面配置镜像加速也很快就拉下了。有一点疑惑的是我在阿里云容器镜像服务里面查询这个容器也没有找到。
2.2安装node节点对象
与master节点对象一样,只是node节点需要少 几个服务只需要 kube-proxy:v1.18.2
pause:3.2
coredns:1.6.7
2.3 部署 keepalive + lvs 对master节点 api-server做高可用
1)部署keepalived+lvs ,在各个master节点操作
yum install -y socat keepalived ipvsadm conntrack
2)配置master1上的keepalived配置
vim /etc/keepalived/keepalived.conf
删除之前配置,粘贴以下配置
global_defs {
router_id LVS_DEVEL
}
vrrp_instance VI_1{
state BACKUP
nopreempt
interface ens33
virtual_router_id 80
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass justokk
}
virtual_ipaddress {
192.168.0.199
}
}
virtual_server 192.168.0.199 6443 {
delay_loop 6
lb_algo loadbalance
lb_kind DR
net_mask 255.255.255.0
persistence_timeout 0
protocol TCP
real_server 192.169.0.6 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connet_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.169.0.16 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connet_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.169.0.26 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connet_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
3)对master2,master3 的keepalived进行配置
仅仅只是修改优先级 priority为 80 ,60,其余保持一致
虚拟IP地址192.168.0.199在整个网段是没有主机在用的
特别提醒
keepalive需要配置backup,而且是非抢占模式nopreempt,意味着每台master都一样的,虽然有优先级的区别,但是不会出现强的转换,例如master1宕机,启动之后apiserver等组件不会立刻运行,服务还在master2或者master3上,如果此时配置为master,它会直接转移到master1上,所以不能配置成非抢占模式。
配置开机自启与启动并查看keepalive运行情况
systemctl enable keepalived && systemctl start keepalived && systemctl status keepalived
active running表示运行成功。
再使用每台主机ping一下192.168.0.199,通的,表示启动成功。
初始化k8s集群
kubeadm init \
--apiserver-advertise-address=192.168.0.6 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.18.2 \
--pod-network-cidr=10.244.0.0/16
#使用kubectl工具:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
#查询kubectl状态
kubectl get cs
-------------------------------
#需要先复制下来,node添加到集群需要用上
kubeadm join 192.168.0.6:6443 --token tnzy95.o2v3a7d5nr5k87dz \
--discovery-token-ca-cert-hash sha256:b76088384580f4dba3c58924e7e56cb44ea48cbf005d0681e548f15b558cc0d7
查看集群状态,notready
加入网络插件calico
kubectl apply -f https://raw.githubusercontent.com/luckylucky421/kubernetes1.17.3/master/calico.yaml
查看集群状态,ready
使用上面复制下来的数据,将node添加集群
至此,完成了k8s单master高可用集群的搭建
安装traefik
1)生成traefik证书,在master1上操作
mkdir ~/ikube/tls/ -p
echo """
[req]
distinguished_name = req_distinguished_name
prompt = yes
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_value = CN
stateOrProvinceName = State orProvince Name (full name)
stateOrProvinceName_value = Beijing
localityName = Locality Name (eg, city)
localityName_value =Haidian
organizationName =Organization Name (eg, company)
organizationName_value = Channelsoft
organizationalUnitName = OrganizationalUnit Name (eg, p)
organizationalUnitName_value = R & D Department
commonName = Common Name (eg, your name or your server\'s hostname)
commonName_value =*.multi.io
emailAddress = Email Address
emailAddress_value =lentil1016@gmail.com
""" > ~/ikube/tls/openssl.cnf
openssl req -newkey rsa:4096 -nodes -config ~/ikube/tls/openssl.cnf -days 3650 -x509 -out ~/ikube/tls/tls.crt -keyout ~/ikube/tls/tls.key
kubectl create -n kube-system secret tls ssl --cert ~/ikube/tls/tls.crt --key ~/ikube/tls/tls.key
2)拉取镜像(每个节点操作)
docker pull emilevauge/traefik
3)创建 traefik.yaml文件(每个节点操作)
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik-ingress-controller
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: traefik-conf
namespace: kube-system
data:
traefik.toml: |
insecureSkipVerify = true
defaultEntryPoints = ["http","https"]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
CertFile = "/ssl/tls.crt"
KeyFile = "/ssl/tls.key"
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: traefik-ingress-controller
namespace: kube-system
labels:
k8s-app: traefik-ingress-lb
spec:
selector:
matchLabels:
k8s-app: traefik-ingress-lb
name: traefik-ingress-lb
template:
metadata:
labels:
k8s-app: traefik-ingress-lb
name: traefik-ingress-lb
spec:
serviceAccountName: traefik-ingress-controller
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
terminationGracePeriodSeconds: 60
hostNetwork: true
volumes:
- name: ssl
secret:
secretName: ssl
- name: config
configMap:
name: traefik-conf
containers:
- image: emilevauge/traefik:latest
name: traefik-ingress-lb
ports:
- name: http
containerPort: 80
hostPort: 80
- name: admin
containerPort: 8080
securityContext:
privileged: true
args:
- --configfile=/config/traefik.toml
- -d
- --web
- --kubernetes
volumeMounts:
- mountPath: "/ssl"
name: "ssl"
- mountPath: "/config"
name: "config"
---
kind: Service
apiVersion: v1
metadata:
name: traefik-ingress-service
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- protocol: TCP
port: 80
name: web
- protocol: TCP
port: 8080
name: admin
- protocol: TCP
port: 443
name: https
type: NodePort
---
apiVersion: v1
kind: Service
metadata:
name: traefik-web-ui
namespace: kube-system
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- port: 80
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: traefik-web-ui
namespace: kube-system
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: ingress.multi.io
http:
paths:
- backend:
serviceName: traefik-web-ui
servicePort: 80
注意:中途在启动kubectl的时候还出现了一个问题 The connection to the server localhost:8080 was refused
将master1上的/etc/kubernetes/admin.conf文件复制其它节点,再配置上环境变量
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> /etc/profile
source /etc/profile
4)执行yaml文件,创建traefik(每个节点操作)
添加tracefik
kubectl apply -f traefik.yaml
查询tracefik是否启动成功
kubectl get pods -n kube-system
安装kubernetes-dashboard-2版本(kubernetes的web ui界面)
查看dashboard是否安装成功:
kubectl get pods -n kubernetes-dashboard
查看dashboard前端的service
kubectl get svc -n kubernetes-dashboard
修改service type类型变成NodePort:
kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard
把type: ClusterIP变成 type: NodePort,保存退出即可。
# 查询服务端口
kubectl get svc -n kubernetes-dashboard
3.1通过yaml文件里指定的默认的token登陆dashboard
1)查看kubernetes-dashboard名称空间下的secret
kubectl get secret -n kubernetes-dashboard
default-token-r9td4 kubernetes.io/service-account-token 3 5d
kubernetes-dashboard-certs Opaque 0 5d
kubernetes-dashboard-csrf Opaque 1 5d
kubernetes-dashboard-key-holder Opaque 2 5d
kubernetes-dashboard-token-blxvz kubernetes.io/service-account-token 3 5d
2)找到对应的带有token的kubernetes-dashboard-token-blxvz
kubectl describe secret kubernetes-dashboard-token-blxvz -n kubernetes-dashboard
复制token登录