基于k8s的nginx双VIP高可用集群

最新推荐文章于 2024-08-19 19:01:53 发布
最新推荐文章于 2024-08-19 19:01:53 发布
阅读量1.2k 收藏 3
点赞数
分类专栏： k8s 文章标签： nginx k8s nfs
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_54470937/article/details/120019702
版权

规划

IpRole
192.168.153.110VIP1
192.168.153.111VIP2
192.168.153.123LB
192.168.153.124LB
192.168.153.125master
192.168.153.126node1
192.168.153.127node2
192.168.153.128node3
192.168.153.129node4、NFS
192.168.153.135client

规划图
请添加图片描述

k8s集群搭建

ansible配置

vim /etc/ansible/hosts

[k8s]
192.168.153.126
192.168.153.127
192.168.153.125
192.168.153.128
192.168.153.129

[master]
192.168.153.125

[node]
192.168.153.126
192.168.153.127
192.168.153.128
192.168.153.129

配置免密

master节点上执行

ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.153.125
ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.153.126
ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.153.127
ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.153.128
ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.153.129

修改主机名

在master和node节点上修改主机名，主机名需唯一
vim /etc/hostnmae

master

vim hosts.sh

cat >> /etc/hosts <<EOF
192.168.153.125 master
192.168.153.126 node1
192.168.153.127 node2
192.168.153.128 node3
192.168.153.129 node4
EOF

允许 iptables 检查桥接流量

编写如下脚本
vim bri-change.sh

# 加载br_netfilter 模块
modprobe br_netfilter

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

ansible k8s -m script -a './bri-change.sh'

检查修改效果

ansible k8s -m shell -a 'cat /etc/sysctl.d/k8s.conf'

192.168.153.128 | CHANGED | rc=0 >>
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
192.168.153.126 | CHANGED | rc=0 >>
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
192.168.153.129 | CHANGED | rc=0 >>
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
192.168.153.125 | CHANGED | rc=0 >>
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
192.168.153.127 | CHANGED | rc=0 >>
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

安装docker

安装必要的一些系统工具

yum install -y yum-utils device-mapper-persistent-data lvm2

下载国内的源

yum-config-manager   --add-repo    https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

安装docker

yum install -y docker-ce docker-ce-cli containerd.io

启动docker

systemctl start docker

配置docker

 cat >> /etc/docker/daemon.json <<EOF
{
registry-mirrors":[
'https://registry.docker-cn.com.
"http://hub-mirror.c.163.com*,
"https://docker.mirrors.ustc.edu.cn"
]
}
EOF

安装 kubeadm、kubelet 和 kubectl

ansible k8s -m script -a './k8s-repo.sh'

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
enable=1
EOF

# 将 SELinux 设置为 permissive 模式（相当于将其禁用）
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0

sudo systemctl enable

初始化Master

初始化

 kubeadm init \
--apiserver-advertise-address=192.168.153.125 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.18.0 \
--service-cidr=10.1.0.0/16\
--pod-network-cidr=10.244.0.0/16

将apiserver-advertise-address修改为本机地址

开始集群

mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

保留节点加入的命令

kubeadm join 192.168.153.125:6443 --token z8weg1.pz9jz431d9gfndmu \
    --discovery-token-ca-cert-hash sha256:b7ba775a440aa595a1c5e3691291d8e2325d535c7397681cbc436e850ea94974

部署CNI网络

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml

kubectl apply -f kube-flannel.yml

稍微等待pod起来

查看node状态

 kubectl get node
NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   34m   v1.18.0

接入节点

使用上面保存节点加入的命令，在各node节点上执行

kubeadm join 192.168.153.125:6443 --token z8weg1.pz9jz431d9gfndmu \
    --discovery-token-ca-cert-hash sha256:b7ba775a440aa595a1c5e3691291d8e2325d535c7397681cbc436e850ea94974

在master上查看node接入情况

kubectl get node
NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   42m   v1.18.0
node1    Ready    <none>   68s   v1.18.0
node2    Ready    <none>   66s   v1.18.0
node3    Ready    <none>   64s   v1.18.0
node4    Ready    <none>   62s   v1.18.0

NFS服务器搭建

安装 nfs 服务器所需的软件包

yum install -y rpcbind nfs-utils

创建 exports 文件

/nfs_data/ *(insecure,rw,sync,no_root_squash)

启动 nfs 服务

mkdir /nfs_data

systemctl enable rpcbind
systemctl enable nfs-server

systemctl start rpcbind
systemctl start nfs-server
exportfs -r

检查配置是否生效

exportfs

# exportfs
/nfs_data    <world>

客户端测试NFS

在k8s集群的所有nfs客户端节点都需执行此命令

yum install -y nfs-utils

检查nfs服务器端是否共享

 showmount -e 192.168.153.129
Export list for 192.168.153.129:
/nfs_data *

挂载

mkdir /mnt/nfsmount
mount -t nfs 192.168.153.129:/nfs_data /mnt/nfsmount

测试是否挂载成功

客户端

echo "hello nfs server" > /mnt/nfsmount/test.txt

nfs服务端查看

cat /nfs_data/test.txt 
hello nfs server

部署nginx服务

方式：helm

使用nodeport暴露服务，将nginx数据挂载到nfs服务器上

安装helm

wget https://get.helm.sh/helm-v3.3.1-linux-amd64.tar.gz 
tar -zxvf helm-v3.3.1-linux-amd64.tar.gz       
mv linux-amd64/helm /usr/local/bin/

查看helm版本

helm version
version.BuildInfo{Version:"v3.3.1", GitCommit:"249e5215cde0c3fa72e27eb7a30e8d55c9696144", GitTreeState:"clean", GoVersion:"go1.14.7"}

helm chart结构

tree
.
├── Chart.yaml
├── README.md
├── templates
│   ├── configmap1.yaml
│   ├── configmap2.yaml
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── NOTES.txt
│   ├── pvc.yaml
│   ├── pv.yaml
│   └── service.yaml
└── values.yaml

helm chart内容

Chart

apiVersion: v2
name: nginx
description: A Helm chart for Kubernetes

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
appVersion: latest

deployment.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: {{ .Release.Name}}
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "nginx.labels" . | nindent 4 }}
    dce.daocloud.io/component: {{ include "nginx.fullname" . }}
    dce.daocloud.io/app: {{ include "nginx.fullname" . }} 
spec:
  selector:
    matchLabels:
        {{- include "nginx.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      name: {{ .Release.Name  }}
      labels: 
        {{- include "nginx.selectorLabels" . | nindent 8 }}
    spec:
      volumes:
        - name: {{ .Values.configmap1.name }}
          configMap:
            name: {{ .Values.configmap1.name }}
            defaultMode: 420
        - name: {{ .Values.configmap2.name }}
          configMap:
            name: {{ .Values.configmap2.name}}
            items:
              - key: nginx.conf
                path: nginx.conf
            defaultMode: 420
        - name: {{ .Values.nfs.name }}
          persistentVolumeClaim:
            claimName: {{ .Release.Name }}-pvc

      containers:
        - name: {{ .Release.Name }}
          image: "{{ .Values.image.repository }}:{{ .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          volumeMounts:
            - name: {{ .Values.configmap1.name }}
              mountPath: {{ .Values.volumemount.configmap1.path }} 
            - name: {{ .Values.configmap2.name }}
              mountPath: {{ .Values.volumemount.configmap2.path }}
              subPath: {{ .Values.volumemount.configmap2.subPath }}
            - name: {{ .Values.nfs.name }}
              mountPath: {{ .Values.nfs.mountpath}}

          resources: 
              {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }} 
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}

value.yaml

# Default values for nginx.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

image:
  repository: nginx
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart version.

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

service:
  type: NodePort

resources:  
   #We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
   limits:
     cpu: 100m
     memory: 128Mi
   requests:
     cpu: 100m
     memory: 128Mi


tolerations: []

affinity: {}


# pv和pvc容量
storage: 10Gi
# pv访问模式
accessModes: ReadWriteOnce


nfs:
  name: nginx-data
  #
  version: 4.1
  # 挂载到容器上的路径
  mountpath: /usr/share/nginx/html
  # nfs服务器上已存在的路径
  path: /nfs_data
  # nfs服务器地址
  server: 192.168.153.129
  pv:
    # 回收策略
    persistentVolumeReclaimPolicy: Retain

configmap1:
  # conf.d的volume名字(由小写字母数字及字符'-'组成)
  name: nginx-conf-d

configmap2:
  # nginx.conf的volume名字(由小写字母数字及字符'-'组成)
  name: nginx-conf
  # nginx.conf的volume路径
  path: /etc/nginx/nginx.conf 

volumemount:
  # conf.d的volume挂载路径
  configmap1: 
    path: /etc/nginx/conf.d
  # nginx.conf的volume挂载路径
  configmap2:
    path: /etc/nginx/nginx.conf
    subPath: nginx.conf

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name  }}
  labels:
    {{- include "nginx.labels" . | nindent 4 }}
    dce.daocloud.io/component: {{ include "nginx.fullname" . }}
    dce.daocloud.io/app: {{ include "nginx.fullname" . }}
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  selector:
    {{- include "nginx.selectorLabels" . | nindent 4 }}

  type: NodePort

configmap1

apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name  }}
  labels:
    {{- include "nginx.labels" . | nindent 4 }}
    dce.daocloud.io/component: {{ include "nginx.fullname" . }}
    dce.daocloud.io/app: {{ include "nginx.fullname" . }}
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  selector:
    {{- include "nginx.selectorLabels" . | nindent 4 }}

  type: NodePort
[root@master nginx]# cat templates/configmap1.yaml 
kind: ConfigMap
apiVersion: v1
metadata:
  name: {{ .Values.configmap1.name }}
data:
  conf.d: |
    server {
         listen       80;
         server_name  localhost;
 
         charset utf-8;
         #access_log  /var/log/nginx/log/host.access.log  main;
 
         location / {
             root   /usr/share/nginx/html;
             index  index.html index.htm;
         }
 
         #error_page  404              /404.html;
 
         # redirect server error pages to the static page /50x.html
         #
         error_page   500 502 503 504  /50x.html;
         location = /50x.html {
             root   /usr/share/nginx/html;
         }
     }

configmap2

kind: ConfigMap
apiVersion: v1
metadata:
  name: {{ .Values.configmap2.name }}
data:
 nginx.conf: |
    user nginx;
    worker_processes auto;
    error_log /var/log/nginx/error.log;
    pid /run/nginx.pid;

    # Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
    include /usr/share/nginx/modules/*.conf;

    events {
        worker_connections 1024;
    }

    http {
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                         '"$http_user_agent" "$http_x_forwarded_for"';

        access_log  /var/log/nginx/access.log  main;

        sendfile            on;
       # tcp_nopush          on;
       # tcp_nodelay         on;
        keepalive_timeout   65;
        types_hash_max_size 4096;

        include             /etc/nginx/mime.types;
        default_type        application/octet-stream;

        # Load modular configuration files from the /etc/nginx/conf.d directory.
        # See http://nginx.org/en/docs/ngx_core_module.html#include
        # for more information.
        include /etc/nginx/conf.d/*.conf;

        server {
            listen       80;
            listen       [::]:80;
            server_name  _;
            location / {
              root         /usr/share/nginx/html;
              index index.html index.htm;
              }
            # Load configuration files for the default server block.
            include /etc/nginx/default.d/*.conf;

            error_page 404 /404.html;
            location = /favicon.ico{
              log_not_found off;
            }
            location = /404.html {
            }

            error_page 500 502 503 504 /50x.html;
            location = /50x.html {
            }
           }
           include /etc/nginx/conf.d/*.conf;
       }

pv

apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ .Values.nfs.name }}-pv
spec:
  capacity:
    storage: {{ .Values.storage }}
  accessModes:
    - {{ .Values.accessModes }}
  storageClassName: nfs
  persistentVolumeReclaimPolicy: {{ .Values.nfs.pv.persistentVolumeReclaimPolicy }}
  mountOptions:
    - hard
    - nfsvers= {{- .Values.nfs.version}}
  nfs:
    path: {{ .Values.nfs.path}}
    server: {{ .Values.nfs.server }}

pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {{ .Release.Name }}-pvc
spec:
  accessModes:
    - {{ .Values.accessModes }}
  storageClassName: nfs
  volumeMode: Filesystem
  resources:
    requests:
      storage: {{ .Values.storage }}
  volumeName: {{  .Values.nfs.name }}-pv

安装nginx

helm install nginx .

查看启动结果

kubectl get po -owide
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
nginx-6848cfd5b8-gdttn   1/1     Running   0          12h   10.244.2.4   node2   <none>           <none>
nginx-6848cfd5b8-mmj2s   1/1     Running   0          12h   10.244.1.5   node1   <none>           <none>

 kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.1.0.1       <none>        443/TCP        37h
nginx        NodePort    10.1.147.111   <none>        80:31982/TCP   12h

访问

在这里插入图片描述

反向代理

keepalived实现双vip

部署keepalived

安装keepalived

在负载均衡器上安装keepalived

 yum install keepalived -y

查看版本

keepalived --version
Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

配置keepalived的参数

LB1

! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
  # vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state MASTER
    interface ens192
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.153.110
    }
}

vrrp_instance VI_2 {
    state BACKUP
    interface ens192
    virtual_router_id 52
    priority 50
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.153.111
    }
}

LB2

! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
  # vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens192
    virtual_router_id 51
    priority 50
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.153.110
    }
}

vrrp_instance VI_2 {
    state MASTER
    interface ens192
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.153.111
    }
}

开启keepalived

[root@localhost ~]systemctl start keepalived
[root@localhost ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-09-08 20:15:14 HKT; 1s ago
  Process: 9234 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 9235 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─9235 /usr/sbin/keepalived -D
           ├─9236 /usr/sbin/keepalived -D
           └─9237 /usr/sbin/keepalived -D

Sep 08 20:15:14 localhost.localdomain Keepalived_vrrp[9237]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
Sep 08 20:15:14 localhost.localdomain Keepalived_vrrp[9237]: Unable to access script `/etc/keepalived/nginx_check.sh`
Sep 08 20:15:14 localhost.localdomain Keepalived_vrrp[9237]: Disabling track script chk_nginx since not found
Sep 08 20:15:14 localhost.localdomain Keepalived_vrrp[9237]: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 08 20:15:14 localhost.localdomain Keepalived_vrrp[9237]: VRRP_Instance(VI_2) removing protocol VIPs.
Sep 08 20:15:14 localhost.localdomain Keepalived_vrrp[9237]: Using LinkWatch kernel netlink reflector...
Sep 08 20:15:14 localhost.localdomain Keepalived_vrrp[9237]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 08 20:15:14 localhost.localdomain Keepalived_vrrp[9237]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Sep 08 20:15:14 localhost.localdomain Keepalived_healthcheckers[9236]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 08 20:15:15 localhost.localdomain Keepalived_vrrp[9237]: VRRP_Instance(VI_2) Transition to MASTER STATE

查看两个LB服务器的ip

LB1

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a8:b0:b5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.153.123/24 brd 192.168.153.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet 192.168.153.110/32 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea8:b0b5/64 scope link 
       valid_lft forever preferred_lft forever

LB2

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a8:42:11 brd ff:ff:ff:ff:ff:ff
    inet 192.168.153.124/24 brd 192.168.153.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet 192.168.153.111/32 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea8:4211/64 scope link 
       valid_lft forever preferred_lft forever

发现VIP成功飘在LB上

部署nginx

安装nginx

nginx编译安装脚本

访问web服务

此处注意需要加载stream模块

安装之后查看nginx版本

nginx -v
nginx version: nginx/1.19.9

修改nginx配置

    upstream myweb{
          least_conn;
          server 192.168.153.125:30710;
          server 192.168.153.126:30710;
          server 192.168.153.127:30710;
          server 192.168.153.128:30710;
          server 192.168.153.129:30710;
         }


    server {
        listen       80;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            proxy_pass http://myweb;
            root   html;
            index  index.html index.htm;
        }

添加upstream配置

添加完之后，在location处加代理
proxy_pass http://myweb;

修改完之后刷新nginx配置

nginx -s reload

编写检测nginx存活脚本

用于keepalived定时检测nginx的服务状态，如果nginx停止了，会尝试重新启动nginx，如果启动失败，会将keepalived进程杀死，将vip漂移到备份机器上

vim nginx-check.sh
#!/bin/sh
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ]
then
  /usr/sbin/nginx
  sleep 1
  A2=`ps -C nginx --no-header |wc -l`
  if [ $A2 -eq 0 ]
  then
    systemctl stop keepalived
  fi
fi

迁移到指定目录下

cp nginx-check.sh /etc/keepalived/

Client访问vip的80端口

# curl 192.168.153.110:80
hello world
# curl 192.168.153.111:80
hello world

模拟LB1宕机

LB1关闭keepalived

systemctl stop keepalived

查看LB2的ip

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a8:42:11 brd ff:ff:ff:ff:ff:ff
    inet 192.168.153.124/24 brd 192.168.153.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet 192.168.153.111/32 scope global ens192
       valid_lft forever preferred_lft forever
    inet 192.168.153.110/32 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea8:4211/64 scope link 
       valid_lft forever preferred_lft forever

发现VIP1成功漂移到LB2上

压测

使用Apachebench进行测试

 ab -c 100 -n 10000 http://192.168.153.110:80/index.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.153.110 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.19.9
Server Hostname:        192.168.153.110
Server Port:            80

Document Path:          /index.html
Document Length:        12 bytes

Concurrency Level:      100
Time taken for tests:   10.135 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      2420000 bytes
HTML transferred:       120000 bytes
Requests per second:    986.68 [#/sec] (mean)
Time per request:       101.350 [ms] (mean)
Time per request:       1.014 [ms] (mean, across all concurrent requests)
Transfer rate:          233.18 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    5   2.0      5      17
Processing:     5   96  27.8     95     383
Waiting:        2   96  27.8     95     383
Total:          7  101  28.2    100     387

Percentage of the requests served within a certain time (ms)
  50%    100
  66%    102
  75%    103
  80%    103
  90%    106
  95%    179
  98%    192
  99%    195
 100%    387 (longest request)

排错

k8s的repo有问题

issue：k8s的repo有问题

failure: repodata/repomd.xml from kubernetes: [Errno 256] No more mirrors to try.
https://mirrors.tuna.tsinghua.edu.cn/kubernetes/yum/repos/kubernetes-el7-/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found

Public key for db7cb5cb0b3f6875f54d10f02e625573988e3e91fd4fc5eef0b1876bb18604ad-kubernetes-cni-0.8.7-0.x86_64.rpm is not installed

fix：换源

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
enable=1
EOF

镜像拉取失败

issue：镜像拉取失败

  [ERROR ImagePull]: failed to pull image registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4: output: Error response from daemon: manifest for registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4 not found: manifest unknown: manifest unknown

fix：拉取其他源的镜像，打tag成指定需要的镜像

docker pull coredns/coredns:1.8.4
docker tag 8d147537fb7d registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4
docker images
REPOSITORY                                                                    TAG       IMAGE ID       CREATED        SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.22.0   838d692cbe28   3 weeks ago    128MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.22.0   5344f96781f4   3 weeks ago    122MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.22.0   3db3d153007f   3 weeks ago    52.7MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.22.0   bbad1636b30d   3 weeks ago    104MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.5.0-0   004811815584   2 months ago   295MB
coredns/coredns                                                               1.8.4     8d147537fb7d   3 months ago   47.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   v1.8.4    8d147537fb7d   3 months ago   47.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.5       ed210e3e4a5b   5 months ago   683kB

master NotReady

issue：cni网络未准备好

kubectl get nodes
NAME     STATUS     ROLES    AGE   VERSION
master   NotReady   master   18m   v1.18.0

fix：

  • 查看pod状态
kubectl get pod -n kube-system
NAME                             READY   STATUS    RESTARTS   AGE
coredns-7ff77c879f-6l6g7         0/1     Pending   0          19m
coredns-7ff77c879f-fxjbm         0/1     Pending   0          19m
  • 查看日志
kubectl logs -n kube-system -p coredns-7ff77c879f-6l6g7

无任何日志输出

  • 查看/var/log/messages的日志
tail  /var/log/messages
Sep  2 22:18:28 master kubelet: W0902 22:18:28.358395   54235 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Sep  2 22:18:29 master kubelet: E0902 22:18:29.880901   54235 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
  • 查看事件
kubectl get events -n kube-system
LAST SEEN   TYPE      REASON              OBJECT                              MESSAGE
2m31s       Warning   FailedScheduling    pod/coredns-7ff77c879f-6l6g7        0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
2m27s       Warning   FailedScheduling    pod/coredns-7ff77c879f-fxjbm        0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

fix：部署CNI网络

Pod Status ContainerCreating

kubectl get po
NAME                         READY   STATUS              RESTARTS   AGE
nginx-869c7bb664-2fb98       1/1     Running             0          5m54s
nginx-869c7bb664-qlm56       0/1     ContainerCreating   0          5m54s
  • 查看日志
kubectl logs  nginx-869c7bb664-qlm56
Error from server (BadRequest): container "nginx" in pod "nginx-869c7bb664-qlm56" is waiting to start: ContainerCreating

没有给出问题的根源

  • 查看pod详细信息
 kubectl describe po nginx-869c7bb664-qlm56


 Warning  FailedMount  2m8s  kubelet, node3     MountVolume.SetUp failed for volume "nginx-data-pv" : mount failed: exit status 32

fix：挂载的volume同名了

需修改helm内容

nfs重复挂载

mount.nfs: Stale file handle

表示挂载到nfs服务器上的目录已经被挂载过

fix：取消此目录挂载

umount -f /mnt/nfsmount

wrong fs type, bad option, bad superblock on 192.168.153.129:/mnt/nfsmount

mount: wrong fs type, bad option, bad superblock on 192.168.153.129:/mnt/nfsmount,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

helm install nginx的时候出现此报错

issue：node节点上未安装nfs-utils

fix：安装nfs-utils

 yum install nfs-utils -y
Vivant-W
关注
专栏目录
基于 kube-vip 部署 kubernetes 高可用集群
IT
08-31 3014
kube-vip 是一个开源项目，旨在简化为 Kubernetes 集群提供负载均衡服务。kube-vip 为 Kubernetes 集群提供虚拟 IP 和负载均衡器，用于控制平面（用于构建高可用集群）和 Kubernetes 服务类型，而无需依赖LoadBalancer任何外部硬件或软件。kube-vip 的最初目的是简化高可用（HA）Kubernetes 集群的构建，当时该集群涉及一些需要管理的组件和配置。
kubeadm安装生产环境多master节点高可用集群
m0_51627713的博客
02-23 664
k8s环境规划：podSubnet（pod网段） 10.244.0.0/16serviceSubnet（service网段）: 10.96.0.0/16操作系统：centos7.6或者centos7.9配置： 4Gib内存/4vCPU/60G硬盘网络：NATkubeadm和二进制安装k8s适用场景分析kubeadm是官方提供的开源工具，是一个开源项目，用于快速搭建kubernetes集群，目前是比较方便和推荐使用的。
基于kube-vip搭建高可用集群k8s
最新发布
qq_36838700的博客
08-19 749
主机名称操作系统Centos 7Centos 7Centos 7Centos 7Centos 7内核版本IP虚拟IP————
K8S高可用集群之负载均衡器VIP（HAProxy、keepalived）
学亮编程手记
04-09 2460
访问以下地址，用户名：admin，密码：password。
搭建高可用k8s集群
qq_36838700的博客
08-13 1279
注意：标题中 没有标记（）的都是全部机器执行主机名称操作系统Centos 7Centos 7Centos 7Centos 7Centos 7内核版本IPNginx1.22.01.22.01.22.0————keepalived主从从————虚拟IP————
kubeadm+kube-vip搭建高可用k8s集群
cbmljs的博客
04-08 1957
通用 安装基本软件包 yum install bash‐comp* vim net‐tools wget ‐y 设置主机名，管理节点设置主机名为node1（node2的设置为node2） hostnamectl set-hostname node102 设置Host解析，编辑/etc/hosts文件，添加域名解析 [root@node1 ~]# vim /etc/hosts 10.168.1.101 node101 10.168.1.102 node102 10.168.1.103
k8s 简易HA
lqbz的总结
05-26 523
在前一章的基础上搭建k8s HA 环境 192.168.201.45 另一台master（apiserver、scheduler、controller-manager） 192.168.201.48 LB_master（nginx、keepalived) 192.168.201.46 LB_backup（nginx、keepalived) 另一台master #scp -r /opt/kubernetes/ root@master2_ip:/opt #在单master基础上继续，将master1上的文件拷贝
K8S高可用集群架构实现
01-27
**K8S高可用集群架构实现** Kubernetes（K8S）是当前最流行的容器编排工具，它提供了一种高效、灵活的方式来管理和部署容器化应用程序。与早期的Docker Swarm相比，K8S提供了更高级别的抽象和自动化，使得应用的可...
k8s下部署nginx集群部署方式(相关yaml文件的编写和说明)
07-31
在Kubernetes（k8s）环境下，部署Nginx集群是一项常见的任务，它涉及到容器编排、服务发现、负载均衡等多个重要概念。本篇将详细阐述如何使用YAML文件进行Nginx集群的配置和部署。 首先，让我们了解YAML文件在k8s中...
基于K8S搭建的一个多web服务的高可用集群
09-21
在本项目中，我们主要关注的是利用Kubernetes（K8S）来构建一个多web服务的高可用集群。Kubernetes是一个开源容器编排系统，能够自动化部署、扩展和管理容器化应用。结合Docker（标签中提及）作为容器化工具，K8S...
部署一套完整的K8s高可用集群（kubeadm-V1.21）
05-09
部署一套完整的 K8s 高可用集群（kubeadm-V1.21） Kubernetes 作为容器集群系统，通过健康检查+重启策略实现了 Pod 故障自我修复能力，通过调度算法实现将 Pod 分布式部署，并保持预期副本数，根据 Node 失效状态...
【云原生】kubeadm搭建企业级K8S高可用集群保姆级攻略
景天科技苑
01-04 1402
我们都知道，k8s中有三位大哥：kubelet, kubeadm, kubectl 其中： kubelet是服务，用来调用下层的container管理器，从而对底层容器进行管理。 kubectl是API，供我们调用，键入命令对k8s资源进行管理。 kubeadm是管理器，我们可以使用它进行k8s节点的管理。 今天，我们就基于kubeadm来详细讲讲怎么部署高可用K8S集群
搭建多主节点k8s高可用集群(三主两从一VIP）
weixin_53667818的博客
09-06 4485
如果使用的是阿里云，kubectl控制端不能放在master节点，因为阿里云SLB有回环问题，也就是说SLB代理的服务器不能反向访问这个问题，但是腾讯云修复了这个问题。通过这些对等连接，每个节点上的路由器可以了解整个网络的拓扑，并决定最佳路径来路由数据包。​ 当k8s集群中某个节点服务器发生故障时，Keepalived服务会自动将失效的节点从正常队列中剔除，并将请求调度到别的正常的节点服务器上，从而保证用户访问不受影响。通过定义网络策略，管理员可以限制容器之间的流量，并确保只有授权的容器才能相互通信。
部署k8s集群：
NancyLCL的博客
11-21 1428
为此，Mirantis和Docker联合创建了cri-dockerd项目，用于为Docker Engine提供一个能够支持到CRI规范的垫片，从而能够让Kubernetes基于CRI控制Docker。7. 配置ipvs功能：在kubernetes中service有两种代理模型，一种是基于iptables的，一种是基于ipvs的两者比较的话，ipvs的性能明显要高一些，但是如果要使用它，需要手动载入ipvs模块。只在master1上执行，因为master1当前是集群的leader。
K8S：二进制部署K8S（两台master+负载均衡nginx+keepalived）
Katie_ff的博客
09-04 1307
本文使用两台master+两台node+两台nginx+keepalived来搭建
Nginx配置
weixin_44602651的博客
09-17 200
序言 Nginx是lgor Sysoev为俄罗斯访问量第二的rambler.ru站点设计开发的。从2004年发布至今，凭借开源的力量，已经接近成熟与完善。 Nginx功能丰富，可作为HTTP服务器，也可作为反向代理服务器，邮件服务器。支持FastCGI、SSL、Virtual Host、URL Rewrite、Gzip等功能。并且支持很多第三方的模块扩展。 Nginx的稳定性、功能集、示例配置文件和低系统资源的消耗让他后来居上，在全球活跃的网站中有12.18%的使用比率，大约为2220万个网站。 ...
K8S------Kubernetes双Master负载均衡集群搭建
下雨天的放羊娃的博客
08-15 4544
目录前言环境准备一.Master02 节点部署1.在Master01节点上拷贝证书文件及服务管理文件2.修改Apiserver配置文件3.启动服务4.查看Node节点状态二.负载均衡部署1.安装Nginx服务2.修改Nginx配置文件，配置反向代理负载均衡3.启动Nginx服务4.部署Keepalived服务5.修改Keepalived配置文件7.创建Nginx状态检查脚本8.启动Keepalived服务9.修改两个Node节点上的配置文件10.测试三.部署 Dashboard UI1.Dashboard
k8s+kube-vip实现k8s高可用
RainMan's love
05-06 5444
kubernetes高可用集群搭建 文章目录kubernetes高可用集群搭建服务器规划安装kubeadm、kubelet、kubectl准备kube-vip相关配置和static-pod.yaml安装集群保证kube-vip的高可用 服务器规划 节点类型 IP master1 172.16.27.10 master2 172.16.27.11 master3 172.16.27.12 slave1 172.16.27.13 slave2 172.16.27.14 vip
Nginx使用keepalived配置VIP
小诸葛的博客
08-21 1762
VIP常用于负载均衡的高可用，使用VIP可以给多个主机绑定一个IP，这样，当某个负载应用挂了之后，可以自动切到另一个负载。我这里是在k8s环境中做的测试，集群中有6个节点，我给140和141两个节点配置VIP。
k8s二进制高可用集群部署
12-20
k8s二进制高可用集群部署是指在生产环境中使用k8s二进制文件手动部署高可用的k8s集群。 首先，需要准备三台或以上的服务器，并为每台服务器安装好操作系统（如Ubuntu、CentOS等）。然后，从kubernetes官网下载对应...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值