kubesphere 3.0集群联邦教程在虚拟机vmware上的安装和使用并且通过家用宽带申请联通外网ip实现连通 第二步 (kubeshpere管理集群安装)

系列文档

• 这里主要讲叙实现原理和步骤，较为简单的步骤就掠过了
• 第一步 集群规划+keepalived以及haproxy安装
• 第二步 kubeshpere管理集群安装以及注意点

下载安装kubeshpere(先安装prod高可用集群)

• 根据自己ip规划修改下面主机名字以及ip地址和密码
• 配置中的组件可按需开启 暂时也可以不用开启后面可修改自动安装

wget -c https://kubesphere.io/download/kubekey-v1.0.0-linux-amd64.tar.gz -O - | tar -xz

vi config-sample.yaml


apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
  name: config-sample
spec:
  hosts:
  - {name: k8s-m1, address: 192.168.3.40, internalAddress: 192.168.3.40, password: xieyyang1111}
  - {name: k8s-m2, address: 192.168.3.41, internalAddress: 192.168.3.41, password: xieyyang1111}
  - {name: k8s-m3, address: 192.168.3.42, internalAddress: 192.168.3.42, password: xieyyang1111}
  - {name: k8s-n1, address: 192.168.3.50, internalAddress: 192.168.3.50, password: xieyyang1111}
  - {name: k8s-n2, address: 192.168.3.51, internalAddress: 192.168.3.51, password: xieyyang290735}
  - {name: k8s-n3, address: 192.168.3.52, internalAddress: 192.168.3.52, password: xieyyang1111}
  roleGroups:
    etcd:
    - k8s-m1
    - k8s-m2
    - k8s-m3
    master:
    - k8s-m1
    - k8s-m2
    - k8s-m3
    worker:
    - k8s-n1
    - k8s-n2
    - k8s-n3
  controlPlaneEndpoint:
    domain: lb.kubesphere.local
    # vip
    address: "192.168.3.37"                    
    port: "6443"
  kubernetes:
    version: v1.17.9
    imageRepo: kubesphere
    clusterName: cluster.local
    masqueradeAll: false  # masqueradeAll tells kube-proxy to SNAT everything if using the pure iptables proxy mode. [Default: false]
    maxPods: 110  # maxPods is the number of pods that can run on this Kubelet. [Default: 110]
    nodeCidrMaskSize: 24  # internal network node size allocation. This is the size allocated to each node on your network. [Default: 24]
    proxyMode: ipvs  # mode specifies which proxy mode to use. [Default: ipvs]
  network:
    plugin: calico
    calico:
      ipipMode: Always  # IPIP Mode to use for the IPv4 POOL created at start up. If set to a value other than Never, vxlanMode should be set to "Never". [Always | CrossSubnet | Never] [Default: Always]
      vxlanMode: Never  # VXLAN Mode to use for the IPv4 POOL created at start up. If set to a value other than Never, ipipMode should be set to "Never". [Always | CrossSubnet | Never] [Default: Never]
      vethMTU: 1440  # The maximum transmission unit (MTU) setting determines the largest packet size that can be transmitted through your network. [Default: 1440]
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
  registry:
    registryMirrors: []
    insecureRegistries: []
    privateRegistry: ""
  storage:
    defaultStorageClass: localVolume
    localVolume:
      storageClassName: local  

---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    version: v3.0.0
spec:
  local_registry: ""
  persistence:
    storageClass: ""
  authentication:
    jwtSecret: ""
  etcd:
    monitoring: true        # Whether to install etcd monitoring dashboard
    endpointIps: 192.168.0.7,192.168.0.8,192.168.0.9  # etcd cluster endpointIps
    port: 2379              # etcd port
    tlsEnable: true
  common:
    mysqlVolumeSize: 20Gi # MySQL PVC size
    minioVolumeSize: 20Gi # Minio PVC size
    etcdVolumeSize: 20Gi  # etcd PVC size
    openldapVolumeSize: 2Gi   # openldap PVC size
    redisVolumSize: 2Gi # Redis PVC size
    es:  # Storage backend for logging, tracing, events and auditing.
      elasticsearchMasterReplicas: 1   # total number of master nodes, it's not allowed to use even number
      elasticsearchDataReplicas: 1     # total number of data nodes
      elasticsearchMasterVolumeSize: 4Gi   # Volume size of Elasticsearch master nodes
      elasticsearchDataVolumeSize: 20Gi    # Volume size of Elasticsearch data nodes
      logMaxAge: 7                     # Log retention time in built-in Elasticsearch, it is 7 days by default.
      elkPrefix: logstash              # The string making up index names. The index name will be formatted as ks-<elk_prefix>-log
      # externalElasticsearchUrl:
      # externalElasticsearchPort:
  console:
    enableMultiLogin: false  # enable/disable multiple sing on, it allows an account can be used by different users at the same time.
    port: 30880
  alerting:                # Whether to install KubeSphere alerting system. It enables Users to customize alerting policies to send messages to receivers in time with different time intervals and alerting levels to choose from.
    enabled: false
  auditing:                # Whether to install KubeSphere audit log system. It provides a security-relevant chronological set of records，recording the sequence of activities happened in platform, initiated by different tenants.
    enabled: false         
  devops:                  # Whether to install KubeSphere DevOps System. It provides out-of-box CI/CD system based on Jenkins, and automated workflow tools including Source-to-Image & Binary-to-Image
    enabled: false
    jenkinsMemoryLim: 2Gi      # Jenkins memory limit
    jenkinsMemoryReq: 1500Mi   # Jenkins memory request
    jenkinsVolumeSize: 8Gi     # Jenkins volume size
    jenkinsJavaOpts_Xms: 512m  # The following three fields are JVM parameters
    jenkinsJavaOpts_Xmx: 512m
    jenkinsJavaOpts_MaxRAM: 2g
  events:                  # Whether to install KubeSphere events system. It provides a graphical web console for Kubernetes Events exporting, filtering and alerting in multi-tenant Kubernetes clusters.
    enabled: false
  logging:                 # Whether to install KubeSphere logging system. Flexible logging functions are provided for log query, collection and management in a unified console. Additional log collectors can be added, such as Elasticsearch, Kafka and Fluentd.
    enabled: false
    logsidecarReplicas: 2
  metrics_server:                    # Whether to install metrics-server. IT enables HPA (Horizontal Pod Autoscaler).
    enabled: true
  monitoring:                        #
    prometheusReplicas: 1            # Prometheus replicas are responsible for monitoring different segments of data source and provide high availability as well.
    prometheusMemoryRequest: 400Mi   # Prometheus request memory
    prometheusVolumeSize: 20Gi       # Prometheus PVC size
    alertmanagerReplicas: 1          # AlertManager Replicas
  multicluster:
    clusterRole: none  # host | member | none  # You can install a solo cluster, or specify it as the role of host or member cluster
  networkpolicy:       # Network policies allow network isolation within the same cluster, which means firewalls can be set up between certain instances (Pods).
    enabled: false     
  notification:        # It supports notification management in multi-tenant Kubernetes clusters. It allows you to set AlertManager as its sender, and receivers include Email, Wechat Work, and Slack.
    enabled: false
  openpitrix:          # Whether to install KubeSphere App Store. It provides an application store for Helm-based applications, and offer application lifecycle management
    enabled: false
  servicemesh:         # Whether to install KubeSphere Service Mesh (Istio-based). It provides fine-grained traffic management, observability and tracing, and offer visualization for traffic topology
    enabled: false
    

使用您在上面自定义的配置文件创建集群：

./kk create cluster -f config-sample.yaml

验证多节点安装

• 通过执行以下命令来检查安装日志：
• 登录控制台
  您将能够使用默认帐户和密码admin/P@88w0rd登录控制台http://{$IP}:30880以浏览KubeSphere。登录后请更改默认密码。

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

dev 和host集群使用all-in-one安装即可，比较简单不再描述，直接查看官方文档 下一篇讲叙集群联邦配置

haproxy 的注意点

• 4层 lvs 内核转发 不和客户端建立连接 可转发大量连接和请求
• 7层转发 先和客户端建立连接，拿到url等信息，根据特定规则路由到不同服务器和服务器再建立连接，做内容交换 比如端口数6万个 则 同时客户端1和后段1一个 6万/2= 3万连接

#### 如果不用了 清理集群
```shell
swapoff -a    #关闭swap交换
kubeadm reset　　#重置K8S配置
systemctl daemon-reload &&s ystemctl restart docker 

rm -rf $HOME/.kube/config　　#删除配置文件
　
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X　　#更新iptables规则
再调用./kk 清理即可

可能会出现的问题解决排查

1. haproxy 重启提示：cannot bind socket（无法绑定socket）

问题：
haproxy 在centos上cannot bind 端口的问题
setsebool -P haproxy_connect_any=1

要不把selinux干掉也行
要临时禁用SELinux，以root用户执行下面的命令：

echo 0 > /selinux/enforce
或者，您可以使用setenforce工具，如下所示：

setenforce 0

vim /etc/selinux/config
然后将配置SELinux=enforcing改为SELinux=disabled，如下图所示。

SELINUX=disabled

然后，保存并退出文件，为了使配置生效，需要重新启动系统，然后使用sestatus命令检查SELinux的状态，如下所示：

sestatus

haproxy配置本机IP或0.0.0.0以外的IP，启动时报错，

报错信息：
[ALERT] 066/104255 (10558) : Starting frontend http-in: cannot bind socket

分析：
高可用虚IP配置后，无法启动。

经查看一篇文章：
https://www.leaseweb.com/labs/2011/09/setting-up-keepalived-on-ubuntu-load-balancing-using-haproxy-on-ubuntu-part-2/

绑定非本机的IP需要在sysctl.conf文件中配置： net.ipv4.ip_nonlocal_bind=1

vi /etc/sysctl.conf
添加：net.ipv4.ip_nonlocal_bind=1

sysctl -p

再重新启动haproxy，发现haproxy已经正常启动，端口也正常绑定。

有任何疑问都可以关注作者公众号点击菜单–>联系作者相互探讨交流