Gluster安装+kubesphere3.0

环境

IP地址(内网)	配置	软件环境
10.0.2.4	4H4G+40G内存+新建虚拟硬盘动态分配64G	GlusterFS,Heketi
10.0.2.15	6H11G+40G内存+新建虚拟硬盘动态分配64G	GlusterFS
10.0.2.5	6H11G+40G内存+新建虚拟硬盘动态分配64G	GlusterFS

前置环境:k8s集群ok ,helm环境ok,安装helm服务端和客户端

目标环境:Gluster环境ok,kubesphere3.0环境ok

1. # yum 安装Gluster
   yum install -y centos-release-gluster 
   yum install -y glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma

   加载内核 

   echo dm_thin_pool | sudo tee -a /etc/modules
   echo dm_snapshot | sudo tee -a /etc/modules
   echo dm_mirror | sudo tee -a /etc/modules

    

   #启动Gluster
   systemctl start glusterd && systemctl enable glusterd && systemctl status glusterd

   主节点执行,发现节点

   gluster peer probe 10.0.2.15
   gluster peer probe 10.0.2.5
   gluster peer probe 10.0.2.4
   
   gluster peer status

   heketi配置

2. 主节点执行

   yum install heketi heketi-client -y

    设置连接

   ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
   chown heketi:heketi  /etc/heketi/heketi*
   ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.0.2.4
   ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.0.2.15
   ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.0.2.5

   验证

   ssh -i /etc/heketi/heketi_key root@10.0.2.5

   如果添加的volume小的话可能会提示No Space，要解决这一问题要在heketi.json添加"brick_min_size_gb" : 1 ，1为1G

   cat <<EOF> /etc/heketi/heketi.json
   {
     "port": "18080",
     "use_auth": false,
   
     "jwt": {
       "admin": {
         "key": "adminSecret"
       },
       "user": {
         "key": "userSecret"
       }
     },
   
     "glusterfs": {
       "executor": "ssh",
       "sshexec": {
         "keyfile": "/etc/heketi/heketi_key",
         "user": "root",
         "port": "22",
         "fstab": "/etc/fstab"
       },
       "brick_min_size_gb" : 1,
       "db": "/var/lib/heketi/heketi.db",
       "loglevel" : "debug"
     }
   }
   EOF

   systemctl start heketi && systemctl enable heketi && systemctl status heketi

   使用拓扑创建存储空间 

   cat <<EOF> /etc/heketi/topolgy_demo.json
   {
     "clusters": [
       {
         "nodes": [
            {
              "node": {
                  "hostnames": {
                      "manage": [
                         "10.0.2.4"
                       ],
                      "storage": [
                      "10.0.2.4"
                       ]
                  },
             "zone": 1
            },
            "devices": [
               "/dev/sdb"
                 ]
              },
   
            {  "node": {
                  "hostnames": {
                      "manage": [
                         "10.0.2.15"
                       ],
                      "storage": [
                      "10.0.2.15"
                       ]
                  },
             "zone": 1
            },
            "devices": [
               "/dev/sdb"
                 ]
              },
            { "node": {
                  "hostnames": {
                      "manage": [
                         "10.0.2.5"
                       ],
                      "storage": [
                      "10.0.2.5"
                       ]
                  },
             "zone": 1
            },
            "devices": [
               "/dev/sdb"
                 ]
              }     
           ]
        }
      ]
   }
   EOF

    

    

   export HEKETI_CLI_SERVER=http://10.0.2.4:18080
   echo $HEKETI_CLI_SERVER
   heketi-cli topology load --json=/etc/heketi/topolgy_demo.json

   #创建一个StorageClass
   cat <<EOF> gluster-storage-class.yaml
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: glusterfs                       #-------------存储类的名字
   provisioner: kubernetes.io/glusterfs
   parameters:
     resturl: "http://10.0.2.4:18080"      #-------------heketi 的ip 和端口
     restuser: "admin"                           #-------------heketi的认证用户，这里随便填，因为没有启用鉴权模式 
     gidMin: "40000"
     gidMax: "50000"
     restauthenabled: "true"
     volumetype: "replicate:3"                 #-------------申请的默认为3副本模式，因为目前有三个gluster节点。
   EOF
   

   kubectl apply -f  gluster-storage-class.yaml

   设置默认存储 

   kubectl patch storageclass glusterfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

   kubectl get sc

   安装kubesphere3.0(最小化安装)

   kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.0.0/kubesphere-installer.yaml
      
   kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.0.0/cluster-configuration.yaml
   

   kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
   

    

   下一步启动devops和告警通知组件

3. 参考文档:https://www.cnblogs.com/breezey/p/8849466.html,https://www.cnblogs.com/fan-gx/p/12101686.html

4. 注意事项

   1. master节点是主节点，是调度的，不需要太高的内存和处理器
   2. node2和node3调整到11g，cpu6核
   3. devops：开启，jekinsmemoryReq大小为1000M，sonar开启
   4. notification：开启
   5. alert： 开启
5. 在启用这三个插件时,ks-minio这个服务会一直创建2个工作负载(这是有问题的,配置文件里面只有1的副本),这个问题会导致无限重启,需要手动删除即可

   kubectl -n kubesphere-system delete job minio-make-bucket-job
   

   kubectl get pod minio-make-bucket-job-jvsrn -n kubesphere-system -o yaml | kubectl replace --force -f -
   

   这个问题最后实在没法解决重启虚拟机就好了,真是奇怪