IP地址(内网) | 配置 | 软件环境 |
---|---|---|
10.0.2.4 | 4H4G+40G内存+新建虚拟硬盘动态分配64G | GlusterFS,Heketi |
10.0.2.15 | 6H11G+40G内存+新建虚拟硬盘动态分配64G | GlusterFS |
10.0.2.5 | 6H11G+40G内存+新建虚拟硬盘动态分配64G | GlusterFS |
前置环境:k8s集群ok ,helm环境ok,安装helm服务端和客户端
目标环境:Gluster环境ok,kubesphere3.0环境ok
-
# yum 安装Gluster yum install -y centos-release-gluster yum install -y glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma
加载内核
echo dm_thin_pool | sudo tee -a /etc/modules echo dm_snapshot | sudo tee -a /etc/modules echo dm_mirror | sudo tee -a /etc/modules
#启动Gluster systemctl start glusterd && systemctl enable glusterd && systemctl status glusterd
主节点执行,发现节点
gluster peer probe 10.0.2.15 gluster peer probe 10.0.2.5 gluster peer probe 10.0.2.4 gluster peer status
heketi配置
- 主节点执行
yum install heketi heketi-client -y
设置连接
ssh-keygen -f /etc/heketi/heketi_key -t rsa -N '' chown heketi:heketi /etc/heketi/heketi* ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.0.2.4 ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.0.2.15 ssh-copy-id -i /etc/heketi/heketi_key.pub root@10.0.2.5
验证
ssh -i /etc/heketi/heketi_key root@10.0.2.5
如果添加的volume小的话可能会提示No Space,要解决这一问题要在heketi.json添加"brick_min_size_gb" : 1 ,1为1G
cat <<EOF> /etc/heketi/heketi.json { "port": "18080", "use_auth": false, "jwt": { "admin": { "key": "adminSecret" }, "user": { "key": "userSecret" } }, "glusterfs": { "executor": "ssh", "sshexec": { "keyfile": "/etc/heketi/heketi_key", "user": "root", "port": "22", "fstab": "/etc/fstab" }, "brick_min_size_gb" : 1, "db": "/var/lib/heketi/heketi.db", "loglevel" : "debug" } } EOF
systemctl start heketi && systemctl enable heketi && systemctl status heketi
使用拓扑创建存储空间
cat <<EOF> /etc/heketi/topolgy_demo.json { "clusters": [ { "nodes": [ { "node": { "hostnames": { "manage": [ "10.0.2.4" ], "storage": [ "10.0.2.4" ] }, "zone": 1 }, "devices": [ "/dev/sdb" ] }, { "node": { "hostnames": { "manage": [ "10.0.2.15" ], "storage": [ "10.0.2.15" ] }, "zone": 1 }, "devices": [ "/dev/sdb" ] }, { "node": { "hostnames": { "manage": [ "10.0.2.5" ], "storage": [ "10.0.2.5" ] }, "zone": 1 }, "devices": [ "/dev/sdb" ] } ] } ] } EOF
export HEKETI_CLI_SERVER=http://10.0.2.4:18080 echo $HEKETI_CLI_SERVER heketi-cli topology load --json=/etc/heketi/topolgy_demo.json
#创建一个StorageClass cat <<EOF> gluster-storage-class.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: glusterfs #-------------存储类的名字 provisioner: kubernetes.io/glusterfs parameters: resturl: "http://10.0.2.4:18080" #-------------heketi 的ip 和端口 restuser: "admin" #-------------heketi的认证用户,这里随便填,因为没有启用鉴权模式 gidMin: "40000" gidMax: "50000" restauthenabled: "true" volumetype: "replicate:3" #-------------申请的默认为3副本模式,因为目前有三个gluster节点。 EOF
kubectl apply -f gluster-storage-class.yaml
设置默认存储
kubectl patch storageclass glusterfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl get sc
安装kubesphere3.0(最小化安装)
kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.0.0/kubesphere-installer.yaml kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.0.0/cluster-configuration.yaml
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
下一步启动devops和告警通知组件
-
参考文档:https://www.cnblogs.com/breezey/p/8849466.html,https://www.cnblogs.com/fan-gx/p/12101686.html
-
注意事项
- master节点是主节点,是调度的,不需要太高的内存和处理器
- node2和node3调整到11g,cpu6核
- devops:开启,jekinsmemoryReq大小为1000M,sonar开启
- notification:开启
- alert: 开启
- 在启用这三个插件时,ks-minio这个服务会一直创建2个工作负载(这是有问题的,配置文件里面只有1的副本),这个问题会导致无限重启,需要手动删除即可
kubectl -n kubesphere-system delete job minio-make-bucket-job
kubectl get pod minio-make-bucket-job-jvsrn -n kubesphere-system -o yaml | kubectl replace --force -f -
这个问题最后实在没法解决重启虚拟机就好了,真是奇怪