506. 【kubernetes】在-k8s-集群上部署-Minio-Operator-和-Minio-Plugin

说明

• 此篇的k8s基础环境搭建方法基于前几篇文章。
• MinIO Kubernetes Plugin提供了使用kubectl MinIO命令在Kubernetes集群上部署和管理MinIO租户的本地支持。
• MinIO Operator 自动生成TLS证书签名请求(Certificate Signing Requests, CSR)，并使用Kubernetes certificates.k8s.ioTLS证书管理API，用于创建已签名的TLS证书。

一、kube-controller-manager 支持创建证书

修改 /etc/kubernetes/controller-manager添加下面两行：

--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt \
--cluster-signing-key-file=/etc/kubernetes/pki/ca.key \

修改后的文件内容如下：

重启 kube-controller-manager服务

systemctl restart kube-controller-manager

二、安装 minio operator 插件

1. 下载最新版的 minio operator 可执行文件：

wget https://github.com/minio/operator/releases/download/v4.5.4/kubectl-minio_4.5.4_linux_amd64

2. 重命名

[root@k8s0 workspace_minio]#  mv kubectl-minio_4.5.4_linux_amd64 kubectl-minio

3. 添加执行权限

[root@k8s0 workspace_minio]#  chmod +x kubectl-minio 

4. 移动到 /usr/local/bin/

[root@k8s0 workspace_minio]#  cp kubectl-minio /usr/local/bin/

5. 验证：

[root@k8s0 workspace_minio]#  kubectl minio version
v4.5.4

• 可以看到，minio operator 已经安装成功。

三、初始化 minio operator

1. 执行 kubectl minio init命令初始化

[root@k8s0 workspace_minio]#  kubectl minio init
namespace/minio-operator created
serviceaccount/minio-operator created
clusterrole.rbac.authorization.k8s.io/minio-operator-role created
clusterrolebinding.rbac.authorization.k8s.io/minio-operator-binding created
customresourcedefinition.apiextensions.k8s.io/tenants.minio.min.io created
service/operator created
deployment.apps/minio-operator created
serviceaccount/console-sa created
secret/console-sa-secret created
clusterrole.rbac.authorization.k8s.io/console-sa-role created
clusterrolebinding.rbac.authorization.k8s.io/console-sa-binding created
configmap/console-env created
service/console created
deployment.apps/console created
-----------------

To open Operator UI, start a port forward using this command:

kubectl minio proxy -n minio-operator 

-----------------

2. 运行 kubectl minio proxy -n minio-operator 命令临时将流量从minio操作员控制台服务转发到本地机器

[root@k8s0 workspace_minio]#  kubectl minio proxy -n minio-operator 
Starting port forward of the Console UI.

To connect open a browser and go to http://localhost:9090

Current JWT to login: eyJhbGciOiJSUzI1NiIsImtpZCI6Inl6cFRBM25HUTJrNUNsck1CR2FpNmVlaHlNaWVHV1lVdEkwOGtrZjBMckkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb25zb2xlLXNhLXNlY3JldCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjb25zb2xlLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZmQ5ZTZjYjMtMDBiOS00NTA5LTgzMzctMzBlNjUyZWNlMDU2Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om1pbmlvLW9wZXJhdG9yOmNvbnNvbGUtc2EifQ.djbnoBplYXwaV2dfUC0BqSZ1rCRFP85SxZdHy3Z1l0rzvKpcUn_B4O6jg6gt4noJh2c8712t_vsHTnlgUKu4ptif2vVFfcCzX21G6ILcZ-X0b6PzQ58rFeFG6RzB-0E1TZXEcaXWrXfXoplwywtVscdyvDNXSXCvwSquyrHs4o4oP2nsxhL0OLblLFT2RiA-6jpwvnaWUEBaO3paZzyvCiUcBoPgQd2HHHRih2MXRWG72w3lCjICa8z60ihKq3NTqbTB5HUJouRoDluI5ykYcajMZ6sXMDFEDG30Wa5q5NE_pXH9XtMOBFXlFiuJZhfq72aD3-snlaGoDQajMuZ_ng

Forwarding from 0.0.0.0:9090 -> 9090

• 这里其实就是创建了一个登录token，并且将临时暴露端口，相当于集成了kubectl create token和 kubectl port-forward 两条命名。
• 复制 token，浏览器输入机器IP:9090，并在登录界面输入 token，即可登录
  
  

四、安装 DirectPV plugin 插件

1. 通过 krew 安装 directpv

[root@k8s0 workspace_krew]#  kubectl krew install directpv
Updated the local copy of plugin index.
Installing plugin: directpv



Installed plugin: directpv
\
 | Use this plugin:
 |      kubectl directpv
 | Documentation:
 |      https://github.com/minio/directpv
/
WARNING: You installed plugin "directpv" from the krew-index plugin repository.
   These plugins are not audited for security by the Krew maintainers.
   Run them at your own risk.

2. 在 kubernetes 安装 directpv 插件

[root@k8s0 workspace_krew]#  kubectl directpv install
I1125 10:32:13.441654   11816 ns.go:65] 'direct.csi.min.io' namespace created
I1125 10:32:13.458859   11816 rbac.go:352] 'direct.csi.min.io' rbac created
I1125 10:32:13.542620   11816 crd.go:135] crds successfully registered
I1125 10:32:13.556357   11816 csidriver.go:148] 'direct.csi.min.io' csidriver created
I1125 10:32:13.583081   11816 storageclass.go:48] 'direct.csi.min.io' storageclass created
I1125 10:32:13.595027   11816 service.go:38] 'direct.csi.min.io' service created
I1125 10:32:13.610337   11816 daemonset.go:41] 'direct.csi.min.io' daemonset created
I1125 10:32:13.632032   11816 deployment.go:289] 'direct.csi.min.io' deployment created

3. 验证安装是否成功

[root@k8s0 tmp]#  kubectl directpv info
 NODE             CAPACITY  ALLOCATED  VOLUMES  DRIVES 
 • 192.168.3.100  -         -          -        -      
 • 192.168.3.101  -         -          -        -      
 • 192.168.3.102  -         -          -        -      
[root@k8s0 tmp]#  date
Sat Nov 26 11:19:07 CST 2022

4. 列出集群中可用的驱动器

[root@k8s0 tmp]#  kubectl directpv drives ls
 DRIVE      CAPACITY  ALLOCATED  FILESYSTEM  VOLUMES  NODE           ACCESS-TIER  STATUS      
 /dev/dm-1  4.0 GiB   -          swap        -        192.168.3.100  -            Available   
 /dev/sdb2  1.4 TiB   -          -           -        192.168.3.100  -            Available   
 /dev/dm-1  4.0 GiB   -          swap        -        192.168.3.101  -            Available   
 /dev/sdb2  1.4 TiB   -          -           -        192.168.3.101  -            Available   
 /dev/dm-1  4.0 GiB   -          swap        -        192.168.3.102  -            Available   
 /dev/sdb2  1.4 TiB   -          -           -        192.168.3.102  -            Available   

5. 选择 directpv 应该管理和格式化的驱动器

kubectl directpv drives format --drives /dev/sdb2

• 该命令标识，格式化所有节点上的 /dev/sdb2

6. 验证

[root@k8s0 tmp]#  kubectl directpv drives ls
 DRIVE      CAPACITY  ALLOCATED  FILESYSTEM  VOLUMES  NODE           ACCESS-TIER  STATUS      
 /dev/dm-1  4.0 GiB   -          swap        -        192.168.3.100  -            Available   
 /dev/sdb2  1.4 TiB   -          xfs         -        192.168.3.100  -            Ready       
 /dev/dm-1  4.0 GiB   -          swap        -        192.168.3.101  -            Available   
 /dev/sdb2  1.4 TiB   -          xfs         -        192.168.3.101  -            Ready       
 /dev/dm-1  4.0 GiB   -          swap        -        192.168.3.102  -            Available   
 /dev/sdb2  1.4 TiB   -          xfs         -        192.168.3.102  -            Ready   

• 可以看到所有节点上的 /dev/sdb2已经格式化完毕了。

7. OK，现在可以 directpv现在可以被指定为PodSpec.VolumeClaimTemplates中的存储类

五、创建高可用 minio 集群

现在可以通过第三步里的“Create”按钮创建集群了。