(k8s)Kubernetes上部署MinIO Operator 对接MinIO

• kubernetes v1.23.8;

• MinIO-Operator v4.5.4;

• minio RELEASE.2022-10-24T18-35-07Z。

部署环境：

序号	主机名	使用磁盘目录
1	k8s3-master	/data/1(租户)、/data/log1(租户)、 /data/minio-tenant-1-log-0(审计数据)、 /data/minio-tenant-1-prometheus(监控数据)
2	k8s3-node1	/data/1(租户)、/data/log1(租户)
3	k8s3-node2	/data/1(租户)、/data/log1(租户)

MinIO Kubernetes架构图

      当前使用3节点kubernetes集群进行部署，生产环境使用本地磁盘作为租户数据存储，测试环境以目录作为模拟。

      3节点部署，租户至少需要6块磁盘(目录)，同时还需要两个磁盘(目录)作为审计日志(audit log)和prometheus监控数据存储，当前环境提供了8个目录，租户共6个，每个10G，审计日志、监控数据各1个，每个5G。

      当前未使用Helm Chart、krew等部署MinIO，直接使用kubectl进行部署，官方使用kubectl部署目前不是很完善，官方默认创建审计日志和监控时会失败，文中已将其完善。

1. Kubernetes TLS Certificate API验证

MinIO Operator需要Kubernetes的kube-controller-manager配置文件包含如下配置:

--cluster-signing-key-file: 指定用于签名群集范围证书的PEM编码的RSA或ECDSA私钥。

--cluster-signing-cert-file: 指定用于颁发群集范围证书的PEM编码的x.509证书颁发机构证书。

验证是否包含cluster-signing-cert-file和cluster-signing-key-file参数：

kubectl get pod kube-controller-manager-$CLUSTERNAME-control-plane -n kube-system -o yaml，如：
[root@k8s3-master ~]# kubectl get pod kube-controller-manager-k8s3-master -n kube-system -o yaml
...
spec:
  containers:
  - command:
...
    - --cluster-name=kubernetes
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
...

配置中包含两个参数，说明支持Kubernetes TLS Certificate API。

2. 部署MinIO Operator

2.1 手动方式安装MinIO Kubernetes Plugin

下载kubectl-minio_4.5.4_linux_amd64并移到指定目录，可安装到多台Kubernetes node。

注意：是kubectl-minio

curl https://github.com/minio/operator/releases/download/v4.5.4/kubectl-minio_4.5.4_linux_amd64 -o kubectl-minio
chmod +x kubectl-minio
mv kubectl-minio /usr/local/bin/

查看MinIO版本：

[root@k8s3-master ~]# kubectl-minio version
v4.5.4

2.2 初始化MinIO Operator

执行初始化:

[root@k8s3-master ~]# kubectl-minio init
Warning: resource namespaces/minio-operator is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/minio-operator configured
serviceaccount/minio-operator created
clusterrole.rbac.authorization.k8s.io/minio-operator-role created
clusterrolebinding.rbac.authorization.k8s.io/minio-operator-binding created
customresourcedefinition.apiextensions.k8s.io/tenants.minio.min.io created
service/operator created
deployment.apps/minio-operator created
serviceaccount/console-sa created
secret/console-sa-secret created
clusterrole.rbac.authorization.k8s.io/console-sa-role created
clusterrolebinding.rbac.authorization.k8s.io/console-sa-binding created
configmap/console-env created
service/console created
deployment.apps/console created
-----------------
To open Operator UI, start a port forward using this command:

kubectl minio proxy -n minio-operator 
-----------------

说明：

operator默认部署到minio-operator命名空间中，如果需要指定命名空间，可使用kubectl minio init --namespace {YOUR-NAMESPACE}。

使用群集。在配置operator的DNS主机名时，将本地作为群集域。指定kubectl minio init --cluster域参数以设置不同的集群域值。

2.3 验证Operator安装

查看各项组件，pod状态均为Running状态，各服务运行正常。

[root@k8s3-master ~]# kubectl get all --namespace minio-operator
NAME                                  READY   STATUS    RESTARTS   AGE
pod/console-68b898c6dc-tq9z2          1/1     Running   0          35s
pod/minio-operator-56c56459bd-dlz66   1/1     Running   0          35s
pod/minio-operator-56c56459bd-hs28d   1/1     Running   0          35s

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/console    ClusterIP   10.105.211.255   <none>        9090/TCP,9443/TCP   35s
service/operator   ClusterIP   10.100.6.186     <none>        4222/TCP,4221/TCP   35s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/console          1/1     1            1           35s
deployment.apps/minio-operator   2/2     2            2           35s
NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/console-68b898c6dc          1         1         1       35s
replicaset.apps/minio-operator-56c56459bd   2         2         2       35s

2.4 打开Operator Console

Operator Console可以web界面进行管理，打开Operator Console，并记录JWT值，作为后续登录使用。

[root@k8s3-master MinIO]# kubectl minio proxy -n minio-operator
Starting port forward of the Console UI.

To connect open a browser and go to http://localhost:9090

Current JWT to login: eyJhbGciOiJSUzI1NiIsImtpZCI6IjBTYnloYVlmbkVReXZ2SG1YaG5Ed0VZYWs2dnQ4T1dlekh3ZEIyUWxCeUkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb25zb2xlLXNhLXRva2VuLXRjeGdmIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNvbnNvbGUtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3N