Kubernetes上部署MinIO Operator 对接MinIO

• kubernetes v1.23.8;

• MinIO-Operator v4.5.4;

• minio RELEASE.2022-10-24T18-35-07Z。

部署环境：

序号	主机名	使用磁盘目录
1	k8s3-master	/data/1(租户)、/data/log1(租户)、 /data/minio-tenant-1-log-0(审计数据)、 /data/minio-tenant-1-prometheus(监控数据)
2	k8s3-node1	/data/1(租户)、/data/log1(租户)
3	k8s3-node2	/data/1(租户)、/data/log1(租户)

MinIO Kubernetes架构图

      当前使用3节点kubernetes集群进行部署，生产环境使用本地磁盘作为租户数据存储，测试环境以目录作为模拟。

      3节点部署，租户至少需要6块磁盘(目录)，同时还需要两个磁盘(目录)作为审计日志(audit log)和prometheus监控数据存储，当前环境提供了8个目录，租户共6个，每个10G，审计日志、监控数据各1个，每个5G。

      当前未使用Helm Chart、krew等部署MinIO，直接使用kubectl进行部署，官方使用kubectl部署目前不是很完善，官方默认创建审计日志和监控时会失败，文中已将其完善。

1. Kubernetes TLS Certificate API验证

MinIO Operator需要Kubernetes的kube-controller-manager配置文件包含如下配置:

--cluster-signing-key-file: 指定用于签名群集范围证书的PEM编码的RSA或ECDSA私钥。

--cluster-signing-cert-file: 指定用于颁发群集范围证书的PEM编码的x.509证书颁发机构证书。

验证是否包含cluster-signing-cert-file和cluster-signing-key-file参数：

kubectl get pod kube-controller-manager-$CLUSTERNAME-control-plane -n kube-system -o yaml，如：
[root@k8s3-master ~]# kubectl get pod kube-controller-manager-k8s3-master -n kube-system -o yaml
...
spec:
  containers:
  - command:
...
    - --cluster-name=kubernetes
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
...

配置中包含两个参数，说明支持Kubernetes TLS Certificate API。

2. 部署MinIO Operator

2.1 手动方式安装MinIO Kubernetes Plugin

下载kubectl-minio_4.5.4_linux_amd64并移到指定目录，可安装到多台Kubernetes node。

curl https://github.com/minio/operator/releases/download/v4.5.4/kubectl-minio_4.5.4_linux_amd64 -o kubectl-minio
chmod +x kubectl-minio
mv kubectl-minio /usr/local/bin/

查看MinIO版本：

[root@k8s3-master ~]# kubectl minio version
v4.5.4

2.2 初始化MinIO Operator

执行初始化:

[root@k8s3-master ~]# kubectl minio init
Warning: resource namespaces/minio-operator is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/minio-operator configured
serviceaccount/minio-operator created
clusterrole.rbac.authorization.k8s.io/minio-operator-role created
clusterrolebinding.rbac.authorization.k8s.io/minio-operator-binding created
customresourcedefinition.apiextensions.k8s.io/tenants.minio.min.io created
service/operator created
deployment.apps/minio-operator created
serviceaccount/console-sa created
secret/console-sa-secret created
clusterrole.rbac.authorization.k8s.io/console-sa-role created
clusterrolebinding.rbac.authorization.k8s.io/console-sa-binding created
configmap/console-env created
service/console created
deployment.apps/console created
-----------------
To open Operator UI, start a port forward using this command:

kubectl minio proxy -n minio-operator 
-----------------

说明：

operator默认部署到minio-operator命名空间中，如果需要指定命名空间，可使用kubectl minio init --namespace {YOUR-NAMESPACE}。

使用群集。在配置operator的DNS主机名时，将本地作为群集域。指定kubectl minio init --cluster域参数以设置不同的集群域值。

2.3 验证Operator安装

查看各项组件，pod状态均为Running状态，各服务运行正常。

[root@k8s3-master ~]# kubectl get all --namespace minio-operator
NAME                                  READY   STATUS    RESTARTS   AGE
pod/console-68b898c6dc-tq9z2          1/1     Running   0          35s
pod/minio-operator-56c56459bd-dlz66   1/1     Running   0          35s
pod/minio-operator-56c56459bd-hs28d   1/1     Running   0          35s

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/console    ClusterIP   10.105.211.255   <none>        9090/TCP,9443/TCP   35s
service/operator   ClusterIP   10.100.6.186     <none>        4222/TCP,4221/TCP   35s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/console          1/1     1            1           35s
deployment.apps/minio-operator   2/2     2            2           35s
NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/console-68b898c6dc          1         1         1       35s
replicaset.apps/minio-operator-56c56459bd   2         2         2       35s

2.4 打开Operator Console

Operator Console可以web界面进行管理，打开Operator Console，并记录JWT值，作为后续登录使用。

[root@k8s3-master MinIO]# kubectl minio proxy -n minio-operator
Starting port forward of the Console UI.

To connect open a browser and go to http://localhost:9090

Current JWT to login: eyJhbGciOiJSUzI1NiIsImtpZCI6IjBTYnloYVlmbkVReXZ2SG1YaG5Ed0VZYWs2dnQ4T1dlekh3ZEIyUWxCeUkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb25zb2xlLXNhLXRva2VuLXRjeGdmIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNvbnNvbGUtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxZmU3ODYyOC0yYzA0LTQ3NWUtYWQ0ZC0xNjRjYzI0OGU4MDgiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6bWluaW8tb3BlcmF0b3I6Y29uc29sZS1zYSJ9.Ypeov7kBvCb0UpHVt0c6dKmszlDR1pda30hg_L7y08I2bw5sZYZcVjc0klkndJVgnOEX0Qw2cwod9rCWnSkR5XIkPeskmIhTomei6TcfwsKPNifw4jnUA4yz_CrTXApRkP13b2jq9s0ySAGxHkiv2MMI4RC501YTrS_jLWFcPOGszze-fheagv4uYQx_j3ZLXqGQtpZM_pZCO2YYy6F3BU6FdQkW_THXtJRMq3yH-Qzk_SoT36yjuDSCmNItCP38VgT-OBbhRlagFkaPu4mT2ME-ovax8AOn10-Eqw5xslxBNaBsPzlGpG_E4wDNXx6ebYK_M1tX7g1tMEIzBg5NTg

Forwarding from 0.0.0.0:9090 -> 9090

使用浏览器登录：

http://K8s-Node-IP:9000，输入上面JWT值登录：

登录后界面：

3. 配置租户所需的存储

MinIO自动为群集中的每个卷生成一个永久卷声明（PVC）。群集必须具有相同数量的永久卷（PV）。MinIO强烈建议使用本地连接的存储以最大化性能和吞吐量。

MinIO可以使用任何支持ReadWriteOnce访问模式的Kubernetes永久卷（PV）。MinIO的一致性保证需要ReadWriteOnce提供的独占存储访问。

MinIO operator为租户中的每个卷生成一个持久卷声明（PVC），再加上两个PVC，以支持收集租户度量和日志。群集必须具有足够的持久卷，以满足租户正确启动每个PVC的容量要求。例如，部署具有16个卷的Tenant需要18（16+2）个卷。如果每个PVC要求1TB容量，那么每个PV也必须提供至少1TB的容量。

以下步骤创建必要的StorageClass和本地永久卷（PV）资源，以便每个MinIO Pod及其关联的存储都位于同一节点的本地。

如果集群已经配置了本地PV资源和供MinIO租户使用的StorageClass，则可以跳过此步骤。

3.1 为MinIO本地卷创建StorageClass

以下YAML描述了一个StorageClass，其中包含用于本地PV的适当字段：

cat >> sc-minio.yaml <<-'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: minio-local-storage     # SC-NAME
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
EOF

StorageClass必须将volumeBindingMode设置为WaitForFirstConsumer，以确保每个pod的持久卷声明（PVC）正确绑定到节点的本地PV。

创建StorageClass:

[root@k8s3-master MinIO]# kubectl apply -f sc-minio.yaml
storageclass.storage.k8s.io/minio-local-storage created

3.2 创建所需的永久卷

以下YAML使用了PV本地容，当前共3个K8s节点k8s3-master、k8s3-node1、k8s3-node2，每个节点两个目录/data/1、/data/log1，分别创建1个PV，3个节点共6个PV。

同时还需要创建审计日志、prometheus监控数据存放，这两处目前官方早期版本不需要，当前版本未提供参考，可使用文中方法实现。

磁盘驱动器(目录)规划:

序号	路径	容量	说明	所在节点
1	/data/1	10G	租户使用	3个节点各1个
2	/data/log1	10G	租户使用	3个节点各1个
3	/data/minio-tenant-1-log-0	5G	审计日志	k8s3-master
4	/data/minio-tenant-1-prometheus	5G	监控数据	k8s3-master

租户容量共60G，审计日志5G，监控数据5G。

1) 目录准备

在3个节点创建租户目录，可自定义，下面是我的目录参考，每个节点至少两个volume:

mkdir -p /data/1 /data/log1

在K8s3-master创建审计日志目录、监控目录：

mkdir -p /data/minio-tenant-1-log-0 /data/minio-tenant-1-prometheus

2) 配置PV、PVC，共8个PV/PVC，以供后续使用

为MinIO租户中的每个卷创建一个PV。例如，假设一个Kubernetes集群有3个节点，每个节点有2个本地连接的驱动器，那么总共创建6个本地PV。

名称、容量大小、目录等可根据个人环境进行配置:

cat >> pv-tenant.yaml <<-'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s3-master-data1  # PV-NAME
spec:
   capacity:
      storage: 10Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s3-master  # NODE-NAME

---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s3-master-log1  # PV-NAME
spec:
   capacity:
      storage: 10Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/log1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s3-master  # NODE-NAME

---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s3-node1-data1  # PV-NAME
spec:
   capacity:
      storage: 10Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s3-node1  # NODE-NAME

---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s3-node1-log1  # PV-NAME
spec:
   capacity:
      storage: 10Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/log1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s3-node1  # NODE-NAME

---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s3-node2-data1  # PV-NAME
spec:
   capacity:
      storage: 10Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s3-node2  # NODE-NAME

---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s3-node2-log1  # PV-NAME
spec:
   capacity:
      storage: 10Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/log1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s3-node2  # NODE-NAME
EOF

创建审计日志PV、PVC配置文件(后续备用)：

cat >> pv-pvc-minio-tenant-1-log-0.yaml <<-'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-tenant-1-log  # PV-NAME
   namespace: minio-tenant-1
spec:
   capacity:
      storage: 5Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/minio-tenant-1-log-0  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s3-master  # NODE-NAME

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    v1.min.io/log-pg: minio-tenant-1-log
  name: minio-tenant-1-log-minio-tenant-1-log-0
  namespace: minio-tenant-1
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: "5368709120"
  storageClassName: minio-local-storage
  volumeMode: Filesystem
  volumeName: minio-tenant-1-log
EOF

创建租户监控prometheus使用的PV、PVC配置文件(后续备用)：

cat >> pv-pvc-minio-tenant-prometheus.yaml <<-'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-tenant-1-prometheus  # PV-NAME
   namespace: minio-tenant-1
spec:
   capacity:
      storage: 5Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/minio-tenant-1-prometheus  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s3-master  # NODE-NAME

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    v1.min.io/prometheus: minio-tenant-1-prometheus
  name: minio-tenant-1-prometheus-minio-tenant-1-prometheus-0
  namespace: minio-tenant-1
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: minio-local-storage  
  volumeMode: Filesystem
  volumeName: minio-tenant-1-prometheus
EOF

创建租户PV：

[root@k8s3-master MinIO]# kubectl apply -f pv-tenant.yaml
persistentvolume/minio-k8s3-master-data1 created
persistentvolume/minio-k8s3-master-log1 created
persistentvolume/minio-k8s3-node1-data1 created
persistentvolume/minio-k8s3-node1-log1 created
persistentvolume/minio-k8s3-node2-data1 created
persistentvolume/minio-k8s3-node2-log1 created

当前只创建租户PV，审计日志和监控数据暂不创建，后续备用。

3) 查看创建的PV

当前已创建6个PV

[root@k8s3-master MinIO]# kubectl get pv | grep minio
minio-k8s3-master-data1                    10Gi       RWX            Retain           Available                                         minio-local-storage            13m
minio-k8s3-master-log1                     10Gi       RWX            Retain           Available                                         minio-local-storage            13m
minio-k8s3-node1-data1                     10Gi       RWX            Retain           Available                                         minio-local-storage            13m
minio-k8s3-node1-log1                      10Gi       RWX            Retain           Available                                         minio-local-storage            13m
minio-k8s3-node2-data1                     10Gi       RWX            Retain           Available                                         minio-local-storage            13m
minio-k8s3-node2-log1                      10Gi       RWX            Retain           Available                                         minio-local-storage            13m
minio-tenant-1-log                         10Gi       RWX            Retain           Available                                         minio-local-storage            13m

 

4. 部署MinIO租户(Tenants)

4.1 为MinIO Tenant创建命名空间

[root@k8s3-master ~]# kubectl create namespace minio-tenant-1
namespace/minio-tenant-1 created

4.2 创建MinIO Tenant租户

使用kubectl minio租户创建命令创建minio租户。该命令始终使用MinIO服务器和MinIO控制台的最新稳定Docker映像。

以下示例创建了一个3节点MinIO部署，6个驱动器的总容量为60Gi。此时会同时创建审计日志、prometheus监控相关pod。

kubectl minio tenant create minio-tenant-1 \
  --servers 3 \
  --volumes 6  \
  --capacity 60Gi \
  --storage-class minio-local-storage \
  --namespace minio-tenant-1

如:

[root@k8s3-master MinIO]# kubectl minio tenant create minio-tenant-1 \
>   --servers 3 \
>   --volumes 6  \
>   --capacity 60Gi \
>   --storage-class minio-local-storage \
>   --namespace minio-tenant-1

Tenant 'minio-tenant-1' created in 'minio-tenant-1' Namespace

  Username: W175STC2EU3QG0KXEEUN 
  Password: BhbYGhIZamMlYQXjYOZaD8TZzJno9zNwXa0VUmrq 
  Note: Copy the credentials to a secure location. MinIO will not display these again.

APPLICATION  SERVICE NAME            NAMESPACE       SERVICE TYPE  SERVICE PORT 
MinIO        minio                   minio-tenant-1  ClusterIP     443           
Console      minio-tenant-1-console  minio-tenant-1  ClusterIP     9443

4.3 状态查看

1) 租户状态查看

当前状态在初始化过程中，等待Log Search就绪：

[root@k8s3-master MinIO]# kubectl get tenants -n minio-tenant-1
NAME             STATE                                     AGE
minio-tenant-1   Waiting for Log Search Pods to be ready   6m12s

租户成功后，状态会显示Initialized：

[root@k8s3-master MinIO]# kubectl -n minio-tenant-1 get tenant
NAME             STATE         AGE
minio-tenant-1   Initialized   100m

2) 各pod、服务等对象成功状态

Pod均为Running状态；

deployment、replicaset、statefulset均达到READY值。

[root@k8s3-master MinIO]# kubectl -n minio-tenant-1 get all
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/minio-tenant-1-log-0                             1/1     Running   0          98m
pod/minio-tenant-1-log-search-api-5c88766db8-ckqjb   1/1     Running   0          62m
pod/minio-tenant-1-prometheus-0                      2/2     Running   0          97m
pod/minio-tenant-1-ss-0-0                            1/1     Running   0          98m
pod/minio-tenant-1-ss-0-1                            1/1     Running   0          98m
pod/minio-tenant-1-ss-0-2                            1/1     Running   0          98m

NAME                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/minio                              ClusterIP   10.109.219.112   <none>        443/TCP    99m
service/minio-tenant-1-console             ClusterIP   10.106.69.239    <none>        9443/TCP   99m
service/minio-tenant-1-hl                  ClusterIP   None             <none>        9000/TCP   99m
service/minio-tenant-1-log-hl-svc          ClusterIP   None             <none>        5432/TCP   98m
service/minio-tenant-1-log-search-api      ClusterIP   10.108.254.178   <none>        8080/TCP   98m
service/minio-tenant-1-prometheus-hl-svc   ClusterIP   None             <none>        9090/TCP   97m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/minio-tenant-1-log-search-api   1/1     1            1           98m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/minio-tenant-1-log-search-api-5c88766db8   1         1         1       98m

NAME                                         READY   AGE
statefulset.apps/minio-tenant-1-log          1/1     98m
statefulset.apps/minio-tenant-1-prometheus   1/1     97m
statefulset.apps/minio-tenant-1-ss-0         3/3     98m

4.4 问题处理

在MinIO-Operator v4.x早期版本中，只创建租户对应的pod，而没有审计日志和prometheus监控。

随之产品逐步完善，增加了这两个对象，而官方并完善底层PV创建的资料，从而导致pod/minio-tenant-1-log-0、minio-tenant-1-log-search-api-xxxxx、minio-tenant-1-prometheus-0这部分对象异常，下面就对问题进行临时处理，非最佳方案。

1) 问题现象

租户初始化后，查看pod未正常

[root@k8s3-master ~]# kubectl get pod -n minio-tenant-1 -o wide
NAME                                             READY   STATUS             RESTARTS      AGE     IP               NODE          NOMINATED NODE   READINESS GATES
minio-tenant-1-log-0                             0/1     Pending            0             2m16s   <none>           <none>        <none>           <none>
minio-tenant-1-log-search-api-5c88766db8-mcccf   0/1     CrashLoopBackOff   4 (38s ago)   2m14s   10.244.203.239   k8s3-node1    <none>           <none>
minio-tenant-1-prometheus-0                      0/2     Pending            0             76s     <none>           <none>        <none>           <none>
minio-tenant-1-ss-0-0                            1/1     Running            0             2m16s   10.244.203.209   k8s3-node1    <none>           <none>
minio-tenant-1-ss-0-1                            1/1     Running            0             2m16s   10.244.219.188   k8s3-master   <none>           <none>
minio-tenant-1-ss-0-2                            1/1     Running            0             2m16s   10.244.41.5      k8s3-node2    <none>           <none>

审计日志、日志搜索、prometheus的pod均为Running

2) 问题原因

查看租户命名空间minio-tenant-1下的pvc，状态为pengding，因PVC未就绪，导致pod不能启动。

[root@k8s3-node1 data]# kubectl get pvc -n minio-tenant-1 
NAME                                                    STATUS    VOLUME                    CAPACITY   ACCESS MODES   STORAGECLASS          AGE
0-minio-tenant-1-ss-0-0                                 Bound     minio-k8s3-node1-data1    10Gi       RWO            minio-local-storage   86s
0-minio-tenant-1-ss-0-1                                 Bound     minio-k8s3-master-data1   10Gi       RWO            minio-local-storage   86s
0-minio-tenant-1-ss-0-2                                 Bound     minio-k8s3-node2-data1    10Gi       RWO            minio-local-storage   86s
1-minio-tenant-1-ss-0-0                                 Bound     minio-k8s3-node1-log1     10Gi       RWO            minio-local-storage   86s
1-minio-tenant-1-ss-0-1                                 Bound     minio-k8s3-master-log1    10Gi       RWO            minio-local-storage   86s
1-minio-tenant-1-ss-0-2                                 Bound     minio-k8s3-node2-log1     10Gi       RWO            minio-local-storage   86s
minio-tenant-1-log-minio-tenant-1-log-0                 Pending                                                                             86s
minio-tenant-1-prometheus-minio-tenant-1-prometheus-0   Pending                                                                             26s

3) 解决方法

创建审计日志、prometheus监控需要的PV和PVC，使用前面章节预留的yaml文件创建。

#1 删除Pengding的两个PVC

[root@k8s3-master MinIO]# kubectl -n minio-tenant-1 delete pvc minio-tenant-1-log-minio-tenant-1-log-0 
persistentvolumeclaim "minio-tenant-1-log-minio-tenant-1-log-0" deleted
[root@k8s3-master MinIO]# kubectl -n minio-tenant-1 delete pvc minio-tenant-1-prometheus-minio-tenant-1-prometheus-0
persistentvolumeclaim "minio-tenant-1-prometheus-minio-tenant-1-prometheus-0" deleted

#2 使用预留的yaml重建两个PVC

[root@k8s3-master MinIO]# kubectl apply -f pv-pvc-minio-tenant-1-log-0.yaml 
persistentvolume/minio-tenant-1-log created
persistentvolumeclaim/minio-tenant-1-log-minio-tenant-1-log-0 created 
[root@k8s3-master MinIO]# kubectl apply -f pv-pvc-minio-tenant-prometheus.yaml 
persistentvolume/minio-tenant-1-prometheus created
persistentvolumeclaim/minio-tenant-1-prometheus-minio-tenant-1-prometheus-0 created

 

#3 查看重建的PVC，状态均为Bound

[root@k8s3-master MinIO]# k -n minio-tenant-1 get pvc
NAME                                                    STATUS   VOLUME                      CAPACITY   ACCESS MODES   STORAGECLASS          AGE
0-minio-tenant-1-ss-0-0                                 Bound    minio-k8s3-node1-data1      10Gi       RWO            minio-local-storage   19m
0-minio-tenant-1-ss-0-1                                 Bound    minio-k8s3-master-data1     10Gi       RWO            minio-local-storage   19m
0-minio-tenant-1-ss-0-2                                 Bound    minio-k8s3-node2-data1      10Gi       RWO            minio-local-storage   19m
1-minio-tenant-1-ss-0-0                                 Bound    minio-k8s3-node1-log1       10Gi       RWO            minio-local-storage   19m
1-minio-tenant-1-ss-0-1                                 Bound    minio-k8s3-master-log1      10Gi       RWO            minio-local-storage   19m
1-minio-tenant-1-ss-0-2                                 Bound    minio-k8s3-node2-log1       10Gi       RWO            minio-local-storage   19m
minio-tenant-1-log-minio-tenant-1-log-0                 Bound    minio-tenant-1-log          5Gi        RWO            minio-local-storage   2m39s
minio-tenant-1-prometheus-minio-tenant-1-prometheus-0   Bound    minio-tenant-1-prometheus   5Gi        RWO            minio-local-storage   14s

4) 查看Pod状态均已Running

[root@k8s3-master ~]# k get pod -o wide
NAME                                             READY   STATUS    RESTARTS        AGE   IP               NODE          NOMINATED NODE   READINESS GATES
minio-tenant-1-log-0                             1/1     Running   0               23m   10.244.219.189   k8s3-master   <none>           <none>
minio-tenant-1-log-search-api-5c88766db8-mcccf   1/1     Running   9 (7m23s ago)   23m   10.244.203.239   k8s3-node1    <none>           <none>
minio-tenant-1-prometheus-0                      2/2     Running   0               22m   10.244.219.190   k8s3-master   <none>           <none>
minio-tenant-1-ss-0-0                            1/1     Running   0               23m   10.244.203.209   k8s3-node1    <none>           <none>
minio-tenant-1-ss-0-1                            1/1     Running   0               23m   10.244.219.188   k8s3-master   <none>           <none>
minio-tenant-1-ss-0-2                            1/1     Running   0               23m   10.244.41.5      k8s3-node2    <none>           <none>

5. 配置访问服务

kubectl minio为minio租户和minio控制台创建了一个服务。kubectl minio租户创建的输出包括两个服务的详细信息。您还可以使用kubectl get-svc来检索服务名称：

[root@k8s3-master ~]# kubectl get svc -n minio-tenant-1 
NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
minio                              ClusterIP   10.109.219.112   <none>        443/TCP    173m
minio-tenant-1-console             ClusterIP   10.106.69.239    <none>        9443/TCP   173m
minio-tenant-1-hl                  ClusterIP   None             <none>        9000/TCP   173m
minio-tenant-1-log-hl-svc          ClusterIP   None             <none>        5432/TCP   172m
minio-tenant-1-log-search-api      ClusterIP   10.108.254.178   <none>        8080/TCP   172m
minio-tenant-1-prometheus-hl-svc   ClusterIP   None             <none>        9090/TCP   171m

minio服务对应于minio租户服务，应用程序应使用此服务对MinIO租户执行操作。

*-console服务对应于MinIO控制台，管理员应使用此服务访问MinIO控制台并对MinIO租户执行管理操作。

其余服务支持租户操作，不供用户或管理员使用。

默认情况下，每个服务仅在Kubernetes集群中可见，部署在集群内的应用程序可以使用cluster-IP访问服务。

Kubernetes集群外部的应用程序可以使用external-IP访问服务，该值仅为为Ingress或类似网络访问服务配置的Kubernetes集群填充。Kubernetes提供了多种配置服务外部访问的选项。

有关配置对服务的外部访问的更多完整信息，请参阅Kubernetes关于Publishing Services（ServiceTypes）和Ingress的文档。

6. 端口转发

可以使用kubectl端口转发程序临时公开每个服务。运行以下示例将流量从运行kubectl的主机转发到Kubernetes集群内运行的服务。

租户端口转发

[root@k8s3-master ~]# kubectl port-forward service/minio 443:443 -n minio-tenant-1 
Forwarding from 127.0.0.1:443 -> 9000
Forwarding from [::1]:443 -> 9000

控制台端口转发

[root@k8s3-master ~]# kubectl port-forward service/minio-tenant-1-console 9443:9443 -n minio-tenant-1 
Forwarding from 127.0.0.1:9443 -> 9443
Forwarding from [::1]:9443 -> 9443

7. 参考

http://docs.minio.org.cn/docs/
http://docs.minio.org.cn/docs/master/deploy-minio-on-kubernetes
https://min.io/docs/minio/kubernetes/upstream/operations/installation.html
https://github.com/minio/operator/blob/master/README.md
http://blog.minio.org.cn/index/news/newsdetails.html?nid=154
http://docs.minio.org.cn/minio/k8s/deployment/deploy-minio-operator.html
https://krew.sigs.k8s.io/docs/user-guide/setup/install/

转载：0195.K Kubernetes上部署MinIO Operator