【Kubernetes】k8s使用minio作为对象存储

白山茶AI

已于 2024-03-04 09:48:22 修改

阅读量1.7k

点赞数 17

分类专栏： Kubernetes 文章标签： kubernetes linux 容器

于 2024-02-27 17:30:27 首次发布

本文链接：https://blog.csdn.net/m0_58096123/article/details/136327990

版权

Kubernetes 专栏收录该内容

4 篇文章 0 订阅

订阅专栏

在这里插入图片描述

一、部署minio operator

k8s version：v1.20.15
minio version ：v4.4.16

（1）安装kubectl-minio插件

自选minio-operaterd的版本下载包

minio-operater plugin

# 上传服务器并解压
unzip kubectl-minio_linux_amd64.zip "kubectl-minio" -d /usr/local/bin/

# 查看是否安装成功。如返回版本信息则安装成功
kubectl minio version

# 初始化
kubectl minio init

# operator默认部署到minio-operator命名空间中，如果需要指定命名空间，可使用kubectl minio init --namespace {YOUR-NAMESPACE}。
# 使用群集。在配置operator的DNS主机名时，将本地作为群集域。指定kubectl minio init --cluster域参数以设置不同的集群域值。

# 验证是否安装成功
[root@k8s-master-4 ~]# kubectl get all -n minio-operator 
NAME                                 READY   STATUS    RESTARTS   AGE
pod/console-5f4f574656-5s2wx         1/1     Running   0          5h52m
pod/minio-operator-c78cb4c65-875rt   1/1     Running   0          5h52m
pod/minio-operator-c78cb4c65-x4869   1/1     Running   0          5h52m

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/console    ClusterIP   10.98.66.116    <none>        9090/TCP,9443/TCP   5h52m
service/operator   ClusterIP   10.96.133.155   <none>        4222/TCP,4221/TCP   5h52m

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/console          1/1     1            1           5h52m
deployment.apps/minio-operator   2/2     2            2           5h52m

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/console-5f4f574656         1         1         1       5h52m
replicaset.apps/minio-operator-c78cb4c65   2         2         2       5h52m


# 访问Operator Console，新开一个终端窗口
[root@k8s-master-4 ~]# kubectl minio proxy -n minio-operator

Starting port forward of the Console UI.

To connect open a browser and go to http://localhost:9090

Current JWT to login: eyJhbGciOiJSUzI1NiIsImtpZCI6Ii1MY0QybWl6YklaUzE1MHZQNzcxcFNpc2JrUk1NdGU1X1MzaHUtY0hmMWcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb25zb2xlLXNhLXRva2VuLThwcTRtIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNvbnNvbGUtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIyMzY4ODJjZS1hMDhhLTQ0MzEtOWY5MC1iN2IxMWJkZWJmMDEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6bWluaW8tb3BlcmF0b3I6Y29uc29sZS1zYSJ9.VpR7HaUT6AU_u61AgGAe5thnXoWcPP7nSK2S7zorUXVShdKfNIO1pdvTz1XJ9X1b7hlk3Yqn6LfLz1WJXxDBhEz_pHSqJ0s-RCv00hEHGRLPzyUdI-s9zjOwwqU7yITxkyNswodiZ6jRCOiMwqPRZEHKetzMZlckK6xvIO6gpcQd0dDwXagNyHGArb8-zvBLzqWhgx0PoB3B-FHuy7VabGzVDoFdMQiIU20KOjvdQEXPPZRfk-RBMM-hy7HdCM-rk4sTYEhTGz9dUzj1_umvQbupZw7kEcw588MMF9Q2Q66hOCOC-6YbQTPJIHzqwuYNv4SOzkj5hds2eUi_1FHHDA

Forwarding from 0.0.0.0:9090 -> 9090
Handling connection for 9090

# JWT的值为后续登陆界面时使用

访问地址：http://ip:9090

二、创建StorageClass

1、sc-minio.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: minio-local-storage     # SC-NAME
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

创建

kubectl apply -f sc-minio.yaml

# 查看是否创建成功
[root@k8s-master-4 ~]# kubectl get sc
NAME                  PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
minio-local-storage   kubernetes.io/no-provisioner   Retain          WaitForFirstConsumer   false                  5h38m

2、创建所需的永久卷

序号	路径	容量 (G)	说明	所在节点
1	/data/1	5	租户使用	3个节点各1个
2	/data/log1	5	租户使用	3个节点各1个
3	/data/minio-tenant-1-log-0	5	审计日志	k8s-master-4
4	/data/minio-tenant-1-prometheus	5	监控数据	k8s-master-4

租户：30G

# 创建目录(三个节点)
mkdir -p /data/1 /data/log1

# 在K8s-master-4创建审计日志目录、监控目录
mkdir -p /data/minio-tenant-1-log-0 /data/minio-tenant-1-prometheus

三、PV && PVC

为MinIO租户中的每个卷创建一个PV。例如，假设一个Kubernetes集群有3个节点，每个节点有2个本地连接的驱动器，那么总共创建6个本地PV。名称、容量大小、目录等可根据个人环境进行配置:

1、pv-tenant.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s-master-4-data1  # PV-NAME
spec:
   capacity:
      storage: 5Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s-master-4  # NODE-NAME
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s-master-4-log1  # PV-NAME
spec:
   capacity:
      storage: 5Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/log1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s-master-4  # NODE-NAME
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s-node1-5-data1  # PV-NAME
spec:
   capacity:
      storage: 5Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s-node1-5  # NODE-NAME
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s-node1-5-log1  # PV-NAME
spec:
   capacity:
      storage: 5Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/log1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s-node1-5  # NODE-NAME

---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s-node2-6-data1  # PV-NAME
spec:
   capacity:
      storage: 5Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s-node2-6  # NODE-NAME
              
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-k8s-node2-6-log1  # PV-NAME
spec:
   capacity:
      storage: 5Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/log1  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s-node2-6  # NODE-NAME

创建审计日志PV、PVC配置文件(后续备用)：

2、pv-pvc-minio-tenant-1-log-0.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-tenant-1-log  # PV-NAME
   namespace: minio-tenant-1
spec:
   capacity:
      storage: 5Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/minio-tenant-1-log-0  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s-master-4  # NODE-NAME

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    v1.min.io/log-pg: minio-tenant-1-log
  name: minio-tenant-1-log-minio-tenant-1-log-0
  namespace: minio-tenant-1
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: "5368709120"
  storageClassName: minio-local-storage
  volumeMode: Filesystem
  volumeName: minio-tenant-1-log

创建租户监控prometheus使用的PV、PVC配置文件(后续备用)：

3、pv-pvc-minio-tenant-prometheus.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
   name: minio-tenant-1-prometheus  # PV-NAME
   namespace: minio-tenant-1
spec:
   capacity:
      storage: 5Gi  # capacity
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce  # default ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage  # SC-NAME
   local:
      path: /data/minio-tenant-1-prometheus  # SC local-path
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - k8s3-master  # NODE-NAME
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    v1.min.io/prometheus: minio-tenant-1-prometheus
  name: minio-tenant-1-prometheus-minio-tenant-1-prometheus-0
  namespace: minio-tenant-1
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: minio-local-storage  
  volumeMode: Filesystem
  volumeName: minio-tenant-1-prometheus

# 为MinIO Tenant创建命名空间
kubectl create namespace minio-tenant-1

# 创建租户pvc
 kubectl apply -f pv-tenant.yaml

# 查看创建的pv
kubectl get pv | grep minio

四、创建MinIO Tenant租户

使用kubectl minio租户创建命令创建minio租户。该命令始终使用MinIO服务器和MinIO控制台的最新稳定Docker映像。

以下示例创建了一个3节点MinIO部署，6个驱动器的总容量为30Gi。此时会同时创建审计日志、prometheus监控相关pod。

# 查看minio支持的api-resource版本
[root@k8s-master-4 ~]# kubectl api-resources 
NAME             SHORTNAMES          APIVERSION                     NAMESPACED            KIND
......
tenants          tenant              minio.min.io/v2                true                  Tenant
......


[root@k8s-master-4 ~]# kubectl minio tenant create minio-tenant-1 \
>  --servers 3 \
>  --volumes 6  \
>  --capacity 30Gi \
>  --storage-class minio-local-storage \
>  --namespace minio-tenant-1
  
Tenant 'minio-tenant-1' created in 'minio-tenant-1' Namespace


  Username: W175STC2EU3QG0KXEEUN 

  Password: BhbYGhIZamMlYQXjYOZaD8TZzJno9zNwXa0VUmrq 

  Note: Copy the credentials to a secure location. MinIO will not display these again.


APPLICATION  SERVICE NAME            NAMESPACE       SERVICE TYPE  SERVICE PORT 

MinIO        minio                   minio-tenant-1  ClusterIP     443           

Console      minio-tenant-1-console  minio-tenant-1  ClusterIP     9443


# 查看租户状态
# 当前状态在初始化过程中，等待Log Search就绪，此过程可能要花费一段时间，租户成功后，状态会显示Initialized：
[root@k8s-master-4 ~]# kubectl get tenants -n minio-tenant-1
NAME             STATE         AGE
minio-tenant-1   Initialized   19h

# 各pod、服务等对象成功状态
kubectl -n minio-tenant-1 get all

# 查看创建的pvc（如图二）
kubectl -n minio-tenant-1 get pvc

# 查看当前租户
[root@k8s-master-4 ~]# kubectl minio tenant list

Tenant 'minio-tenant-1', Namespace 'minio-tenant-1', Total capacity 30 GiB

  Current status: Initialized 
  MinIO version: minio/minio:RELEASE.2022-04-16T04-26-02Z 
  
  


# 可以使用kubectl端口转发程序临时公开每个服务。运行以下示例将流量从运行kubectl的主机转发到Kubernetes集群内运行的服务。
# 租户端口转发
[root@k8s-master-4 ~]# kubectl port-forward service/minio 443:443 -n minio-tenant-1 
Forwarding from 127.0.0.1:443 -> 9000
Forwarding from [::1]:443 -> 9000

[root@k8s-master-4 ~]# kubectl port-forward service/minio-tenant-1-console 9443:9443 -n minio-tenant-1 
Forwarding from 127.0.0.1:9443 -> 9443

现在我们进入minio的控制台查看

租户状态已经ok了

五、其他操作

# 删除minio-operater
kubectl minio delete

# 删除tenant
kubectl minio tenant delete minio-tenant-1 --namespace minio-tenant-1

# 删除pvc、pv
kubectl delete pvc --all -n minio-tenant-1 
kubectl delete pv --all -n minio-tenant-1 

# 删除命名空间
kubectl delete ns minio-tenant-1