minio-operator部署minio服务
本文介绍如何用minio-operator来部署minio服务,并浏览器和Java客户端来访问minio上的数据。
kubernetes环境
确保kubenetes环境是v1.19+ 环境
# kubelet 版本
[root@devops-yanxiao-1-161]# kubelet --version
Kubernetes v1.19.3
# kubectl 版本
kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
安装minio-operator
安装minio-operator有两种方式
第一种(不推荐):可以用刚才安装的krew插件来安装kubectl minio命令插件,然后用插件来安装。
安装krew插件
脚本命令安装
# 下载并安装krew
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)
# 为了操作方便也可以将命令放入PATH路径
vim /etc/profile
# 在文件最后一行加入如下脚本
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
# 当命令生效
[root@devops-yanxiao-1-161]# source /etc/profile
# 查看krew版本,看是否安装成功并生效
[root@devops-yanxiao-1-161]# kubectl krew version
OPTION VALUE
GitTag v0.4.3
GitCommit dbfefa5
IndexURI https://github.com/kubernetes-sigs/krew-index.git
BasePath /root/.krew
IndexPath /root/.krew/index/default
InstallPath /root/.krew/store
BinPath /root/.krew/bin
DetectedPlatform linux/amd64
[root@devops-yanxiao-1-161]#
[root@devops-yanxiao-1-161]# kubectl krew update
[root@devops-yanxiao-1-161]# kubectl krew install minio
警告:用这种方式安装的minio-operator在安装minio服务时,因为安装的是最新的minio-operator版本,因此可能会出现与kubernetes版本兼容的问题。
第二种(推荐):下载指定版本脚本安装
地址: https://github.com/minio/operator/tags
本文档安装的是v4.2.2版本
[root@devops-yanxiao-1-161]# wget https://github.com/minio/operator/releases/download/v4.2.2/kubectl-minio_4.2.2_linux_amd64 -O kubectl-minio
[root@devops-yanxiao-1-161]# chmod +x kubectl-minio
[root@devops-yanxiao-1-161]# mv kubectl-minio /usr/local/bin/
检查命令是否安装成功
[root@devops-yanxiao-1-161]# kubectl minio version
v4.2.2
初始化minio-operator,初始化时会将服务部署到minio-operator命名空间
[root@devops-yanxiao-1-161]# kubectl minio init
查看服务是否部署成功
[root@devops-yanxiao-1-161]# kubectl get all --namespace minio-operator
NAME READY STATUS RESTARTS AGE
pod/console-7c855c9789-k4wt2 1/1 Running 1 5d17h
pod/minio-operator-66849f98b7-7wq27 1/1 Running 1 5d17h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/console ClusterIP 10.110.18.0 <none> 9090/TCP,9443/TCP 5d17h
service/operator ClusterIP 10.110.81.114 <none> 4222/TCP,4233/TCP 5d17h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/console 1/1 1 1 5d17h
deployment.apps/minio-operator 1/1 1 1 5d17h
NAME DESIRED CURRENT READY AGE
replicaset.apps/console-7c855c9789 1 1 1 5d17h
replicaset.apps/minio-operator-66849f98b7 1 1 1 5d17h
[root@devops-yanxiao-1-161]#
安装minio tenant存储服务
tenant就是实际上的minio对象存储服务,如果minio-operator安装成功后就可以来初始化minio tenant服务了,在初始化minio tenant服务前需要创建local pv,但是官方强烈推荐MinIO DirectCSI 来支撑minio的高性能对象存储服务
DirectPV 是直接附加存储的 CSI 驱动程序。 在更简单的意义上,它是一个分布式持久卷管理器,而不是像 SAN 或 NAS 这样的存储系统。 跨服务器发现、格式化、安装、调度和监控驱动器很有用。https://github.com/minio/directpv
directpv 安装
# 安装 kubectl directpv 插件
kubectl krew install directpv
# 用这个插件初始化directpv到kubenetes集群
kubectl directpv install
# 确保directpv成功安装
kubectl directpv info
# 列表kubernetes集群所有磁盘驱动
kubectl directpv drives ls --all
# 将磁盘驱动format以及管理起来,请将drives参数和modes参数替换为kubernetes集群中实际的值
kubectl directpv drives format --drives /dev/sd{a...f} --nodes directpv-1,directpv2
directpv安装成功后,会创建storage-class
[root@devops-yanxiao-1-161 krew]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
direct-csi-min-io direct-csi-min-io Delete WaitForFirstConsumer false 3d23h
directpv-min-io direct-csi-min-io Delete WaitForFirstConsumer false 3d23h
初始化 tenant服务
# 创建namespace
[root@devops-yanxiao-1-161 krew]# kubectl create ns minio-tenant-1
# 初始化tenant集群,容量为1G,初始化命令提交后会返回访问tenant集群的账号和密码
[root@devops-yanxiao-1-161 krew]# kubectl minio tenant create minio-tenant-1 --storage-class direct-csi-min-io --namespace minio-tenant-1 --servers 1 --volumes 4 --capacity 1Gi
# 查看tenant集群创建状态,如果没查到数据请稍等一会儿
[root@devops-yanxiao-1-161 krew]# kubectl get all -n minio-tenant-1
NAME READY STATUS RESTARTS AGE
pod/minio-tenant-1-ss-0-0 1/1 Running 0 3d15h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/minio NodePort 10.100.108.23 <none> 443:30280/TCP 3d15h
service/minio-tenant-1-console NodePort 10.107.12.82 <none> 9443:32000/TCP 3d15h
service/minio-tenant-1-hl ClusterIP None <none> 9000/TCP 3d15h
NAME READY AGE
statefulset.apps/minio-tenant-1-ss-0 1/1 3d15h
[root@devops-yanxiao-1-161 krew]
外网访问tenant服务
说明: 在minio tenant服务集群创建后,minio以及minio-tenant的service type都是ClusterIp,默认只能在kubernetes集群内部访问,如果想在k8s集群外部访问需要编辑对应的svc,将type改为NodePort,并指定nodePort端口
# 修改svc type类型为NodePort,并指定端口
[root@devops-yanxiao-1-161 krew]# kubectl edit svc minio -n minio-tenant-1
# 修改svc type类型为NodePort,并指定端口
[root@devops-yanxiao-1-161 krew]# kubectl edit svc minio-tenant-1-console -n minio-tenant-1
[root@devops-yanxiao-1-161 krew]# kubectl get svc -n minio-tenant-1
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
minio NodePort 10.100.108.23 <none> 443:30280/TCP 3d16h
minio-tenant-1-console NodePort 10.107.12.82 <none> 9443:32000/TCP 3d16h
minio-tenant-1-hl ClusterIP None <none> 9000/TCP 3d16h
[root@devops-yanxiao-1-161 krew]#
说明:修改minio svc为了让客户端访问minio对象存储服务,修改minio-tenant-1-console svc 是为了能在浏览器登录管理后台访问minio 服务
浏览器访问tenant服务 https://192.168.1.162:32000/login
账号密码是在初始化minio tenant服务时,返回的
Java客户端连接tenant服务
说明:如果是浏览器访问的话,需要访问minio-tenant-1-console svc 的ip和端口,但如果是用客户端来连接minio,那么需要访问minio svc的ip和端口。如果你minio svc的type是ClusterIp,请先修改为NodePort类型。
[root@devops-yanxiao-1-161 krew]# kubectl get svc minio -n minio-tenant-1
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
minio NodePort 10.100.108.23 <none> 443:30280/TCP 3d16h
[root@devops-yanxiao-1-161 krew]#
minio Java客户端访问minio
引入依赖
<dependency>
<groupId>io.minio</groupId>
<artifactId>minio</artifactId>
<version>8.2.1</version>
</dependency>
Java代码
import io.minio.MinioClient;
import io.minio.messages.Bucket;
import okhttp3.OkHttpClient;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.cert.X509Certificate;
import java.util.List;
import java.util.Objects;
public class MinioTest {
public static OkHttpClient getUnsafeOkHttpsClient() {
try {
final TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
@Override
public void checkClientTrusted(X509Certificate[] x509Certificates, String s) {}
@Override
public void checkServerTrusted(X509Certificate[] x509Certificates, String s) {}
@Override
public X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[]{};
}
}
};
final SSLContext sslContext = SSLContext.getInstance("SSL");
sslContext.init(null, trustAllCerts, new SecureRandom());
final SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();
OkHttpClient.Builder builder = new OkHttpClient.Builder();
builder.sslSocketFactory(sslSocketFactory);
builder.hostnameVerifier((s, sslSession) -> true);
return builder.build();
} catch (NoSuchAlgorithmException | KeyManagementException e) {
e.printStackTrace();
}
return null;
}
public static void main(String[] args) throws Exception {
MinioClient admin = MinioClient.builder()
.endpoint("https://192.168.1.162:30280")
.credentials("admin", "6d24b54e-68c4-46b7-9128-a623058c7556")
.httpClient(Objects.requireNonNull(getUnsafeOkHttpsClient()))
.build();
List<Bucket> buckets = admin.listBuckets();
for (Bucket bucket : buckets) {
System.out.println(bucket.name());
}
}
}
注意:连接的ip和端口,一定要是minio svc的ip和端口