【数据平台】Minio-超大文件类断点续传实战

我们都知道，MinIO 是一个基于Apache License v2.0开源协议的对象存储服务文件系统。它基于云原生文件系统生态，兼容亚马逊S3云存储服务接口，非常适合于存储大容量非结构化的数据。例如：视频、图片、数据文件、日志文件、备份数据和容器/虚拟机镜像等，而一个对象文件可以是任意大小，文件大小从几kb到最大5T不等。本节将说明如何将大文件续传到minio文件服务器。

putObject(String bucketName, String objectName, InputStream stream, long size, String contentType, SecretKey key)
public void putObject(String bucketName, String objectName, InputStream stream, long size, String contentType, SecretKey key)

通过文件路径拿到流的数据，使用随机生成的content key进行加密，并上传到已指定的存储桶中。同时将加密后的content key和iv做为加密对象有header也上传到存储桶中。content key使用传入到该方法的master key进行加密。如果上传对象大于5MB，客户端会自动进行multi part上传。

参数说明：

示例：

`引入相关依赖`
<dependency>
  <groupId>io.minio</groupId>
  <artifactId>minio</artifactId>
  <version>7.1.4</version>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot</artifactId>
</dependency>

`application.yml文件配置`
minio:
  access-key: minio
  security-key: minio
  endpoint: http://127.0.0.1:9000
  bucket-name: bucketMinio
  path-name: pathDir/

@ConfigurationProperties(prefix = "minio")
@Component
@Data
public class MinioProperties {

    private String accessKey;

    private String securityKey;

    private String endpoint;

    private String bucketName;

    private String pathName;
}

`实战代码`
public class BreakpointContinue{
    /*** file part size*/
    private static final int FILE_PART_SIZE = 1024 * 1024 * 10;
    private final MinioProperties properties;
    private MinioClient minioClient;

    public MinioLoaderTask(MinioProperties properties) {
        this.properties = properties;
        minioClient = MinioClient.builder()
                .credentials(properties.getAccessKey(), properties.getSecurityKey())
                .endpoint(properties.getEndpoint())
                .build();
    }
    
    /**
     * 文件上传
     * @param dataFilePath 文件路径
     */
    public void breakpointUpload(String dataFilePath) {
        try {
            boolean isExist = minioClient.bucketExists(BucketExistsArgs.builder().bucket(properties.getBucketName()).build());
            if (!isExist) {
                throw new IllegalArgumentException("bucket is not exist :" + properties.getBucketName());
            }
            long objectSize = Files.size(Paths.get(dataFilePath));
            String objectName = properties.getPathName() + getFileName(dataFilePath);
            try (InputStream in = new FileInputStream(dataFilePath)) {
                minioClient.putObject(PutObjectArgs.builder()
                        .bucket(properties.getBucketName())
                        .object(objectName)
                        .stream(in, objectSize, FILE_PART_SIZE)
                        .build());
            }
        } catch (Exception e) {
            log.error("上传文件异常", e);
        }
    }

    private static String getFileName(String filePath) {
        int lastIndex = filePath.lastIndexOf(File.separator);
        if (lastIndex > -1) {
            return filePath.substring(lastIndex + 1);
        }
        return null;
    }
}