SpringBoot通过Minio实现大文件分片上传

最近开发功能需要大文件上传，正常会在业务代码中封装一层minio的上传，为了避免中转，实现了minio直传的方式。实现过程中也确实碰到了不少坑，只能查询文档，本着开箱即用，简单方便的原则，java版本的Minio sdk默认是不允许单独调用分片的相关方法，但是升级到8.0.3后可以通过继承MinioClient实现分片方法的使用，后来又碰到官方Minio版本的bug，提交issue，还好Minio github开发者处理非常迅捷，再次感谢，2021-02-04修复的，使用的开发者需要注意下。 大致描述下流程：

1. 用户调用初始化接口，后端调用minio初始化，得到uploadId，生成每个分片的minio上传url
2. 用户调用对应分片的上传地址，多次上传会覆盖
3. 调用完成接口，后端查询所有上传的分片并合并

• 自定义minioClient

public class CustomMinioClient extends MinioClient {

    protected CustomMinioClient(MinioClient client) {
        super(client);
    }

    public String initMultiPartUpload(String bucket, String region, String object, Multimap<String, String> headers, Multimap<String, String> extraQueryParams) throws IOException, InvalidKeyException, NoSuchAlgorithmException, InsufficientDataException, ServerException, InternalException, XmlParserException, InvalidResponseException, ErrorResponseException {
        CreateMultipartUploadResponse response = this.createMultipartUpload(bucket, region, object, headers, extraQueryParams);

        return response.result().uploadId();
    }

    public ObjectWriteResponse mergeMultipartUpload(String bucketName, String region, String objectName, String uploadId, Part[] parts, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams) throws IOException, InvalidKeyException, NoSuchAlgorithmException, InsufficientDataException, ServerException, InternalException, XmlParserException, InvalidResponseException, ErrorResponseException {

        return this.completeMultipartUpload(bucketName, region, objectName, uploadId, parts, extraHeaders, extraQueryParams);
    }

    public ListPartsResponse listMultipart(String bucketName, String region, String objectName, Integer maxParts, Integer partNumberMarker, String uploadId, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams) throws NoSuchAlgorithmException, InsufficientDataException, IOException, InvalidKeyException, ServerException, XmlParserException, ErrorResponseException, InternalException, InvalidResponseException {
        return this.listParts(bucketName, region, objectName, maxParts, partNumberMarker, uploadId, extraHeaders, extraQueryParams);
    }
}

• 后端处理类

public Map<String, Object> initMultiPartUpload(String bucketName, String objectName, int totalPart) {
        Map<String, Object> result = new HashMap<>();
        try {
            String uploadId = customMinioClient.initMultiPartUpload(bucketName, null, objectName, null, null);

            result.put("uploadId", uploadId);
            List<String> partList = new ArrayList<>();

            Map<String, String> reqParams = new HashMap<>();
            //reqParams.put("response-content-type", "application/json");
            reqParams.put("uploadId", uploadId);
            for (int i = 1; i <= totalPart; i++) {
                reqParams.put("partNumber", String.valueOf(i));
                String uploadUrl = minioClient.getPresignedObjectUrl(
                        GetPresignedObjectUrlArgs.builder()
                                .method(Method.PUT)
                                .bucket(bucketName)
                                .object(objectName)
                                .expiry(1, TimeUnit.DAYS)
                                .extraQueryParams(reqParams)
                                .build());
                partList.add(uploadUrl);
            }
            result.put("uploadUrls", partList);
        } catch (Exception e) {
            logger.error("error: {}", e.getMessage(), e);
            return null;
        }

        return result;
    }

    public boolean mergeMultipartUpload(String bucketName, String objectName, String uploadId) {
        try {
            Part[] parts = new Part[1000];
            //此方法注意2020.02.04之前的minio服务端有bug
            ListPartsResponse partResult = customMinioClient.listMultipart(bucketName, null, objectName, 1000, 0, uploadId, null, null);
            int partNumber = 1;
            for (Part part : partResult.result().partList()) {
                parts[partNumber - 1] = new Part(partNumber, part.etag());
                partNumber++;
            }
            customMinioClient.mergeMultipartUpload(bucketName, null, objectName, uploadId, parts, null, null);
        } catch (Exception e) {
            logger.error("error: {}", e.getMessage(), e);
            return false;
        }

        return true;
    }

注意：此处做了很多简单的操作还需自行优化，比如：

• 设定了最大仅允许1000个分片
• 初始化生成上传签名地址header头相关操作
• 等

整个流程看起来很简单，开始做的时候查询资料网上没有这么操作的，再加上又碰到了官方小bug（bug:listParts），还好社区活跃，不然只能老实的在业务中自己实现分片了。

完整的Demo地址：minio-multipart-upload，仅为实现流程，实现比较糙，需要进行优化改造。

参考文档：

AWS-CreateMultipartUpload

阿里-分片上传