Minio实现大文件切片上传，断点续传，秒传功能

一、 概述

1. 切片上传：在上传时指将一个大文件分成多个小块（chunk），逐块上传到服务器，服务器在接收到所有的块后，再将它们拼接成完整的文件。
2. 断点续传：上传过程中如果因为网络故障等原因中断，可以从中断的地方继续上传，而不用重新上传整个文件。
3. 秒传：在上传文件之前，先检查该文件是否已经存在于服务器上。如果存在，则直接返回上传成功的结果，而不需要再次上传文件。

二、 具体思路

前端通过Web Worker（浏览器提供的一种多线程机制）来实现文件切片，获取文件的MD5。通过>MD5校验文件是否已经上传，或已上传的部分片段，连接后端对应上传文件，从而实现上述功能。

三、 代码实现

worker.js：将一个文件分成多个块，并计算每个块的MD5哈希值。在onmessage事件处理程序中，接收包含文件、起始位置、结束位置和块大小的数据。然后，通过调用createChunk函数生成文件块并计算其哈希值。最后，将结果发送回主线程。
createChunk函数负责实际的文件块生成和哈希计算。它使用file.slice方法从文件中切割出特定大小的块，并使用SparkMD5.ArrayBuffer()类计算块的MD5哈希值。计算哈希的过程是将块读取为ArrayBuffer，然后使用spark.append方法追加到哈希对象中，并调用spark.end()方法获取最终的哈希值。最后，将块的起始位置、结束位置、索引、哈希值和重新创建的Blob对象一起作为结果返回。

import SparkMD5 from 'spark-md5';

onmessage = async (e) => {
    const {
        file,
        start,
        end,
        CHUNK_SIZE
    } = e.data;
    const result = [];
    for (let i = start; i < end; i++) {
        const promise = createChunk(file, i, CHUNK_SIZE);
        result.push(promise);
    }
    const chunks = await Promise.all(result);
    postMessage(chunks);
};

async function createChunk(file, index, chunkSize) {
    return new Promise((resolve, reject) => {
        const start = index * chunkSize;
        const end = Math.min(start + chunkSize, file.size);

        const reader = file.slice(start, end).stream().getReader();
        const spark = new SparkMD5.ArrayBuffer();
        const chunks = [];
        let size = 0;

        reader.read().then(function processText({ done, value }) {
            if (done) {
                // Convert chunks to ArrayBuffer and calculate hash
                const arrayBuffer = new Uint8Array(size).buffer;
                let offset = 0;
                for (const chunk of chunks) {
                    new Uint8Array(arrayBuffer, offset, chunk.length).set(chunk);
                    offset += chunk.length;
                }

                spark.append(arrayBuffer);
                const hash = spark.end();
                // Recreate the Blob from the accumulated chunks
                const blob = new Blob(chunks);
                resolve({ start, end, index, hash, blob });
                return;
            }
            chunks.push(value);
            size += value.length;
            return reader.read().then(processText);
        }).catch(reject);
    });
}

index.vue：实现对文件上传的灵活处理和进度显示。同时，通过Web Worker技术加速文件切片，提高效率

<template>
   <div>
       <el-upload
           drag
           :auto-upload="false"
           :on-change="handleChange"
           :limit="1"
           ref="uploadRef">
           <el-button type="primary">选择文件</el-button>
           <div slot="tip" class="el-upload__tip">只能选择一个文件进行上传</div>
       </el-upload>
       <el-button type="success" @click="uploadFile" v-if="MD5Status" :loading-icon="Eleme" loading>
           解析资源中，请稍等...
       </el-button>
       <el-button type="success" @click="uploadFile" v-else>上传</el-button>
       <el-progress
           v-if="state.uploadProgress > 0"
           :percentage="state.uploadProgress"
           status="success">
       </el-progress>
   </div>
</template>

<script setup lang="ts">
import { reactive, ref } from 'vue';
import SparkMD5 from 'spark-md5';
import { checkMd5, merge, upload } from '@/api/upload';
import { UploadProps } from 'element-plus';
import { Eleme } from '@element-plus/icons-vue';

const state = reactive({
   file: null as File | null,
   md5: '',
   uploadProgress: 0
});

interface fileBlob {
   start: number,
   end: number,
   index: number,
   hash: string,
   blob: Blob
}

const uploadRef = ref();
const MD5Status = ref(false);
const CHUNK_SIZE = 1024 * 1024 * 5; // 最少5M,minio合并最低要求
const THREAD_COUNT = navigator.hardwareConcurrency || 4;
const chunks = ref<fileBlob[]>([]);

const handleChange: UploadProps['onChange'] = (uploadFile, uploadFiles) => {
   MD5Status.value = true;
   state.file = uploadFile.raw as File;
   cutFile(state.file).then((res: any) => {
       chunks.value = res;
       const md5Values = res.map((it: any) => it.hash).join('');
       state.md5 = SparkMD5.hash(md5Values);
       MD5Status.value = false;
   });
};

const uploadFile = async () => {
   if (!state.file) return;
   state.uploadProgress = 0;

   const file = state.file;
   const md5 = state.md5;

   // 先询问后端哪些chunk已经上传过
   let uploadedChunks = new Set();
   await checkMd5(md5).then((response: any) => {
       uploadedChunks = new Set(response.msg.split(':').map(Number));
   });


   for (let i = 0; i < chunks.value.length; i++) {
       // 不包含片段
       if (!uploadedChunks.has(i + 1)) {
           await uploadChunk(chunks.value[i].blob, i + 1, md5);
       }
       state.uploadProgress = Math.floor(((i + 1) / chunks.value.length) * 100);
       if (state.uploadProgress === 100) {
           uploadRef.value.clearFiles();
       }
   }

   // 通知后端进行合并
   await merge(null, {
       params: {
           md5,
           fileName: file.name
       }
   }).then(url => {
       console.log('文件url', url.msg);
   });
};

const uploadChunk = async (chunk: Blob, chunkNumber: number, md5: string) => {
   const formData = new FormData();
   formData.append('chunk', chunk);
   formData.append('chunkNumber', chunkNumber.toString());
   formData.append('md5', md5);

   await upload(formData, {
       headers: {
           'Content-Type': 'multipart/form-data'
       }
   });
};


async function cutFile(file: File) {
   return new Promise((resolve, reject) => {
       const chunks = Math.ceil(file.size / CHUNK_SIZE);
       const threadChunkCount = Math.ceil(chunks / THREAD_COUNT);
       const result: any[] = [];
       let finishCount = 0;

       for (let i = 0; i < THREAD_COUNT; i++) {
           // 分配线程任务
           const worker = new Worker(new URL('./worker.js', import.meta.url), {
               type: 'module'
           });
           const start = i * threadChunkCount;
           let end = (i + 1) * threadChunkCount;
           if (end > chunks) {
               end = chunks;
           }
           worker.postMessage({
               file,
               start,
               end,
               CHUNK_SIZE
           });
           worker.onmessage = e => {
               try {
                   result[i] = e.data;
                   worker.terminate();
                   finishCount++;
                   if (finishCount === THREAD_COUNT) {
                       resolve(result.flat());
                   }
               } catch (error) {
                   reject(`Error processing the file in worker: ${error}`);
               }
           };
           worker.onerror = (error) => {
               reject(`Worker encountered an error: ${error}`);
           };
       }
   });
}
</script>
>>> 基于`axios`封装请求
export const checkMd5 = (md5: string) => {
   return http.get('/file/check/' + md5);
};

export function merge(data: any, config: any) {
   return http.post('/file/merge', data, config);
}

export function upload(data: any, config: any) {
   return http.post('/file/upload', data, config);
}

引入MinIo依赖

<dependency>
     <groupId>io.minio</groupId>
     <artifactId>minio</artifactId>
     <version>8.2.2</version>
</dependency>

MinioConfig ：Minio 配置类

import io.minio.MinioClient;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * Minio 配置信息
 */
@Data
@Configuration
@ConfigurationProperties(prefix = "minio")
public class MinioConfig {
    /**
     * 服务地址
     */
    private String url;

    /**
     * 用户名
     */
    private String accessKey;

    /**
     * 密码
     */
    private String secretKey;

    /**
     * 存储桶名称
     */
    private String bucketName;

    @Bean
    public MinioClient getMinioClient() {
        return MinioClient.builder().endpoint(url).credentials(accessKey, secretKey).build();
    }
}

file_upload_detail：数据库表

CREATE TABLE `file_upload_detail`  (
  `id` bigint NOT NULL AUTO_INCREMENT COMMENT '主键',
  `username` varchar(64) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '上传文件的用户账号',
  `file_name` varchar(64) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '上传文件名',
  `md5` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '上传文件的MD5值',
  `is_uploaded` int NULL DEFAULT 0 COMMENT '是否完整上传过 0：否 1：是',
  `has_been_uploaded` longtext CHARACTER SET utf8 COLLATE utf8_general_ci NULL COMMENT '曾经上传过的分片号',
  `url` longtext CHARACTER SET utf8 COLLATE utf8_general_ci NULL COMMENT '存储的url，或者是本机的url地址',
  `create_time` datetime NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '本条记录创建时间',
  `update_time` datetime NULL DEFAULT NULL COMMENT '本条记录更新时间',
  `total_chunks` int NULL DEFAULT NULL COMMENT '文件的总分片数',
  PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 0 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;

FileUploadDetail：实体类

import com.fasterxml.jackson.annotation.JsonFormat;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.io.Serial;
import java.io.Serializable;
import java.util.Date;

@Data
@AllArgsConstructor
@NoArgsConstructor
public class FileUploadDetail implements Serializable {
    @Serial
    private static final long serialVersionUID = 1L;

    /**
     * 主键
     * */
    private Long id;

    /**
     * 用户
     * */
    private String username;

    /**
     * 上传文件名
     * */
    private String fileName;

    /**
     * 上传文件的MD5值
     * */
    private String md5;

    /**
     * 是否完整上传过 0：否 1：是
     * */
    private int isUploaded;

    /**
     * 曾经上传过的分片号
     * */
    private String hasBeenUploaded;

    /**
     * 存储的url
     * */
    private String url;

    /**
     * 创建时间
     * */
    @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss")
    private Date createTime;

    /**
     * 更新时间
     * */
    @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss")
    private Date updateTime;

    /**
     * 文件的总分片数
     * */
    private int totalChunks;
}

FileUploadController：文件上传控制器，用于处理文件上传相关的请求

import com.ruoyi.stock.service.impl.FileUploadService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import com.ruoyi.common.core.domain.AjaxResult;

import java.util.Set;
import java.util.stream.Collectors;

/**
 * 文件上传控制器
 * 处理文件上传相关的请求
 */
@RestController
@RequestMapping("/file")
public class FileUploadController {

    @Autowired
    private FileUploadService fileUploadService;

    /**
     * 上传文件块
     *
     * @param chunk       文件块
     * @param chunkNumber 当前块的编号
     * @param md5         文件的MD5值
     * @return 上传结果
     */
    @PostMapping("/upload")
    public AjaxResult uploadChunk(
            @RequestParam("chunk") MultipartFile chunk,
            @RequestParam("chunkNumber") int chunkNumber,
            @RequestParam("md5") String md5) {
        try {
            fileUploadService.uploadChunk(chunk, chunkNumber, md5);
            return AjaxResult.success();
        } catch (Exception e) {
            return AjaxResult.error(e.getMessage());
        }
    }


    /**
     * 检查文件是否已上传
     *
     * @param md5 文件的MD5值
     * @return 已上传的文件块编号
     */
    @GetMapping("/check/{md5}")
    public AjaxResult checkFile(@PathVariable String md5) {
        Set<Integer> uploadedChunks = fileUploadService.getUploadedChunks(md5);
        return AjaxResult.success(uploadedChunks.stream()
                .map(String::valueOf)
                .collect(Collectors.joining(":")));
    }


    /**
     * 合并文件块
     *
     * @param md5     文件的MD5值
     * @param fileName 文件名
     * @return 合并结果
     */
    @PostMapping("/merge")
    public AjaxResult mergeChunks(@RequestParam("md5") String md5, @RequestParam("fileName") String fileName) {
        try {
            return AjaxResult.success(fileUploadService.mergeChunks(md5, fileName));
        } catch (Exception e) {
            return AjaxResult.error(e.getMessage());
        }
    }
}

FileUploadService：相关逻辑实现

import com.ruoyi.common.exception.ServiceException;
import com.ruoyi.common.utils.DateUtils;
import com.ruoyi.common.utils.SecurityUtils;
import com.ruoyi.stock.domain.FileUploadDetail;
import com.ruoyi.stock.mapper.FileUploadDetailMapper;
import io.minio.*;
import io.minio.messages.DeleteError;
import io.minio.messages.DeleteObject;
import io.minio.messages.Item;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;
import io.minio.http.Method;

import java.io.InputStream;
import java.util.*;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.stream.Collectors;

@Service
public class FileUploadService {

    private static final Logger log = LoggerFactory.getLogger(FileUploadService.class);

    @Autowired
    private FileUploadDetailMapper fileUploadDetailMapper;

    @Autowired
    private MinioClient minioClient;


    @Value("${minio.bucketName}")
    private String bucketName;

    /**
     * 上传文件块
     *
     * @param chunk       文件块
     * @param chunkNumber 当前块的编号
     * @param md5         文件的MD5值
     */
    public void uploadChunk(MultipartFile chunk, int chunkNumber, String md5) throws Exception {
        // 将文件块上传到MinIO
        String chunkName = md5 + "-" + chunkNumber;
        try (InputStream inputStream = chunk.getInputStream()) {
            minioClient.putObject(
                    PutObjectArgs.builder().bucket(bucketName).object(chunkName)
                            .stream(inputStream, chunk.getSize(), -1)
                            .contentType(chunk.getContentType())
                            .build()
            );
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        }


        // 更新数据库
        FileUploadDetail fileUploadDetail = fileUploadDetailMapper.selectFileUploadDetailByMd5(md5);
        if (fileUploadDetail == null) {
            // 如果文件上传详细信息不存在，则创建新的记录
            fileUploadDetail = new FileUploadDetail();
            fileUploadDetail.setUsername(SecurityUtils.getUsername());
            fileUploadDetail.setMd5(md5);
            fileUploadDetail.setHasBeenUploaded(String.valueOf(chunkNumber));
            fileUploadDetail.setCreateTime(DateUtils.getNowDate());
            fileUploadDetail.setTotalChunks(1);
            fileUploadDetailMapper.insertFileUploadDetail(fileUploadDetail);
        } else {
            // 如果文件上传详细信息已存在，则更新记录
            Set<Integer> uploadedChunks = new HashSet<>();
            if (fileUploadDetail.getHasBeenUploaded() != null) {
                // 将已上传的块编号从字符串转换为集合
                uploadedChunks = Arrays.stream(fileUploadDetail.getHasBeenUploaded().split(":"))
                        .map(Integer::parseInt)
                        .collect(Collectors.toSet());
            }
            // 添加当前已上传的块编号
            uploadedChunks.add(chunkNumber);

            // 更新已上传的块编号字符串
            String updatedChunks = uploadedChunks.stream()
                    .map(String::valueOf)
                    .collect(Collectors.joining(":"));
            fileUploadDetail.setHasBeenUploaded(updatedChunks);
            fileUploadDetail.setUpdateTime(DateUtils.getNowDate());
            fileUploadDetail.setTotalChunks(uploadedChunks.size());
            fileUploadDetailMapper.updateFileUploadDetail(fileUploadDetail);
        }

    }


    /**
     * 根据MD5获取已上传的文件块编号
     *
     * @param md5 文件的MD5值
     * @return 已上传的文件块编号
     */
    public Set<Integer> getUploadedChunks(String md5) {
        FileUploadDetail detail = fileUploadDetailMapper.selectFileUploadDetailByMd5(md5);
        if (detail == null || detail.getHasBeenUploaded() == null) {
            return Collections.emptySet();
        }
        return Arrays.stream(detail.getHasBeenUploaded().split(":"))
                .map(Integer::parseInt)
                .collect(Collectors.toSet());
    }


    /**
     * 合并文件块
     *
     * @param md5     文件的MD5值
     * @param fileName 文件名
     * @return 合并结果
     */
    public String mergeChunks(String md5, String fileName) throws Exception {
        CopyOnWriteArrayList<ComposeSource> sourceObjectList = new CopyOnWriteArrayList<>();
        // 从数据库中获取文件上传详细信息
        FileUploadDetail fileUploadDetail = fileUploadDetailMapper.selectFileUploadDetailByMd5(md5);
        if (fileUploadDetail == null) {
            throw new Exception("File not found");
        }
        // 获取所有已上传的块编号
        String[] chunkNames = fileUploadDetail.getHasBeenUploaded().split(":");

        for (String chunkName : chunkNames) {
            try {
                // 验证资源是否存在,释放流,避免资源占用
                try (InputStream stream = minioClient.getObject(
                        GetObjectArgs.builder().bucket(bucketName).object(md5 + "-" + chunkName).build());) {
                    // 添加到合并列表
                    sourceObjectList.add(ComposeSource. builder().bucket(bucketName).object(md5 + "-" + chunkName).build());
                } catch (Exception e) {
                    // 抛出异常,后续处理
                    throw new ServiceException("Chunk not found");
                }
            } catch (Exception e) {
                // 处理MinIO资源不存在的情况
                try {
                    List<DeleteObject> deleteObjects = new ArrayList<>();
                    // 列出所有匹配前缀的文件并添加到删除列表
                    Iterable<Result<Item>> findMD5Files = minioClient.listObjects(ListObjectsArgs.builder().bucket(bucketName)
                            .prefix(md5)
                            .recursive(true)
                            .build());
                    for (Result<Item> findMD5File : findMD5Files) {
                        Item item = findMD5File.get();
                        deleteObjects.add(new DeleteObject(item.objectName()));
                    }
                    deleteObjects.add(new DeleteObject(fileUploadDetail.getFileName()));
                    // 执行批量删除
                    Iterable<Result<DeleteError>> results = minioClient.removeObjects(
                            RemoveObjectsArgs.builder()
                                    .bucket(bucketName)
                                    .objects(deleteObjects)
                                    .build()
                    );
                    for (Result<DeleteError> result : results) {
                        DeleteError error = result.get();
                        log.error("Error in deleting object {}; {}", error.objectName(), error.message());
                    }
                    // 从数据库中删除文件上传详细信息
                    fileUploadDetailMapper.deleteFileUploadDetail(md5);
                } catch (Exception deleteError) {
                    log.error(deleteError.getMessage());
                }
                throw new ServiceException("操作失败，请重试！");
            }
        }

        // 合并文件块
        minioClient.composeObject(
                ComposeObjectArgs.builder()
                        .bucket(bucketName)
                        .object(fileName)
                        .sources(sourceObjectList).build()
        );

        // 生成文件的预签名URL
        String url = minioClient.getPresignedObjectUrl(
                GetPresignedObjectUrlArgs.builder()
                        .method(Method.GET)
                        .bucket(bucketName)
                        .object(fileName)
                        .build()
        );

        // 更新数据库
        fileUploadDetail.setIsUploaded(1);
        fileUploadDetail.setUrl(url);
        fileUploadDetail.setFileName(fileName);
        fileUploadDetail.setUpdateTime(DateUtils.getNowDate());
        fileUploadDetailMapper.updateFileUploadDetail(fileUploadDetail);
        // todo 之前对应的md5文件的url变更

        return url;
    }
}

FileUploadDetailMapper

import com.ruoyi.stock.domain.FileUploadDetail;

public interface FileUploadDetailMapper {
    int insertFileUploadDetail(FileUploadDetail detail);
    int updateFileUploadDetail(FileUploadDetail detail);
    FileUploadDetail selectFileUploadDetailByMd5(String md5);
    int deleteFileUploadDetail(String md5);
}

FileUploadDetailMapper.xml

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.ruoyi.stock.mapper.FileUploadDetailMapper">

    <resultMap id="BaseResultMap" type="com.ruoyi.stock.domain.FileUploadDetail">
        <id column="id" property="id" jdbcType="BIGINT"/>
        <result column="username" property="username" jdbcType="VARCHAR"/>
        <result column="file_name" property="fileName" jdbcType="VARCHAR"/>
        <result column="md5" property="md5" jdbcType="VARCHAR"/>
        <result column="is_uploaded" property="isUploaded" jdbcType="INTEGER"/>
        <result column="has_been_uploaded" property="hasBeenUploaded" jdbcType="VARCHAR"/>
        <result column="url" property="url" jdbcType="VARCHAR"/>
        <result column="create_time" property="createTime" />
        <result column="update_time" property="updateTime" />
        <result column="total_chunks" property="totalChunks" jdbcType="INTEGER"/>
    </resultMap>

    <insert id="insertFileUploadDetail" parameterType="com.ruoyi.stock.domain.FileUploadDetail">
        INSERT INTO file_upload_detail (username, file_name, md5, is_uploaded, has_been_uploaded, url, create_time, update_time, total_chunks)
        VALUES (#{username}, #{fileName}, #{md5}, #{isUploaded}, #{hasBeenUploaded}, #{url}, #{createTime}, #{updateTime}, #{totalChunks})
    </insert>

    <update id="updateFileUploadDetail" parameterType="com.ruoyi.stock.domain.FileUploadDetail">
        UPDATE file_upload_detail
        SET username=#{username}, file_name=#{fileName}, md5=#{md5}, is_uploaded=#{isUploaded}, has_been_uploaded=#{hasBeenUploaded}, url=#{url}, create_time=#{createTime}, update_time=#{updateTime}, total_chunks=#{totalChunks}
        WHERE id=#{id}
    </update>

    <select id="selectFileUploadDetailByMd5" parameterType="String" resultMap="BaseResultMap">
        SELECT * FROM file_upload_detail WHERE md5=#{md5}
    </select>

    <delete id="deleteFileUploadDetail" parameterType="String">
        delete from file_upload_detail where md5=#{md5}
    </delete>

</mapper>

以上为全部内容，感谢阅览！