一、 概述
切片上传:
在上传时指将一个大文件分成多个小块(chunk),逐块上传到服务器,服务器在接收到所有的块后,再将它们拼接成完整的文件。断点续传:
上传过程中如果因为网络故障等原因中断,可以从中断的地方继续上传,而不用重新上传整个文件。秒传:
在上传文件之前,先检查该文件是否已经存在于服务器上。如果存在,则直接返回上传成功的结果,而不需要再次上传文件。
二、 具体思路
前端通过Web Worker(浏览器提供的一种多线程机制)来实现文件切片,获取文件的MD5。通过>MD5校验文件是否已经上传,或已上传的部分片段,连接后端对应上传文件,从而实现上述功能。
三、 代码实现
worker.js:
将一个文件分成多个块,并计算每个块的MD5哈希值。在onmessage事件处理程序中,接收包含文件、起始位置、结束位置和块大小的数据。然后,通过调用createChunk函数生成文件块并计算其哈希值。最后,将结果发送回主线程。
createChunk函数负责实际的文件块生成和哈希计算。它使用file.slice方法从文件中切割出特定大小的块,并使用SparkMD5.ArrayBuffer()类计算块的MD5哈希值。计算哈希的过程是将块读取为ArrayBuffer,然后使用spark.append方法追加到哈希对象中,并调用spark.end()方法获取最终的哈希值。最后,将块的起始位置、结束位置、索引、哈希值和重新创建的Blob对象一起作为结果返回。
import SparkMD5 from 'spark-md5';
onmessage = async (e) => {
const {
file,
start,
end,
CHUNK_SIZE
} = e.data;
const result = [];
for (let i = start; i < end; i++) {
const promise = createChunk(file, i, CHUNK_SIZE);
result.push(promise);
}
const chunks = await Promise.all(result);
postMessage(chunks);
};
async function createChunk(file, index, chunkSize) {
return new Promise((resolve, reject) => {
const start = index * chunkSize;
const end = Math.min(start + chunkSize, file.size);
const reader = file.slice(start, end).stream().getReader();
const spark = new SparkMD5.ArrayBuffer();
const chunks = [];
let size = 0;
reader.read().then(function processText({ done, value }) {
if (done) {
// Convert chunks to ArrayBuffer and calculate hash
const arrayBuffer = new Uint8Array(size).buffer;
let offset = 0;
for (const chunk of chunks) {
new Uint8Array(arrayBuffer, offset, chunk.length).set(chunk);
offset += chunk.length;
}
spark.append(arrayBuffer);
const hash = spark.end();
// Recreate the Blob from the accumulated chunks
const blob = new Blob(chunks);
resolve({ start, end, index, hash, blob });
return;
}
chunks.push(value);
size += value.length;
return reader.read().then(processText);
}).catch(reject);
});
}
index.vue:
实现对文件上传的灵活处理和进度显示。同时,通过Web Worker技术加速文件切片,提高效率
<template>
<div>
<el-upload
drag
:auto-upload="false"
:on-change="handleChange"
:limit="1"
ref="uploadRef">
<el-button type="primary">选择文件</el-button>
<div slot="tip" class="el-upload__tip">只能选择一个文件进行上传</div>
</el-upload>
<el-button type="success" @click="uploadFile" v-if="MD5Status" :loading-icon="Eleme" loading>
解析资源中,请稍等...
</el-button>
<el-button type="success" @click="uploadFile" v-else>上传</el-button>
<el-progress
v-if="state.uploadProgress > 0"
:percentage="state.uploadProgress"
status="success">
</el-progress>
</div>
</template>
<script setup lang="ts">
import { reactive, ref } from 'vue';
import SparkMD5 from 'spark-md5';
import { checkMd5, merge, upload } from '@/api/upload';
import { UploadProps } from 'element-plus';
import { Eleme } from '@element-plus/icons-vue';
const state = reactive({
file: null as File | null,
md5: '',
uploadProgress: 0
});
interface fileBlob {
start: number,
end: number,
index: number,
hash: string,
blob: Blob
}
const uploadRef = ref();
const MD5Status = ref(false);
const CHUNK_SIZE = 1024 * 1024 * 5; // 最少5M,minio合并最低要求
const THREAD_COUNT = navigator.hardwareConcurrency || 4;
const chunks = ref<fileBlob[]>([]);
const handleChange: UploadProps['onChange'] = (uploadFile, uploadFiles) => {
MD5Status.value = true;
state.file = uploadFile.raw as File;
cutFile(state.file).then((res: any) => {
chunks.value = res;
const md5Values = res.map((it: any) => it.hash).join('');
state.md5 = SparkMD5.hash(md5Values);
MD5Status.value = false;
});
};
const uploadFile = async () => {
if (!state.file) return;
state.uploadProgress = 0;
const file = state.file;
const md5 = state.md5;
// 先询问后端哪些chunk已经上传过
let uploadedChunks = new Set();
await checkMd5(md5).then((response: any) => {
uploadedChunks = new Set(response.msg.split(':').map(Number));
});
for (let i = 0; i < chunks.value.length; i++) {
// 不包含片段
if (!uploadedChunks.has(i + 1)) {
await uploadChunk(chunks.value[i].blob, i + 1, md5);
}
state.uploadProgress = Math.floor(((i + 1) / chunks.value.length) * 100);
if (state.uploadProgress === 100) {
uploadRef.value.clearFiles();
}
}
// 通知后端进行合并
await merge(null, {
params: {
md5,
fileName: file.name
}
}).then(url => {
console.log('文件url', url.msg);
});
};
const uploadChunk = async (chunk: Blob, chunkNumber: number, md5: string) => {
const formData = new FormData();
formData.append('chunk', chunk);
formData.append('chunkNumber', chunkNumber.toString());
formData.append('md5', md5);
await upload(formData, {
headers: {
'Content-Type': 'multipart/form-data'
}
});
};
async function cutFile(file: File) {
return new Promise((resolve, reject) => {
const chunks = Math.ceil(file.size / CHUNK_SIZE);
const threadChunkCount = Math.ceil(chunks / THREAD_COUNT);
const result: any[] = [];
let finishCount = 0;
for (let i = 0; i < THREAD_COUNT; i++) {
// 分配线程任务
const worker = new Worker(new URL('./worker.js', import.meta.url), {
type: 'module'
});
const start = i * threadChunkCount;
let end = (i + 1) * threadChunkCount;
if (end > chunks) {
end = chunks;
}
worker.postMessage({
file,
start,
end,
CHUNK_SIZE
});
worker.onmessage = e => {
try {
result[i] = e.data;
worker.terminate();
finishCount++;
if (finishCount === THREAD_COUNT) {
resolve(result.flat());
}
} catch (error) {
reject(`Error processing the file in worker: ${error}`);
}
};
worker.onerror = (error) => {
reject(`Worker encountered an error: ${error}`);
};
}
});
}
</script>
>>> 基于`axios`封装请求
export const checkMd5 = (md5: string) => {
return http.get('/file/check/' + md5);
};
export function merge(data: any, config: any) {
return http.post('/file/merge', data, config);
}
export function upload(data: any, config: any) {
return http.post('/file/upload', data, config);
}
引入MinIo依赖
<dependency>
<groupId>io.minio</groupId>
<artifactId>minio</artifactId>
<version>8.2.2</version>
</dependency>
MinioConfig :
Minio 配置类
import io.minio.MinioClient;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* Minio 配置信息
*/
@Data
@Configuration
@ConfigurationProperties(prefix = "minio")
public class MinioConfig {
/**
* 服务地址
*/
private String url;
/**
* 用户名
*/
private String accessKey;
/**
* 密码
*/
private String secretKey;
/**
* 存储桶名称
*/
private String bucketName;
@Bean
public MinioClient getMinioClient() {
return MinioClient.builder().endpoint(url).credentials(accessKey, secretKey).build();
}
}
file_upload_detail:
数据库表
CREATE TABLE `file_upload_detail` (
`id` bigint NOT NULL AUTO_INCREMENT COMMENT '主键',
`username` varchar(64) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '上传文件的用户账号',
`file_name` varchar(64) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '上传文件名',
`md5` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '上传文件的MD5值',
`is_uploaded` int NULL DEFAULT 0 COMMENT '是否完整上传过 0:否 1:是',
`has_been_uploaded` longtext CHARACTER SET utf8 COLLATE utf8_general_ci NULL COMMENT '曾经上传过的分片号',
`url` longtext CHARACTER SET utf8 COLLATE utf8_general_ci NULL COMMENT '存储的url,或者是本机的url地址',
`create_time` datetime NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '本条记录创建时间',
`update_time` datetime NULL DEFAULT NULL COMMENT '本条记录更新时间',
`total_chunks` int NULL DEFAULT NULL COMMENT '文件的总分片数',
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 0 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;
FileUploadDetail:
实体类
import com.fasterxml.jackson.annotation.JsonFormat;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.io.Serial;
import java.io.Serializable;
import java.util.Date;
@Data
@AllArgsConstructor
@NoArgsConstructor
public class FileUploadDetail implements Serializable {
@Serial
private static final long serialVersionUID = 1L;
/**
* 主键
* */
private Long id;
/**
* 用户
* */
private String username;
/**
* 上传文件名
* */
private String fileName;
/**
* 上传文件的MD5值
* */
private String md5;
/**
* 是否完整上传过 0:否 1:是
* */
private int isUploaded;
/**
* 曾经上传过的分片号
* */
private String hasBeenUploaded;
/**
* 存储的url
* */
private String url;
/**
* 创建时间
* */
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss")
private Date createTime;
/**
* 更新时间
* */
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss")
private Date updateTime;
/**
* 文件的总分片数
* */
private int totalChunks;
}
FileUploadController:
文件上传控制器,用于处理文件上传相关的请求
import com.ruoyi.stock.service.impl.FileUploadService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import com.ruoyi.common.core.domain.AjaxResult;
import java.util.Set;
import java.util.stream.Collectors;
/**
* 文件上传控制器
* 处理文件上传相关的请求
*/
@RestController
@RequestMapping("/file")
public class FileUploadController {
@Autowired
private FileUploadService fileUploadService;
/**
* 上传文件块
*
* @param chunk 文件块
* @param chunkNumber 当前块的编号
* @param md5 文件的MD5值
* @return 上传结果
*/
@PostMapping("/upload")
public AjaxResult uploadChunk(
@RequestParam("chunk") MultipartFile chunk,
@RequestParam("chunkNumber") int chunkNumber,
@RequestParam("md5") String md5) {
try {
fileUploadService.uploadChunk(chunk, chunkNumber, md5);
return AjaxResult.success();
} catch (Exception e) {
return AjaxResult.error(e.getMessage());
}
}
/**
* 检查文件是否已上传
*
* @param md5 文件的MD5值
* @return 已上传的文件块编号
*/
@GetMapping("/check/{md5}")
public AjaxResult checkFile(@PathVariable String md5) {
Set<Integer> uploadedChunks = fileUploadService.getUploadedChunks(md5);
return AjaxResult.success(uploadedChunks.stream()
.map(String::valueOf)
.collect(Collectors.joining(":")));
}
/**
* 合并文件块
*
* @param md5 文件的MD5值
* @param fileName 文件名
* @return 合并结果
*/
@PostMapping("/merge")
public AjaxResult mergeChunks(@RequestParam("md5") String md5, @RequestParam("fileName") String fileName) {
try {
return AjaxResult.success(fileUploadService.mergeChunks(md5, fileName));
} catch (Exception e) {
return AjaxResult.error(e.getMessage());
}
}
}
FileUploadService:
相关逻辑实现
import com.ruoyi.common.exception.ServiceException;
import com.ruoyi.common.utils.DateUtils;
import com.ruoyi.common.utils.SecurityUtils;
import com.ruoyi.stock.domain.FileUploadDetail;
import com.ruoyi.stock.mapper.FileUploadDetailMapper;
import io.minio.*;
import io.minio.messages.DeleteError;
import io.minio.messages.DeleteObject;
import io.minio.messages.Item;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;
import io.minio.http.Method;
import java.io.InputStream;
import java.util.*;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.stream.Collectors;
@Service
public class FileUploadService {
private static final Logger log = LoggerFactory.getLogger(FileUploadService.class);
@Autowired
private FileUploadDetailMapper fileUploadDetailMapper;
@Autowired
private MinioClient minioClient;
@Value("${minio.bucketName}")
private String bucketName;
/**
* 上传文件块
*
* @param chunk 文件块
* @param chunkNumber 当前块的编号
* @param md5 文件的MD5值
*/
public void uploadChunk(MultipartFile chunk, int chunkNumber, String md5) throws Exception {
// 将文件块上传到MinIO
String chunkName = md5 + "-" + chunkNumber;
try (InputStream inputStream = chunk.getInputStream()) {
minioClient.putObject(
PutObjectArgs.builder().bucket(bucketName).object(chunkName)
.stream(inputStream, chunk.getSize(), -1)
.contentType(chunk.getContentType())
.build()
);
} catch (Exception e) {
log.error(e.getMessage(), e);
}
// 更新数据库
FileUploadDetail fileUploadDetail = fileUploadDetailMapper.selectFileUploadDetailByMd5(md5);
if (fileUploadDetail == null) {
// 如果文件上传详细信息不存在,则创建新的记录
fileUploadDetail = new FileUploadDetail();
fileUploadDetail.setUsername(SecurityUtils.getUsername());
fileUploadDetail.setMd5(md5);
fileUploadDetail.setHasBeenUploaded(String.valueOf(chunkNumber));
fileUploadDetail.setCreateTime(DateUtils.getNowDate());
fileUploadDetail.setTotalChunks(1);
fileUploadDetailMapper.insertFileUploadDetail(fileUploadDetail);
} else {
// 如果文件上传详细信息已存在,则更新记录
Set<Integer> uploadedChunks = new HashSet<>();
if (fileUploadDetail.getHasBeenUploaded() != null) {
// 将已上传的块编号从字符串转换为集合
uploadedChunks = Arrays.stream(fileUploadDetail.getHasBeenUploaded().split(":"))
.map(Integer::parseInt)
.collect(Collectors.toSet());
}
// 添加当前已上传的块编号
uploadedChunks.add(chunkNumber);
// 更新已上传的块编号字符串
String updatedChunks = uploadedChunks.stream()
.map(String::valueOf)
.collect(Collectors.joining(":"));
fileUploadDetail.setHasBeenUploaded(updatedChunks);
fileUploadDetail.setUpdateTime(DateUtils.getNowDate());
fileUploadDetail.setTotalChunks(uploadedChunks.size());
fileUploadDetailMapper.updateFileUploadDetail(fileUploadDetail);
}
}
/**
* 根据MD5获取已上传的文件块编号
*
* @param md5 文件的MD5值
* @return 已上传的文件块编号
*/
public Set<Integer> getUploadedChunks(String md5) {
FileUploadDetail detail = fileUploadDetailMapper.selectFileUploadDetailByMd5(md5);
if (detail == null || detail.getHasBeenUploaded() == null) {
return Collections.emptySet();
}
return Arrays.stream(detail.getHasBeenUploaded().split(":"))
.map(Integer::parseInt)
.collect(Collectors.toSet());
}
/**
* 合并文件块
*
* @param md5 文件的MD5值
* @param fileName 文件名
* @return 合并结果
*/
public String mergeChunks(String md5, String fileName) throws Exception {
CopyOnWriteArrayList<ComposeSource> sourceObjectList = new CopyOnWriteArrayList<>();
// 从数据库中获取文件上传详细信息
FileUploadDetail fileUploadDetail = fileUploadDetailMapper.selectFileUploadDetailByMd5(md5);
if (fileUploadDetail == null) {
throw new Exception("File not found");
}
// 获取所有已上传的块编号
String[] chunkNames = fileUploadDetail.getHasBeenUploaded().split(":");
for (String chunkName : chunkNames) {
try {
// 验证资源是否存在,释放流,避免资源占用
try (InputStream stream = minioClient.getObject(
GetObjectArgs.builder().bucket(bucketName).object(md5 + "-" + chunkName).build());) {
// 添加到合并列表
sourceObjectList.add(ComposeSource. builder().bucket(bucketName).object(md5 + "-" + chunkName).build());
} catch (Exception e) {
// 抛出异常,后续处理
throw new ServiceException("Chunk not found");
}
} catch (Exception e) {
// 处理MinIO资源不存在的情况
try {
List<DeleteObject> deleteObjects = new ArrayList<>();
// 列出所有匹配前缀的文件并添加到删除列表
Iterable<Result<Item>> findMD5Files = minioClient.listObjects(ListObjectsArgs.builder().bucket(bucketName)
.prefix(md5)
.recursive(true)
.build());
for (Result<Item> findMD5File : findMD5Files) {
Item item = findMD5File.get();
deleteObjects.add(new DeleteObject(item.objectName()));
}
deleteObjects.add(new DeleteObject(fileUploadDetail.getFileName()));
// 执行批量删除
Iterable<Result<DeleteError>> results = minioClient.removeObjects(
RemoveObjectsArgs.builder()
.bucket(bucketName)
.objects(deleteObjects)
.build()
);
for (Result<DeleteError> result : results) {
DeleteError error = result.get();
log.error("Error in deleting object {}; {}", error.objectName(), error.message());
}
// 从数据库中删除文件上传详细信息
fileUploadDetailMapper.deleteFileUploadDetail(md5);
} catch (Exception deleteError) {
log.error(deleteError.getMessage());
}
throw new ServiceException("操作失败,请重试!");
}
}
// 合并文件块
minioClient.composeObject(
ComposeObjectArgs.builder()
.bucket(bucketName)
.object(fileName)
.sources(sourceObjectList).build()
);
// 生成文件的预签名URL
String url = minioClient.getPresignedObjectUrl(
GetPresignedObjectUrlArgs.builder()
.method(Method.GET)
.bucket(bucketName)
.object(fileName)
.build()
);
// 更新数据库
fileUploadDetail.setIsUploaded(1);
fileUploadDetail.setUrl(url);
fileUploadDetail.setFileName(fileName);
fileUploadDetail.setUpdateTime(DateUtils.getNowDate());
fileUploadDetailMapper.updateFileUploadDetail(fileUploadDetail);
// todo 之前对应的md5文件的url变更
return url;
}
}
FileUploadDetailMapper
import com.ruoyi.stock.domain.FileUploadDetail;
public interface FileUploadDetailMapper {
int insertFileUploadDetail(FileUploadDetail detail);
int updateFileUploadDetail(FileUploadDetail detail);
FileUploadDetail selectFileUploadDetailByMd5(String md5);
int deleteFileUploadDetail(String md5);
}
FileUploadDetailMapper.xml
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.ruoyi.stock.mapper.FileUploadDetailMapper">
<resultMap id="BaseResultMap" type="com.ruoyi.stock.domain.FileUploadDetail">
<id column="id" property="id" jdbcType="BIGINT"/>
<result column="username" property="username" jdbcType="VARCHAR"/>
<result column="file_name" property="fileName" jdbcType="VARCHAR"/>
<result column="md5" property="md5" jdbcType="VARCHAR"/>
<result column="is_uploaded" property="isUploaded" jdbcType="INTEGER"/>
<result column="has_been_uploaded" property="hasBeenUploaded" jdbcType="VARCHAR"/>
<result column="url" property="url" jdbcType="VARCHAR"/>
<result column="create_time" property="createTime" />
<result column="update_time" property="updateTime" />
<result column="total_chunks" property="totalChunks" jdbcType="INTEGER"/>
</resultMap>
<insert id="insertFileUploadDetail" parameterType="com.ruoyi.stock.domain.FileUploadDetail">
INSERT INTO file_upload_detail (username, file_name, md5, is_uploaded, has_been_uploaded, url, create_time, update_time, total_chunks)
VALUES (#{username}, #{fileName}, #{md5}, #{isUploaded}, #{hasBeenUploaded}, #{url}, #{createTime}, #{updateTime}, #{totalChunks})
</insert>
<update id="updateFileUploadDetail" parameterType="com.ruoyi.stock.domain.FileUploadDetail">
UPDATE file_upload_detail
SET username=#{username}, file_name=#{fileName}, md5=#{md5}, is_uploaded=#{isUploaded}, has_been_uploaded=#{hasBeenUploaded}, url=#{url}, create_time=#{createTime}, update_time=#{updateTime}, total_chunks=#{totalChunks}
WHERE id=#{id}
</update>
<select id="selectFileUploadDetailByMd5" parameterType="String" resultMap="BaseResultMap">
SELECT * FROM file_upload_detail WHERE md5=#{md5}
</select>
<delete id="deleteFileUploadDetail" parameterType="String">
delete from file_upload_detail where md5=#{md5}
</delete>
</mapper>
以上为全部内容,感谢阅览!