本博客将介绍如何进行文件的分块上传。如果读者还想了解文件的“分块”下载相关内容可以去参考博客《
Java 服务器端支持断点续传的源代码【支持快车、迅雷】
》。
本文侧重介绍服务器端,客户端端请参考本篇博客的姊妹篇《 Java 文件分块上传客户端源代码 》,关于分块上传的思想及其流程,已在该博客中进行了详细说明,这里不再赘述。
OK,分块上传的服务器、客户端源代码及其工作流程至此已全部介绍完毕,以上源代码全部是经过项目实践过的,大部分现在仍运行于一些项目之中。有兴趣的朋友可以自己动手,将以上代码自行改造,看看能否运行成功。如果遇到问题可以在本博客下跟帖留言,大家一起讨论讨论。
本文侧重介绍服务器端,客户端端请参考本篇博客的姊妹篇《 Java 文件分块上传客户端源代码 》,关于分块上传的思想及其流程,已在该博客中进行了详细说明,这里不再赘述。
直接上代码。接收客户端 HTTP 分块上传请求的 Spring MVC 控制器源代码如下:
- @Controller
- public class UploadController extends BaseController {
- private static final Log log = LogFactory.getLog(UploadController.class);
- private UploadService uploadService;
- private AuthService authService;
- /**
- * 大文件分成小文件块上传,一次传递一块,最后一块上传成功后,将合并所有已经上传的块,保存到File Server
- * 上相应的位置,并返回已经成功上传的文件的详细属性. 当最后一块上传完毕,返回上传成功的信息。此时用getFileList查询该文件,
- * 该文件的uploadStatus为2。client请自行处理该状态下文件如何显示。(for UPS Server)
- *
- */
- @RequestMapping("/core/v1/file/upload")
- @ResponseBody
- public Object upload(HttpServletResponse response,
- @RequestParam(value = "client_id", required = false) String appkey,
- @RequestParam(value = "sig", required = false) String appsig,
- @RequestParam(value = "token", required = false) String token,
- @RequestParam(value = "uuid", required = false) String uuid,
- @RequestParam(value = "block", required = false) String blockIndex,
- @RequestParam(value = "file", required = false) MultipartFile multipartFile,
- @RequestParam Map<String, String> parameters) {
- checkEmpty(appkey, BaseException.ERROR_CODE_16002);
- checkEmpty(token, BaseException.ERROR_CODE_16007);
- checkEmpty(uuid, BaseException.ERROR_CODE_20016);
- checkEmpty(blockIndex, BaseException.ERROR_CODE_20006);
- checkEmpty(appsig, BaseException.ERROR_CODE_10010);
- if (multipartFile == null) {
- throw new BaseException(BaseException.ERROR_CODE_20020);// 上传文件不存在
- }
- Long uuidL = parseLong(uuid, BaseException.ERROR_CODE_20016);
- Integer blockIndexI = parseInt(blockIndex, BaseException.ERROR_CODE_20006);
- Map<String, Object> appMap = getAuthService().validateSigature(parameters);
- AccessToken accessToken = CasUtil.checkAccessToken(token, appMap);
- Long uid = accessToken.getUid();
- String bucketUrl = accessToken.getBucketUrl();
- // 从上传目录拷贝文件到工作目录
- String fileAbsulutePath = null;
- try {
- fileAbsulutePath = this.copyFile(multipartFile.getInputStream(), multipartFile.getOriginalFilename());
- } catch (IOException ioe) {
- log.error(ioe.getMessage(), ioe);
- throw new BaseException(BaseException.ERROR_CODE_20020);// 上传文件不存在
- }
- File uploadedFile = new File(Global.UPLOAD_TEMP_DIR + fileAbsulutePath);
- checkEmptyFile(uploadedFile);// file 非空验证
- Object rs = uploadService.upload(uuidL, blockIndexI, uid, uploadedFile, bucketUrl);
- setHttpStatusOk(response);
- return rs;
- }
- // TODO 查看下这里是否有问题
- // 上传文件非空验证
- private void checkEmptyFile(File file) {
- if (file == null || file.getAbsolutePath() == null) {
- throw new BaseException(BaseException.ERROR_CODE_20020);// 上传文件不存在
- }
- }
- /**
- * 写文件到本地文件夹
- *
- * @throws IOException
- * 返回生成的文件名
- */
- private String copyFile(InputStream inputStream, String fileName) {
- OutputStream outputStream = null;
- String tempFileName = null;
- int pointPosition = fileName.lastIndexOf(".");
- if (pointPosition < 0) {// myvedio
- tempFileName = UUID.randomUUID().toString();// 94d1d2e0-9aad-4dd8-a0f6-494b0099ff26
- } else {// myvedio.flv
- tempFileName = UUID.randomUUID() + fileName.substring(pointPosition);// 94d1d2e0-9aad-4dd8-a0f6-494b0099ff26.flv
- }
- try {
- outputStream = new FileOutputStream(Global.UPLOAD_TEMP_DIR + tempFileName);
- int readBytes = 0;
- byte[] buffer = new byte[10000];
- while ((readBytes = inputStream.read(buffer, 0, 10000)) != -1) {
- outputStream.write(buffer, 0, readBytes);
- }
- return tempFileName;
- } catch (IOException ioe) {
- // log.error(ioe.getMessage(), ioe);
- throw new BaseException(BaseException.ERROR_CODE_20020);// 上传文件不存在
- } finally {
- if (outputStream != null) {
- try {
- outputStream.close();
- } catch (IOException e) {
- }
- }
- if (inputStream != null) {
- try {
- inputStream.close();
- } catch (IOException e) {
- }
- }
- }
- }
- /**
- * 测试此服务是否可用
- *
- * @param response
- * @return
- * @author zwq7978
- */
- @RequestMapping("/core/v1/file/testServer")
- @ResponseBody
- public Object testServer(HttpServletResponse response) {
- setHttpStatusOk(response);
- return Global.SUCCESS_RESPONSE;
- }
- public UploadService getUploadService() {
- return uploadService;
- }
- public void setUploadService(UploadService uploadService) {
- this.uploadService = uploadService;
- }
- public void setAuthService(AuthService authService) {
- this.authService = authService;
- }
- public AuthService getAuthService() {
- return authService;
- }
- }
比如要上传的文件是 test450k.mp4。对照《 Java 文件分块上传客户端源代码 》中分块上传服务器对分块文件参数定义的名字"file",upload 方法里使用的是 MultipartFile 接收该对象。对于每次的 HTTP 请求,使用 copyFile 方法将文件流输出到服务器本地的一个临时文件夹里,比如作者的是 D:/defonds/syncPath/uploadTemp,该文件下会有 50127019-b63b-4a54-8f53-14efd1e58ada.mp4 临时文件生成用于保存上传文件流。
分块依次上传。当所有块都上传完毕之后,将这些临时文件都转移到服务器指定目录中,比如作者的这个目录是 D:/defonds/syncPath/file,在该文件夹下会有/1/temp_dir_5_1 目录生成,而 uploadTemp 的临时文件则被挨个转移到这个文件夹下,生成形如 5.part0001 的文件。以下是文件转移的源代码:
- /**
- * 把所有块从临时文件目录移到指定本地目录或S2/S3
- *
- * @param preUpload
- */
- private void moveBlockFiles(BlockPreuploadFileInfo preUpload) {
- @SuppressWarnings("unchecked")
- String[] s3BlockUrl=new String[preUpload.getBlockNumber()];
- String[] localBlockUrl=new String[preUpload.getBlockNumber()];//本地的块文件路径 以便以后删除
- List<BlockUploadInfo> blocks = (List<BlockUploadInfo>) getBaseDao().queryForList(
- "upload.getBlockUploadFileByUuid", preUpload.getUuid());
- String tempDirName = SyncUtil.getTempDirName(preUpload.getUuid(), preUpload.getUid());
- String parentPath = Global.UPLOAD_ABSOLUTE_PAHT_ + Global.PATH_SEPARATIVE_SIGN
- + String.valueOf(preUpload.getUid());
- String dirPath = parentPath + Global.PATH_SEPARATIVE_SIGN + tempDirName;
- new File(dirPath).mkdirs();//创建存放块文件的文件夹 (本地)
- int j=0;
- for (BlockUploadInfo info : blocks) {
- try {
- String strBlockIndex = createStrBlockIndex(info.getBlockIndex());
- String suffixPath = preUpload.getUuid() + ".part" + strBlockIndex;
- String tempFilePath = info.getTempFile();
- File tempFile = new File(tempFilePath);
- File tmpFile = new File(dirPath + suffixPath);
- if (tmpFile.exists()) {
- FileUtils.deleteQuietly(tmpFile);
- }
- FileUtils.moveFile(tempFile, tmpFile);
- localBlockUrl[j]=dirPath + suffixPath;
- j++;
- info.setStatus(Global.MOVED_TO_NEWDIR);
- getBaseDao().update("upload.updateBlockUpload", info);
- if (log.isInfoEnabled())
- log.info(preUpload.getUuid() + " " + info.getBuId() + " moveBlockFiles");
- } catch (IOException e) {
- log.error(e.getMessage(), e);
- throw new BaseException("file not found");
- }
- }
- preUpload.setLocalBlockUrl(localBlockUrl);
- preUpload.setDirPath(dirPath);
- preUpload.setStatus(Global.MOVED_TO_NEWDIR);
- getBaseDao().update("upload.updatePreUploadInfo", preUpload);
- }
- private String createStrBlockIndex(int blockIndex) {
- String strBlockIndex;
- if (blockIndex < 10) {
- strBlockIndex = "000" + blockIndex;
- } else if (10 <= blockIndex && blockIndex < 100) {
- strBlockIndex = "00" + blockIndex;
- } else if (100 <= blockIndex && blockIndex < 1000) {
- strBlockIndex = "0" + blockIndex;
- } else {
- strBlockIndex = "" + blockIndex;
- }
- return strBlockIndex;
- }
最后是文件的组装源代码:
- /**
- * 组装文件
- *
- */
- private void assembleFileWithBlock(BlockPreuploadFileInfo preUpload) {
- String dirPath = preUpload.getDirPath();
- // 开始在指定目录组装文件
- String uploadedUrl = null;
- String[] separatedFiles;
- String[][] separatedFilesAndSize;
- int fileNum = 0;
- File file = new File(dirPath);
- separatedFiles = file.list();
- separatedFilesAndSize = new String[separatedFiles.length][2];
- Arrays.sort(separatedFiles);
- fileNum = separatedFiles.length;
- for (int i = 0; i < fileNum; i++) {
- separatedFilesAndSize[i][0] = separatedFiles[i];
- String fileName = dirPath + separatedFiles[i];
- File tmpFile = new File(fileName);
- long fileSize = tmpFile.length();
- separatedFilesAndSize[i][1] = String.valueOf(fileSize);
- }
- RandomAccessFile fileReader = null;
- RandomAccessFile fileWrite = null;
- long alreadyWrite = 0;
- int len = 0;
- byte[] buf = new byte[1024];
- try {
- uploadedUrl = Global.UPLOAD_ABSOLUTE_PAHT_ + Global.PATH_SEPARATIVE_SIGN + preUpload.getUid() + Global.PATH_SEPARATIVE_SIGN + preUpload.getUuid();
- fileWrite = new RandomAccessFile(uploadedUrl, "rw");
- for (int i = 0; i < fileNum; i++) {
- fileWrite.seek(alreadyWrite);
- // 读取
- fileReader = new RandomAccessFile((dirPath + separatedFilesAndSize[i][0]), "r");
- // 写入
- while ((len = fileReader.read(buf)) != -1) {
- fileWrite.write(buf, 0, len);
- }
- fileReader.close();
- alreadyWrite += Long.parseLong(separatedFilesAndSize[i][1]);
- }
- fileWrite.close();
- preUpload.setStatus(Global.ASSEMBLED);
- preUpload.setServerPath(uploadedUrl);
- getBaseDao().update("upload.updatePreUploadInfo", preUpload);
- if(Global.BLOCK_UPLOAD_TO!=Global.BLOCK_UPLOAD_TO_LOCAL)
- {
- //组装完毕没有问题 删除掉S2/S3上的block
- String[] path=preUpload.getS3BlockUrl();
- for (String string : path) {
- try {
- if(Global.BLOCK_UPLOAD_TO==Global.BLOCK_UPLOAD_TO_S2)
- {
- S2Util.deleteFile(preUpload.getBucketUrl(), string);
- }else
- {
- S3Util.deleteFile(preUpload.getBucketUrl(), string);
- }
- } catch (Exception e) {
- log.error(e.getMessage(), e);
- }
- }
- }
- if (log.isInfoEnabled())
- log.info(preUpload.getUuid() + " assembleFileWithBlock");
- } catch (IOException e) {
- log.error(e.getMessage(), e);
- try {
- if (fileReader != null) {
- fileReader.close();
- }
- if (fileWrite != null) {
- fileWrite.close();
- }
- } catch (IOException ex) {
- log.error(e.getMessage(), e);
- }
- }
- }
BlockPreuploadFileInfo 是我们自定义的业务文件处理 bean。
OK,分块上传的服务器、客户端源代码及其工作流程至此已全部介绍完毕,以上源代码全部是经过项目实践过的,大部分现在仍运行于一些项目之中。有兴趣的朋友可以自己动手,将以上代码自行改造,看看能否运行成功。如果遇到问题可以在本博客下跟帖留言,大家一起讨论讨论。