AWS之S3大文件复制（Java SDK V1）

问题

需要在s3中复制大文件，s3复制大文件，如果文件超过5G就需要使用分块上传的API进行大文件复制。s3对分块api约束如下：

由此可知道，最大块为5G，而且不能超过10000块，也就说s3单个文件最大5T。

解决

使用s3的分块上传api进行s3的大文件复制。主要就是3个步骤：

• 1.启动复制任务，获取任务id；
• 2.对大文件进行分块，并获取分块任务id
• 3.合并分块任务，完成大文件复制任务。

Java

String sourceBucketName = "xxx";
    String sourceObjectKey = "xxx/xxx/xxx.mov";
    String destBucketName = "xxx";
    String destObjectKey = "xxx/xxx/xxx2.mov";

    Regions clientRegion = Regions.AP_SOUTHEAST_1;
    try {
      BasicAWSCredentials awsCreds =
          new BasicAWSCredentials(
              "xxx", "xxx");

      AmazonS3 s3Client =
          AmazonS3ClientBuilder.standard()
              .withCredentials(new AWSStaticCredentialsProvider(awsCreds))
              .withRegion(clientRegion)
              .build();

      // Initiate the multipart upload.初始化复制任务
      InitiateMultipartUploadRequest initRequest =
          new InitiateMultipartUploadRequest(destBucketName, destObjectKey);
      InitiateMultipartUploadResult initResult = s3Client.initiateMultipartUpload(initRequest);

      // Get the object size to track the end of the copy operation.获取文件大小
      GetObjectMetadataRequest metadataRequest =
          new GetObjectMetadataRequest(sourceBucketName, sourceObjectKey);
      ObjectMetadata metadataResult = s3Client.getObjectMetadata(metadataRequest);
      long objectSize = metadataResult.getContentLength();

      // Copy the object using 5 GB parts.使用5GB块进行分块，java中long乘法一定要加L
      long partSize = 5L * 1024L * 1024L * 1024L;
      long bytePosition = 0L;
      int partNum = 1;// 分块只能从2号开始，不能从1号块
      List<PartETag> etags = new ArrayList<>();
      long startSUM = 0L;
      while (bytePosition < objectSize) {
        // The last part might be smaller than partSize, so check to make sure
        // that lastByte isn't beyond the end of the object.
        long lastByte = Math.min(bytePosition + partSize - 1, objectSize - 1);

        // Copy this part.
        CopyPartRequest copyRequest =
            new CopyPartRequest()
                .withSourceBucketName(sourceBucketName)
                .withSourceKey(sourceObjectKey)
                .withDestinationBucketName(destBucketName)
                .withDestinationKey(destObjectKey)
                .withUploadId(initResult.getUploadId())
                .withFirstByte(bytePosition)
                .withLastByte(lastByte)
                .withPartNumber(partNum++);
        long start = System.currentTimeMillis();
        if (2 == partNum) {
          startSUM = start;
        }
        CopyPartResult response = s3Client.copyPart(copyRequest);
        System.out.printf("第%d块复制消耗: %d ms", partNum, System.currentTimeMillis() - start);
        etags.add(new PartETag(response.getPartNumber(), response.getETag()));
        bytePosition += partSize;
      }

      // Complete the upload request to concatenate all uploaded parts and make the copied object
      // available.
      CompleteMultipartUploadRequest completeRequest =
          new CompleteMultipartUploadRequest(
              destBucketName, destObjectKey, initResult.getUploadId(), etags);
      s3Client.completeMultipartUpload(completeRequest);
      System.out.printf("总消耗: %d ms", System.currentTimeMillis() - startSUM);
    } catch (SdkClientException e) {
      // The call was transmitted successfully, but Amazon S3 couldn't process
      // it, so it returned an error response.
      e.printStackTrace();
    }

这样就能够进行s3大文件复制了。

总结

注意这里使用的AWS S3 Java SDK为V1，不是V2，V2版本没有尝试成功。AWS限制单个文件最大5T，分块最大5G，最多分10000块，分块序号只能从1到10000之间的整数，也就意味着只能从编号为2号块开始上传。

参考：

• Amazon S3 分段上传限制
• 使用分段上传复制对象