问题
需要在s3中复制大文件,s3复制大文件,如果文件超过5G就需要使用分块上传的API进行大文件复制。s3对分块api约束如下:
由此可知道,最大块为5G,而且不能超过10000块,也就说s3单个文件最大5T。
解决
使用s3的分块上传api进行s3的大文件复制。主要就是3个步骤:
- 1.启动复制任务,获取任务id;
- 2.对大文件进行分块,并获取分块任务id
- 3.合并分块任务,完成大文件复制任务。
Java
String sourceBucketName = "xxx";
String sourceObjectKey = "xxx/xxx/xxx.mov";
String destBucketName = "xxx";
String destObjectKey = "xxx/xxx/xxx2.mov";
Regions clientRegion = Regions.AP_SOUTHEAST_1;
try {
BasicAWSCredentials awsCreds =
new BasicAWSCredentials(
"xxx", "xxx");
AmazonS3 s3Client =
AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
.withRegion(clientRegion)
.build();
// Initiate the multipart upload.初始化复制任务
InitiateMultipartUploadRequest initRequest =
new InitiateMultipartUploadRequest(destBucketName, destObjectKey);
InitiateMultipartUploadResult initResult = s3Client.initiateMultipartUpload(initRequest);
// Get the object size to track the end of the copy operation.获取文件大小
GetObjectMetadataRequest metadataRequest =
new GetObjectMetadataRequest(sourceBucketName, sourceObjectKey);
ObjectMetadata metadataResult = s3Client.getObjectMetadata(metadataRequest);
long objectSize = metadataResult.getContentLength();
// Copy the object using 5 GB parts.使用5GB块进行分块,java中long乘法一定要加L
long partSize = 5L * 1024L * 1024L * 1024L;
long bytePosition = 0L;
int partNum = 1;// 分块只能从2号开始,不能从1号块
List<PartETag> etags = new ArrayList<>();
long startSUM = 0L;
while (bytePosition < objectSize) {
// The last part might be smaller than partSize, so check to make sure
// that lastByte isn't beyond the end of the object.
long lastByte = Math.min(bytePosition + partSize - 1, objectSize - 1);
// Copy this part.
CopyPartRequest copyRequest =
new CopyPartRequest()
.withSourceBucketName(sourceBucketName)
.withSourceKey(sourceObjectKey)
.withDestinationBucketName(destBucketName)
.withDestinationKey(destObjectKey)
.withUploadId(initResult.getUploadId())
.withFirstByte(bytePosition)
.withLastByte(lastByte)
.withPartNumber(partNum++);
long start = System.currentTimeMillis();
if (2 == partNum) {
startSUM = start;
}
CopyPartResult response = s3Client.copyPart(copyRequest);
System.out.printf("第%d块复制消耗: %d ms", partNum, System.currentTimeMillis() - start);
etags.add(new PartETag(response.getPartNumber(), response.getETag()));
bytePosition += partSize;
}
// Complete the upload request to concatenate all uploaded parts and make the copied object
// available.
CompleteMultipartUploadRequest completeRequest =
new CompleteMultipartUploadRequest(
destBucketName, destObjectKey, initResult.getUploadId(), etags);
s3Client.completeMultipartUpload(completeRequest);
System.out.printf("总消耗: %d ms", System.currentTimeMillis() - startSUM);
} catch (SdkClientException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process
// it, so it returned an error response.
e.printStackTrace();
}
这样就能够进行s3大文件复制了。
总结
注意这里使用的AWS S3 Java SDK为V1,不是V2,V2版本没有尝试成功。AWS限制单个文件最大5T,分块最大5G,最多分10000块,分块序号只能从1到10000之间的整数,也就意味着只能从编号为2号块开始上传。