在使用对象存储的时候,遇到超大文件上传的应用场景,可以通过前后端联动的方式,实现超大文件从前端直接向对象存储服务进行分片上传,具体实现过程如下:
Maven依赖:
<dependency><groupId>software.amazon.awssdk</groupId><artifactId>s3</artifactId><version>2.23.3</version><scope>test</scope>
</dependency>
- 由后端生成超大文件分片后的多个预签上传链接给前端
@Test
@DisplayName("测试生成分片上传预签地址")
public void testMultipartUpload() {String bucket = "bucketName";String object = "objectName";CreateMultipartUploadRequest multipartUploadRequest = CreateMultipartUploadRequest.builder().bucket(bucket).key(object).build();CreateMultipartUploadResponse multipartUploadResponse = s3Client.createMultipartUpload(multipartUploadRequest);String uploadId = multipartUploadResponse.uploadId();System.out.println("UploadId: " + uploadId);generateUploadPresignedUrls(presigner, bucket, object, uploadId, 2).stream().forEach(System.out::println);
}private List<String> generateUploadPresignedUrls(S3Presigner presigner, String bucketName, String keyName, String uploadId, int partCount) {List<String> presignedUrls = new ArrayList<>();for (int partNumber = 0; partNumber < partCount; partNumber++) {UploadPartRequest uploadPartRequest = UploadPartRequest.builder().bucket(bucketName).key(keyName).uploadId(uploadId).partNumber(partNumber).build();UploadPartPresignRequest uploadPartPresignRequest = UploadPartPresignRequest.builder().signatureDuration(Duration.ofMinutes(10)).uploadPartRequest(uploadPartRequest).build();PresignedUploadPartRequest presignedUploadPartRequest = presigner.presignUploadPart(uploadPartPresignRequest);URL url = presignedUploadPartRequest.url();presignedUrls.add(url.toString());}return presignedUrls;
}
得到UploadId和预签上传链接如下:
UploadId: ef32490d4801d55
https://xxx.xxxcloud.com.cn/oss/test1/2e48eb6f502xxxxxxxx7aae0fa0e69?partNumber=0&uploadId=ef32490d4801d55&X-Amz-Security-Token=&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250307T064845Z&X-Amz-SignedHeaders=host&X-Amz-Credential=GMKFIWVHTQXGHXVFRSCY%2F20250307%2Flaz01%2Fs3%2Faws4_request&X-Amz-Expires=600&X-Amz-Signature=387c4f5d0a3c73b8333d1e872359798d9cbd4f3aa563fe9d6ced31dd9d4b1e2e
https://xxx.xxxcloud.com.cn/oss/test1/2e48eb6f502cxxxxxxa27aae0fa0e69?partNumber=1&uploadId=ef32490d4801d55&X-Amz-Security-Token=&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250307T064845Z&X-Amz-SignedHeaders=host&X-Amz-Credential=GMKFIWVHTQXGHXVFRSCY%2F20250307%2Flaz01%2Fs3%2Faws4_request&X-Amz-Expires=600&X-Amz-Signature=95a232824ff032a4845b2311832746e11dd6a4b20a3eafe364fe3b985b5ffa66
- 预签上传链接返回给前端,对文件进行分片上传
// 前端通过PUT请求调用预签上传链接,上传文件流
...省略前端代码...
分片上传成功后对象存储服务会在响应请求头中返回ETag值,需要把ETag值保存下来传给后端作为完成分片上传请求的参数,如下:
3. 前端上传完所有分片后通过后端调用完成分片上传接口
@Test
@DisplayName("测试合并文件")
public void testMergeObjects() {List<CompletedPart> completedParts = new ArrayList<>();completedParts.add(CompletedPart.builder().partNumber(0).eTag("b6076b40b4538a0b6005cf1cdd8e80ce").build());completedParts.add(CompletedPart.builder().partNumber(1).eTag("0c17c47cf572116b95a481bed625cb0f").build());CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder().parts(completedParts).build();CompleteMultipartUploadRequest request = CompleteMultipartUploadRequest.builder().bucket("bucketName").key("objectName").uploadId("uploadId").multipartUpload(completedMultipartUpload).build();s3Client.completeMultipartUpload(request);}