我正在尝试从s3下载zip文件,解压缩它,然后尝试使用multipart upload将它上载到s3。裁判
我尝试了多个文件,它的工作非常好。它的失败只是因为 .txt
.
代码段
private void process(final S3ObjectSummary s3ObjectSummary) {
try {
@Cleanup S3Object s3Object = s3Accessor
.getObjectFromBucket(new GetObjectRequest(sourceBucketName, s3ObjectSummary.getKey()));
@Cleanup ZipInputStream zipInputStream = new ZipInputStream(s3Object.getObjectContent());
ZipEntry zipEntry;
while (null != (zipEntry = zipInputStream.getNextEntry())) {
String name = zipEntry.getName();
if (zipEntry.isDirectory()) {
// skip the directory
continue;
}
// fetch content type
String contentType = getContentType(name);
// take the copy of the stream and re-write it to an InputStream
@Cleanup PipedOutputStream out = new PipedOutputStream();
@Cleanup PipedInputStream in = new PipedInputStream(out);
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType(contentType);
executorService.submit(() -> {
Upload upload = transferManager.upload(destBucketName, name, in, metadata);
try {
upload.waitForCompletion();
} catch (InterruptedException e) {
log.error("Error while uploading files {} to destination", name, e);
}
});
// copy zipInputStream to PipeOutputStream
copy(zipInputStream, out);
out.flush();
out.close();
}
} catch (RuntimeException | IOException e) {
throw new Exception("Error while processing the downloaded s3 content", e);
}
}
例外情况
[java] 05 Jan 2021 20:24:10,448 [ERROR] (pool-7-thread-1) com.amazon.s3ziputils.utils.UnzipHelper: Error while uploading files airport/atestingfile.txt to destination
[java] com.amazonaws.SdkClientException: Unable to create HTTP entity: Pipe closed
[java] at com.amazonaws.http.apache.utils.ApacheUtils.newBufferedHttpEntity(ApacheUtils.java:145) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.apache.request.impl.ApacheHttpRequestFactory.createHttpEntityForNonPostVerbs(ApacheHttpRequestFactory.java:227) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.apache.request.impl.ApacheHttpRequestFactory.wrapEntity(ApacheHttpRequestFactory.java:188) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.apache.request.impl.ApacheHttpRequestFactory.createApacheRequest(ApacheHttpRequestFactory.java:144) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.apache.request.impl.ApacheHttpRequestFactory.create(ApacheHttpRequestFactory.java:94) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.apache.request.impl.ApacheHttpRequestFactory.create(ApacheHttpRequestFactory.java:51) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.AmazonHttpClient$RequestExecutor$ExecOneRequestParams.newApacheRequest(AmazonHttpClient.java:1966) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1312) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5247) ~[AWSS3JavaClient-1.11.x.jar:?]
[java] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5194) ~[AWSS3JavaClient-1.11.x.jar:?]
[java] at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:415) ~[AWSS3JavaClient-1.11.x.jar:?]
[java] at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6308) ~[AWSS3JavaClient-1.11.x.jar:?]
[java] at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1840) ~[AWSS3JavaClient-1.11.x.jar:?]
[java] at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1800) ~[AWSS3JavaClient-1.11.x.jar:?]
[java] at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:168) ~[AWSS3JavaClient-1.11.x.jar:?]
[java] at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:148) ~[AWSS3JavaClient-1.11.x.jar:?]
[java] at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:115) ~[AWSS3JavaClient-1.11.x.jar:?]
[java] at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:45) ~[AWSS3JavaClient-1.11.x.jar:?]
[java] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_275]
[java] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
[java] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
[java] at java.lang.Thread.run(Thread.java:749) [?:1.8.0_275]
[java] Caused by: java.io.IOException: Pipe closed
[java] at java.io.PipedInputStream.read(PipedInputStream.java:307) ~[?:1.8.0_275]
[java] at java.io.PipedInputStream.read(PipedInputStream.java:377) ~[?:1.8.0_275]
[java] at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90) ~[AWSJavaClientCore-1.11.x.jar:?]
[java] at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90) ~[AWSJavaClientCore-1.11.x.jar:?]
[java] at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.read(MD5DigestCalculatingInputStream.java:128) ~[AWSS3JavaClient-1.11.x.jar:?]
[java] at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_275]
[java] at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_275]
[java] at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_275]
[java] at com.amazonaws.internal.SdkBufferedInputStream.read(SdkBufferedInputStream.java:76) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90) ~[AWSJavaClientCore-1.11.x.jar:?]
[java] at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90) ~[AWSJavaClientCore-1.11.x.jar:?]
[java] at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.8.0_275]
[java] at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:133) ~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
[java] at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:160) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] at org.apache.http.entity.BufferedHttpEntity.<init>(BufferedHttpEntity.java:62) ~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
[java] at com.amazonaws.http.apache.utils.ApacheUtils.newBufferedHttpEntity(ApacheUtils.java:140) ~[AWSJavaClientRuntime-1.11.x.jar:?]
[java] ... 29 more
有什么需要帮忙的吗?
我试着在单线程中编辑和上传文件,这样它就不会关闭流。即使这样,我也面临着文本文件的问题。
日志:
[java] 05 Jan 2021 22:01:14,768 [WARN] (s3-transfer-manager-worker-2) com.amazonaws.services.s3.AmazonS3Client: No content length specified for stream data. Stream contents will be buffered in memory and could result in out of memory errors.
[java] 05 Jan 2021 22:01:14,768 [DEBUG] (s3-transfer-manager-worker-2) com.amazonaws.request: Sending Request: PUT https://zipreview.s3.us-west-2.amazonaws.com /airport/atestingfile.txt Headers: (amz-sdk-invocation-id: 8b784eca-754c-4dd1-f693-06373cba916f, Content-Type: text/plain, User-Agent: aws-internal/3 aws-sdk-java/1.11.922 Linux/4.9.230-0.1.ac.223.84.332.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.275-b01 java/1.8.0_275 vendor/Oracle_Corporation com.amazonaws.services.s3.transfer.TransferManager/1.11.922, )
[java] 05 Jan 2021 22:01:14,768 [DEBUG] (s3-transfer-manager-worker-2) com.amazonaws.auth.AWS4Signer: AWS4 Canonical Request: '"PUT
[java] /airport/atestingfile.txt
[java]
[java] amz-sdk-invocation-id:8b784eca-754c-4dd1-f693-06373cba916f
[java] amz-sdk-request:attempt=1;max=4
[java] amz-sdk-retry:0/0/500
[java] content-type:text/plain
[java] host:zipreview.s3.us-west-2.amazonaws.com
[java] user-agent:aws-internal/3 aws-sdk-java/1.11.922 Linux/4.9.230-0.1.ac.223.84.332.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.275-b01 java/1.8.0_275 vendor/Oracle_Corporation com.amazonaws.services.s3.transfer.TransferManager/1.11.922
[java] x-amz-content-sha256:UNSIGNED-PAYLOAD
[java] x-amz-date:20210105T220114Z
[java]
[java] amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
[java] UNSIGNED-PAYLOAD"
[java] 05 Jan 2021 22:01:14,768 [DEBUG] (s3-transfer-manager-worker-2) com.amazonaws.auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
[java] 20210105T220114Z
[java] 20210105/us-west-2/s3/aws4_request
[java] 77ce57c7d607766d66cac9408f9639f8a336a26a0bac898f93bd8dab8a8bcf4e"
暂无答案!
目前还没有任何答案,快来回答吧!