s3流上载.txt文件失败,但可用于图像

bmp9r5qi  于 2021-06-26  发布在  Java
关注(0)|答案(0)|浏览(229)

我正在尝试从s3下载zip文件,解压缩它,然后尝试使用multipart upload将它上载到s3。裁判
我尝试了多个文件,它的工作非常好。它的失败只是因为 .txt .

代码段

private void process(final S3ObjectSummary s3ObjectSummary) {
        try {
            @Cleanup S3Object s3Object = s3Accessor
                    .getObjectFromBucket(new GetObjectRequest(sourceBucketName, s3ObjectSummary.getKey()));
            @Cleanup ZipInputStream zipInputStream = new ZipInputStream(s3Object.getObjectContent());
            ZipEntry zipEntry;

            while (null != (zipEntry = zipInputStream.getNextEntry())) {
                String name = zipEntry.getName();
                if (zipEntry.isDirectory()) {
                    // skip the directory
                    continue;
                }

               // fetch content type
                String contentType = getContentType(name);

                // take the copy of the stream and re-write it to an InputStream
                @Cleanup PipedOutputStream out = new PipedOutputStream();
                @Cleanup PipedInputStream in = new PipedInputStream(out);

                ObjectMetadata metadata = new ObjectMetadata();
                metadata.setContentType(contentType);

                executorService.submit(() -> {
                    Upload upload = transferManager.upload(destBucketName, name, in, metadata);
                    try {
                        upload.waitForCompletion();
                    } catch (InterruptedException e) {
                        log.error("Error while uploading files {} to destination", name, e);
                    }
                });

              // copy zipInputStream to PipeOutputStream
               copy(zipInputStream, out);

                out.flush();
                out.close();
            }
        } catch (RuntimeException | IOException e) {
            throw new Exception("Error while processing the downloaded s3 content", e);
        }
    }

例外情况

[java] 05 Jan 2021 20:24:10,448 [ERROR]  (pool-7-thread-1) com.amazon.s3ziputils.utils.UnzipHelper: Error while uploading files airport/atestingfile.txt to destination
     [java] com.amazonaws.SdkClientException: Unable to create HTTP entity: Pipe closed
     [java]     at com.amazonaws.http.apache.utils.ApacheUtils.newBufferedHttpEntity(ApacheUtils.java:145) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.apache.request.impl.ApacheHttpRequestFactory.createHttpEntityForNonPostVerbs(ApacheHttpRequestFactory.java:227) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.apache.request.impl.ApacheHttpRequestFactory.wrapEntity(ApacheHttpRequestFactory.java:188) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.apache.request.impl.ApacheHttpRequestFactory.createApacheRequest(ApacheHttpRequestFactory.java:144) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.apache.request.impl.ApacheHttpRequestFactory.create(ApacheHttpRequestFactory.java:94) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.apache.request.impl.ApacheHttpRequestFactory.create(ApacheHttpRequestFactory.java:51) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.AmazonHttpClient$RequestExecutor$ExecOneRequestParams.newApacheRequest(AmazonHttpClient.java:1966) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1312) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5247) ~[AWSS3JavaClient-1.11.x.jar:?]
     [java]     at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5194) ~[AWSS3JavaClient-1.11.x.jar:?]
     [java]     at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:415) ~[AWSS3JavaClient-1.11.x.jar:?]
     [java]     at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6308) ~[AWSS3JavaClient-1.11.x.jar:?]
     [java]     at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1840) ~[AWSS3JavaClient-1.11.x.jar:?]
     [java]     at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1800) ~[AWSS3JavaClient-1.11.x.jar:?]
     [java]     at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:168) ~[AWSS3JavaClient-1.11.x.jar:?]
     [java]     at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:148) ~[AWSS3JavaClient-1.11.x.jar:?]
     [java]     at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:115) ~[AWSS3JavaClient-1.11.x.jar:?]
     [java]     at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:45) ~[AWSS3JavaClient-1.11.x.jar:?]
     [java]     at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_275]
     [java]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
     [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
     [java]     at java.lang.Thread.run(Thread.java:749) [?:1.8.0_275]
     [java] Caused by: java.io.IOException: Pipe closed
     [java]     at java.io.PipedInputStream.read(PipedInputStream.java:307) ~[?:1.8.0_275]
     [java]     at java.io.PipedInputStream.read(PipedInputStream.java:377) ~[?:1.8.0_275]
     [java]     at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90) ~[AWSJavaClientCore-1.11.x.jar:?]
     [java]     at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90) ~[AWSJavaClientCore-1.11.x.jar:?]
     [java]     at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.read(MD5DigestCalculatingInputStream.java:128) ~[AWSS3JavaClient-1.11.x.jar:?]
     [java]     at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_275]
     [java]     at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_275]
     [java]     at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_275]
     [java]     at com.amazonaws.internal.SdkBufferedInputStream.read(SdkBufferedInputStream.java:76) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90) ~[AWSJavaClientCore-1.11.x.jar:?]
     [java]     at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90) ~[AWSJavaClientCore-1.11.x.jar:?]
     [java]     at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.8.0_275]
     [java]     at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:133) ~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
     [java]     at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:160) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     at org.apache.http.entity.BufferedHttpEntity.<init>(BufferedHttpEntity.java:62) ~[Apache-HttpComponents-HttpCore-4.4.x.jar:?]
     [java]     at com.amazonaws.http.apache.utils.ApacheUtils.newBufferedHttpEntity(ApacheUtils.java:140) ~[AWSJavaClientRuntime-1.11.x.jar:?]
     [java]     ... 29 more

有什么需要帮忙的吗?
我试着在单线程中编辑和上传文件,这样它就不会关闭流。即使这样,我也面临着文本文件的问题。

日志:

[java] 05 Jan 2021 22:01:14,768 [WARN]  (s3-transfer-manager-worker-2) com.amazonaws.services.s3.AmazonS3Client: No content length specified for stream data.  Stream contents will be buffered in memory and could result in out of memory errors.
     [java] 05 Jan 2021 22:01:14,768 [DEBUG]  (s3-transfer-manager-worker-2) com.amazonaws.request: Sending Request: PUT https://zipreview.s3.us-west-2.amazonaws.com /airport/atestingfile.txt Headers: (amz-sdk-invocation-id: 8b784eca-754c-4dd1-f693-06373cba916f, Content-Type: text/plain, User-Agent: aws-internal/3 aws-sdk-java/1.11.922 Linux/4.9.230-0.1.ac.223.84.332.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.275-b01 java/1.8.0_275 vendor/Oracle_Corporation com.amazonaws.services.s3.transfer.TransferManager/1.11.922, ) 
     [java] 05 Jan 2021 22:01:14,768 [DEBUG]  (s3-transfer-manager-worker-2) com.amazonaws.auth.AWS4Signer: AWS4 Canonical Request: '"PUT
     [java] /airport/atestingfile.txt
     [java] 
     [java] amz-sdk-invocation-id:8b784eca-754c-4dd1-f693-06373cba916f
     [java] amz-sdk-request:attempt=1;max=4
     [java] amz-sdk-retry:0/0/500
     [java] content-type:text/plain
     [java] host:zipreview.s3.us-west-2.amazonaws.com
     [java] user-agent:aws-internal/3 aws-sdk-java/1.11.922 Linux/4.9.230-0.1.ac.223.84.332.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.275-b01 java/1.8.0_275 vendor/Oracle_Corporation com.amazonaws.services.s3.transfer.TransferManager/1.11.922
     [java] x-amz-content-sha256:UNSIGNED-PAYLOAD
     [java] x-amz-date:20210105T220114Z
     [java] 
     [java] amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
     [java] UNSIGNED-PAYLOAD"
     [java] 05 Jan 2021 22:01:14,768 [DEBUG]  (s3-transfer-manager-worker-2) com.amazonaws.auth.AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
     [java] 20210105T220114Z
     [java] 20210105/us-west-2/s3/aws4_request
     [java] 77ce57c7d607766d66cac9408f9639f8a336a26a0bac898f93bd8dab8a8bcf4e"

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题