如何使用Java读取AWS S3文件?

dzjeubhm  于 2023-02-14  发布在  Java
关注(0)|答案(6)|浏览(767)

我尝试从AWS S3读取一个文件到我的Java代码:

File file = new File("s3n://mybucket/myfile.txt");
  FileInputStream fileInput = new FileInputStream(file);

然后我得到一个错误:

java.io.FileNotFoundException: s3n:/mybucket/myfile.txt (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)

有没有办法从AWS S3打开/读取文件?非常感谢!

mkshixfv

mkshixfv1#

Java中的“File”类无法理解S3的存在。Here's an example of reading a file from the AWS documentation

AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());        
S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, key));
InputStream objectData = object.getObjectContent();
// Process the objectData stream.
objectData.close();
j0pj023g

j0pj023g2#

在2019年,有一种更优化、更稳健的方式来从S3读取文件:

private final AmazonS3 amazonS3Client = AmazonS3ClientBuilder.standard().build();

private Collection<String> loadFileFromS3() {
    try (final S3Object s3Object = amazonS3Client.getObject(BUCKET_NAME,
                                                            FILE_NAME);
        final InputStreamReader streamReader = new InputStreamReader(s3Object.getObjectContent(), StandardCharsets.UTF_8);
        final BufferedReader reader = new BufferedReader(streamReader)) {
        return reader.lines().collect(Collectors.toSet());
    } catch (final IOException e) {
        log.error(e.getMessage(), e)
        return Collections.emptySet();
    }
}

一定要注意关于这段代码的内存使用的注解。

eqoofvh9

eqoofvh93#

我们也可以使用software.amazon.awssdk:s3

//Assuming the credentials are read from Environment Variables, so no hardcoding here

    S3Client client = S3Client.builder()
                        .region(regionSelected)
                        .build();
    
    GetObjectRequest getObjectRequest = GetObjectRequest.builder()
                    .bucket(bucketName)
                    .key(fileName)
                    .build();
    
    ResponseInputStream<GetObjectResponse> responseInputStream = client.getObject(getObjectRequest);

    InputStream stream = new ByteArrayInputStream(responseInputStream.readAllBytes());
    
    
    System.out.println("Content :"+ new String(responseInputStream.readAllBytes(), StandardCharsets.UTF_8));
ar7v8xwq

ar7v8xwq4#

在java中读取S3文件的步骤可以是:
1.创建AmazonS3客户端。
1.使用存储桶名称和键创建S3Object。
1.使用S3Object创建缓冲区读取器并逐行读取文件。
1〉〉〉

BasicAWSCredentials awsCreds = new BasicAWSCredentials("accessKey", "secretKey");
    AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
            .withCredentials(new AWSStaticCredentialsProvider(awsCreds))
            .withRegion("region_name_here").build();

2〉〉〉

S3Object object = s3Client.getObject(new GetObjectRequest("bucketName", "key"));

3〉〉〉

BufferedReader reader = new BufferedReader(new InputStreamReader(object.getObjectContent()));

    String s = null;
    while ((s = reader.readLine()) != null)
    {
        System.out.println(s);
        //your business logic here
    }

谢谢。

ifmq2ha2

ifmq2ha25#

如果文件的内容是字符串,则可以使用getObjectAsString,否则可以使用getObjectContent()上的IOUtils.toByteArray将文件的内容读入字节数组。
显然,这些最好用在比较小的S3对象上,这样可以很容易地放入内存。

private final AmazonS3 amazonS3Client = AmazonS3ClientBuilder.standard().build();

private String loadStringFromS3() {
    try {
        return amazonS3Client.getObjectAsString(BUCKET_NAME, FILE_NAME);
    } catch (final IOException e) {
        log.error(e.getMessage(), e)
        return null;
    }
}

private byte[] loadDataFromS3() {
    try (final S3Object s3Object = amazonS3Client.getObject(BUCKET_NAME, FILE_NAME)) {
        return IOUtils.toByteArray(s3Object.getObjectContent());
    } catch (final IOException e) {
        log.error(e.getMessage(), e)
        return null;
    } finally {
        IOUtils.closeQuietly(object, log);
    }
}
wtlkbnrh

wtlkbnrh6#

这是我的解决方案。我使用的是 Spring Boot 2.4.3
创建amazon s3客户端

AmazonS3 amazonS3Client = AmazonS3ClientBuilder
                .standard()
                .withRegion("your-region")
                .withCredentials(
                        new AWSStaticCredentialsProvider(
                            new BasicAWSCredentials("your-access-key", "your-secret-access-key")))
                .build();

创建一个亚马逊转账客户端

TransferManager transferManagerClient = TransferManagerBuilder.standard()
                .withS3Client(amazonS3Client)
                .build();

在**/tmp/{your-s3-key}**中创建一个 * 临时文件 *,以便我们可以将下载的文件放在此文件中。

File file = new File(System.getProperty("java.io.tmpdir"), "your-s3-key"); 

try {
    file.createNewFile(); // Create temporary file
} catch (IOException e) {
    e.printStackTrace();
}

file.mkdirs();  // Create the directory of the temporary file

然后,我们使用传输管理器客户端从s3下载文件

// Note that in this line the s3 file downloaded has been transferred in to the temporary file that we created
Download download = transferManagerClient.download(
               new GetObjectRequest("your-s3-bucket-name", "your-s3-key"), file); 

// This line blocks the thread until the download is finished
download.waitForCompletion();

现在,s3文件已经成功地传输到我们创建的 * 临时文件 * 中,我们可以获取 * 临时文件 * 的InputStream。

InputStream input = new DataInputStream(new FileInputStream(file));

因为不再需要 * 临时文件 *,所以我们将其删除。

file.delete();

相关问题