flink应用程序写入s3a文件系统失败,因为在任何加载程序中都找不到aws凭据

yk9xbfzb  于 2021-06-25  发布在  Flink
关注(0)|答案(0)|浏览(320)

我正在尝试为我的flink应用程序从s3n迁移到s3a。我已经编写了用于将数据写入awss3的自定义滚动接收器。它接受url作为基本路径,如s3a://。它适用于s3n,当我为s3a做以下更改时,它抛出以下异常。似乎core-site.xml在s3a for aws凭证的情况下不起作用。它使用aws默认加载程序加载凭证。请让我知道这个缺少什么配置。现在我运行在eclipse开发环境中,它是如何在flink集群模式下工作的?
例外情况是:

com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
                at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
                at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
                at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
                at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
                at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2316)
                at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
                at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
                at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
                at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
                at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
                at com.abc.abc.common.sinks.CustomRollingSink.open(CustomRollingSink.java:260)

我正在使用flink1.0.0版本,并将以下依赖项放入projectpom文件和core site.xml作为aws凭证。

<property>
<name>fs.s3a.awsAccessKeyId</name>  
<value>value..</value>
</property>
<property>
<name>fs.s3a.awsSecretAccessKey</name>
<value>values...</value>
</property>
<property>
    <name>fs.s3a.impl</name>        
    <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>

pom文件:

<dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-aws</artifactId>
            <version>2.7.2</version>
        </dependency>
         <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.2.5</version>
        </dependency> 
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.2.5</version>
        </dependency>   
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.6.3</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>2.6.3</version>
        </dependency>

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题