我正在尝试为我的flink应用程序从s3n迁移到s3a。我已经编写了用于将数据写入awss3的自定义滚动接收器。它接受url作为基本路径,如s3a://。它适用于s3n,当我为s3a做以下更改时,它抛出以下异常。似乎core-site.xml在s3a for aws凭证的情况下不起作用。它使用aws默认加载程序加载凭证。请让我知道这个缺少什么配置。现在我运行在eclipse开发环境中,它是如何在flink集群模式下工作的?
例外情况是:
com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2316)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at com.abc.abc.common.sinks.CustomRollingSink.open(CustomRollingSink.java:260)
我正在使用flink1.0.0版本,并将以下依赖项放入projectpom文件和core site.xml作为aws凭证。
<property>
<name>fs.s3a.awsAccessKeyId</name>
<value>value..</value>
</property>
<property>
<name>fs.s3a.awsSecretAccessKey</name>
<value>values...</value>
</property>
<property>
<name>fs.s3a.impl</name>
<value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>
pom文件:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>2.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.2.5</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.2.5</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.6.3</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.6.3</version>
</dependency>
暂无答案!
目前还没有任何答案,快来回答吧!