我试图通过intellij和aws执行hadoop操作,但是hadoop有一个问题-无法访问我的凭证密钥。我的intellij帐户已经连接到aws,我可以使用这些服务,但是在hadoop中,我遇到了以下问题。
这是密码-
public static void main(String[] args) throws Exception {
String s3Path = "adoop-map-reduce-collocation-extraction/corpus-cardinality-by-decade";
String output = "s3a://adoop-map-reduce-collocation-extraction/statistic-extraction";
Configuration conf = new Configuration();
conf.set("CorpusCardinalityFilePath", s3Path);
Job job = Job.getInstance(conf, "Appearances Calculation");
job.setJarByClass(AppearancesCalculation.class);
job.setMapperClass(AC_Mapper.class);
job.setCombinerClass(AC_Combiner.class);
job.setReducerClass(AC_Reducer.class);
job.setOutputKeyClass(nGramKey.class);
job.setOutputValueClass(LongWritable.class);
job.setMapOutputValueClass(LongWritable.class);
job.setMapOutputKeyClass(nGramKey.class);
job.setOutputValueClass(nGramValue.class);
FileInputFormat.addInputPath(job, new Path(inPath));
FileOutputFormat.setOutputPath(job, new Path(output));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
以及所有依赖关系-
3.0.0
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.531</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>amazon-kinesis-client</artifactId>
<version>1.2.1</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.jetbrains</groupId>
<artifactId>annotations-java5</artifactId>
<version>17.0.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
</dependency>
</dependencies>
这就是整个错误-
线程“main”java.lang.runtimeexception:java.io.interruptedioexception:DoesBucketExiston on adoop map reduce配置提取:com.amazonaws.amazonclientexception:basicawscredentialsprovider环境VariableCredentialsProvider instanceprofilecredentialsprovider未提供aws凭据:com.amazonaws.sdkclientexception:无法从位于org.apache.hadoop.mapreduce.lib.output.fileoutputformat.setoutputpath(fileoutputformat)的服务端点加载凭据。java:182)在外观上。java:153)原因:java.io.interruptedioexception:DoesBucketExiston on adoop map reduce配置提取:com.amazonaws.amazonclientexception:basicawscredentialsprovider环境variableCredentialProvider instanceprofilecredentialsprovider未提供aws凭据:com.amazonaws.sdkClientxception:无法从位于org.apache.hadoop.fs.s3a.s3autils.translateexception(s3autils)的服务端点加载凭据。java:141)在org.apache.hadoop.fs.s3a.s3afilesystem.verifybucketexists(s3afilesystem。java:332)在org.apache.hadoop.fs.s3a.s3afilesystem.initialize(s3afilesystem。java:275)在org.apache.hadoop.fs.filesystem.createfilesystem(filesystem。java:3303)在org.apache.hadoop.fs.filesystem.access$200(文件系统)。java:124)在org.apache.hadoop.fs.filesystem$cache.getinternal(文件系统)。java:3352)在org.apache.hadoop.fs.filesystem$cache.get(filesystem。java:3320)在org.apache.hadoop.fs.filesystem.get(filesystem。java:479)在org.apache.hadoop.fs.path.getfilesystem(path。java:361)在org.apache.hadoop.mapreduce.lib.output.fileoutputformat.setoutputpath(fileoutputformat)。java:178) ... 1个以上原因:com.amazonaws.amazonclientexception:basicawscredentialsprovider环境未提供aws凭据VariableCredentialsProvider instanceprofilecredentialsprovider:com.amazonaws.sdkclientexception:无法从位于的服务终结点加载凭据org.apache.hadoop.fs.s3a.awscredentialproviderlist.getcredentials(awscredentialproviderlist)。java:151)在com.amazonaws.http.amazonhttpclient$requestexecutor.getcredentialsfromcontext(amazonhttpclient)上。java:1225)在com.amazonaws.http.amazonhttpclient$requestexecutor.runbeforerequesthandlers(amazonhttpclient)上。java:801)在amazonaws.http.amazonhttpclient$requestexecutor.doexecute(amazonhttpclient。java:751)在com.amazonaws.http.amazonhttpclient$requestexecutor.executewithtimer(amazonhttpclient)上。java:744)在com.amazonaws.http.amazonhttpclient$requestexecutor.execute(amazonhttpclient。java:726)在amazonaws.http.amazonhttpclient$requestexecutor.access$500(amazonhttpclient。java:686)在com.amazonaws.http.amazonhttpclient$requestexecutionbuilderimpl.execute(amazonhttpclient)。java:668)在com.amazonaws.http.amazonhttpclient.execute(amazonhttpclient。java:532)在com.amazonaws.http.amazonhttpclient.execute(amazonhttpclient。java:512)在com.amazonaws.services.s3.amazon3client.invoke(amazon3client。java:4914)在com.amazonaws.services.s3.amazon3client.getbucketregionviaheadrequest(amazon3client。java:5693)在com.amazonaws.services.s3.amazon3client.fetchregionfromcache(amazon3client。java:5666)在com.amazonaws.services.s3.amazon3client.invoke(amazon3client。java:4898)在com.amazonaws.services.s3.amazon3client.invoke(amazon3client。java:4860)在com.amazonaws.services.s3.amazon3client.headbucket(amazon3client。java:1394)在com.amazonaws.services.s3.amazons3client.doesbucketexist(amazons3client。java:1333)在org.apache.hadoop.fs.s3a.s3afilesystem.verifybucketexists(s3afilesystem。java:320) ... 9其他原因:com.amazonaws.sdkclientexception:无法从位于com.amazonaws.auth.ec2credentialsfetcher.handleerror(ec2credentialsfetcher)的服务端点加载凭据。java:183)访问com.amazonaws.auth.ec2credentialsfetcher.fetchcredentials(ec2credentialsfetcher。java:162)在com.amazonaws.auth.ec2credentialsfetcher.getcredentials(ec2credentialsfetcher。java:82)访问com.amazonaws.auth.instanceprofilecredentialsprovider.getcredentials(instanceprofilecredentialsprovider)。java:172)在org.apache.hadoop.fs.s3a.awscredentialproviderlist.getcredentials(awscredentialproviderlist)上。java:129) ... 26更多原因:java.net.sockettimeoutexception:connect在java.base/java.net.dualstackplainsocketimpl.waitforconnect(本机方法)在java.base/java.net.dualstackplainsocketimpl.socketconnect(dualstackplainsocketimpl)处超时。java:85)位于java.base/java.net.abstractplainsocketimpl.doconnect(abstractplainsocketimpl。java:400)在java.base/java.net.abstractplainsocketimpl.connecttoaddress(abstractplainsocketimpl。java:243)位于java.base/java.net.abstractplainsocketimpl.connect(abstractplainsocketimpl。java:225)在java.base/java.net.plainsocketimpl.connect(plainsocketimpl。java:148)在java.base/java.net.socket.connect(socket。java:591)在java.base/sun.net.networkclient.doconnect(网络客户端。java:177)位于java.base/sun.net。www.http.httpclient.openserver(httpclient。java:474)位于java.base/sun.net。www.http.httpclient.openserver(httpclient。java:569)位于java.base/sun.net。www.http.httpclient.(httpclient。java:242)位于java.base/sun.net。www.http.httpclient.new(httpclient。java:341)在java.base/sun.net。www.http.httpclient.new(httpclient。java:362)位于java.base/sun.net。www.protocol.http.httpurlconnection.getnewhttpclient(httpurlconnection。java:1224)位于java.base/sun.net。www.protocol.http.httpurlconnection.plainconnect0(httpurlconnection。java:1203)在java.base/sun.net。www.protocol.http.httpurlconnection.plainconnect(httpurlconnection。java:1057)位于java.base/sun.net。www.protocol.http.httpurlconnection.connect(httpurlconnection。java:991)在com.amazonaws.internal.connectionutils.connecttoendpoint(connectionutils。java:54)在com.amazonaws.internal.ec2credentialsutils.readresource(ec2credentialsutils。java:113)在com.amazonaws.internal.ec2credentialsutils.readresource(ec2credentialsutils。java:82)在com.amazonaws.auth.instanceprofilecredentialsprovider$instancemetadatacredentialsendpointprovider.getcredentialsendpoint(instanceprofilecredentialsprovider)。java:197)在amazonaws.auth.ec2credentialsfetcher.fetchcredentials(ec2credentialsfetcher。java:122) ... 29更多与目标vm断开连接,地址:'127.0.0.1:60790',传输:'socket'
进程已完成,退出代码为1
1条答案
按热度按时间omjgkv6w1#
查看错误,我认为您需要将core-site.xml文件添加到具有属性的资源中
fs.s3a.access.key
,fs.s3a.secret.key
以及fs.s3a.endpoint